chore: import upstream snapshot with attribution

tools/Polygraphy/examples/cli/inspect/01_inspecting_a_tensorrt_network/README.md

@@ -0,0 +1,37 @@
+# Inspecting A TensorRT Network
+
+
+## Introduction
+
+The `inspect model` subtool can automatically convert supported formats
+into TensorRT networks, and then display them.
+
+
+## Running The Example
+
+1. Display the TensorRT network after parsing an ONNX model:
+
+    ```bash
+    polygraphy inspect model identity.onnx \
+        --show layers --display-as=trt
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== TensorRT Network ====
+        Name: Unnamed Network 0 | Explicit Batch Network
+
+        ---- 1 Network Input(s) ----
+        {x [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 1 Network Output(s) ----
+        {y [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 1 Layer(s) ----
+        Layer 0    | node_of_y [Op: LayerType.IDENTITY]
+            {x [dtype=float32, shape=(1, 1, 2, 2)]}
+             -> {y [dtype=float32, shape=(1, 1, 2, 2)]}
+    ```
+
+    It is also possible to show detailed layer information, including layer attributes, using `--show layers attrs weights`.

tools/Polygraphy/examples/cli/inspect/01_inspecting_a_tensorrt_network/identity.onnx

@@ -0,0 +1,15 @@
+backend-test:[
+
+
x
y"Identity
+test_identityZ
+
x
+

+

+

+
+b
+
y
+

+

+

+

tools/Polygraphy/examples/cli/inspect/02_inspecting_a_tensorrt_engine/README.md

@@ -0,0 +1,82 @@
+# Inspecting A TensorRT Engine
+
+
+## Introduction
+
+The `inspect model` subtool can load and display information
+about TensorRT engines, i.e. plan files:
+
+
+## Running The Example
+
+1. Generate an engine with dynamic shapes and 2 profiles:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt \
+        --trt-min-shapes X:[1,2,1,1] --trt-opt-shapes X:[1,2,3,3] --trt-max-shapes X:[1,2,5,5] \
+        --trt-min-shapes X:[1,2,2,2] --trt-opt-shapes X:[1,2,4,4] --trt-max-shapes X:[1,2,6,6] \
+        --save-engine dynamic_identity.engine
+    ```
+
+    You can also dump unfused intermediate tensors by adding `--mark-unfused-tensors-as-debug-tensors` and 
+        `--save-outputs output.json` options. Later, this tensor information can be combined with the inspector output.
+
+2. Inspect the engine:
+
+    ```bash
+    polygraphy inspect model dynamic_identity.engine \
+        --show layers
+    ```
+
+    NOTE: `--show layers` only works if the engine was built with a `profiling_verbosity` other than `NONE`.
+        Higher verbosities make more per-layer information available.
+
+    This will display something like:
+
+    ```
+    [I] ==== TensorRT Engine ====
+        Name: Unnamed Network 0 | Explicit Batch Engine
+
+        ---- 1 Engine Input(s) ----
+        {X [dtype=float32, shape=(1, 2, -1, -1)]}
+
+        ---- 1 Engine Output(s) ----
+        {Y [dtype=float32, shape=(1, 2, -1, -1)]}
+
+        ---- Memory ----
+        Device Memory: 0 bytes
+
+        ---- 2 Profile(s) (2 Tensor(s) Each) ----
+        - Profile: 0
+            Tensor: X          (Input), Index: 0 | Shapes: min=(1, 2, 1, 1), opt=(1, 2, 3, 3), max=(1, 2, 5, 5)
+            Tensor: Y         (Output), Index: 1 | Shape: (1, 2, -1, -1)
+
+        - Profile: 1
+            Tensor: X          (Input), Index: 0 | Shapes: min=(1, 2, 2, 2), opt=(1, 2, 4, 4), max=(1, 2, 6, 6)
+            Tensor: Y         (Output), Index: 1 | Shape: (1, 2, -1, -1)
+
+        ---- 1 Layer(s) Per Profile ----
+        - Profile: 0
+            Layer 0    | node_of_Y [Op: Reformat]
+                {X [shape=(1, 2, -1, -1)]}
+                 -> {Y [shape=(1, 2, -1, -1)]}
+
+        - Profile: 1
+            Layer 0    | node_of_Y [profile 1] [Op: MyelinReformat]
+                {X [profile 1] [shape=(1, 2, -1, -1)]}
+                 -> {Y [profile 1] [shape=(1, 2, -1, -1)]}
+    ```
+
+    It is also possible to show more detailed layer information using `--show layers attrs`.
+
+    You can also combine tensor value statistics using `--combine-tensor-info output.json` where the JSON file is got from
+        `--mark-unfused-tensors-as-debug-tensors` and `--save-outputs output.json`.
+    
+    The statistics will be added to the input and output tensors of each layer:
+
+    <!-- Polygraphy Test: Ignore Start -->
+    ```
+        {X [dtype=float32, shape=(1, 2, -1, -1), Format: Float, min=0.42, max=0.72, avg=0.57]}
+         -> {Y [dtype=float32, shape=(1, 2, -1, -1), Format: Float, min=0.42, max=0.72, avg=0.57]}
+    ```
+    <!-- Polygraphy Test: Ignore End -->

tools/Polygraphy/examples/cli/inspect/02_inspecting_a_tensorrt_engine/dynamic_identity.onnx

@@ -0,0 +1,15 @@
+backend_test:y
+
+
X
Y"Identityonnx_dynamic_identityZ&
+
X!
+

+

+
+height
+widthb&
+
Y!
+

+

+
+height
+widthB

tools/Polygraphy/examples/cli/inspect/03_inspecting_an_onnx_model/README.md

@@ -0,0 +1,38 @@
+# Inspecting An ONNX Model
+
+
+## Introduction
+
+The `inspect model` subtool can display ONNX models.
+
+
+## Running The Example
+
+1. Inspect the ONNX model:
+
+    ```bash
+    polygraphy inspect model identity.onnx --show layers
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== ONNX Model ====
+        Name: test_identity | ONNX Opset: 8
+
+        ---- 1 Graph Input(s) ----
+        {x [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 1 Graph Output(s) ----
+        {y [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 0 Initializer(s) ----
+        {}
+
+        ---- 1 Node(s) ----
+        Node 0    |  [Op: Identity]
+            {x [dtype=float32, shape=(1, 1, 2, 2)]}
+             -> {y [dtype=float32, shape=(1, 1, 2, 2)]}
+    ```
+
+    It is also possible to show detailed layer information, including layer attributes, using `--show layers attrs weights`.

tools/Polygraphy/examples/cli/inspect/03_inspecting_an_onnx_model/identity.onnx

@@ -0,0 +1,15 @@
+backend-test:[
+
+
x
y"Identity
+test_identityZ
+
x
+

+

+

+
+b
+
y
+

+

+

+

tools/Polygraphy/examples/cli/inspect/04_inspecting_a_tensorflow_graph/README.md

@@ -0,0 +1,28 @@
+# Inspecting A TensorFlow Graph
+
+
+## Introduction
+
+The `inspect model` subtool can display TensorFlow graphs.
+
+
+## Running The Example
+
+1. Inspect a TensorFlow frozen model:
+
+    ```bash
+    polygraphy inspect model identity.pb --model-type=frozen
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== TensorFlow Graph ====
+        ---- 1 Graph Inputs ----
+        {Input:0 [dtype=float32, shape=(1, 15, 25, 30)]}
+
+        ---- 1 Graph Outputs ----
+        {Identity_2:0 [dtype=float32, shape=(1, 15, 25, 30)]}
+
+        ---- 4 Nodes ----
+    ```

tools/Polygraphy/examples/cli/inspect/04_inspecting_a_tensorflow_graph/identity.pb

@@ -0,0 +1,17 @@
+
+>
+InputPlaceholder*
+dtype0
*
+shape:

+$
+IdentityIdentityInput*
+
T0

+)
+
+Identity_1IdentityIdentity*
+
T0

++
+
+Identity_2Identity
+Identity_1*
+
T0
"

tools/Polygraphy/examples/cli/inspect/05_inspecting_inference_outputs/README.md

@@ -0,0 +1,34 @@
+# Inspecting Inference Outputs
+
+
+## Introduction
+
+The `inspect data` subtool can display information about the
+`RunResults` object generated by `Comparator.run()`, which represents inference outputs.
+
+
+## Running The Example
+
+1. Generate some inference outputs using ONNX-Runtime:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt --save-outputs outputs.json
+    ```
+
+2. Inspect the results:
+
+    ```bash
+    polygraphy inspect data outputs.json --show-values
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== Run Results (1 runners) ====
+
+        ---- onnxrt-runner-N0-07/15/21-10:46:07 (1 iterations) ----
+
+        y [dtype=float32, shape=(1, 1, 2, 2)] | Stats: mean=0.35995, std-dev=0.25784, var=0.066482, median=0.35968, min=0.00011437 at (0, 0, 1, 0), max=0.72032 at (0, 0, 0, 1), avg-magnitude=0.35995, p90=0.62933, p95=0.67483, p99=0.71123
+            [[[[4.17021990e-01 7.20324516e-01]
+               [1.14374816e-04 3.02332580e-01]]]]
+    ```

tools/Polygraphy/examples/cli/inspect/05_inspecting_inference_outputs/identity.onnx

@@ -0,0 +1,15 @@
+backend-test:[
+
+
x
y"Identity
+test_identityZ
+
x
+

+

+

+
+b
+
y
+

+

+

+

tools/Polygraphy/examples/cli/inspect/06_inspecting_input_data/README.md

@@ -0,0 +1,31 @@
+# Inspecting Input Data
+
+
+## Introduction
+
+The `inspect data` subtool can display information about input data generated
+by a data loader.
+
+
+## Running The Example
+1. Generate some input data by running inference:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt --save-inputs inputs.json
+    ```
+
+2. Inspect the input data:
+
+    ```bash
+    polygraphy inspect data inputs.json --show-values
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== Data (1 iterations) ====
+
+        x [dtype=float32, shape=(1, 1, 2, 2)] | Stats: mean=0.35995, std-dev=0.25784, var=0.066482, median=0.35968, min=0.00011437 at (0, 0, 1, 0), max=0.72032 at (0, 0, 0, 1), avg-magnitude=0.35995, p90=0.62933, p95=0.67483, p99=0.71123
+            [[[[4.17021990e-01 7.20324516e-01]
+               [1.14374816e-04 3.02332580e-01]]]]
+    ```

tools/Polygraphy/examples/cli/inspect/06_inspecting_input_data/identity.onnx

@@ -0,0 +1,15 @@
+backend-test:[
+
+
x
y"Identity
+test_identityZ
+
x
+

+

+

+
+b
+
y
+

+

+

+

tools/Polygraphy/examples/cli/inspect/07_inspecting_tactic_replays/README.md

@@ -0,0 +1,31 @@
+# Inspecting Tactic Replay Files
+
+
+## Introduction
+
+The `inspect tactics` subtool can display information about TensorRT tactic replay
+files generated by Polygraphy.
+
+
+## Running The Example
+
+1. Generate a tactic replay file:
+
+    ```bash
+    polygraphy run model.onnx --trt --save-tactics replay.json
+    ```
+
+2. Inspect the tactic replay:
+
+    ```bash
+    polygraphy inspect tactics replay.json
+    ```
+
+    This will display something like:
+
+    ```
+    [I] Layer: ONNXTRT_Broadcast
+            Algorithm: (Implementation: 2147483661, Tactic: 0) | Inputs: (('DataType.FLOAT'),) | Outputs: (('DataType.FLOAT'),)
+        Layer: node_of_z
+            Algorithm: (Implementation: 2147483651, Tactic: 1) | Inputs: (('DataType.FLOAT'), ('DataType.FLOAT')) | Outputs: (('DataType.FLOAT'),)
+    ```

tools/Polygraphy/examples/cli/inspect/08_inspecting_tensorrt_onnx_support/README.md

@@ -0,0 +1,34 @@
+# Inspecting TensorRT ONNX Support
+
+## Introduction
+
+The `inspect capability` subtool provides detailed information on TensorRT's ONNX operator support for a given ONNX graph.
+It is also able to partition and save supported and unsupported subgraphs from the original model in order to report all the dynamically checked errors with a given model.
+
+## Running The Example
+
+1. Generate the capability report
+
+    ```bash
+    polygraphy inspect capability --with-partitioning model.onnx
+    ```
+
+2. This should display a summary table like:
+
+    ```
+    [I] ===== Summary =====
+        Operator | Count   | Reason                                                                                                                                                                    | Nodes
+        -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+        Fake     |       1 | In node 0 with name:  and operator: Fake (checkFallbackPluginImporter): INVALID_NODE: creator && "Plugin not found, are the plugin name, version, and namespace correct?" | [[2, 3]]
+    ```
+
+## Understanding The Output
+
+In this example, `model.onnx` contains a `Fake` node that is not supported by TensorRT.
+The summary table shows the unsupported operator, the reason it's unsupported, how many times it appears in the graph,
+and the index range of these nodes in the graph in case there are multiple unsupported nodes in a row.
+Note that this range uses an inclusive start index and an exclusive end index.
+
+It is important to note that the graph partitioning logic (`--with-partitioning`) currently does not support surfacing issues with nodes inside local functions (`FunctionProto`s). See the description of the default flow (without `--with-partitioning` option, described in the example `09_inspecting_tensorrt_static_onnx_support`) for static error reporting that properly handles nodes inside local functions.
+
+For more information and options, see `polygraphy inspect capability --help`.

tools/Polygraphy/examples/cli/inspect/09_inspecting_tensorrt_static_onnx_support/README.md

@@ -0,0 +1,31 @@
+# Inspecting TensorRT ONNX Support
+
+## Introduction
+
+The `inspect capability` subtool provides detailed information on TensorRT's ONNX operator support for a given ONNX graph.
+It is also able to partition and save supported and unsupported subgraphs from the original model in order to report all the dynamically checked errors with a given model (see the example `08_inspecting_tensorrt_onnx_support`).
+
+## Running The Example
+
+1. Generate the capability report
+
+    ```bash
+    polygraphy inspect capability nested_local_function.onnx
+    ```
+
+2. This should display a summary table like:
+
+    ```
+    [I] ===== Summary =====
+        Stack trace                                                                             | Operator  | Node               | Reason
+        -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+        onnx_graphsurgeon_node_1 (OuterFunction) -> onnx_graphsurgeon_node_1 (NestedLocalFake2) | Fake_2    | nested_node_fake_2 | In node 0 with name: nested_node_fake_2 and operator: Fake_2 (checkFallbackPluginImporter): INVALID_NODE: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
+        onnx_graphsurgeon_node_1 (OuterFunction)                                                | Fake_1    | nested_node_fake_1 | In node 0 with name: nested_node_fake_1 and operator: Fake_1 (checkFallbackPluginImporter): INVALID_NODE: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
+    ```
+
+## Understanding The Output
+
+In this example, `nested_local_function.onnx` contains `Fake_1` and `Fake_2` nodes that are not supported by TensorRT. `Fake_1` node is located inside a local function `OuterFunction` and `Fake_2` node is located inside a nested local function, `NestedLocalFake2`.
+The summary table shows the current stack trace consisting of local functions, the operator in which the error occurred and the reason it's unsupported.
+
+For more information and options, see `polygraphy inspect capability --help`.