chore: import upstream snapshot with attribution

2026-07-13 13:36:55 +08:00
commit c8a779b1bb
1887 changed files with 3245738 additions and 0 deletions
@@ -0,0 +1,6 @@
+# Polygraphy CLI Examples
+
+This directory includes examples that use the Polygraphy CLI.
+For examples of the Python API, see the [api](../api/) directory instead.
+
+You may find the [CLI User Guide](../../polygraphy/tools/) useful to navigate the CLI examples.
@@ -0,0 +1,125 @@
+# checking An ONNX Model
+
+
+## Introduction
+
+The `check lint` subtool validates ONNX Models and generates a JSON report detailing any bad/unused nodes or model errors.
+
+## Running The Example
+
+### Lint the ONNX model:
+
+<!-- Polygraphy Test: XFAIL Start -->
+```bash
+polygraphy check lint bad_graph.onnx -o report.json
+```
+<!-- Polygraphy Test: XFAIL End -->
+The output should look something like this:
+```bash
+[I] RUNNING | Command: polygraphy check lint bad_graph.onnx -o report.json
+[I] Loading model: bad_graph.onnx
+[E] LINT | Field 'name' of 'graph' is required to be non-empty.
+[I] Will generate inference input data according to provided TensorMetadata: {E [dtype=float32, shape=(1, 4)],
+     F [dtype=float32, shape=(4, 1)],
+     G [dtype=int64, shape=(4, 4)],
+     D [dtype=float32, shape=(4, 1)],
+     C [dtype=float32, shape=(3, 4)],
+     A [dtype=float32, shape=(1, 3)],
+     B [dtype=float32, shape=(4, 4)]}
+[E] LINT | Name: MatMul_3, Op: MatMul |  Incompatible dimensions for matrix multiplication
+[E] LINT | Name: Add_0, Op: Add |  Incompatible dimensions
+[E] LINT | Name: MatMul_0, Op: MatMul |  Incompatible dimensions for matrix multiplication
+[W] LINT | Input: 'A' does not affect outputs, can be removed.
+[W] LINT | Input: 'B' does not affect outputs, can be removed.
+[W] LINT | Name: MatMul_0, Op: MatMul | Does not affect outputs, can be removed.
+[I] Saving linting report to report.json
+[E] FAILED | Runtime: 1.006s | Command: polygraphy check lint bad_graph.onnx -o report.json
+```
+
+- This will create a `report.json` that contains information about what's wrong with the model.
+- The above example uses a faulty ONNX Model `bad_graph.onnx` that has multiple errors/warnings captured by the linter.
+The errors are:
+    1. Model has an empty name.
+    2. Nodes `Add_0`, `MatMul_0` and `MatMul_3` have incompatible input shapes.
+The warnings are:
+    1. Inputs `A` and `B` are unused output.
+    2. Node `MatMul_0` is unused by output.
+
+### Example Report:
+
+The generated report looks as follows:
+
+<!-- Polygraphy Test: Ignore Start -->
+```json
+{
+    "summary": {
+        "passing": [
+            "MatMul_1",
+            "cast_to_int64",
+            "NonZero"
+        ],
+        "failing": [
+            "MatMul_0",
+            "MatMul_3",
+            "Add_0"
+        ]
+    },
+    "lint_entries": [
+        {
+            "level": "exception",
+            "source": "onnx_checker",
+            "message": "Field 'name' of 'graph' is required to be non-empty."
+        },
+        {
+            "level": "exception",
+            "source": "onnxruntime",
+            "message": " Incompatible dimensions for matrix multiplication",
+            "nodes": [
+                "MatMul_3"
+            ]
+        },
+        {
+            "level": "exception",
+            "source": "onnxruntime",
+            "message": " Incompatible dimensions",
+            "nodes": [
+                "Add_0"
+            ]
+        },
+        {
+            "level": "exception",
+            "source": "onnxruntime",
+            "message": " Incompatible dimensions for matrix multiplication",
+            "nodes": [
+                "MatMul_0"
+            ]
+        },
+        {
+            "level": "warning",
+            "source": "onnx_graphsurgeon",
+            "message": "Input: 'A' does not affect outputs, can be removed."
+        },
+        {
+            "level": "warning",
+            "source": "onnx_graphsurgeon",
+            "message": "Input: 'B' does not affect outputs, can be removed."
+        },
+        {
+            "level": "warning",
+            "source": "onnx_graphsurgeon",
+            "message": "Does not affect outputs, can be removed.",
+            "nodes": [
+                "MatMul_0"
+            ]
+        }
+    ]
+}
+```
+<!-- Polygraphy Test: Ignore End -->
+
+### Notes
+Since it runs ONNX Runtime under the hood, it is possible to specify execution providers using `--providers`. Defaults to CPU.
+
+It is also possible to override the input shapes using `--input-shapes`, or provide custom input data. For more details, refer [how-to/use_custom_input_data](../../../../how-to/use_custom_input_data.md).
+
+For more information on usage, use `polygraphy check lint --help`.
@@ -0,0 +1,52 @@
+# Int8 Calibration In TensorRT
+
+
+## Introduction
+
+In [API example 04](../../../api/04_int8_calibration_in_tensorrt/), we saw how we can leverage
+Polygraphy's included calibrator to easily run int8 calibration with TensorRT.
+
+But what if we wanted to do the same thing on the command-line?
+
+To do this, we need a way to supply custom input data to our command-line tools.
+Polygraphy provides multiple ways to do so, which are detailed [here](../../../../how-to/use_custom_input_data.md).
+
+In this example, we'll use a data loader script by defining a `load_data` function in a Python
+script called `data_loader.py` and then use `polygraphy convert` to build the TensorRT engine.
+
+*TIP: We can use a similar approach with `polygraphy run` to build and run the engine.*
+
+## Running The Example
+
+1. Convert the model, using the custom data loader script to supply calibration data,
+    saving a calibration cache for future use:
+
+    ```bash
+    polygraphy convert identity.onnx --int8 \
+        --data-loader-script ./data_loader.py \
+        --calibration-cache identity_calib.cache \
+        -o identity.engine
+    ```
+
+2. **[Optional]** Rebuild the engine using the cache to skip calibration:
+
+    ```bash
+    polygraphy convert identity.onnx --int8 \
+        --calibration-cache identity_calib.cache \
+        -o identity.engine
+    ```
+
+    Since the calibration cache is already populated, calibration will be skipped.
+    Hence, we do *not* need to supply input data.
+
+
+3. **[Optional]** Use the data loader directly from the API example.
+
+    The method outlined here is so flexible that we can even use the data loader we defined in the API example!
+    We just need to specify the function name since the example does not call it `load_data`:
+
+    ```bash
+    polygraphy convert identity.onnx --int8 \
+        --data-loader-script ../../../api/04_int8_calibration_in_tensorrt/example.py:calib_data \
+        -o identity.engine
+    ```
@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Defines a `load_data` function that returns a generator yielding
+feed_dicts so that this script can be used as the argument for
+the --data-loader-script command-line parameter.
+"""
+import numpy as np
+
+INPUT_SHAPE = (1, 1, 2, 2)
+
+
+def load_data():
+    for _ in range(5):
+        yield {
+            "x": np.ones(shape=INPUT_SHAPE, dtype=np.float32)
+        }  # Still totally real data
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,41 @@
+# Deterministic Engine Building In TensorRT
+
+**NOTE: This example requires TensorRT 8.7 or newer.**
+
+## Introduction
+
+During engine building, TensorRT runs and times several kernels in order to select
+the most optimal ones. Since kernel timings may vary slightly from run to run, this
+process is inherently non-deterministic.
+
+In many cases, deterministic engine builds may be desirable. One way of achieving this
+is to use a timing cache to ensure the same kernels are picked each time.
+
+## Running The Example
+
+1. Build an engine and save a timing cache:
+
+    ```bash
+    polygraphy convert identity.onnx \
+        --save-timing-cache timing.cache \
+        -o 0.engine
+    ```
+
+2. Use the timing cache for another engine build:
+
+    ```bash
+    polygraphy convert identity.onnx \
+        --load-timing-cache timing.cache --error-on-timing-cache-miss \
+        -o 1.engine
+    ```
+
+    We specify `--error-on-timing-cache-miss` so that we can be sure that the new engine
+    used the entries from the timing cache for each layer.
+
+3. Verify that the engines are exactly the same:
+
+    <!-- Polygraphy Test: Ignore Start -->
+    ```bash
+    diff <(polygraphy inspect model 0.engine --show layers attrs) <(polygraphy inspect model 1.engine --show layers attrs)
+    ```
+    <!-- Polygraphy Test: Ignore End -->
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,42 @@
+# Working With Models With Dynamic Shapes In TensorRT
+
+## Introduction
+
+In order to use dynamic input shapes with TensorRT, we have to specify a range
+(or multiple ranges) of possible shapes when we build the engine.
+For details on how this works, refer to
+[API example 07](../../../api/07_tensorrt_and_dynamic_shapes/).
+
+When using the CLI, we can specify the per-input minimum, optimum, and maximum
+shapes one or more times. If shapes are specified more than
+once per input, multiple optimization profiles are created.
+
+## Running The Example
+
+1. Build an engine with 3 separate profiles:
+
+    ```bash
+    polygraphy convert dynamic_identity.onnx -o dynamic_identity.engine \
+        --trt-min-shapes X:[1,3,28,28] --trt-opt-shapes X:[1,3,28,28] --trt-max-shapes X:[1,3,28,28] \
+        --trt-min-shapes X:[1,3,28,28] --trt-opt-shapes X:[4,3,28,28] --trt-max-shapes X:[32,3,28,28] \
+        --trt-min-shapes X:[128,3,28,28] --trt-opt-shapes X:[128,3,28,28] --trt-max-shapes X:[128,3,28,28]
+    ```
+
+    For models with multiple inputs, simply provide multiple arguments to each `--trt-*-shapes` parameter.
+    For example: `--trt-min-shapes input0:[10,10] input1:[10,10] input2:[10,10] ...`
+
+    *TIP: If we want to use only a single profile where min == opt == max, we can leverage the runtime input*
+        *shapes option: `--input-shapes` as a conveneint shorthand instead of setting min/opt/max separately.*
+
+
+2. **[Optional]** Inspect the resulting engine:
+
+    ```bash
+    polygraphy inspect model dynamic_identity.engine
+    ```
+
+
+## Further Reading
+
+For more information on using dynamic shapes with TensorRT, see the
+[developer guide](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#work_dynamic_shapes)
@@ -0,0 +1,12 @@
+:[
+
+XY"Identityonnx_dynamic_identityZ%
+X 
+
+
+batch_size
+
+
+b	
+Y
+B
@@ -0,0 +1,49 @@
+# Converting ONNX Models To FP16
+
+## Introduction
+
+When debugging accuracy issues with using TensorRT reduced precision
+optimizations (`--fp16` and `--tf32` flags) on an ONNX model trained in FP32,
+it can be helpful to convert the model to FP16 and run it under ONNX-Runtime
+to check if there are might be problems inherent to running the model
+with reduced precision.
+
+## Running The Example
+
+1. Convert the model to FP16:
+
+   ```bash
+   polygraphy convert --fp-to-fp16 -o identity_fp16.onnx identity.onnx
+   ```
+
+2. **[Optional]** Inspect the resulting model:
+
+   ```bash
+   polygraphy inspect model identity_fp16.onnx
+   ```
+
+3. **[Optional]** Run the FP32 and FP16 models under ONNX-Runtime and then compare the results:
+
+   ```bash
+   polygraphy run --onnxrt identity.onnx \
+      --save-inputs inputs.json --save-outputs outputs_fp32.json
+   ```
+
+   ```bash
+   polygraphy run --onnxrt identity_fp16.onnx \
+      --load-inputs inputs.json --load-outputs outputs_fp32.json \
+      --atol 0.001 --rtol 0.001
+   ```
+
+4. **[Optional]** Check if any intermediate outputs of the FP16 model
+   contain NaN or infinity (see [Checking for Intermediate NaN or Infinities](../../../../examples/cli/run/07_checking_nan_inf)):
+
+   ```bash
+   polygraphy run --onnxrt identity_fp16.onnx --onnx-outputs mark all --validate
+   ```
+
+## See Also
+
+* [Comparing Across Runs](../../../../examples/cli/run/02_comparing_across_runs)
+* [Checking for Intermediate NaN or Infinities](../../../../examples/cli/run/07_checking_nan_inf)
+* [Debugging TensorRT Accuracy Issues](../../../../how-to/debug_accuracy.md)
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,94 @@
+# Debugging Flaky TensorRT Tactics
+
+**IMPORTANT: This example no longer works reliably for newer versions of TensorRT, since they make some**
+    **tactic choices that are not exposed via the IAlgorithmSelector interface (Deprecated in TensorRT 10.8.**
+    **Please use editable mode in ITimingCache instead). Thus, the approach outlined below**
+    **cannot guarantee deterministic engine builds. With TensorRT 8.7 and newer, you can use the**
+    **tactic timing cache (`--save-timing-cache` and `--load-timing-cache` in Polygraphy) to ensure**
+    **determinism, but these files are opaque and thus cannot be interpreted by `inspect diff-tactics`**
+
+## Introduction
+
+Sometimes, a tactic in TensorRT may produce incorrect results, or have
+otherwise buggy behavior. Since the TensorRT builder relies on timing
+tactics, engine builds are non-deterministic, which can make tactic bugs
+manifest as flaky/intermittent failures.
+
+One approach to tackling the problem is to run the builder several times,
+saving tactic replay files from each run. Once we have a set of known-good and
+known-bad tactics, we can compare them to determine which tactic
+is likely to be the source of error.
+
+The `debug build` subtool allows you to automate this process.
+
+For more details on how the `debug` tools work, see the help output:
+`polygraphy debug -h` and `polygraphy debug build -h`.
+
+
+## Running The Example
+
+1. Generate golden outputs from ONNX-Runtime:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt \
+        --save-outputs golden.json
+    ```
+
+2. Use `debug build` to repeatedly build TensorRT engines and compare results against the golden outputs,
+    saving a tactic replay file each time:
+
+    ```bash
+    polygraphy debug build identity.onnx --fp16 --save-tactics replay.json \
+        --artifacts-dir replays --artifacts replay.json --until=10 \
+        --check polygraphy run polygraphy_debug.engine --trt --load-outputs golden.json
+    ```
+
+    Let's break this down:
+
+    - Like other `debug` subtools, `debug build` generates an intermediate artifact each iteration
+        (`./polygraphy_debug.engine` by default). This artifact in this case is a TensorRT engine.
+
+        *TIP: `debug build` supports all the TensorRT builder configuration options supported*
+            *by other tools, like `convert` or `run`.*
+
+    - In order for `debug build` to determine whether each engine fails or passes,
+        we provide a `--check` command. Since we're looking at a (fake) accuracy issue,
+        we can use `polygraphy run` to compare the outputs of the engine to our golden values.
+
+        *TIP: Like other `debug` subtools, an interactive mode is also supported, which you can*
+            *use simply by omitting the `--check` argument.*
+
+    - Unlike other `debug` subtools, `debug build` has no automatic terminating condition, so we need
+        to provide the `--until` option so that the tool knows when to stop. This can either be a number
+        of iterations, or `"good"` or `"bad"`. In the latter case, the tool will stop after finding the
+        first passing or failing iteration respectively.
+
+    - Since we eventually want to compare the good and bad tactic replays, we specify `--save-tactics`
+        to save tactic replay files from each iteration, then use `--artifacts` to tell `debug build`
+        to manage them, which involves sorting them into `good` and `bad` subdirectories under the
+        main artifacts directory, specified with `--artifacts-dir`.
+
+
+3. Use `inspect diff-tactics` to determine which tactics could be bad:
+
+    ```bash
+    polygraphy inspect diff-tactics --dir replays
+    ```
+
+    *NOTE: This last step should report that it could not determine potentially bad tactics since*
+        *our `bad` directory should be empty at this point (please file a TensorRT issue otherwise!):*
+
+    <!-- Polygraphy Test: Ignore Start -->
+    ```
+    [I] Loaded 2 good tactic replays.
+    [I] Loaded 0 bad tactic replays.
+    [I] Could not determine potentially bad tactics. Try generating more tactic replay files?
+    ```
+    <!-- Polygraphy Test: Ignore End -->
+
+
+## Further Reading
+
+For more information on the `debug` tool, as well as tips and tricks applicable
+to all `debug` subtools, see the
+[how-to guide for `debug` subtools](../../../../how-to/use_debug_subtools_effectively.md).
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,155 @@
+# Reducing Failing ONNX Models
+
+## Introduction
+
+When a model fails for any reason (for example, an accuracy issue in TensorRT) it is often
+useful to reduce it to the smallest possible subgraph that triggers the failure. That makes
+it easier to pinpoint the cause of the failure.
+
+One approach to doing so is to generate successively smaller subgraphs of the original ONNX model.
+At each iteration, we can check whether the subgraph works or is still failing; once we have a working
+subgraph, we know that the subgraph generated by the previous iteration is the smallest failing
+subgraph.
+
+The `debug reduce` subtool allows us to automate this process.
+
+
+## Running The Example
+
+For the sake of this example, we'll assume our model (`./model.onnx`) has accuracy issues
+in TensorRT. Since the model actually does work in TensorRT (please report a bug if not!),
+we'll outline the commands that you would normally run followed by commands you can run to
+simulate a failure to get a feel for how the tool looks in practice.
+
+Our simulated failures will trigger whenever there's a `Mul` node in the model:
+
+![./model.png](./model.png)
+
+Hence, the final reduced model should contain just the `Mul` node (since the other nodes don't cause a failure).
+
+1. For models that use dynamic input shapes or contain shape operations, freeze the input
+    shapes and fold shape operations with:
+
+    ```bash
+    polygraphy surgeon sanitize model.onnx -o folded.onnx --fold-constants \
+        --override-input-shapes x0:[1,3,224,224] x1:[1,3,224,224]
+    ```
+
+2. Let's assume ONNX-Runtime gives us correct outputs. We'll start by generating golden
+    values for every tensor in the network. We'll also save the inputs we use:
+
+    ```bash
+    polygraphy run folded.onnx --onnxrt \
+        --save-inputs inputs.json \
+        --onnx-outputs mark all --save-outputs layerwise_golden.json
+    ```
+
+    Then we'll combine the inputs and layerwise outputs into a single layerwise inputs file
+    using the `data to-input` subtool (we'll see why this is necessary in the next step):
+
+    ```bash
+    polygraphy data to-input inputs.json layerwise_golden.json -o layerwise_inputs.json
+    ```
+
+
+3. Next, we'll use `debug reduce` in `bisect` mode:
+
+    ```bash
+    polygraphy debug reduce folded.onnx -o initial_reduced.onnx --mode=bisect --load-inputs layerwise_inputs.json \
+        --check polygraphy run polygraphy_debug.onnx --trt \
+                --load-inputs layerwise_inputs.json --load-outputs layerwise_golden.json
+    ```
+
+    Let's break this down:
+
+    - Like the other `debug` subtools, `debug reduce` generates an intermediate artifact each iteration
+        (`./polygraphy_debug.onnx` by default). The artifact in this case is some subgraph of the original ONNX model.
+
+    - In order for `debug reduce` to determine whether each subgraph fails or passes,
+        we provide a `--check` command. Since we're looking into an accuracy issue,
+        we can use `polygraphy run` to compare against our golden outputs from before.
+
+        *TIP: Like other `debug` subtools, an interactive mode is also supported, which you can*
+            *use simply by omitting the `--check` argument.*
+
+    - In the `--check` command, we provide the layerwise inputs via `--load-inputs`, since otherwise, `polygraphy run`
+        would generate new inputs for the subgraph tensors, which may not match the values those tensors
+        had when we generated our golden data. An alternative approach is to run the reference implementation
+        (ONNX-Runtime here) during each iteration of `debug reduce` rather than ahead of time.
+
+    - Since we're using non-default input data, we also provide the layerwise inputs via `--load-inputs` directly to the
+        `debug reduce` command (in addition to providing it to the `--check` command).
+        This is important in models with multiple parallel branches (*referring to paths in the model rather than control flow*) like:
+        <!-- Polygraphy Test: Ignore Start -->
+        ```
+         inp0  inp1
+          |     |
+         Abs   Abs
+            \ /
+            Sum
+             |
+            out
+        ```
+        In such cases, `debug reduce` needs to be able to replace one branch with a constant.
+        To do so, it needs to know the input data you are using so that it can replace it with the correct values.
+        Though we're using a file here, input data can be provided via any other Polygraphy data loader argument covered in
+        [the CLI user guide](../../../../how-to/use_custom_input_data.md).
+
+        In case you're not sure whether you need to provide a data loader,
+        `debug reduce` will emit a warning like this when it tries to replace a branch:
+        ```
+        [W]     This model includes multiple branches/paths. In order to continue reducing, one branch needs to be folded away.
+                Please ensure that you have provided a data loader argument to `debug reduce` if your `--check` command is using a non-default data loader.
+                Not doing so may result in false negatives!
+        ```
+        <!-- Polygraphy Test: Ignore End -->
+
+    - We specify the `-o` option so that the reduced model will be written to `initial_reduced.onnx`.
+
+    **To Simulate A Failure:** We can use `polygraphy inspect model` in conjunction with `--fail-regex` to trigger
+    a failure whenever the model contains a `Mul` node:
+
+    ```bash
+    polygraphy debug reduce folded.onnx -o initial_reduced.onnx --mode=bisect \
+        --fail-regex "Op: Mul" \
+        --check polygraphy inspect model polygraphy_debug.onnx --show layers
+    ```
+
+4. **[Optional]** As a sanity check, we can inspect our reduced model to ensure that it does contain the `Mul` node:
+
+    ```bash
+    polygraphy inspect model initial_reduced.onnx --show layers
+    ```
+
+5. Since we used `bisect` mode in the previous step, the model may not be as minimal as it could be.
+    To further refine it, we'll run `debug reduce` again in `linear` mode:
+
+    ```bash
+    polygraphy debug reduce initial_reduced.onnx -o final_reduced.onnx --mode=linear --load-inputs layerwise_inputs.json \
+        --check polygraphy run polygraphy_debug.onnx --trt \
+                --load-inputs layerwise_inputs.json --load-outputs layerwise_golden.json
+    ```
+
+    **To Simulate A Failure:** We'll use the same technique as before:
+
+    ```bash
+    polygraphy debug reduce initial_reduced.onnx -o final_reduced.onnx --mode=linear \
+        --fail-regex "Op: Mul" \
+        --check polygraphy inspect model polygraphy_debug.onnx --show layers
+    ```
+
+6. **[Optional]** At this stage, `final_reduced.onnx` should contain just the failing node - the `Mul`.
+    We can verify this with `inspect model`:
+
+    ```bash
+    polygraphy inspect model final_reduced.onnx --show layers
+    ```
+
+
+## Further Reading
+
+- For more details on how the `debug` tools work, see the help output:
+    `polygraphy debug -h` and `polygraphy debug reduce -h`.
+
+- Also see the [`debug reduce` how-to guide](../../../../how-to/use_debug_reduce_effectively.md)
+    for more information, tips, and tricks.
@@ -0,0 +1,37 @@
+# Inspecting A TensorRT Network
+
+
+## Introduction
+
+The `inspect model` subtool can automatically convert supported formats
+into TensorRT networks, and then display them.
+
+
+## Running The Example
+
+1. Display the TensorRT network after parsing an ONNX model:
+
+    ```bash
+    polygraphy inspect model identity.onnx \
+        --show layers --display-as=trt
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== TensorRT Network ====
+        Name: Unnamed Network 0 | Explicit Batch Network
+
+        ---- 1 Network Input(s) ----
+        {x [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 1 Network Output(s) ----
+        {y [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 1 Layer(s) ----
+        Layer 0    | node_of_y [Op: LayerType.IDENTITY]
+            {x [dtype=float32, shape=(1, 1, 2, 2)]}
+             -> {y [dtype=float32, shape=(1, 1, 2, 2)]}
+    ```
+
+    It is also possible to show detailed layer information, including layer attributes, using `--show layers attrs weights`.
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,82 @@
+# Inspecting A TensorRT Engine
+
+
+## Introduction
+
+The `inspect model` subtool can load and display information
+about TensorRT engines, i.e. plan files:
+
+
+## Running The Example
+
+1. Generate an engine with dynamic shapes and 2 profiles:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt \
+        --trt-min-shapes X:[1,2,1,1] --trt-opt-shapes X:[1,2,3,3] --trt-max-shapes X:[1,2,5,5] \
+        --trt-min-shapes X:[1,2,2,2] --trt-opt-shapes X:[1,2,4,4] --trt-max-shapes X:[1,2,6,6] \
+        --save-engine dynamic_identity.engine
+    ```
+
+    You can also dump unfused intermediate tensors by adding `--mark-unfused-tensors-as-debug-tensors` and 
+        `--save-outputs output.json` options. Later, this tensor information can be combined with the inspector output.
+
+2. Inspect the engine:
+
+    ```bash
+    polygraphy inspect model dynamic_identity.engine \
+        --show layers
+    ```
+
+    NOTE: `--show layers` only works if the engine was built with a `profiling_verbosity` other than `NONE`.
+        Higher verbosities make more per-layer information available.
+
+    This will display something like:
+
+    ```
+    [I] ==== TensorRT Engine ====
+        Name: Unnamed Network 0 | Explicit Batch Engine
+
+        ---- 1 Engine Input(s) ----
+        {X [dtype=float32, shape=(1, 2, -1, -1)]}
+
+        ---- 1 Engine Output(s) ----
+        {Y [dtype=float32, shape=(1, 2, -1, -1)]}
+
+        ---- Memory ----
+        Device Memory: 0 bytes
+
+        ---- 2 Profile(s) (2 Tensor(s) Each) ----
+        - Profile: 0
+            Tensor: X          (Input), Index: 0 | Shapes: min=(1, 2, 1, 1), opt=(1, 2, 3, 3), max=(1, 2, 5, 5)
+            Tensor: Y         (Output), Index: 1 | Shape: (1, 2, -1, -1)
+
+        - Profile: 1
+            Tensor: X          (Input), Index: 0 | Shapes: min=(1, 2, 2, 2), opt=(1, 2, 4, 4), max=(1, 2, 6, 6)
+            Tensor: Y         (Output), Index: 1 | Shape: (1, 2, -1, -1)
+
+        ---- 1 Layer(s) Per Profile ----
+        - Profile: 0
+            Layer 0    | node_of_Y [Op: Reformat]
+                {X [shape=(1, 2, -1, -1)]}
+                 -> {Y [shape=(1, 2, -1, -1)]}
+
+        - Profile: 1
+            Layer 0    | node_of_Y [profile 1] [Op: MyelinReformat]
+                {X [profile 1] [shape=(1, 2, -1, -1)]}
+                 -> {Y [profile 1] [shape=(1, 2, -1, -1)]}
+    ```
+
+    It is also possible to show more detailed layer information using `--show layers attrs`.
+
+    You can also combine tensor value statistics using `--combine-tensor-info output.json` where the JSON file is got from
+        `--mark-unfused-tensors-as-debug-tensors` and `--save-outputs output.json`.
+    
+    The statistics will be added to the input and output tensors of each layer:
+
+    <!-- Polygraphy Test: Ignore Start -->
+    ```
+        {X [dtype=float32, shape=(1, 2, -1, -1), Format: Float, min=0.42, max=0.72, avg=0.57]}
+         -> {Y [dtype=float32, shape=(1, 2, -1, -1), Format: Float, min=0.42, max=0.72, avg=0.57]}
+    ```
+    <!-- Polygraphy Test: Ignore End -->
@@ -0,0 +1,15 @@
+backend_test:y
+
+XY"Identityonnx_dynamic_identityZ&
+X!
+
+
+
+height
+widthb&
+Y!
+
+
+
+height
+widthB
@@ -0,0 +1,38 @@
+# Inspecting An ONNX Model
+
+
+## Introduction
+
+The `inspect model` subtool can display ONNX models.
+
+
+## Running The Example
+
+1. Inspect the ONNX model:
+
+    ```bash
+    polygraphy inspect model identity.onnx --show layers
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== ONNX Model ====
+        Name: test_identity | ONNX Opset: 8
+
+        ---- 1 Graph Input(s) ----
+        {x [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 1 Graph Output(s) ----
+        {y [dtype=float32, shape=(1, 1, 2, 2)]}
+
+        ---- 0 Initializer(s) ----
+        {}
+
+        ---- 1 Node(s) ----
+        Node 0    |  [Op: Identity]
+            {x [dtype=float32, shape=(1, 1, 2, 2)]}
+             -> {y [dtype=float32, shape=(1, 1, 2, 2)]}
+    ```
+
+    It is also possible to show detailed layer information, including layer attributes, using `--show layers attrs weights`.
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,28 @@
+# Inspecting A TensorFlow Graph
+
+
+## Introduction
+
+The `inspect model` subtool can display TensorFlow graphs.
+
+
+## Running The Example
+
+1. Inspect a TensorFlow frozen model:
+
+    ```bash
+    polygraphy inspect model identity.pb --model-type=frozen
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== TensorFlow Graph ====
+        ---- 1 Graph Inputs ----
+        {Input:0 [dtype=float32, shape=(1, 15, 25, 30)]}
+
+        ---- 1 Graph Outputs ----
+        {Identity_2:0 [dtype=float32, shape=(1, 15, 25, 30)]}
+
+        ---- 4 Nodes ----
+    ```
@@ -0,0 +1,17 @@
+
+>
+InputPlaceholder*
+dtype0*
+shape:
+$
+IdentityIdentityInput*
+T0
+)
+
+Identity_1IdentityIdentity*
+T0
+
+
+Identity_2Identity
+Identity_1*
+T0"
@@ -0,0 +1,34 @@
+# Inspecting Inference Outputs
+
+
+## Introduction
+
+The `inspect data` subtool can display information about the
+`RunResults` object generated by `Comparator.run()`, which represents inference outputs.
+
+
+## Running The Example
+
+1. Generate some inference outputs using ONNX-Runtime:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt --save-outputs outputs.json
+    ```
+
+2. Inspect the results:
+
+    ```bash
+    polygraphy inspect data outputs.json --show-values
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== Run Results (1 runners) ====
+
+        ---- onnxrt-runner-N0-07/15/21-10:46:07 (1 iterations) ----
+
+        y [dtype=float32, shape=(1, 1, 2, 2)] | Stats: mean=0.35995, std-dev=0.25784, var=0.066482, median=0.35968, min=0.00011437 at (0, 0, 1, 0), max=0.72032 at (0, 0, 0, 1), avg-magnitude=0.35995, p90=0.62933, p95=0.67483, p99=0.71123
+            [[[[4.17021990e-01 7.20324516e-01]
+               [1.14374816e-04 3.02332580e-01]]]]
+    ```
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,31 @@
+# Inspecting Input Data
+
+
+## Introduction
+
+The `inspect data` subtool can display information about input data generated
+by a data loader.
+
+
+## Running The Example
+1. Generate some input data by running inference:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt --save-inputs inputs.json
+    ```
+
+2. Inspect the input data:
+
+    ```bash
+    polygraphy inspect data inputs.json --show-values
+    ```
+
+    This will display something like:
+
+    ```
+    [I] ==== Data (1 iterations) ====
+
+        x [dtype=float32, shape=(1, 1, 2, 2)] | Stats: mean=0.35995, std-dev=0.25784, var=0.066482, median=0.35968, min=0.00011437 at (0, 0, 1, 0), max=0.72032 at (0, 0, 0, 1), avg-magnitude=0.35995, p90=0.62933, p95=0.67483, p99=0.71123
+            [[[[4.17021990e-01 7.20324516e-01]
+               [1.14374816e-04 3.02332580e-01]]]]
+    ```
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,31 @@
+# Inspecting Tactic Replay Files
+
+
+## Introduction
+
+The `inspect tactics` subtool can display information about TensorRT tactic replay
+files generated by Polygraphy.
+
+
+## Running The Example
+
+1. Generate a tactic replay file:
+
+    ```bash
+    polygraphy run model.onnx --trt --save-tactics replay.json
+    ```
+
+2. Inspect the tactic replay:
+
+    ```bash
+    polygraphy inspect tactics replay.json
+    ```
+
+    This will display something like:
+
+    ```
+    [I] Layer: ONNXTRT_Broadcast
+            Algorithm: (Implementation: 2147483661, Tactic: 0) | Inputs: (('DataType.FLOAT'),) | Outputs: (('DataType.FLOAT'),)
+        Layer: node_of_z
+            Algorithm: (Implementation: 2147483651, Tactic: 1) | Inputs: (('DataType.FLOAT'), ('DataType.FLOAT')) | Outputs: (('DataType.FLOAT'),)
+    ```
@@ -0,0 +1,34 @@
+# Inspecting TensorRT ONNX Support
+
+## Introduction
+
+The `inspect capability` subtool provides detailed information on TensorRT's ONNX operator support for a given ONNX graph.
+It is also able to partition and save supported and unsupported subgraphs from the original model in order to report all the dynamically checked errors with a given model.
+
+## Running The Example
+
+1. Generate the capability report
+
+    ```bash
+    polygraphy inspect capability --with-partitioning model.onnx
+    ```
+
+2. This should display a summary table like:
+
+    ```
+    [I] ===== Summary =====
+        Operator | Count   | Reason                                                                                                                                                                    | Nodes
+        -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+        Fake     |       1 | In node 0 with name:  and operator: Fake (checkFallbackPluginImporter): INVALID_NODE: creator && "Plugin not found, are the plugin name, version, and namespace correct?" | [[2, 3]]
+    ```
+
+## Understanding The Output
+
+In this example, `model.onnx` contains a `Fake` node that is not supported by TensorRT.
+The summary table shows the unsupported operator, the reason it's unsupported, how many times it appears in the graph,
+and the index range of these nodes in the graph in case there are multiple unsupported nodes in a row.
+Note that this range uses an inclusive start index and an exclusive end index.
+
+It is important to note that the graph partitioning logic (`--with-partitioning`) currently does not support surfacing issues with nodes inside local functions (`FunctionProto`s). See the description of the default flow (without `--with-partitioning` option, described in the example `09_inspecting_tensorrt_static_onnx_support`) for static error reporting that properly handles nodes inside local functions.
+
+For more information and options, see `polygraphy inspect capability --help`.
@@ -0,0 +1,31 @@
+# Inspecting TensorRT ONNX Support
+
+## Introduction
+
+The `inspect capability` subtool provides detailed information on TensorRT's ONNX operator support for a given ONNX graph.
+It is also able to partition and save supported and unsupported subgraphs from the original model in order to report all the dynamically checked errors with a given model (see the example `08_inspecting_tensorrt_onnx_support`).
+
+## Running The Example
+
+1. Generate the capability report
+
+    ```bash
+    polygraphy inspect capability nested_local_function.onnx
+    ```
+
+2. This should display a summary table like:
+
+    ```
+    [I] ===== Summary =====
+        Stack trace                                                                             | Operator  | Node               | Reason
+        -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+        onnx_graphsurgeon_node_1 (OuterFunction) -> onnx_graphsurgeon_node_1 (NestedLocalFake2) | Fake_2    | nested_node_fake_2 | In node 0 with name: nested_node_fake_2 and operator: Fake_2 (checkFallbackPluginImporter): INVALID_NODE: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
+        onnx_graphsurgeon_node_1 (OuterFunction)                                                | Fake_1    | nested_node_fake_1 | In node 0 with name: nested_node_fake_1 and operator: Fake_1 (checkFallbackPluginImporter): INVALID_NODE: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
+    ```
+
+## Understanding The Output
+
+In this example, `nested_local_function.onnx` contains `Fake_1` and `Fake_2` nodes that are not supported by TensorRT. `Fake_1` node is located inside a local function `OuterFunction` and `Fake_2` node is located inside a nested local function, `NestedLocalFake2`.
+The summary table shows the current stack trace consisting of local functions, the operator in which the error occurred and the reason it's unsupported.
+
+For more information and options, see `polygraphy inspect capability --help`.
@@ -0,0 +1,36 @@
+# Using Shard To Convert a SD Model to MD
+
+
+## Introduction
+
+The `shard` tool can be used to convert single-device (SD) models containing attention layers into multi-device (MD) models intended to be run on multiple GPUs using a hints file.
+
+In this example, we'll show how to shard a simple model containing an attention layer
+
+![./model.png](./model.png)
+
+## Hint Configuration
+
+For this example we'll be using [this hints file](./hint.json).
+
+See the [Shard README](../../../../polygraphy/tools/multi_device/README.md#sharding-hints-file-format) for an explanation of the hints file format.
+
+
+## Running the Example
+
+```bash
+polygraphy multi-device shard \
+    ../attention.onnx \
+    -s hint.json \
+    -o attention_md.onnx
+```
+
+Looking at the result, we can now see the model is ready to be run on multiple GPUs through TensorRT
+
+![./model_md.png](./model_md.png)
+
+
+### A Note On Gathering Q
+
+If we changed `gather_q` in the hints to `true` the model effectively becomes SD, and a final all-gather will not be inserted. All attention layers must have Q consistently sharded, as it affects whether or not to place an all-gather at the output of the model
+
@@ -0,0 +1,35 @@
+{
+    "parallelism": "CP",
+    "group_size": 0,
+    "root": 0,
+    "groups": [],
+    "attention_layers": [
+        {
+            "q": "q",
+            "gather_kv": true,
+            "gather_q": false,
+            "polygraphy_class": "AttentionLayerHint"
+        }
+    ],
+    "inputs": [
+        {
+            "name": "input",
+            "seq_len_idx": 0,
+            "rank": 3,
+            "polygraphy_class": "ShardTensor"
+        }
+    ],
+    "outputs": [
+        {
+            "name": "output",
+            "seq_len_idx": 0,
+            "rank": 3,
+            "polygraphy_class": "ShardTensor"
+        }
+    ],
+    "k_seq_len_idx": 0,
+    "v_seq_len_idx": 0,
+    "kv_rank": null,
+    "reduce_scatter_reduce_op": "max",
+    "polygraphy_class": "ShardHints"
+}
@@ -0,0 +1,126 @@
+# Matching and replacing a subgraph with a plugin in an onnx model
+
+## Introduction
+
+The `plugin` tool offers subtools to find and replace subgraphs in an onnx model.
+
+Subgraph substition is a three-step process:
+
+1. Find matching subgraphs based on the plugin's graph pattern (pattern.py) and list the potential substitutions in a user-editable intermediate file (config.yaml)
+2. Review and edit (if necessary) the list of potential substitutions (config.yaml)
+3. Replace subgraphs with plugins based on the list of potential substitutions (config.yaml)
+
+`original.onnx` -------> `match` -------> `config.yaml` -------> `replace` -------> `replaced.onnx`
+`plugins` ----------------^ `usr input`---^ `plugins`--------^
+
+## Details
+
+### Match
+
+Finding matchings subgraphs in a model is done based on a graph pattern description (`pattern.py`) provided by the plugins.
+The graph pattern description (`pattern.py`) contains information about the topology and additional constraints for the graph nodes, and a way to calculate the plugin's attributes based on the matching subgraph.
+Only plugins which provide a graph pattern description (pattern.py) are considered for matching.
+
+The result of the matching is stored in an intermediate file called `config.yaml`.
+The user should review and edit this file, as it serves as a TODO list for the replacement step. For example, if there are 2 matching subgraphs, but only one should be substituted, the result can be removed from the file.
+
+As a preview/dry-run step, the `plugin list` subtool can show the list of potential substitutions without generating an intermediate file.
+
+### Replace
+
+Replacement of subgraphs with plugins uses the `config.yaml` file generated in the matching stage. Any matching subgraph listed in this file is going to be removed and replaced with a single node representing the plugin. The original file is kept, and a new file is saved where the replacements are done. This file by default is called `replaced.onnx`.
+
+### Compare
+
+The original and the replaced model can be compared to check if they behave the same way before and after plugin substitution:
+`polygraphy run original.onnx --trt --save-outputs model_output.json`
+`polygraphy run replaced.onnx --trt --load-outputs model_output.json`
+
+## Running The Example
+
+1. Find and save matches of toyPlugin in the example network:
+
+   ```bash
+   polygraphy plugin match toy_subgraph.onnx \
+       --plugin-dir ./plugins -o config.yaml
+   ```
+
+   <!-- Polygraphy Test: Ignore Start -->
+
+   This will display something like:
+
+   ```
+   checking toyPlugin in model
+   [I] Start a subgraph matching...
+   [I] 	Checking node: n1 against pattern node: Anode.
+   [I] 	No match because: Op did not match. Node op was: O but pattern op was: A.
+   [I] Start a subgraph matching...
+   [I] Found a matched subgraph!
+   [I] Start a subgraph matching...
+   ```
+
+   The resulting config.yaml will look like:
+
+   ```
+   name: toyPlugin
+   instances:
+   - inputs:
+   - i1
+   - i1
+   outputs:
+   - o1
+   - o2
+   attributes:
+       x: 1
+   ```
+
+   <!-- Polygraphy Test: Ignore End -->
+
+2. **[Optional]** List matches of toyPlugin in the example network, without saving config.yaml:
+
+   ```bash
+   polygraphy plugin list toy_subgraph.onnx \
+       --plugin-dir ./plugins
+   ```
+
+   <!-- Polygraphy Test: Ignore Start -->
+
+   This will display something like:
+
+   ```
+   checking toyPlugin in model
+   [I] Start a subgraph matching...
+   [I] 	Checking node: n1 against pattern node: Anode.
+   [I] 	No match because: Op did not match. Node op was: O but pattern op was: A.
+   [I] Start a subgraph matching...
+   ...
+   [I] Found a matched subgraph!
+   [I] Start a subgraph matching...
+   [I] 	Checking node: n6 against pattern node: Anode.
+   [I] 	No match because: Op did not match. Node op was: E but pattern op was: A.
+   the following plugins would be used:
+   {'toyPlugin': 1}
+   ```
+
+   There will be no resulting config.yaml, as this command is only for printing the number of matches per plugin
+   <!-- Polygraphy Test: Ignore End -->
+
+The `plugin replace` subtool replaces subgraphs in an onnx model with plugins
+
+3. Replace parts of the example network with toyPlugin:
+
+   ```bash
+   polygraphy plugin replace toy_subgraph.onnx \
+       --plugin-dir ./plugins --config config.yaml -o replaced.onnx
+   ```
+
+   <!-- Polygraphy Test: Ignore Start -->
+
+   This will display something like:
+
+   ```
+   [I] Loading model: toy_subgraph.onnx
+   ```
+
+   The result file is replaced.onnx, where a subgraph in the example network is replaced by toyPlugin
+   <!-- Polygraphy Test: Ignore End -->
@@ -0,0 +1,48 @@
+from polygraphy import mod
+gs = mod.lazy_import("onnx_graphsurgeon>=0.5.0")
+from typing import List,Dict
+
+def get_plugin_pattern():
+    """
+    Toy plugin pattern:
+        A     B
+        \   /
+          C, attrs['x'] < 2.0
+        /   \
+        D     E
+    """
+    pattern = gs.GraphPattern()
+    in_0 = pattern.variable()
+    in_1 = pattern.variable()
+    a_out = pattern.add("Anode", "A", inputs=[in_0])
+    b_out = pattern.add("Bnode", "B", inputs=[in_1])
+    check_function = lambda node : node.attrs["x"] < 2.0
+    c_out = pattern.add("Cnode", "C", inputs=[a_out, b_out], check_func=check_function)
+    d_out = pattern.add("Dnode", "D", inputs=[c_out])
+    e_out = pattern.add("Enode", "E", inputs=[c_out])
+    pattern.set_output_tensors([d_out, e_out])
+
+    return pattern
+
+def get_matching_subgraphs(graph) -> List[Dict[str,str]]:
+    gp = get_plugin_pattern()
+    matches = gp.match_all(graph)
+    ans = []
+    for m in matches:
+        # save the input and output tensor names of the matching subgraph(s)
+        input_tensors = list(set([ip_tensor.name for ip_tensor in m.inputs]))
+        output_tensors = list(set([op_tensor.name for op_tensor in m.outputs]))
+
+        attrs = {"ToyX": int(m.get("Cnode").attrs["x"]) * 2}
+        ioa = {
+            'inputs':input_tensors,
+            'outputs':output_tensors,
+            'attributes':attrs
+        }
+        ans.append(ioa)
+    return ans
+
+def get_plugin_metadata() -> Dict[str,str]:
+    return {'name':'toyPlugin',
+            'op':'CustomToyPlugin',
+            }
@@ -0,0 +1,153 @@
+# Comparing Frameworks
+
+## Introduction
+
+You can use the `run` subtool to compare a model between different frameworks.
+In the simplest case, you can supply a model, and one or more framework flags.
+By default, it will generate synthetic input data, run inference using the
+specified frameworks, then compare outputs of the specified frameworks.
+
+## Running The Example
+
+In this example, we'll outline various common use-cases for the `run` subtool:
+
+- [Comparing TensorRT And ONNX-Runtime Outputs](#comparing-tensorrt-and-onnx-runtime-outputs)
+- [Comparing TensorRT Precisions](#comparing-tensorrt-precisions)
+- [Changing Tolerances](#changing-tolerances)
+- [Changing Comparison Metrics](#changing-comparison-metrics)
+- [Comparing Per-Layer Outputs Between ONNX-Runtime And TensorRT](#comparing-per-layer-outputs-between-onnx-runtime-and-tensorrt)
+
+### Comparing TensorRT And ONNX-Runtime Outputs
+
+To run the model in Polygraphy with both frameworks and perform an output
+comparison:
+
+```bash
+polygraphy run dynamic_identity.onnx --trt --onnxrt
+```
+
+The `dynamic_identity.onnx` model has dynamic input shapes. By default,
+Polygraphy will override any dynamic input dimensions in the model to
+`constants.DEFAULT_SHAPE_VALUE` (defined as `1`) and warn you:
+
+<!-- Polygraphy Test: Ignore Start -->
+```
+[W]     Input tensor: X (dtype=DataType.FLOAT, shape=(1, 2, -1, -1)) | No shapes provided; Will use shape: [1, 2, 1, 1] for min/opt/max in profile.
+[W]     This will cause the tensor to have a static shape. If this is incorrect, please set the range of shapes for this input tensor.
+```
+<!-- Polygraphy Test: Ignore End -->
+
+In order to suppress this message and explicitly provide input shapes to
+Polygraphy, use the `--input-shapes` option:
+
+```
+polygraphy run dynamic_identity.onnx --trt --onnxrt \
+    --input-shapes X:[1,2,4,4]
+```
+
+### Comparing TensorRT Precisions
+
+To build a TensorRT engine with reduced precision layers for comparison against
+ONNXRT, use one of the supported precision flags (e.g. `--tf32`, `--fp16`,`--int8`, etc.).
+For example:
+
+```bash
+polygraphy run dynamic_identity.onnx --trt --fp16 --onnxrt \
+    --input-shapes X:[1,2,4,4]
+```
+
+> :warning: Getting acceptable accuracy with INT8 precision typically requires an additional calibration step:
+  see the [developer guide](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#working-with-int8)
+  and instructions on [how to do calibration](../../../../examples/cli/convert/01_int8_calibration_in_tensorrt)
+  with Polygraphy on the command line.
+
+### Changing Tolerances
+
+The default tolerances used by `run` are usually appropriate for FP32 precision
+but may not be appropriate for reduced precisions. In order to relax tolerances,
+you can use the `--atol` and `--rtol` options to set absolute and relative
+tolerance respectively.
+
+### Changing Comparison Metrics
+
+You can use the `--check-error-stat` option to change the metric used for
+comparison. By default, Polygraphy uses an "elementwise" metric
+(`--check-error-stat elemwise`).
+
+Other possible metrics for `--check-error-stat` are `mean`, `median`, and `max`, which
+compares the mean, median, and maximum absolute/relative error across the tensor, respectively.
+
+To better understand this, suppose we are
+comparing two outputs `out0` and `out1`. Polygraphy takes
+the elementwise absolute and relative difference of these tensors:
+
+<!-- Polygraphy Test: Ignore Start -->
+```
+absdiff = out0 - out1
+reldiff = absdiff / abs(out1)
+```
+<!-- Polygraphy Test: Ignore End -->
+
+Then, for each index `i` in the output, Polygraphy checks whether
+`absdiff[i] > atol and reldiff[i] > rtol`. If any index satisfies this,
+then the comparison will fail.  This is less stringent than comparing the maximum
+absolute and relative error across the entire tensor (`--check-error-stat max`) since if
+*different* indices `i` and `j` satisfy `absdiff[i] > atol` and `reldiff[j] > rtol`,
+then the `max` comparison will fail but the `elemwise` comparison may
+pass.
+
+Putting it all together, the below example runs a `median` comparison between
+TensorRT using FP16 and ONNX-Runtime, using absolute and relative tolerances of `0.001`:
+
+```bash
+polygraphy run dynamic_identity.onnx --trt --fp16 --onnxrt \
+    --input-shapes X:[1,2,4,4] \
+    --atol 0.001 --rtol 0.001 --check-error-stat median
+```
+
+> You can also specify per-output values for `--atol`/`--rtol`/`--check-error-stat`.
+  See the help output of the `run` subtool for more information.
+
+### Comparing Per-Layer Outputs Between ONNX-Runtime And TensorRT
+
+When network outputs do not match, it can be useful to compare per-layer outputs
+to see where the error is introduced. To do so, you can use the `--trt-outputs`
+and `--onnx-outputs` options respectively. These options accept one or more
+output names as their arguments. The special value `mark all` indicates that all
+tensors in the model should be compared:
+
+```bash
+ polygraphy run dynamic_identity.onnx --trt --onnxrt \
+     --trt-outputs mark all \
+     --onnx-outputs mark all
+```
+
+To find the first mismatched output more easily, you can use the `--fail-fast`
+option which will cause the tool to exit after the first mismatch between
+outputs.
+
+Note that use of `--trt-outputs mark all` can sometimes perturb the generated
+engine due to differences in timing, layer fusion choices, and format
+constraints, which can hide the failure.  In that case, you may have to use a
+more sophisticated approach to bisect the failing model and generate a reduced
+test case that reproduces the error. See [Reducing Failing ONNX
+Models](../../../../examples/cli/debug/02_reducing_failing_onnx_models) for a tutorial on
+how to do this with Polygraphy.
+
+## Further Reading
+
+* In some cases you may need to do comparisons across multiple Polygraphy runs
+  (for example, when comparing the output of a pre-built TensorRT engine or
+  [Polygraphy network script](../../../../examples/cli/run/04_defining_a_tensorrt_network_or_config_manually)
+  against ONNX-Runtime). See [Comparing Across Runs](../../../../examples/cli/run/02_comparing_across_runs) for a tutorial on how to
+  accomplish this.
+
+* For more details on working with dynamic shapes in TensorRT:
+  * See [Dynamic Shapes in TensorRT](../../../../examples/cli/convert/03_dynamic_shapes_in_tensorrt/) for how to specify
+    optimization profiles for use with the engine using the Polygraphy CLI
+  * See [TensorRT and Dynamic Shapes](../../../../examples/api/07_tensorrt_and_dynamic_shapes/) for details on
+    how to do this with the Polygraphy API
+
+* For details on how to supply real input data, see [Comparing with Custom Input Data](../05_comparing_with_custom_input_data/).
+
+* See [Debugging TensorRT Accuracy Issues](../../../../how-to/debug_accuracy.md) for a broader tutorial on how to debug accuracy failures using Polygraphy.
@@ -0,0 +1,19 @@
+:·
+
+Xintermediate"Identity
+
+intermediateY"Identityonnx_dynamic_identityZ&
+X!
+
+
+
+height
+widthb	
+Y
+j1
+intermediate!
+
+
+
+height
+widthB
@@ -0,0 +1,72 @@
+# Comparing Across Runs
+
+## Prerequisites
+For a general overview of how to use `polygraphy run` to compare the outputs of
+different frameworks, see the example on [Comparing Frameworks](../../../../examples/cli/run/01_comparing_frameworks).
+
+## Introduction
+
+There are situations where you may need to compare results across different invocations
+of the `polygraphy run` command.  Some examples of this include:
+
+* Comparing results across different platforms
+* Comparing results across different versions of TensorRT
+* Comparing different model types with compatible input(s)/output(s)
+
+In this example, we'll demonstrate how to accomplish this with Polygraphy.
+
+## Running The Example
+
+### Comparing Across Runs
+
+1. Save the input and output values from the first run:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt \
+        --save-inputs inputs.json --save-outputs run_0_outputs.json
+    ```
+
+2. Run the model again, this time loading the saved inputs and outputs from
+    the first run.  The saved inputs will be used as inputs for the current run, and
+    the saved outputs will be used to compare against the first run.
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt \
+        --load-inputs inputs.json --load-outputs run_0_outputs.json
+    ```
+
+    The `--atol/--rtol/--check-error-stat` options all work the same as in the
+    [Comparing Frameworks](../../../../examples/cli/run/01_comparing_frameworks) example:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt \
+        --load-inputs inputs.json --load-outputs run_0_outputs.json \
+        --atol 0.001 --rtol 0.001 --check-error-stat median
+    ```
+
+### Comparing Different Models
+
+We can also use this technique to compare different models, like TensorRT engines
+and ONNX modles (if they have matching outputs).
+
+1. Convert the ONNX model to a TensorRT engine and save it to disk:
+
+    ```bash
+    polygraphy convert identity.onnx -o identity.engine
+    ```
+
+2. Run the saved engine in Polygraphy, using the saved inputs from the ONNX-Runtime run as
+    inputs to the engine, and compare the engine's outputs to the saved ONNX-Runtime outputs:
+
+    ```bash
+    polygraphy run --trt identity.engine --model-type=engine \
+        --load-inputs inputs.json --load-outputs run_0_outputs.json
+    ```
+
+
+## Further Reading
+
+For details on how to access and work with the saved outputs
+using the Python API, refer to [API example 08](../../../api/08_working_with_run_results_and_saved_inputs_manually/).
+
+For information on comparing against custom outputs, refer to [`run` example 06](../06_comparing_with_custom_output_data/).
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,27 @@
+
+# Generating A Script For Advanced Comparisons
+
+
+## Introduction
+
+For more advanced requirements, you may want to use the [API](../../../../polygraphy).
+Instead of writing a script from scratch, you can use `run`'s `--gen-script` option
+to create a Python script that you can use as a starting point.
+
+
+## Running The Example
+
+1. Generate a comparison script:
+
+    ```bash
+    polygraphy run identity.onnx --trt --onnxrt \
+        --gen-script=compare_trt_onnxrt.py
+    ```
+
+    The generated script will do exactly what the `run` command would otherwise do.
+
+2. Run the comparison script, optionally after modifying it:
+
+    ```bash
+    python3 compare_trt_onnxrt.py
+    ```
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,71 @@
+# Defining A TensorRT Network Or Config Manually
+
+
+## Introduction
+
+In some cases, it can be useful to define a TensorRT network from scratch using the Python API,
+or modify a network created by other means (e.g. a parser). Normally, this would restrict you
+from using CLI tools, at least until you build an engine, since the network cannot be serialized
+to disk and loaded on the command-line.
+
+Polygraphy CLI tools provide a work-around for this - if your Python script defines a function
+named `load_network`, which takes no parameters and returns a TensorRT builder, network,
+and optionally parser, then you can provide your Python script in place of a model argument.
+
+Similarly, we can create a custom TensorRT builder configuration using a script that defines
+a function called `load_config` which accepts a builder and network and returns a builder configuration.
+
+In this example, the included `define_network.py` script parses an ONNX model and appends an identity
+layer to it. Since it returns the builder, network, and parser in a function called `load_network`,
+we can build and run a TensorRT engine from it using just a single command. The `create_config.py`
+script creates a new TensorRT builder configuration and enables FP16 mode.
+
+
+### TIP: Generating Script Templates Automatically
+
+Instead of writing the network script from scratch, you can use
+`polygraphy template trt-network` to give you a starting point:
+
+```bash
+polygraphy template trt-network -o my_define_network.py
+```
+
+If you want to start from a model and modify the resulting TensorRT network instead
+of creating one from scratch, simply provide the model as an argument to `template trt-network`:
+
+```bash
+polygraphy template trt-network identity.onnx -o my_define_network.py
+```
+
+Similarly, you can generate a template script for the config using `polygraphy template trt-config`:
+
+```bash
+polygraphy template trt-config -o my_create_config.py
+```
+
+You can also specify builder configuration options to pre-populate the script.
+For example, to enable FP16 mode:
+
+```bash
+polygraphy template trt-config --fp16 -o my_create_config.py
+```
+
+
+## Running The Example
+
+1. Run the network defined in `define_network.py`:
+
+    ```bash
+    polygraphy run --trt define_network.py --model-type=trt-network-script
+    ```
+
+2. Run the network from step (1) using the builder configuration defined in `create_config.py`:
+
+    ```bash
+    polygraphy run --trt define_network.py --model-type=trt-network-script --trt-config-script=create_config.py
+    ```
+
+    Note that we could have defined both `load_network` and `load_config` in the same script.
+    In fact, we could have retrieved these functions from arbitrary scripts, or even modules.
+
+*TIP: We can use the same approach with `polygraphy convert` to build, but not run, the engine.*
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+Creates a TensorRT builder configuration and enables FP16 tactics.
+"""
+import tensorrt as trt
+from polygraphy import func
+from polygraphy.backend.trt import CreateConfig
+
+
+# If we define a function called `load_config`, polygraphy can use it to
+# create the builder configuration.
+#
+# TIP: If our function isn't called `load_config`, we can explicitly specify
+# the name with the script argument, separated by a colon. For example: `create_config.py:my_func`.
+@func.extend(CreateConfig())
+def load_config(config):
+    # NOTE: func.extend() causes the signature of this function to be `(builder, network) -> config`
+    # For details on how this works, see examples/api/03_interoperating_with_tensorrt
+
+    config.set_flag(trt.BuilderFlag.FP16)
+
+    # Notice that we don't need to return anything - `extend()` takes care of that for us!
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+Parses an ONNX model, and then extends it with an Identity layer.
+"""
+from polygraphy import func
+from polygraphy.backend.trt import NetworkFromOnnxPath
+
+parse_onnx = NetworkFromOnnxPath("identity.onnx")
+
+
+# If we define a function called `load_network`, polygraphy can
+# use it directly in place of using a model file.
+#
+# TIP: If our function isn't called `load_network`, we can explicitly specify
+# the name with the model argument, separated by a colon. For example, `define_network.py:my_func`.
+@func.extend(parse_onnx)
+def load_network(builder, network, parser):
+    # NOTE: func.extend() causes the signature of this function to be `() -> (builder, network, parser)`
+    # For details on how this works, see examples/api/03_interoperating_with_tensorrt
+
+    # Append an identity layer to the network
+    prev_output = network.get_output(0)
+    network.unmark_output(prev_output)
+
+    output = network.add_identity(prev_output).get_output(0)
+    network.mark_output(output)
+
+    # Notice that we don't need to return anything - `extend()` takes care of that for us!
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,56 @@
+# Comparing With Custom Input Data
+
+## Introduction
+
+In some cases, we may want to run comparisons using custom input data.
+Polygraphy provides multiple ways to do so, which are detailed [here](../../../../how-to/use_custom_input_data.md).
+
+In this example, we'll demonstrate 2 different approaches:
+
+1. Using a data loader script by defining a `load_data()` function in a Python script (`data_loader.py`).
+    Polygraphy will use `load_data()` to generate inputs at runtime.
+
+2. Using a JSON file containing pre-generated inputs.
+    For convenience, we'll use our script from above (`data_loader.py`) to save the inputs
+    generated by `load_data()` to a file called `custom_inputs.json`.
+
+*TIP: Generally, a data loader script is preferrable when working with large amounts of input data*
+    *as it avoids the need to write to the disk.*
+    *On the other hand, JSON files may be more portable and can help ensure reproducibility.*
+
+Finally, we'll supply our custom input data to `polygraphy run` and compare outputs between
+ONNX-Runtime and TensorRT.
+
+Since our model has dynamic shapes, we'll need to set up a TensorRT Optimization Profile.
+For details on how we can do this via the command-line,
+see [`convert` example 03](../../convert/03_dynamic_shapes_in_tensorrt).
+For simplicitly, we'll create a profile where `min` == `opt` == `max`.
+
+*NOTE: It is important that our optimization profile works with the shapes provided by our*
+    *custom data loader. In our very simple case, the data loader always generates inputs of*
+    *shape (1, 2, 28, 28), so we just need to ensure this falls within [`min`, `max`].*
+
+## Running The Example
+
+1. Run the script to save input data to the disk.
+    *NOTE: This is only necessary for option 2.*
+    ```bash
+    python3 data_loader.py
+    ```
+
+2. Run the model with TensorRT and ONNX-Runtime using custom input data:
+    - Option 1: Using the data loader script:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt --onnxrt \
+        --trt-min-shapes X:[1,2,28,28] --trt-opt-shapes X:[1,2,28,28] --trt-max-shapes X:[1,2,28,28] \
+        --data-loader-script data_loader.py
+    ```
+
+    - Option 2: Using the JSON file containing the saved inputs:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt --onnxrt \
+        --trt-min-shapes X:[1,2,28,28] --trt-opt-shapes X:[1,2,28,28] --trt-max-shapes X:[1,2,28,28] \
+        --load-inputs custom_inputs.json
+    ```
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Demonstrates two methods of loading custom input data in Polygraphy:
+
+Option 1: Defines a `load_data` function that returns a generator yielding
+    feed_dicts so that this script can be used as the argument for
+    the --data-loader-script command-line parameter.
+
+Option 2: Writes input data to a JSON file that can be used as the argument for
+    the --load-inputs command-line parameter.
+"""
+import numpy as np
+from polygraphy.json import save_json
+
+INPUT_SHAPE = (1, 2, 28, 28)
+
+
+# Option 1: Define a function that will yield feed_dicts (i.e. Dict[str, np.ndarray])
+def load_data():
+    for _ in range(5):
+        yield {
+            "x": np.ones(shape=INPUT_SHAPE, dtype=np.float32)
+        }  # Still totally real data
+
+
+# Option 2: Create a JSON file containing the input data using the `save_json()` helper.
+#   The input to `save_json()` should have type: List[Dict[str, np.ndarray]].
+#   For convenience, we'll reuse our `load_data()` implementation to generate the list.
+input_data = list(load_data())
+save_json(input_data, "custom_inputs.json", description="custom input data")
@@ -0,0 +1,15 @@
+backend_test:y
+
+XY"Identityonnx_dynamic_identityZ&
+X!
+
+
+
+height
+widthb&
+Y!
+
+
+
+height
+widthB
@@ -0,0 +1,43 @@
+# Comparing With Custom Output Data
+
+## Introduction
+
+In some cases, it may be useful to compare against output values generated outside Polygraphy.
+The simplest way to do so is to create a `RunResults` object and save it to a file.
+
+This example illustrates how you can generate custom input and output data outside of Polygraphy
+and seamlessly load it into Polygraphy for comparison.
+
+## Running The Example
+
+1. Generate the input and output data:
+
+    ```bash
+    python3 generate_data.py
+    ```
+
+2. **[Optional]** Inspect the data.
+    For inputs:
+
+    ```bash
+    polygraphy inspect data custom_inputs.json
+    ```
+
+    For outputs:
+
+    ```bash
+    polygraphy inspect data custom_outputs.json
+    ```
+
+3. Run inference with the generated input data and then compare outputs against the custom outputs:
+
+    ```bash
+    polygraphy run identity.onnx --trt \
+        --load-inputs custom_inputs.json \
+        --load-outputs custom_outputs.json
+    ```
+
+## Further Reading
+
+For details on how to access and work with the outputs stored in `RunResults` objects
+using the Python API, refer to [API example 08](../../../api/08_working_with_run_results_and_saved_inputs_manually/).
@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Generates input and output data for an identity model and saves it to disk.
+"""
+import numpy as np
+from polygraphy.comparator import RunResults
+from polygraphy.json import save_json
+
+INPUT_SHAPE = (1, 1, 2, 2)
+
+
+# We'll generate arbitrary input data and then "compute" the expected output data before saving both to disk.
+# In order for Polygraphy to load the input and output data, they must be in the following format:
+#   - Input Data: List[Dict[str, np.ndarray]] (A list of feed_dicts)
+#   - Output Data: RunResults
+
+
+# Generate arbitrary input data compatible with the model.
+#
+# TIP: We could have alternatively used a generator as in `run` example 05 (05_comparing_with_custom_input_data).
+#   In that case, we would simply provide this script to `--data-loader-script` instead of saving the inputs here
+#   and then using `--load-inputs`.
+input_data = {"x": np.ones(shape=INPUT_SHAPE, dtype=np.float32)}
+
+# NOTE: Input data must be in a list (to support multiple sets of inputs), so we create one before saving it.
+#   The `description` argument is optional:
+save_json([input_data], "custom_inputs.json", description="custom input data")
+
+
+# "Compute" the outputs based on the input data. Since this is an identity model, we can just copy the inputs.
+output_data = {"y": input_data["x"]}
+
+# To save output data, we can create a RunResults object:
+custom_outputs = RunResults()
+
+# The `add()` helper function allows us to easily add entries.
+#
+# NOTE: As with input data, output data must be in a list, so we create one before saving it.
+#
+# TIP: Alternatively, we can manually add entries using an approach like:
+#   runner_name = "custom_runner"
+#   custom_outputs[runner_name] = [IterationResult(output_data, runner_name=runner_name), ...]
+#
+# TIP: To store outputs from multiple different implementations, you can specify different `runner_name`s to `add()`.
+#   If `runner_name` is omitted, a default is used.
+custom_outputs.add([output_data], runner_name="custom_runner")
+custom_outputs.save("custom_outputs.json")
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,34 @@
+# Checking for Intermediate NaN or Infinities
+
+## Introduction
+
+When debugging model accuracy issues in Polygraphy, it can be helpful to check layerwise outputs for potential problems. Polygraphy's `run` subtool provides a helpful flag `--validate` which can quickly diagnose problematic intermediate outputs.
+
+This example demonstrates use of this flag with a model which intentionally generates an
+infinite output by adding infinity to the input tensor.
+
+## Running The Example
+
+ <!-- Polygraphy Test: XFAIL Start -->
+```bash
+polygraphy run add_infinity.onnx --onnx-outputs mark all --onnxrt --validate
+```
+ <!-- Polygraphy Test: XFAIL End -->
+
+ <!-- Polygraphy Test: Ignore Start -->
+You should see output like:
+```
+[I] onnxrt-runner-N0-05/13/22-22:35:48  | Completed 1 iteration(s) in 0.1326 ms | Average inference time: 0.1326 ms.
+[I] Output Validation | Runners: ['onnxrt-runner-N0-05/13/22-22:35:48']
+[I]     onnxrt-runner-N0-05/13/22-22:35:48  | Validating output: B (check_inf=True, check_nan=True)
+[I]         mean=inf, std-dev=nan, var=nan, median=inf, min=inf at (0,), max=inf at (0,), avg-magnitude=inf
+[E]         Inf Detected | One or more non-finite values were encountered in this output
+[I]         Note: Use -vv or set logging verbosity to EXTRA_VERBOSE to display non-finite values
+[E]         FAILED | Errors detected in output: B
+[E]     FAILED | Output Validation
+```
+ <!-- Polygraphy Test: Ignore End -->
+
+## See Also
+
+* [Debugging TensorRT Accuracy Issues](../../../../how-to/debug_accuracy.md)
@@ -0,0 +1,138 @@
+# Adding Precision Constraints
+
+## Introduction
+
+When a model trained in FP32 is used to build a TensorRT engine that leverages
+[reduced precision optimizations](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#reduced-precision),
+certain layers in the model may need to be constrained to run in FP32 to
+preserve acceptable accuracy.
+
+The following example demonstrates how to selectively constrain precisions of
+specified layers in a network. The provided ONNX model does the following:
+
+1. Flips its input horizontally by right-multiplying by a 90 degree rotated identity matrix,
+2. Adds `FP16_MAX` to the flipped input, then subtracts `FP16_MAX` from the result,
+3. Flips the output of the subtraction horizontally by right-multiplying by the rotated identity.
+
+If `x` is positive, step (2) in this procedure needs to be done in FP32 in
+order to achieve acceptable accuracy since values will exceed FP16 representable
+range (by design).  However, when FP16 optimizations are enabled without
+constraints, TensorRT, having no knowledge of what range of values will be used
+for `x`, will usually choose to run all steps in this process in FP16:
+
+* The GEMM operations in steps (1) and (3) will run faster in FP16 than in FP32
+    (for large enough problem sizes)
+
+* The pointwise operations in step (2) will run faster in FP16, and leaving the
+    data in FP16 eliminates the need for additional reformats to/from FP32.
+
+Hence, you need to constrain the allowed precisions in the TensorRT network in
+order for TensorRT to make appropriate choices when assigning layer precisions
+in the engine.
+
+Polygraphy command-line tools provide multiple methods of constraining layer precisions:
+
+1. The `--layer-precisions` option allows you to set precisions for individual layers.
+
+2. A network post-processing script allows you to programmatically modify a TensorRT network
+    parsed or otherwise generated by Polygraphy.
+
+3. A network loader script allows you to construct the entire TensorRT network manually using the
+    TensorRT Python API. During network construction, you can set layer precisions as desired.
+
+
+## Running The Example
+
+**Warning:** _This example requires TensorRT 8.4 or later._
+
+### Using The `--layer-precisions` Option
+
+Run the following command to compare running the model with TensorRT using FP16
+optimizations against ONNX-Runtime in FP32:
+
+<!-- Polygraphy Test: XFAIL Start -->
+```bash
+polygraphy run needs_constraints.onnx \
+    --trt --fp16 --onnxrt --val-range x:[1,2] \
+    --layer-precisions Add:float16 Sub:float32 --precision-constraints prefer \
+    --check-error-stat median
+```
+<!-- Polygraphy Test: XFAIL End -->
+
+To increase the chances that this command fails for the reasons outlined above,
+we'll force the `Add` to run in FP16 precision and the subsequent `Sub` to run in FP32.
+This will prevent them from being fused and cause the outputs of `Add` to overflow the FP16 range.
+
+
+### Using a Network Postprocessing Script to Constrain Precisions
+
+Another option is to use a TensorRT network postprocessing script to apply precisions on the parsed network.
+
+Use the provided network postprocessing script [add_constraints.py](./add_constraints.py) to constrain precisions in the model:
+
+```
+polygraphy run needs_constraints.onnx --onnxrt --trt --fp16 --precision-constraints obey \
+    --val-range x:[1,2] --check-error-stat median \
+    --trt-network-postprocess-script ./add_constraints.py
+```
+
+*TIP: You can use `--trt-npps` as shorthand for `--trt-network-postprocess-script`.*
+
+By default Polygraphy looks for a function called `postprocess` in the script to execute.  To specify
+a different function to use, suffix the script name with a colon followed by the function name, e.g.
+
+<!-- Polygraphy Test: Ignore Start -->
+```
+polygraphy run ... --trt-npps my_script.py:custom_func
+```
+<!-- Polygraphy Test: Ignore End -->
+
+
+### Using A Network Loader Script To Constrain Precisions
+
+Alternatively, you can use a network loader script to define the entire network manually,
+as a part of which you can set layer precisions.
+
+The below section assumes you have read through the example on
+[Defining a TensorRT Network or Config Manually](../../../../examples/cli/run/04_defining_a_tensorrt_network_or_config_manually)
+and have a basic understanding of how to use the [TensorRT Python API](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/python-api/index.html).
+
+First, run ONNX-Runtime on the model to generate reference inputs and golden outputs:
+
+```bash
+polygraphy run needs_constraints.onnx --onnxrt --val-range x:[1,2] \
+    --save-inputs inputs.json --save-outputs golden_outputs.json
+```
+
+Next, run the provided network loader script
+[constrained_network.py](./constrained_network.py) which constrains precisions
+in the model, forcing TensorRT to obey constraints, using the saved input and comparing against the saved golden output:
+
+```bash
+polygraphy run constrained_network.py --precision-constraints obey \
+    --trt --fp16 --load-inputs inputs.json --load-outputs golden_outputs.json \
+    --check-error-stat median
+```
+
+Note that TensorRT may choose to run other layers in the network in FP32 besides
+the explicitly constrained layers if doing so would result in higher overall
+engine performance.
+
+**[Optional]**: Run the network script but allow TensorRT to ignore precision
+constraints if necessary. This may be required to run the network if TensorRT
+has no layer implementation that satisfies the requested precision constraints:
+
+```
+polygraphy run constrained_network.py --precision-constraints prefer \
+    --trt --fp16 --load-inputs inputs.json --load-outputs golden_outputs.json \
+    --check-error-stat median
+```
+
+
+## See Also
+
+* [Working with Reduced Precision](../../../../how-to/work_with_reduced_precision.md) for a more general guide on how to debug
+  reduced precision optimizations using Polygraphy.
+* [Defining a TensorRT Network or Config Manually](../../../../examples/cli/run/04_defining_a_tensorrt_network_or_config_manually) for
+  instructions on how to create network script templates.
+* [TensorRT Python API Reference](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/python-api/index.html)
@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Postprocessing script to add precision constraints to a TensorRT network.
+"""
+
+import tensorrt as trt
+
+
+def postprocess(network):
+    """
+    Traverses the parsed network and constrains precisions
+    for specific layers to FP32.
+
+    Args:
+        network (trt.INetworkDefinition): The network to modify.
+
+    Returns:
+        None
+    """
+    for layer in network:
+        # Set computation precision for Add and Sub layer to FP32
+        if layer.name in ("Add", "Sub"):
+            layer.precision = trt.float32
+
+        # Set the output precision for the Add layer to FP32.  Without this,
+        # the intermediate output data of the Add may be stored as FP16 even
+        # though the computation itself is performed in FP32.
+        if layer.name == "Add":
+            layer.set_output_type(0, trt.float32)
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Parses an ONNX model, then adds precision constraints so specific layers run in FP32.
+"""
+
+from polygraphy import func
+from polygraphy.backend.trt import NetworkFromOnnxPath
+import tensorrt as trt
+
+# Load the model, which implements the following network:
+#
+# x -> MatMul (I_rot90) -> Add (FP16_MAX) -> Sub (FP16_MAX) -> MatMul (I_rot90) -> out
+#
+# Without constraining the subgraph (Add -> Sub) to FP32, this model may
+# produce incorrect results when run with FP16 optimziations enabled.
+parse_network_from_onnx = NetworkFromOnnxPath("./needs_constraints.onnx")
+
+
+@func.extend(parse_network_from_onnx)
+def load_network(builder, network, parser):
+    """The below function traverses the parsed network and constrains precisions
+    for specific layers to FP32.
+
+    See examples/cli/run/04_defining_a_tensorrt_network_or_config_manually
+    for more examples using network scripts in Polygraphy.
+    """
+    for layer in network:
+        # Set computation precision for Add and Sub layer to FP32
+        if layer.name in ("Add", "Sub"):
+            layer.precision = trt.float32
+
+        # Set the output precision for the Add layer to FP32.  Without this,
+        # the intermediate output data of the Add may be stored as FP16 even
+        # though the computation itself is performed in FP32.
+        if layer.name == "Add":
+            layer.set_output_type(0, trt.float32)
@@ -0,0 +1,65 @@
+# Using Extract To Isolate A Subgraph
+
+
+## Introduction
+
+The `surgeon extract` subtool can be used to extract a subgraph from a model with a single command.
+
+In this example, we'll extract a subgraph from a model that computes `Y = x0 + (a * x1 + b)`:
+
+![./model.png](./model.png)
+
+Let's assume that we want to isolate the subgraph that computes `(a * x1 + b)`, and that we've
+used `polygraphy inspect model model.onnx --show layers` to determine the names of the input/output tensors
+of this subgraph, but that we don't know the shapes or data types of any of the tensors involved.
+
+When shapes and data types are unknown, you can use `auto` to indicate that Polygraphy should
+attempt to automatically determine these.
+For inputs, we must specify both shape and data type, whereas outputs only require the data
+type - hence `--inputs` requires 2 `auto`s and `--outputs` requires only 1.
+
+
+## Running The Example
+
+1. Extract the subgraph:
+
+    ```bash
+    polygraphy surgeon extract model.onnx \
+        --inputs x1:auto:auto \
+        --outputs add_out:auto \
+        -o subgraph.onnx
+    ```
+
+    If we knew the shapes and/or data types, we could instead write, for example:
+
+    ```bash
+    polygraphy surgeon extract model.onnx \
+        --inputs x1:[1,3,224,224]:float32 \
+        --outputs add_out:float32 \
+        -o subgraph.onnx
+    ```
+
+    The resulting subgraph will look like this:
+
+    ![./subgraph.png](./subgraph.png)
+
+2. **[Optional]** At this point, the model is ready for use. You can use `inspect model`
+    to confirm whether it looks correct:
+
+    ```bash
+    polygraphy inspect model subgraph.onnx --show layers
+    ```
+
+## A Note On `auto`
+
+When `auto` is specified as a shape or data type, Polygraphy relies on ONNX shape
+inference to determine the shapes and data types of intermediate tensors.
+
+In cases where ONNX shape inference cannot determine shapes, Polygraphy
+will run inference on the model using ONNX-Runtime with synthetic input data
+You can control the shape of this input data using the `--model-inputs` argument
+and the contents using the `Data Loader` options.
+
+This will cause the inputs of the resulting subgraph to have fixed shapes. You can change
+these back to dynamic by using the extract command again on the subgraph, and specifying
+the same inputs, but using shapes with dynamic dimensions, e.g. `--inputs identity_out_0:[-1,-1]:auto`
@@ -0,0 +1,41 @@
+# Using Sanitize To Fold Constants
+
+
+## Introduction
+
+The `surgeon sanitize` subtool can be used to fold constants in graphs,
+remove unused nodes, and topologically sort nodes. In cases where shapes
+are statically known, it can also simplify subgraphs involving shape operations.
+
+In this example, we'll fold constants in a graph that computes `output = input + ((a + b) + d)`,
+where `a`, `b`, and `d` are constants:
+
+![./model.png](./model.png)
+
+
+## Running The Example
+
+1. Fold constants with:
+
+    ```bash
+    polygraphy surgeon sanitize model.onnx \
+        --fold-constants \
+        -o folded.onnx
+    ```
+
+    This collapses `a`, `b`, and `d` into a constant tensor, and the resulting graph
+    computes `output = input + e`:
+
+    ![./folded.png](./folded.png)
+
+    *TIP: Sometimes, models include operations like `Tile` or `ConstantOfShape`, that may*
+        *generate large constant tensors. Folding these can bloat the model size*
+        *to an undesirable degree. You can use the `--fold-size-threshold` to control*
+        *the maximum size, in bytes, for which to fold tensors. Any nodes that generate*
+        *tensors over this limit will not be folded, but instead computed at runtime.*
+
+2. **[Optional]** You can use `inspect model` to confirm whether it looks correct:
+
+    ```bash
+    polygraphy inspect model folded.onnx --show layers
+    ```
@@ -0,0 +1,31 @@
+# Modifying Input Shapes
+
+
+## Introduction
+
+The `surgeon sanitize` subtool can be used to modify the input shapes of an ONNX model.
+This does not change the intermediate layers of the model, and as such, may cause issues if
+the model makes assumptions about the input shapes (for example, a `Reshape` node with a hard-coded
+new shape).
+
+Output shapes can be inferred and so these are not modified (nor do they need to be).
+
+*NOTE: Re-exporting the ONNX model with the desired shapes is strongly recommended.*
+    *The method shown here should only be used when doing so is not possible.*
+
+## Running The Example
+
+1. Change the input shape of the model to a shape with a dynamic batch dimension,
+    keeping other dimensions the same:
+
+    ```bash
+    polygraphy surgeon sanitize identity.onnx \
+        --override-input-shapes x:['batch',1,2,2] \
+        -o dynamic_identity.onnx
+    ```
+
+2. **[Optional]** You can use `inspect model` to confirm whether it looks correct:
+
+    ```bash
+    polygraphy inspect model dynamic_identity.onnx --show layers
+    ```
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,56 @@
+# Using Sanitize To Set Upper Bounds For Unbounded Data-Dependent Shapes (DDS)
+
+
+## Introduction
+
+The `surgeon sanitize` subtool can be used to set upper bounds for unbounded Data-Dependent Shapes (DDS).
+When the shape of a tensor depends on the runtime value of another tensor, such shape is called DDS.
+Some DDS has a limited upper bound. For example, the output shape of a `NonZero` operator is a DDS, but its output shape will not exceed the shape of its input.
+While, some other DDS has no upper bound. For example, the output of a `Range` operator has an unbounded DDS when the `limit` input is a runtime tensor.
+Tensors with unbounded DDS are difficult for TensorRT to optimize inference performance and memory usage at builder stage. 
+In the worst case, they can cause TensorRT engine building failures. 
+
+In this example, we'll use polygraphy to set upper bounds for an unbounded DDS in a graph:
+
+![./model.png](./model.png)
+
+
+## Running The Example
+
+1. Run constant folding for the model first:
+
+    ```bash
+    polygraphy surgeon sanitize model.onnx -o folded.onnx --fold-constants
+    ```
+
+    Note that const folding and symbolic shape inference are required for listing unbounded DDS and setting upper bounds.
+
+2. Find tensors with unbounded DDS with:
+
+    ```bash
+    polygraphy inspect model folded.onnx --list-unbounded-dds
+    ```
+
+    Polygraphy will show all tensors with unbounded DDS.
+
+3. Set upper bounds for unbounded DDS with:
+
+    ```bash
+    polygraphy surgeon sanitize folded.onnx --set-unbounded-dds-upper-bound 1000 -o modified.onnx 
+    ```
+
+    Polygraphy will first search all tensors with unbounded DDS. 
+    Then it will insert min operators with the provided upper bound values to limit the DDS tensor size.
+    In this example, a min operator is inserted before the `Range` operator.
+    With the modified model, TensorRT will know that the output shape of the `Range` operator will not exceed 1000.
+    Thus more kernels can be selected for the following layers.
+
+    ![./modified.png](./modified.png)
+
+4. Check that there is no tensors with unbounded DDS now:
+
+    ```bash
+    polygraphy inspect model modified.onnx --list-unbounded-dds
+    ```
+
+    The modified.onnx should contain no unbounded DDS now.