chore: import upstream snapshot with attribution

2026-07-13 13:36:55 +08:00
commit c8a779b1bb
1887 changed files with 3245738 additions and 0 deletions
@@ -0,0 +1,153 @@
+# Comparing Frameworks
+
+## Introduction
+
+You can use the `run` subtool to compare a model between different frameworks.
+In the simplest case, you can supply a model, and one or more framework flags.
+By default, it will generate synthetic input data, run inference using the
+specified frameworks, then compare outputs of the specified frameworks.
+
+## Running The Example
+
+In this example, we'll outline various common use-cases for the `run` subtool:
+
+- [Comparing TensorRT And ONNX-Runtime Outputs](#comparing-tensorrt-and-onnx-runtime-outputs)
+- [Comparing TensorRT Precisions](#comparing-tensorrt-precisions)
+- [Changing Tolerances](#changing-tolerances)
+- [Changing Comparison Metrics](#changing-comparison-metrics)
+- [Comparing Per-Layer Outputs Between ONNX-Runtime And TensorRT](#comparing-per-layer-outputs-between-onnx-runtime-and-tensorrt)
+
+### Comparing TensorRT And ONNX-Runtime Outputs
+
+To run the model in Polygraphy with both frameworks and perform an output
+comparison:
+
+```bash
+polygraphy run dynamic_identity.onnx --trt --onnxrt
+```
+
+The `dynamic_identity.onnx` model has dynamic input shapes. By default,
+Polygraphy will override any dynamic input dimensions in the model to
+`constants.DEFAULT_SHAPE_VALUE` (defined as `1`) and warn you:
+
+<!-- Polygraphy Test: Ignore Start -->
+```
+[W]     Input tensor: X (dtype=DataType.FLOAT, shape=(1, 2, -1, -1)) | No shapes provided; Will use shape: [1, 2, 1, 1] for min/opt/max in profile.
+[W]     This will cause the tensor to have a static shape. If this is incorrect, please set the range of shapes for this input tensor.
+```
+<!-- Polygraphy Test: Ignore End -->
+
+In order to suppress this message and explicitly provide input shapes to
+Polygraphy, use the `--input-shapes` option:
+
+```
+polygraphy run dynamic_identity.onnx --trt --onnxrt \
+    --input-shapes X:[1,2,4,4]
+```
+
+### Comparing TensorRT Precisions
+
+To build a TensorRT engine with reduced precision layers for comparison against
+ONNXRT, use one of the supported precision flags (e.g. `--tf32`, `--fp16`,`--int8`, etc.).
+For example:
+
+```bash
+polygraphy run dynamic_identity.onnx --trt --fp16 --onnxrt \
+    --input-shapes X:[1,2,4,4]
+```
+
+> :warning: Getting acceptable accuracy with INT8 precision typically requires an additional calibration step:
+  see the [developer guide](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#working-with-int8)
+  and instructions on [how to do calibration](../../../../examples/cli/convert/01_int8_calibration_in_tensorrt)
+  with Polygraphy on the command line.
+
+### Changing Tolerances
+
+The default tolerances used by `run` are usually appropriate for FP32 precision
+but may not be appropriate for reduced precisions. In order to relax tolerances,
+you can use the `--atol` and `--rtol` options to set absolute and relative
+tolerance respectively.
+
+### Changing Comparison Metrics
+
+You can use the `--check-error-stat` option to change the metric used for
+comparison. By default, Polygraphy uses an "elementwise" metric
+(`--check-error-stat elemwise`).
+
+Other possible metrics for `--check-error-stat` are `mean`, `median`, and `max`, which
+compares the mean, median, and maximum absolute/relative error across the tensor, respectively.
+
+To better understand this, suppose we are
+comparing two outputs `out0` and `out1`. Polygraphy takes
+the elementwise absolute and relative difference of these tensors:
+
+<!-- Polygraphy Test: Ignore Start -->
+```
+absdiff = out0 - out1
+reldiff = absdiff / abs(out1)
+```
+<!-- Polygraphy Test: Ignore End -->
+
+Then, for each index `i` in the output, Polygraphy checks whether
+`absdiff[i] > atol and reldiff[i] > rtol`. If any index satisfies this,
+then the comparison will fail.  This is less stringent than comparing the maximum
+absolute and relative error across the entire tensor (`--check-error-stat max`) since if
+*different* indices `i` and `j` satisfy `absdiff[i] > atol` and `reldiff[j] > rtol`,
+then the `max` comparison will fail but the `elemwise` comparison may
+pass.
+
+Putting it all together, the below example runs a `median` comparison between
+TensorRT using FP16 and ONNX-Runtime, using absolute and relative tolerances of `0.001`:
+
+```bash
+polygraphy run dynamic_identity.onnx --trt --fp16 --onnxrt \
+    --input-shapes X:[1,2,4,4] \
+    --atol 0.001 --rtol 0.001 --check-error-stat median
+```
+
+> You can also specify per-output values for `--atol`/`--rtol`/`--check-error-stat`.
+  See the help output of the `run` subtool for more information.
+
+### Comparing Per-Layer Outputs Between ONNX-Runtime And TensorRT
+
+When network outputs do not match, it can be useful to compare per-layer outputs
+to see where the error is introduced. To do so, you can use the `--trt-outputs`
+and `--onnx-outputs` options respectively. These options accept one or more
+output names as their arguments. The special value `mark all` indicates that all
+tensors in the model should be compared:
+
+```bash
+ polygraphy run dynamic_identity.onnx --trt --onnxrt \
+     --trt-outputs mark all \
+     --onnx-outputs mark all
+```
+
+To find the first mismatched output more easily, you can use the `--fail-fast`
+option which will cause the tool to exit after the first mismatch between
+outputs.
+
+Note that use of `--trt-outputs mark all` can sometimes perturb the generated
+engine due to differences in timing, layer fusion choices, and format
+constraints, which can hide the failure.  In that case, you may have to use a
+more sophisticated approach to bisect the failing model and generate a reduced
+test case that reproduces the error. See [Reducing Failing ONNX
+Models](../../../../examples/cli/debug/02_reducing_failing_onnx_models) for a tutorial on
+how to do this with Polygraphy.
+
+## Further Reading
+
+* In some cases you may need to do comparisons across multiple Polygraphy runs
+  (for example, when comparing the output of a pre-built TensorRT engine or
+  [Polygraphy network script](../../../../examples/cli/run/04_defining_a_tensorrt_network_or_config_manually)
+  against ONNX-Runtime). See [Comparing Across Runs](../../../../examples/cli/run/02_comparing_across_runs) for a tutorial on how to
+  accomplish this.
+
+* For more details on working with dynamic shapes in TensorRT:
+  * See [Dynamic Shapes in TensorRT](../../../../examples/cli/convert/03_dynamic_shapes_in_tensorrt/) for how to specify
+    optimization profiles for use with the engine using the Polygraphy CLI
+  * See [TensorRT and Dynamic Shapes](../../../../examples/api/07_tensorrt_and_dynamic_shapes/) for details on
+    how to do this with the Polygraphy API
+
+* For details on how to supply real input data, see [Comparing with Custom Input Data](../05_comparing_with_custom_input_data/).
+
+* See [Debugging TensorRT Accuracy Issues](../../../../how-to/debug_accuracy.md) for a broader tutorial on how to debug accuracy failures using Polygraphy.
@@ -0,0 +1,19 @@
+:·
+
+Xintermediate"Identity
+
+intermediateY"Identityonnx_dynamic_identityZ&
+X!
+
+
+
+height
+widthb	
+Y
+j1
+intermediate!
+
+
+
+height
+widthB
@@ -0,0 +1,72 @@
+# Comparing Across Runs
+
+## Prerequisites
+For a general overview of how to use `polygraphy run` to compare the outputs of
+different frameworks, see the example on [Comparing Frameworks](../../../../examples/cli/run/01_comparing_frameworks).
+
+## Introduction
+
+There are situations where you may need to compare results across different invocations
+of the `polygraphy run` command.  Some examples of this include:
+
+* Comparing results across different platforms
+* Comparing results across different versions of TensorRT
+* Comparing different model types with compatible input(s)/output(s)
+
+In this example, we'll demonstrate how to accomplish this with Polygraphy.
+
+## Running The Example
+
+### Comparing Across Runs
+
+1. Save the input and output values from the first run:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt \
+        --save-inputs inputs.json --save-outputs run_0_outputs.json
+    ```
+
+2. Run the model again, this time loading the saved inputs and outputs from
+    the first run.  The saved inputs will be used as inputs for the current run, and
+    the saved outputs will be used to compare against the first run.
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt \
+        --load-inputs inputs.json --load-outputs run_0_outputs.json
+    ```
+
+    The `--atol/--rtol/--check-error-stat` options all work the same as in the
+    [Comparing Frameworks](../../../../examples/cli/run/01_comparing_frameworks) example:
+
+    ```bash
+    polygraphy run identity.onnx --onnxrt \
+        --load-inputs inputs.json --load-outputs run_0_outputs.json \
+        --atol 0.001 --rtol 0.001 --check-error-stat median
+    ```
+
+### Comparing Different Models
+
+We can also use this technique to compare different models, like TensorRT engines
+and ONNX modles (if they have matching outputs).
+
+1. Convert the ONNX model to a TensorRT engine and save it to disk:
+
+    ```bash
+    polygraphy convert identity.onnx -o identity.engine
+    ```
+
+2. Run the saved engine in Polygraphy, using the saved inputs from the ONNX-Runtime run as
+    inputs to the engine, and compare the engine's outputs to the saved ONNX-Runtime outputs:
+
+    ```bash
+    polygraphy run --trt identity.engine --model-type=engine \
+        --load-inputs inputs.json --load-outputs run_0_outputs.json
+    ```
+
+
+## Further Reading
+
+For details on how to access and work with the saved outputs
+using the Python API, refer to [API example 08](../../../api/08_working_with_run_results_and_saved_inputs_manually/).
+
+For information on comparing against custom outputs, refer to [`run` example 06](../06_comparing_with_custom_output_data/).
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,27 @@
+
+# Generating A Script For Advanced Comparisons
+
+
+## Introduction
+
+For more advanced requirements, you may want to use the [API](../../../../polygraphy).
+Instead of writing a script from scratch, you can use `run`'s `--gen-script` option
+to create a Python script that you can use as a starting point.
+
+
+## Running The Example
+
+1. Generate a comparison script:
+
+    ```bash
+    polygraphy run identity.onnx --trt --onnxrt \
+        --gen-script=compare_trt_onnxrt.py
+    ```
+
+    The generated script will do exactly what the `run` command would otherwise do.
+
+2. Run the comparison script, optionally after modifying it:
+
+    ```bash
+    python3 compare_trt_onnxrt.py
+    ```
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,71 @@
+# Defining A TensorRT Network Or Config Manually
+
+
+## Introduction
+
+In some cases, it can be useful to define a TensorRT network from scratch using the Python API,
+or modify a network created by other means (e.g. a parser). Normally, this would restrict you
+from using CLI tools, at least until you build an engine, since the network cannot be serialized
+to disk and loaded on the command-line.
+
+Polygraphy CLI tools provide a work-around for this - if your Python script defines a function
+named `load_network`, which takes no parameters and returns a TensorRT builder, network,
+and optionally parser, then you can provide your Python script in place of a model argument.
+
+Similarly, we can create a custom TensorRT builder configuration using a script that defines
+a function called `load_config` which accepts a builder and network and returns a builder configuration.
+
+In this example, the included `define_network.py` script parses an ONNX model and appends an identity
+layer to it. Since it returns the builder, network, and parser in a function called `load_network`,
+we can build and run a TensorRT engine from it using just a single command. The `create_config.py`
+script creates a new TensorRT builder configuration and enables FP16 mode.
+
+
+### TIP: Generating Script Templates Automatically
+
+Instead of writing the network script from scratch, you can use
+`polygraphy template trt-network` to give you a starting point:
+
+```bash
+polygraphy template trt-network -o my_define_network.py
+```
+
+If you want to start from a model and modify the resulting TensorRT network instead
+of creating one from scratch, simply provide the model as an argument to `template trt-network`:
+
+```bash
+polygraphy template trt-network identity.onnx -o my_define_network.py
+```
+
+Similarly, you can generate a template script for the config using `polygraphy template trt-config`:
+
+```bash
+polygraphy template trt-config -o my_create_config.py
+```
+
+You can also specify builder configuration options to pre-populate the script.
+For example, to enable FP16 mode:
+
+```bash
+polygraphy template trt-config --fp16 -o my_create_config.py
+```
+
+
+## Running The Example
+
+1. Run the network defined in `define_network.py`:
+
+    ```bash
+    polygraphy run --trt define_network.py --model-type=trt-network-script
+    ```
+
+2. Run the network from step (1) using the builder configuration defined in `create_config.py`:
+
+    ```bash
+    polygraphy run --trt define_network.py --model-type=trt-network-script --trt-config-script=create_config.py
+    ```
+
+    Note that we could have defined both `load_network` and `load_config` in the same script.
+    In fact, we could have retrieved these functions from arbitrary scripts, or even modules.
+
+*TIP: We can use the same approach with `polygraphy convert` to build, but not run, the engine.*
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+Creates a TensorRT builder configuration and enables FP16 tactics.
+"""
+import tensorrt as trt
+from polygraphy import func
+from polygraphy.backend.trt import CreateConfig
+
+
+# If we define a function called `load_config`, polygraphy can use it to
+# create the builder configuration.
+#
+# TIP: If our function isn't called `load_config`, we can explicitly specify
+# the name with the script argument, separated by a colon. For example: `create_config.py:my_func`.
+@func.extend(CreateConfig())
+def load_config(config):
+    # NOTE: func.extend() causes the signature of this function to be `(builder, network) -> config`
+    # For details on how this works, see examples/api/03_interoperating_with_tensorrt
+
+    config.set_flag(trt.BuilderFlag.FP16)
+
+    # Notice that we don't need to return anything - `extend()` takes care of that for us!
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""
+Parses an ONNX model, and then extends it with an Identity layer.
+"""
+from polygraphy import func
+from polygraphy.backend.trt import NetworkFromOnnxPath
+
+parse_onnx = NetworkFromOnnxPath("identity.onnx")
+
+
+# If we define a function called `load_network`, polygraphy can
+# use it directly in place of using a model file.
+#
+# TIP: If our function isn't called `load_network`, we can explicitly specify
+# the name with the model argument, separated by a colon. For example, `define_network.py:my_func`.
+@func.extend(parse_onnx)
+def load_network(builder, network, parser):
+    # NOTE: func.extend() causes the signature of this function to be `() -> (builder, network, parser)`
+    # For details on how this works, see examples/api/03_interoperating_with_tensorrt
+
+    # Append an identity layer to the network
+    prev_output = network.get_output(0)
+    network.unmark_output(prev_output)
+
+    output = network.add_identity(prev_output).get_output(0)
+    network.mark_output(output)
+
+    # Notice that we don't need to return anything - `extend()` takes care of that for us!
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,56 @@
+# Comparing With Custom Input Data
+
+## Introduction
+
+In some cases, we may want to run comparisons using custom input data.
+Polygraphy provides multiple ways to do so, which are detailed [here](../../../../how-to/use_custom_input_data.md).
+
+In this example, we'll demonstrate 2 different approaches:
+
+1. Using a data loader script by defining a `load_data()` function in a Python script (`data_loader.py`).
+    Polygraphy will use `load_data()` to generate inputs at runtime.
+
+2. Using a JSON file containing pre-generated inputs.
+    For convenience, we'll use our script from above (`data_loader.py`) to save the inputs
+    generated by `load_data()` to a file called `custom_inputs.json`.
+
+*TIP: Generally, a data loader script is preferrable when working with large amounts of input data*
+    *as it avoids the need to write to the disk.*
+    *On the other hand, JSON files may be more portable and can help ensure reproducibility.*
+
+Finally, we'll supply our custom input data to `polygraphy run` and compare outputs between
+ONNX-Runtime and TensorRT.
+
+Since our model has dynamic shapes, we'll need to set up a TensorRT Optimization Profile.
+For details on how we can do this via the command-line,
+see [`convert` example 03](../../convert/03_dynamic_shapes_in_tensorrt).
+For simplicitly, we'll create a profile where `min` == `opt` == `max`.
+
+*NOTE: It is important that our optimization profile works with the shapes provided by our*
+    *custom data loader. In our very simple case, the data loader always generates inputs of*
+    *shape (1, 2, 28, 28), so we just need to ensure this falls within [`min`, `max`].*
+
+## Running The Example
+
+1. Run the script to save input data to the disk.
+    *NOTE: This is only necessary for option 2.*
+    ```bash
+    python3 data_loader.py
+    ```
+
+2. Run the model with TensorRT and ONNX-Runtime using custom input data:
+    - Option 1: Using the data loader script:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt --onnxrt \
+        --trt-min-shapes X:[1,2,28,28] --trt-opt-shapes X:[1,2,28,28] --trt-max-shapes X:[1,2,28,28] \
+        --data-loader-script data_loader.py
+    ```
+
+    - Option 2: Using the JSON file containing the saved inputs:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt --onnxrt \
+        --trt-min-shapes X:[1,2,28,28] --trt-opt-shapes X:[1,2,28,28] --trt-max-shapes X:[1,2,28,28] \
+        --load-inputs custom_inputs.json
+    ```
@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Demonstrates two methods of loading custom input data in Polygraphy:
+
+Option 1: Defines a `load_data` function that returns a generator yielding
+    feed_dicts so that this script can be used as the argument for
+    the --data-loader-script command-line parameter.
+
+Option 2: Writes input data to a JSON file that can be used as the argument for
+    the --load-inputs command-line parameter.
+"""
+import numpy as np
+from polygraphy.json import save_json
+
+INPUT_SHAPE = (1, 2, 28, 28)
+
+
+# Option 1: Define a function that will yield feed_dicts (i.e. Dict[str, np.ndarray])
+def load_data():
+    for _ in range(5):
+        yield {
+            "x": np.ones(shape=INPUT_SHAPE, dtype=np.float32)
+        }  # Still totally real data
+
+
+# Option 2: Create a JSON file containing the input data using the `save_json()` helper.
+#   The input to `save_json()` should have type: List[Dict[str, np.ndarray]].
+#   For convenience, we'll reuse our `load_data()` implementation to generate the list.
+input_data = list(load_data())
+save_json(input_data, "custom_inputs.json", description="custom input data")
@@ -0,0 +1,15 @@
+backend_test:y
+
+XY"Identityonnx_dynamic_identityZ&
+X!
+
+
+
+height
+widthb&
+Y!
+
+
+
+height
+widthB
@@ -0,0 +1,43 @@
+# Comparing With Custom Output Data
+
+## Introduction
+
+In some cases, it may be useful to compare against output values generated outside Polygraphy.
+The simplest way to do so is to create a `RunResults` object and save it to a file.
+
+This example illustrates how you can generate custom input and output data outside of Polygraphy
+and seamlessly load it into Polygraphy for comparison.
+
+## Running The Example
+
+1. Generate the input and output data:
+
+    ```bash
+    python3 generate_data.py
+    ```
+
+2. **[Optional]** Inspect the data.
+    For inputs:
+
+    ```bash
+    polygraphy inspect data custom_inputs.json
+    ```
+
+    For outputs:
+
+    ```bash
+    polygraphy inspect data custom_outputs.json
+    ```
+
+3. Run inference with the generated input data and then compare outputs against the custom outputs:
+
+    ```bash
+    polygraphy run identity.onnx --trt \
+        --load-inputs custom_inputs.json \
+        --load-outputs custom_outputs.json
+    ```
+
+## Further Reading
+
+For details on how to access and work with the outputs stored in `RunResults` objects
+using the Python API, refer to [API example 08](../../../api/08_working_with_run_results_and_saved_inputs_manually/).
@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Generates input and output data for an identity model and saves it to disk.
+"""
+import numpy as np
+from polygraphy.comparator import RunResults
+from polygraphy.json import save_json
+
+INPUT_SHAPE = (1, 1, 2, 2)
+
+
+# We'll generate arbitrary input data and then "compute" the expected output data before saving both to disk.
+# In order for Polygraphy to load the input and output data, they must be in the following format:
+#   - Input Data: List[Dict[str, np.ndarray]] (A list of feed_dicts)
+#   - Output Data: RunResults
+
+
+# Generate arbitrary input data compatible with the model.
+#
+# TIP: We could have alternatively used a generator as in `run` example 05 (05_comparing_with_custom_input_data).
+#   In that case, we would simply provide this script to `--data-loader-script` instead of saving the inputs here
+#   and then using `--load-inputs`.
+input_data = {"x": np.ones(shape=INPUT_SHAPE, dtype=np.float32)}
+
+# NOTE: Input data must be in a list (to support multiple sets of inputs), so we create one before saving it.
+#   The `description` argument is optional:
+save_json([input_data], "custom_inputs.json", description="custom input data")
+
+
+# "Compute" the outputs based on the input data. Since this is an identity model, we can just copy the inputs.
+output_data = {"y": input_data["x"]}
+
+# To save output data, we can create a RunResults object:
+custom_outputs = RunResults()
+
+# The `add()` helper function allows us to easily add entries.
+#
+# NOTE: As with input data, output data must be in a list, so we create one before saving it.
+#
+# TIP: Alternatively, we can manually add entries using an approach like:
+#   runner_name = "custom_runner"
+#   custom_outputs[runner_name] = [IterationResult(output_data, runner_name=runner_name), ...]
+#
+# TIP: To store outputs from multiple different implementations, you can specify different `runner_name`s to `add()`.
+#   If `runner_name` is omitted, a default is used.
+custom_outputs.add([output_data], runner_name="custom_runner")
+custom_outputs.save("custom_outputs.json")
@@ -0,0 +1,15 @@
+backend-test:[
+
+xy"Identity
+test_identityZ
+x
+
+
+
+
+b
+y
+
+
+
+
@@ -0,0 +1,34 @@
+# Checking for Intermediate NaN or Infinities
+
+## Introduction
+
+When debugging model accuracy issues in Polygraphy, it can be helpful to check layerwise outputs for potential problems. Polygraphy's `run` subtool provides a helpful flag `--validate` which can quickly diagnose problematic intermediate outputs.
+
+This example demonstrates use of this flag with a model which intentionally generates an
+infinite output by adding infinity to the input tensor.
+
+## Running The Example
+
+ <!-- Polygraphy Test: XFAIL Start -->
+```bash
+polygraphy run add_infinity.onnx --onnx-outputs mark all --onnxrt --validate
+```
+ <!-- Polygraphy Test: XFAIL End -->
+
+ <!-- Polygraphy Test: Ignore Start -->
+You should see output like:
+```
+[I] onnxrt-runner-N0-05/13/22-22:35:48  | Completed 1 iteration(s) in 0.1326 ms | Average inference time: 0.1326 ms.
+[I] Output Validation | Runners: ['onnxrt-runner-N0-05/13/22-22:35:48']
+[I]     onnxrt-runner-N0-05/13/22-22:35:48  | Validating output: B (check_inf=True, check_nan=True)
+[I]         mean=inf, std-dev=nan, var=nan, median=inf, min=inf at (0,), max=inf at (0,), avg-magnitude=inf
+[E]         Inf Detected | One or more non-finite values were encountered in this output
+[I]         Note: Use -vv or set logging verbosity to EXTRA_VERBOSE to display non-finite values
+[E]         FAILED | Errors detected in output: B
+[E]     FAILED | Output Validation
+```
+ <!-- Polygraphy Test: Ignore End -->
+
+## See Also
+
+* [Debugging TensorRT Accuracy Issues](../../../../how-to/debug_accuracy.md)
@@ -0,0 +1,138 @@
+# Adding Precision Constraints
+
+## Introduction
+
+When a model trained in FP32 is used to build a TensorRT engine that leverages
+[reduced precision optimizations](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#reduced-precision),
+certain layers in the model may need to be constrained to run in FP32 to
+preserve acceptable accuracy.
+
+The following example demonstrates how to selectively constrain precisions of
+specified layers in a network. The provided ONNX model does the following:
+
+1. Flips its input horizontally by right-multiplying by a 90 degree rotated identity matrix,
+2. Adds `FP16_MAX` to the flipped input, then subtracts `FP16_MAX` from the result,
+3. Flips the output of the subtraction horizontally by right-multiplying by the rotated identity.
+
+If `x` is positive, step (2) in this procedure needs to be done in FP32 in
+order to achieve acceptable accuracy since values will exceed FP16 representable
+range (by design).  However, when FP16 optimizations are enabled without
+constraints, TensorRT, having no knowledge of what range of values will be used
+for `x`, will usually choose to run all steps in this process in FP16:
+
+* The GEMM operations in steps (1) and (3) will run faster in FP16 than in FP32
+    (for large enough problem sizes)
+
+* The pointwise operations in step (2) will run faster in FP16, and leaving the
+    data in FP16 eliminates the need for additional reformats to/from FP32.
+
+Hence, you need to constrain the allowed precisions in the TensorRT network in
+order for TensorRT to make appropriate choices when assigning layer precisions
+in the engine.
+
+Polygraphy command-line tools provide multiple methods of constraining layer precisions:
+
+1. The `--layer-precisions` option allows you to set precisions for individual layers.
+
+2. A network post-processing script allows you to programmatically modify a TensorRT network
+    parsed or otherwise generated by Polygraphy.
+
+3. A network loader script allows you to construct the entire TensorRT network manually using the
+    TensorRT Python API. During network construction, you can set layer precisions as desired.
+
+
+## Running The Example
+
+**Warning:** _This example requires TensorRT 8.4 or later._
+
+### Using The `--layer-precisions` Option
+
+Run the following command to compare running the model with TensorRT using FP16
+optimizations against ONNX-Runtime in FP32:
+
+<!-- Polygraphy Test: XFAIL Start -->
+```bash
+polygraphy run needs_constraints.onnx \
+    --trt --fp16 --onnxrt --val-range x:[1,2] \
+    --layer-precisions Add:float16 Sub:float32 --precision-constraints prefer \
+    --check-error-stat median
+```
+<!-- Polygraphy Test: XFAIL End -->
+
+To increase the chances that this command fails for the reasons outlined above,
+we'll force the `Add` to run in FP16 precision and the subsequent `Sub` to run in FP32.
+This will prevent them from being fused and cause the outputs of `Add` to overflow the FP16 range.
+
+
+### Using a Network Postprocessing Script to Constrain Precisions
+
+Another option is to use a TensorRT network postprocessing script to apply precisions on the parsed network.
+
+Use the provided network postprocessing script [add_constraints.py](./add_constraints.py) to constrain precisions in the model:
+
+```
+polygraphy run needs_constraints.onnx --onnxrt --trt --fp16 --precision-constraints obey \
+    --val-range x:[1,2] --check-error-stat median \
+    --trt-network-postprocess-script ./add_constraints.py
+```
+
+*TIP: You can use `--trt-npps` as shorthand for `--trt-network-postprocess-script`.*
+
+By default Polygraphy looks for a function called `postprocess` in the script to execute.  To specify
+a different function to use, suffix the script name with a colon followed by the function name, e.g.
+
+<!-- Polygraphy Test: Ignore Start -->
+```
+polygraphy run ... --trt-npps my_script.py:custom_func
+```
+<!-- Polygraphy Test: Ignore End -->
+
+
+### Using A Network Loader Script To Constrain Precisions
+
+Alternatively, you can use a network loader script to define the entire network manually,
+as a part of which you can set layer precisions.
+
+The below section assumes you have read through the example on
+[Defining a TensorRT Network or Config Manually](../../../../examples/cli/run/04_defining_a_tensorrt_network_or_config_manually)
+and have a basic understanding of how to use the [TensorRT Python API](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/python-api/index.html).
+
+First, run ONNX-Runtime on the model to generate reference inputs and golden outputs:
+
+```bash
+polygraphy run needs_constraints.onnx --onnxrt --val-range x:[1,2] \
+    --save-inputs inputs.json --save-outputs golden_outputs.json
+```
+
+Next, run the provided network loader script
+[constrained_network.py](./constrained_network.py) which constrains precisions
+in the model, forcing TensorRT to obey constraints, using the saved input and comparing against the saved golden output:
+
+```bash
+polygraphy run constrained_network.py --precision-constraints obey \
+    --trt --fp16 --load-inputs inputs.json --load-outputs golden_outputs.json \
+    --check-error-stat median
+```
+
+Note that TensorRT may choose to run other layers in the network in FP32 besides
+the explicitly constrained layers if doing so would result in higher overall
+engine performance.
+
+**[Optional]**: Run the network script but allow TensorRT to ignore precision
+constraints if necessary. This may be required to run the network if TensorRT
+has no layer implementation that satisfies the requested precision constraints:
+
+```
+polygraphy run constrained_network.py --precision-constraints prefer \
+    --trt --fp16 --load-inputs inputs.json --load-outputs golden_outputs.json \
+    --check-error-stat median
+```
+
+
+## See Also
+
+* [Working with Reduced Precision](../../../../how-to/work_with_reduced_precision.md) for a more general guide on how to debug
+  reduced precision optimizations using Polygraphy.
+* [Defining a TensorRT Network or Config Manually](../../../../examples/cli/run/04_defining_a_tensorrt_network_or_config_manually) for
+  instructions on how to create network script templates.
+* [TensorRT Python API Reference](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/python-api/index.html)
@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Postprocessing script to add precision constraints to a TensorRT network.
+"""
+
+import tensorrt as trt
+
+
+def postprocess(network):
+    """
+    Traverses the parsed network and constrains precisions
+    for specific layers to FP32.
+
+    Args:
+        network (trt.INetworkDefinition): The network to modify.
+
+    Returns:
+        None
+    """
+    for layer in network:
+        # Set computation precision for Add and Sub layer to FP32
+        if layer.name in ("Add", "Sub"):
+            layer.precision = trt.float32
+
+        # Set the output precision for the Add layer to FP32.  Without this,
+        # the intermediate output data of the Add may be stored as FP16 even
+        # though the computation itself is performed in FP32.
+        if layer.name == "Add":
+            layer.set_output_type(0, trt.float32)
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+#
+# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Parses an ONNX model, then adds precision constraints so specific layers run in FP32.
+"""
+
+from polygraphy import func
+from polygraphy.backend.trt import NetworkFromOnnxPath
+import tensorrt as trt
+
+# Load the model, which implements the following network:
+#
+# x -> MatMul (I_rot90) -> Add (FP16_MAX) -> Sub (FP16_MAX) -> MatMul (I_rot90) -> out
+#
+# Without constraining the subgraph (Add -> Sub) to FP32, this model may
+# produce incorrect results when run with FP16 optimziations enabled.
+parse_network_from_onnx = NetworkFromOnnxPath("./needs_constraints.onnx")
+
+
+@func.extend(parse_network_from_onnx)
+def load_network(builder, network, parser):
+    """The below function traverses the parsed network and constrains precisions
+    for specific layers to FP32.
+
+    See examples/cli/run/04_defining_a_tensorrt_network_or_config_manually
+    for more examples using network scripts in Polygraphy.
+    """
+    for layer in network:
+        # Set computation precision for Add and Sub layer to FP32
+        if layer.name in ("Add", "Sub"):
+            layer.precision = trt.float32
+
+        # Set the output precision for the Add layer to FP32.  Without this,
+        # the intermediate output data of the Add may be stored as FP16 even
+        # though the computation itself is performed in FP32.
+        if layer.name == "Add":
+            layer.set_output_type(0, trt.float32)