wehub/nvidia--tensorrt
1
0
Fork 0
Code Issues Pull Requests Actions 27 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Docker Image CI / build-ubuntu2004 (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:36:55 +08:00
commit c8a779b1bb
1887 changed files with 3245738 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/Polygraphy/examples/cli/debug/01_debugging_flaky_trt_tactics/README.md
+94
View File
@@ -0,0 +1,94 @@
# Debugging Flaky TensorRT Tactics
**IMPORTANT: This example no longer works reliably for newer versions of TensorRT, since they make some**
**tactic choices that are not exposed via the IAlgorithmSelector interface (Deprecated in TensorRT 10.8.**
**Please use editable mode in ITimingCache instead). Thus, the approach outlined below**
**cannot guarantee deterministic engine builds. With TensorRT 8.7 and newer, you can use the**
**tactic timing cache (`--save-timing-cache` and `--load-timing-cache` in Polygraphy) to ensure**
**determinism, but these files are opaque and thus cannot be interpreted by `inspect diff-tactics`**
## Introduction
Sometimes, a tactic in TensorRT may produce incorrect results, or have
otherwise buggy behavior. Since the TensorRT builder relies on timing
tactics, engine builds are non-deterministic, which can make tactic bugs
manifest as flaky/intermittent failures.
One approach to tackling the problem is to run the builder several times,
saving tactic replay files from each run. Once we have a set of known-good and
known-bad tactics, we can compare them to determine which tactic
is likely to be the source of error.
The `debug build` subtool allows you to automate this process.
For more details on how the `debug` tools work, see the help output:
`polygraphy debug -h` and `polygraphy debug build -h`.
## Running The Example
1. Generate golden outputs from ONNX-Runtime:
```bash
polygraphy run identity.onnx --onnxrt \
--save-outputs golden.json
```
2. Use `debug build` to repeatedly build TensorRT engines and compare results against the golden outputs,
saving a tactic replay file each time:
```bash
polygraphy debug build identity.onnx --fp16 --save-tactics replay.json \
--artifacts-dir replays --artifacts replay.json --until=10 \
--check polygraphy run polygraphy_debug.engine --trt --load-outputs golden.json
```
Let's break this down:
- Like other `debug` subtools, `debug build` generates an intermediate artifact each iteration
(`./polygraphy_debug.engine` by default). This artifact in this case is a TensorRT engine.
*TIP: `debug build` supports all the TensorRT builder configuration options supported*
*by other tools, like `convert` or `run`.*
- In order for `debug build` to determine whether each engine fails or passes,
we provide a `--check` command. Since we're looking at a (fake) accuracy issue,
we can use `polygraphy run` to compare the outputs of the engine to our golden values.
*TIP: Like other `debug` subtools, an interactive mode is also supported, which you can*
*use simply by omitting the `--check` argument.*
- Unlike other `debug` subtools, `debug build` has no automatic terminating condition, so we need
to provide the `--until` option so that the tool knows when to stop. This can either be a number
of iterations, or `"good"` or `"bad"`. In the latter case, the tool will stop after finding the
first passing or failing iteration respectively.
- Since we eventually want to compare the good and bad tactic replays, we specify `--save-tactics`
to save tactic replay files from each iteration, then use `--artifacts` to tell `debug build`
to manage them, which involves sorting them into `good` and `bad` subdirectories under the
main artifacts directory, specified with `--artifacts-dir`.
3. Use `inspect diff-tactics` to determine which tactics could be bad:
```bash
polygraphy inspect diff-tactics --dir replays
```
*NOTE: This last step should report that it could not determine potentially bad tactics since*
*our `bad` directory should be empty at this point (please file a TensorRT issue otherwise!):*
<!-- Polygraphy Test: Ignore Start -->
```
[I] Loaded 2 good tactic replays.
[I] Loaded 0 bad tactic replays.
[I] Could not determine potentially bad tactics. Try generating more tactic replay files?
```
<!-- Polygraphy Test: Ignore End -->
## Further Reading
For more information on the `debug` tool, as well as tips and tricks applicable
to all `debug` subtools, see the
[how-to guide for `debug` subtools](../../../../how-to/use_debug_subtools_effectively.md).
tools/Polygraphy/examples/cli/debug/01_debugging_flaky_trt_tactics/identity.onnx
+15
View File
@@ -0,0 +1,15 @@
 backend-test:[

xy"Identity
test_identityZ
x




b
y




tools/Polygraphy/examples/cli/debug/02_reducing_failing_onnx_models/README.md
+155
View File
@@ -0,0 +1,155 @@
# Reducing Failing ONNX Models
## Introduction
When a model fails for any reason (for example, an accuracy issue in TensorRT) it is often
useful to reduce it to the smallest possible subgraph that triggers the failure. That makes
it easier to pinpoint the cause of the failure.
One approach to doing so is to generate successively smaller subgraphs of the original ONNX model.
At each iteration, we can check whether the subgraph works or is still failing; once we have a working
subgraph, we know that the subgraph generated by the previous iteration is the smallest failing
subgraph.
The `debug reduce` subtool allows us to automate this process.
## Running The Example
For the sake of this example, we'll assume our model (`./model.onnx`) has accuracy issues
in TensorRT. Since the model actually does work in TensorRT (please report a bug if not!),
we'll outline the commands that you would normally run followed by commands you can run to
simulate a failure to get a feel for how the tool looks in practice.
Our simulated failures will trigger whenever there's a `Mul` node in the model:
![./model.png](./model.png)
Hence, the final reduced model should contain just the `Mul` node (since the other nodes don't cause a failure).
1. For models that use dynamic input shapes or contain shape operations, freeze the input
shapes and fold shape operations with:
```bash
polygraphy surgeon sanitize model.onnx -o folded.onnx --fold-constants \
--override-input-shapes x0:[1,3,224,224] x1:[1,3,224,224]
```
2. Let's assume ONNX-Runtime gives us correct outputs. We'll start by generating golden
values for every tensor in the network. We'll also save the inputs we use:
```bash
polygraphy run folded.onnx --onnxrt \
--save-inputs inputs.json \
--onnx-outputs mark all --save-outputs layerwise_golden.json
```
Then we'll combine the inputs and layerwise outputs into a single layerwise inputs file
using the `data to-input` subtool (we'll see why this is necessary in the next step):
```bash
polygraphy data to-input inputs.json layerwise_golden.json -o layerwise_inputs.json
```
3. Next, we'll use `debug reduce` in `bisect` mode:
```bash
polygraphy debug reduce folded.onnx -o initial_reduced.onnx --mode=bisect --load-inputs layerwise_inputs.json \
--check polygraphy run polygraphy_debug.onnx --trt \
--load-inputs layerwise_inputs.json --load-outputs layerwise_golden.json
```
Let's break this down:
- Like the other `debug` subtools, `debug reduce` generates an intermediate artifact each iteration
(`./polygraphy_debug.onnx` by default). The artifact in this case is some subgraph of the original ONNX model.
- In order for `debug reduce` to determine whether each subgraph fails or passes,
we provide a `--check` command. Since we're looking into an accuracy issue,
we can use `polygraphy run` to compare against our golden outputs from before.
*TIP: Like other `debug` subtools, an interactive mode is also supported, which you can*
*use simply by omitting the `--check` argument.*
- In the `--check` command, we provide the layerwise inputs via `--load-inputs`, since otherwise, `polygraphy run`
would generate new inputs for the subgraph tensors, which may not match the values those tensors
had when we generated our golden data. An alternative approach is to run the reference implementation
(ONNX-Runtime here) during each iteration of `debug reduce` rather than ahead of time.
- Since we're using non-default input data, we also provide the layerwise inputs via `--load-inputs` directly to the
`debug reduce` command (in addition to providing it to the `--check` command).
This is important in models with multiple parallel branches (*referring to paths in the model rather than control flow*) like:
<!-- Polygraphy Test: Ignore Start -->
```
inp0 inp1
| |
Abs Abs
\ /
Sum
|
out
```
In such cases, `debug reduce` needs to be able to replace one branch with a constant.
To do so, it needs to know the input data you are using so that it can replace it with the correct values.
Though we're using a file here, input data can be provided via any other Polygraphy data loader argument covered in
[the CLI user guide](../../../../how-to/use_custom_input_data.md).
In case you're not sure whether you need to provide a data loader,
`debug reduce` will emit a warning like this when it tries to replace a branch:
```
[W] This model includes multiple branches/paths. In order to continue reducing, one branch needs to be folded away.
Please ensure that you have provided a data loader argument to `debug reduce` if your `--check` command is using a non-default data loader.
Not doing so may result in false negatives!
```
<!-- Polygraphy Test: Ignore End -->
- We specify the `-o` option so that the reduced model will be written to `initial_reduced.onnx`.
**To Simulate A Failure:** We can use `polygraphy inspect model` in conjunction with `--fail-regex` to trigger
a failure whenever the model contains a `Mul` node:
```bash
polygraphy debug reduce folded.onnx -o initial_reduced.onnx --mode=bisect \
--fail-regex "Op: Mul" \
--check polygraphy inspect model polygraphy_debug.onnx --show layers
```
4. **[Optional]** As a sanity check, we can inspect our reduced model to ensure that it does contain the `Mul` node:
```bash
polygraphy inspect model initial_reduced.onnx --show layers
```
5. Since we used `bisect` mode in the previous step, the model may not be as minimal as it could be.
To further refine it, we'll run `debug reduce` again in `linear` mode:
```bash
polygraphy debug reduce initial_reduced.onnx -o final_reduced.onnx --mode=linear --load-inputs layerwise_inputs.json \
--check polygraphy run polygraphy_debug.onnx --trt \
--load-inputs layerwise_inputs.json --load-outputs layerwise_golden.json
```
**To Simulate A Failure:** We'll use the same technique as before:
```bash
polygraphy debug reduce initial_reduced.onnx -o final_reduced.onnx --mode=linear \
--fail-regex "Op: Mul" \
--check polygraphy inspect model polygraphy_debug.onnx --show layers
```
6. **[Optional]** At this stage, `final_reduced.onnx` should contain just the failing node - the `Mul`.
We can verify this with `inspect model`:
```bash
polygraphy inspect model final_reduced.onnx --show layers
```
## Further Reading
- For more details on how the `debug` tools work, see the help output:
`polygraphy debug -h` and `polygraphy debug reduce -h`.
- Also see the [`debug reduce` how-to guide](../../../../how-to/use_debug_reduce_effectively.md)
for more information, tips, and tricks.
tools/Polygraphy/examples/cli/debug/02_reducing_failing_onnx_models/model.onnx
BIN
View File
Binary file not shown.
tools/Polygraphy/examples/cli/debug/02_reducing_failing_onnx_models/model.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 48 KiB