wehub/nvidia--tensorrt
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Docker Image CI / build-ubuntu2004 (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:36:55 +08:00
commit c8a779b1bb
1887 changed files with 3245738 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
samples/trtexec/CMakeLists.txt
+82
View File
@@ -0,0 +1,82 @@
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
add_executable(trtexec trtexec_main.cpp trtexec.cpp)
target_link_libraries(trtexec PRIVATE trt_samples_common)
if (TRT_BUILD_SAMPLES)
add_dependencies(tensorrt_samples trtexec)
endif()
# Windows-specific configuration for long path support
if(MSVC)
# For long path support to work on Windows, users must
# additionally enable a system-level setting.
# Adding manifest as a source file works on CMake 3.4+.
target_sources(trtexec PRIVATE trtexec.manifest)
endif()
installLibraries(
TARGETS trtexec
OPTIONAL
COMPONENT external
EXPORT TensorRT
)
# When statically linked, trtexec requires the plugins library.
# Otherwise it will resolve it via dlopen and doesn't need a link-time dependency.
if(${TRT_BUILD_SAMPLES_LINK_STATIC_TRT})
target_link_libraries(trtexec PRIVATE
$<LINK_LIBRARY:WHOLE_ARCHIVE,tensorrt_plugins_static>
)
endif()
# Change the file name if TRT_WINML variable is set
set(sample_name "trtexec")
if (${TRT_BUILD_WINML})
set(sample_name "tensorrt_rtx")
endif()
set_target_properties(trtexec
PROPERTIES
OUTPUT_NAME ${sample_name}
)
# In this mode, we build an additional binary trtexec_static that always links tensorrt_static.
if(${TRT_BUILD_TRTEXEC_STATIC})
add_executable(trtexec_static trtexec_main.cpp trtexec.cpp)
target_link_libraries(trtexec_static PRIVATE trt_samples_common)
if (TRT_BUILD_SAMPLES)
add_dependencies(tensorrt_samples trtexec_static)
endif()
installLibraries(
TARGETS trtexec_static
OPTIONAL
COMPONENT internal
)
target_link_libraries(trtexec_static PRIVATE
$<LINK_LIBRARY:WHOLE_ARCHIVE,tensorrt_static>
$<LINK_LIBRARY:WHOLE_ARCHIVE,tensorrt_plugins_static>
)
# Link as PIE and the binary will be loaded at a high address.
# Then, avoid conflicts with ASan's shadow memory range.
# MSVC does not support -pie.
if(NOT MSVC)
target_link_options(trtexec_static PRIVATE -pie)
endif()
endif()
samples/trtexec/README.md
+469
View File
@@ -0,0 +1,469 @@
# TensorRT Command-Line Wrapper: trtexec
**Table Of Contents**
- [TensorRT Command-Line Wrapper: trtexec](#tensorrt-command-line-wrapper-trtexec)
- [Description](#description)
- [Building `trtexec`](#building-trtexec)
- [Using `trtexec`](#using-trtexec)
- [Example 1: Profiling a custom layer](#example-1-profiling-a-custom-layer)
- [Example 2: Running a network on DLA](#example-2-running-a-network-on-dla)
- [Example 3: Running an ONNX model with full dimensions and dynamic shapes](#example-3-running-an-onnx-model-with-full-dimensions-and-dynamic-shapes)
- [Example 4: Collecting and printing a timing trace](#example-4-collecting-and-printing-a-timing-trace)
- [Example 5: Tune throughput with multi-streaming](#example-5-tune-throughput-with-multi-streaming)
- [Example 6: Create a strongly typed plan file](#example-6-create-a-strongly-typed-plan-file)
- [Example 7: Global performance tuner](#example-7-global-performance-tuner)
- [7.1: Discovering tunable knobs](#71-discovering-tunable-knobs)
- [7.2: Building one specific configuration](#72-building-one-specific-configuration)
- [7.3: Sweeping a configuration space](#73-sweeping-a-configuration-space)
- [7.4: Choosing a search algorithm](#74-choosing-a-search-algorithm)
- [7.5: Accuracy-aware tuning](#75-accuracy-aware-tuning)
- [7.6: The tuning cache (and resuming)](#76-the-tuning-cache-and-resuming)
- [7.7: Other useful flags](#77-other-useful-flags)
- [7.8: Caveats of tuning](#78-caveats-of-tuning)
- [Tool command line arguments](#tool-command-line-arguments)
- [Additional resources](#additional-resources)
- [License](#license)
- [Changelog](#changelog)
- [Known issues](#known-issues)
## Description
Included in the `samples` directory is a command line wrapper tool, called `trtexec`. `trtexec` is a tool to quickly utilize TensorRT without having to develop your own application. The `trtexec` tool has two main purposes:
- It’s useful for benchmarking networks on random or user-provided input data.
- It’s useful for generating serialized engines from models.
**Benchmarking network** - If you have a model saved as an ONNX file, you can use the `trtexec` tool to test the performance of running inference on your network using TensorRT. The `trtexec` tool has many options for specifying inputs and outputs, iterations for performance timing, precision allowed, and other options.
**Serialized engine generation** - If you generate a saved serialized engine file, you can pull it into another application that runs inference. For example, you can use the [TensorRT Laboratory](https://github.com/NVIDIA/tensorrt-laboratory) to run the engine with multiple execution contexts from multiple threads in a fully pipelined asynchronous way to test parallel inference performance. Also, in INT8 mode, random weights are used.
**Using custom input data** - By default trtexec will run inference with randomly generated inputs. To provide custom inputs for an inference run, trtexec expects a binary file containing the data for each input tensor. It is recommended that this binary file be generated through `numpy`. For example, to create custom data of all ones to an ONNX model with one input named `data` with shape `(1,3,244,244)` and type `FLOAT`:
```
import numpy as np
data = np.ones((1,3,244,244), dtype=np.float32)
data.tofile("data.bin")
```
This binary file can be be loaded by trtexec during inference by using the `--loadInputs` flag:
```
./trtexec --onnx=model.onnx --loadInputs=data:data.bin
```
The name of the input can be optionally wrapped in single quotes to support absolute paths on Windows:
```
.\trtexec.exe --onnx=model.onnx --loadInputs='data':C:\Users\TRT\data.bin
```
## Building `trtexec`
`trtexec` can be used to build engines, using different TensorRT features (see command line arguments), and run inference. `trtexec` also measures and reports execution time and can be used to understand performance and possibly locate bottlenecks.
Compile the sample by following build instructions in [TensorRT README](https://github.com/NVIDIA/TensorRT/).
## Using `trtexec`
`trtexec` can build engines from models in ONNX format.
### Example 1: Profiling a custom layer
You can profile a custom layer, implemented as a [TensorRT plugin](https://github.com/NVIDIA/TensorRT/tree/main/plugin#tensorrt-plugins), by leveraging `trtexec`. Plugins need to be registered in the plugin registry (instance of `IPluginRegistry`) to be visible to TensorRT. `trtexec` will load the TensorRT standard plugin library (`libnvinfer_plugin.so` / `nvinfer_plugin.dll`) that provides plugin support to TensorRT. Checkout the [Non-Zero Plugins Sample](../sampleNonZeroPlugin/) for a quick sample, or the [Plugins section](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#extending) of the TensorRT Developer Guide for a more detailed walkthrough.
Plugins can be used with `trtexec` in the following 2 ways:
<details>
<summary> Using TensorRT-shipped Plugins </summary>
- If you are using TensorRT-shipped plugins (included in `libnvinfer_plugin.so` / `nvinfer_plugin.dll`), no extra steps are required from the user as these plugins are pre-registered with the plugin registry.
</details>
<details>
<summary> Using your own Plugin </summary>
- If you want to define your own plugin and have `trtexec` use it as part of the network, you should define your own _Plugin Shared library_ with specific entry-points recognized by TensorRT. Then, provide the shared plugin library path to `trtexec` using the `--dynamicPlugins` flag.
- More information on Plugin Shared Libraries and how to define them can be seen in the [Plugin Shared Libraries](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#plugin-serialization) section of the [TensorRT Developer Guide](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html).
In summary, there are two methods:
1. The `REGISTER_TENSORRT_PLUGIN` macro can be applied to the plugin creator for each plugin that needs to be statically registered. i.e. Registered at load-time of the plugin library.
2. For dynamic registration, the plugin shared library must expose the below symbols which will be the entry points for TensorRT:
```cpp
extern "C" void setLoggerFinder(ILoggerFinder* finder);
extern "C" IPluginCreatorInterface* const* getCreators(int32_t& nbCreators)
```
In the above, `setLoggerFinder()` should accept a pointer to an `ILoggerFinder`, through which an `ILogger` instance can be retrieved for the purpose of logging inside the library code. `getCreators()` should return an array of plugin creators the library contains. Example implementations of these entry points can be found in [plugin/vc/vfcCommon.cpp](../../plugin/vc/vfcCommon.cpp) and [plugin/vc/vfcCommon.h](../../plugin/vc/vfcCommon.h).
**Note**: Usage of `getPluginCreators` instead of `getCreators` is also valid, but deprecated.
- If the user wants to build a TensorRT engine first and run later, the user has the option to serialize the shared plugin library as part of the engine itself by specifying `--setPluginsToSerialize`. By doing so, the user does not have to specify `--dynamicPlugins` to `trtexec` when running the built engine.
- For more information on these flags, run `./trtexec --help`.
</details>
### Example 2: Running a network on DLA
To run the MNIST network on NVIDIA DLA (Deep Learning Accelerator) using `trtexec` in FP16 mode, issue:
```
./trtexec --onnx=data/mnist/mnist.onnx --useDLACore=1 --fp16 --allowGPUFallback
```
To run the MNIST network on DLA using `trtexec` in INT8 mode, issue:
```
./trtexec --onnx=data/mnist/mnist.onnx --useDLACore=1 --int8 --allowGPUFallback
```
To run the MNIST network on DLA using `trtexec`, issue:
```
./trtexec --onnx=data/mnist/mnist.onnx --useDLACore=0 --fp16 --allowGPUFallback
```
For more information about DLA, see [Working With DLA](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#dla_topic).
### Example 3: Running an ONNX model with full dimensions and dynamic shapes
To run an ONNX model in full-dimensions mode with static input shapes:
```
./trtexec --onnx=model.onnx
```
The following examples assumes an ONNX model with one dynamic input with name `input` and dimensions `[-1, 3, 244, 244]`
To run an ONNX model in full-dimensions mode with an given input shape:
```
./trtexec --onnx=model.onnx --shapes=input:32x3x244x244
```
To benchmark your ONNX model with a range of possible input shapes:
```
./trtexec --onnx=model.onnx --minShapes=input:1x3x244x244 --optShapes=input:16x3x244x244 --maxShapes=input:32x3x244x244 --shapes=input:5x3x244x244
```
### Example 4: Collecting and printing a timing trace
When running, `trtexec` prints the measured performance, but can also export the measurement trace to a json file:
```
./trtexec --onnx=data/mnist/mnist.onnx --exportTimes=trace.json
```
Once the trace is stored in a file, it can be printed using the `tracer.py` utility. This tool prints timestamps and duration of input, compute, and output, in different forms:
```
./tracer.py trace.json
```
Similarly, profiles can also be printed and stored in a json file. The utility `profiler.py` can be used to read and print the profile from a json file.
### Example 5: Tune throughput with multi-streaming
Tuning throughput may require running multiple concurrent streams of execution. This is the case for example when the latency achieved is well within the desired
threshold, and we can increase the throughput, even at the expense of some latency. For example, saving engines with different precisions and assume that both
execute within 2ms, the latency threshold:
```
trtexec --onnx=resnet50.onnx --saveEngine=g1.trt --int8 --skipInference
trtexec --onnx=resnet50.onnx --saveEngine=g2.trt --best --skipInference
```
Now, the saved engines can be tried to find the combination precision/streams below 2 ms that maximizes the throughput:
```
trtexec --loadEngine=g1.trt --streams=2
trtexec --loadEngine=g1.trt --streams=3
trtexec --loadEngine=g1.trt --streams=4
trtexec --loadEngine=g2.trt --streams=2
```
### Example 6: Create a strongly typed plan file
This flag will create a network with the `NetworkDefinitionCreationFlag::kSTRONGLY_TYPED` flag where tensor data types are inferred from network input types
and operator type specification. Use of specific builder precision flags such as `--int8` or `--best` with this option is not allowed.
```
./trtexec --onnx=model.onnx --stronglyTyped
```
### Example 7: Global performance tuner
TensorRT exposes a number of internal builder knobs — heuristics, layer
selections, codegen toggles — that change how an engine is built. A specific
combination of values for these knobs is called a **build route**. Different
routes produce engines with different performance and (occasionally) different
numerical accuracy on the same model.
`trtexec` can drive a tuning loop that sweeps a set of build routes,
benchmarks each, optionally validates accuracy against reference outputs, and
records the best configuration. Any single iteration in a sweep is fully
reproducible by re-running `trtexec` with `--setBuildRoute=<route>` and the
same model — see [7.2](#72-building-one-specific-configuration).
Tuning is not supported on Windows.
This functionality is exposed through a small group of related flags. The
sub-sections below cover each in turn.
#### 7.1: Discovering tunable knobs
`--helpBuildRoute` prints the knob database as JSON. Each entry lists the
knob name, the allowed values, and a default. This is the source of truth
for what you can put inside a `--setBuildRoute` or `--tuneBuildRoutes`
expression.
```
./trtexec --helpBuildRoute
```
Filter to a single knob — the leading dash is optional:
```
./trtexec --helpBuildRoute=match_ragged_mha
./trtexec --helpBuildRoute=-match_ragged_mha
```
If the named knob does not exist, `trtexec` exits with a non-zero status
and a message pointing to the unfiltered listing. `--helpBuildRoute` does
not require `--onnx` and ignores other build flags; `--help` takes
precedence when both are passed.
#### 7.2: Building one specific configuration
`--setBuildRoute=<route>` builds a single engine with a chosen route. The
route is a space-separated list of `-knob=value` tokens (note the leading
dash on each knob):
```
./trtexec --onnx=model.onnx \
--setBuildRoute="-match_ragged_mha=on -copy_ppg=off" \
--saveEngine=model.plan
```
This is the easiest way to reproduce or debug a specific result from a
tuning sweep: take the route reported in the sweep's log for an iteration
of interest, plug it into `--setBuildRoute`, and run again.
#### 7.3: Sweeping a configuration space
`--tuneBuildRoutes=<expr>` runs an autotuning loop over an expression that
describes a set of routes. The expression uses two forms:
- `-knob=[a|b|c]` — variable knob; the loop iterates over each listed value.
- `-knob=fixed` — fixed value pinned across every iteration.
Multiple tokens are space-separated. The expression is typically quoted to
protect the brackets from the shell. `--saveEngine=<path>` is optional —
without it the sweep still benchmarks every route but no engine is written
to disk. `--saveEngine` becomes required when `--loadRefOutputs` is also
set (see [7.5](#75-accuracy-aware-tuning)).
```
./trtexec --onnx=model.onnx \
--tuneBuildRoutes="-match_ragged_mha=[on|off] -copy_ppg=[on|off]" \
--saveEngine=best.plan
```
For long expressions, put them in a file (one token per line) and pass the
file with `--tuneBuildRouteFile`:
```
$ cat routes.txt
-match_ragged_mha=[on|off]
-copy_ppg=[on|off]
$ ./trtexec --onnx=model.onnx --tuneBuildRouteFile=routes.txt --saveEngine=best.plan
```
`--tuneBuildRoutes` and `--tuneBuildRouteFile` are mutually exclusive.
#### 7.4: Choosing a search algorithm
`--tuningSearch=<spec>` controls how the expression is expanded into the
list of routes that the loop will try:
| Value | Behavior |
|---------|----------|
| `fast` | (default) baseline run with each knob at its default, plus one-off variations that change one knob at a time. Linear in the number of variable knobs. |
| `full` | Cartesian product over every variable knob. Exponential — use for small expressions. |
| `mixed` | A `fast` scan first to identify which knob values improve performance, then a full sweep over only those "positive" knobs. A pragmatic middle ground for larger spaces. |
To make the difference concrete, take an expression with three binary
knobs `-A=[on|off] -B=[on|off] -C=[on|off]` and defaults `A=on, B=on, C=on`:
- **`full`** generates **8 routes** — every combination of A × B × C.
- **`fast`** generates **4 routes** — the baseline `A=on B=on C=on`, plus
three one-off variations that flip exactly one knob at a time
(`A=off B=on C=on`, `A=on B=off C=on`, `A=on B=on C=off`).
- **`mixed`** runs the 4 `fast` routes first; whichever knobs improved
performance over the baseline are then explored exhaustively in a
second pass (e.g. if A and C improved, phase 2 sweeps the 4 routes of
A × C with B pinned to its default).
`--dryRun` enumerates the route list and exits without building any engine —
useful for sanity-checking a large expression before paying for it.
`--dryRun` cannot be combined with `--tuningSearch=mixed`.
```
./trtexec --onnx=model.onnx \
--tuneBuildRoutes="-match_ragged_mha=[on|off] -copy_ppg=[on|off]" \
--tuningSearch=full --dryRun
```
#### 7.5: Accuracy-aware tuning
The tuning loop can validate each iteration's output against a reference
(typically a CPU/FP32 capture from another framework) and discard any engine
that drifts beyond a threshold. Combine `--loadInputs`, `--loadRefOutputs`,
and `--accuracyThreshold` with the tuning flags. When `--loadRefOutputs` is
present, `--accuracyThreshold` is required.
```
./trtexec --onnx=model.onnx \
--tuneBuildRoutes="-match_ragged_mha=[on|off]" \
--loadInputs=input:input.bin \
--loadRefOutputs=output:ref_output.bin \
--accuracyThreshold=0.5 \
--saveEngine=best.plan
```
**Multiple input/output pairs.** Tuning often needs to be validated across
several inputs (e.g. different batch sizes or representative samples).
Group each `(input, reference-output)` pair behind `--refPair=N` and pass
multiple pairs; every iteration is validated against all of them.
```
./trtexec --onnx=model.onnx \
--tuneBuildRoutes="-match_ragged_mha=[on|off]" \
--refPair=0 --loadInputs=input:in0.bin --loadRefOutputs=output:ref0.bin \
--refPair=1 --loadInputs=input:in1.bin --loadRefOutputs=output:ref1.bin \
--accuracyThreshold=0.5 --saveEngine=best.plan
```
**Choosing the loss metric.** `--accuracyAlgorithm=<spec>` selects how
the loss is computed per output tensor. All five metrics are non-negative
(lower is better; `0.0` is a perfect match):
| Spec | Metric |
|--------|--------|
| `l0` | (default) fraction of elements outside `atol + rtol · abs(ref)`; tweak with `--atol` / `--rtol`. |
| `l1` | mean absolute error. |
| `l2` | mean squared error. |
| `lInf` | maximum absolute error. |
| `cos` | `1 − cosine_similarity(actual, reference)`. |
```
./trtexec --onnx=model.onnx \
--tuneBuildRoutes="-match_ragged_mha=[on|off]" \
--loadInputs=input:input.bin --loadRefOutputs=output:ref.bin \
--accuracyAlgorithm=l2 --accuracyThreshold=0.01 \
--saveEngine=best.plan
```
Iterations that fail the accuracy check are recorded in the cache but are
excluded from "best engine" selection.
#### 7.6: The tuning cache (and resuming)
`--tuningCacheFile=<path>` writes a JSON record of the sweep: one line
describing the run, followed by one line per completed iteration with the
build route, GPU time, and per-output accuracy loss. This file is both a
human-readable record of the sweep and the input for resume.
```
./trtexec --onnx=model.onnx \
--tuneBuildRoutes="-match_ragged_mha=[on|off] -copy_ppg=[on|off]" \
--tuningCacheFile=tune.jsonl --saveEngine=best.plan
```
If the run is interrupted (Ctrl-C, OOM, timeout), resume it with:
```
./trtexec --continue --tuningCacheFile=tune.jsonl
```
`--continue` accepts **only** `--tuningCacheFile=<path>` — passing
`--onnx`, `--tuneBuildRoutes`, `--accuracyThreshold`, or any other flag
alongside it is rejected. The cache file carries everything needed to
continue the original run, and existing iteration results in it are
kept. The sweep picks up at the next iteration after the last one
already recorded.
#### 7.7: Other useful flags
- `--tuningTimeOut=<seconds>` — stop the loop after N elapsed seconds (the
current iteration finishes first). `-1` (default) disables the timeout.
Useful for capping a large `full` sweep at a deadline.
- `--saveAllEngines` — in addition to the best engine at `--saveEngine=<p>`,
write every iteration's engine to `<p>.iter<N>`. Requires `--saveEngine`.
Disk-heavy; intended for debugging accuracy regressions across iterations.
- `--setBuildRoute=<route>` — see [7.2](#72-building-one-specific-configuration).
Useful for replaying any single iteration from a sweep by hand.
For the complete list with one-line descriptions, run `./trtexec --help`
and look at the **Build Route Tuning Options** section.
#### 7.8: Caveats of tuning
A few things to keep in mind when relying on a tuning result in production:
- **Improvement is opportunistic.** The default build route may already
be the fastest one for the (model, hardware) combination you're tuning
on. Treat any speedup as a bonus, not an expected outcome.
- **An explicit knob value may be overridden during compilation.** Even
when you pin a knob with `--setBuildRoute=<route>`, the compiler is
free to change that value internally if the network requires it. The
engine that `--saveEngine` records is the source of truth — not the
route string. Ship the saved engine when exact behavior matters.
- **Re-running a route doesn't reproduce engine bytes.** Engine builds
are not bit-deterministic; kernel timings vary, the builder breaks
ties accordingly, and the serialized engine reflects those picks.
Re-running with the same `--setBuildRoute` on the same model and the
same machine produces an engine with the same knob choices, but the
bytes may differ slightly. Again — ship the saved engine, not the
route string, when you need the exact engine that won the sweep.
- **Tuner version matters.** Across TensorRT releases, the set of
tunable knobs and their default values can change. A route or cache
produced by one version may reference knobs that no longer exist in
another, and even the default route is not stable across versions.
- **Pick a sensible `--accuracyThreshold`.** A threshold set too tight
will reject every iteration, and the sweep will report no winner. If
you don't have a prior calibration, start loose and ratchet down.
- **Results are model-specific.** The optimal route depends on the
exact ONNX. A different model — or even the same model rebuilt with
different shapes or precision flags — invalidates a previously-saved
result.
- **Results are hardware-specific.** The same model tuned on different
GPU SKUs can pick different "best" routes. Re-tune when you move to a
different target.
- **Re-tune after TensorRT / tuner upgrades.** A previously-tuned
non-default build route is **not** guaranteed to keep its advantage
after an upgrade — and may even regress relative to the new default.
Re-tune whenever the tuner version or TensorRT version changes. Only
performance regressions on the **default** build route are tracked as
TensorRT performance bugs; non-default routes are best-effort.
## Tool command line arguments
To see the full list of available options and their descriptions, issue the `./trtexec --help` command.
**Note:** Specifying the `--safe` parameter turns the safety mode switch `ON`. By default, the `--safe` parameter is not specified; the safety mode switch is `OFF`. The layers and parameters that are contained within the `--safe` subset are restricted if the switch is set to `ON`. The switch is used for prototyping the safety restricted flows until the TensorRT safety runtime is made available. This parameter is required when loading or saving safe engines with the standard TensorRT package. For more information, see the [Working With Automotive Safety section in the TensorRT Developer Guide](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#working_auto_safety).
## Additional resources
The following resources provide more details about `trtexec`:
**Documentation**
- [NVIDIA trtexec](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#trtexec)
- [TensorRT Sample Support Guide](https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html)
- [NVIDIA’s TensorRT Documentation Library](https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/index.html)
# License
For terms and conditions for use, reproduction, and distribution, see the [TensorRT Software License Agreement](https://docs.nvidia.com/deeplearning/sdk/tensorrt-sla/index.html)
documentation.
# Changelog
April 2019
This is the first release of this `README.md` file.
# Known issues
There are no known issues in this sample.
samples/trtexec/prn_utils.py
Executable
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
Simple printing utils
Utils to print traces and profiles in CSV format
"""
from __future__ import print_function
def combineDescriptions(prolog, features, descriptions):
"""Combine features with their descriptions"""
fullDescription = prolog
sep = " "
for feature, description in zip(features, descriptions):
fullDescription += sep + feature + " (" + description + ")"
sep = ", "
return fullDescription
def printHeader(allFeatures, selection, gp=False, count=False):
"""Print table header"""
if gp:
sep = "#"
if count:
sep += "count, "
else:
sep = ""
for feature in allFeatures:
if feature in selection:
print(sep + feature, end="")
sep = ", "
print("")
def printCsv(data, count=False):
"""Print trace in CSV format"""
c = 0
for row in data:
if count:
print(c, end="")
c += 1
sep = ", "
else:
sep = ""
for r in row:
if isinstance(r, str):
print(sep + r, end="")
else:
print("{}{:.6}".format(sep, float(r)), end="")
sep = ", "
print("")
def filterData(data, allFeatures, selection):
"""Drop features not in the given set"""
filteredData = []
for d in data:
row = []
for f in allFeatures:
if f in selection:
if f in d:
row.append(d[f])
else:
row.append("")
filteredData.append(row)
return filteredData
samples/trtexec/profiler.py
Executable
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
Print a trtexec profile from a JSON file
Given a JSON file containing a trtexec profile,
this program prints the profile in CSV table format.
Each row represents a layer in the profile.
The output format can be optionally converted to a
format suitable for GNUPlot.
"""
import sys
import json
import argparse
import prn_utils as pu
allFeatures = ["name", "timeMs", "averageMs", "percentage"]
defaultFeatures = ",".join(allFeatures)
descriptions = ["layer name", "total layer time", "average layer time", "percentage of total time"]
featuresDescription = pu.combineDescriptions("Features are (times in ms):", allFeatures, descriptions)
def hasNames(features):
"""Check if the name is included in the set"""
return "name" in features
def totalData(features, profile):
"""Add row at the bottom with the total"""
accumulator = {}
for f in features:
accumulator[f] = 0
accumulator["name"] = "total"
for row in profile:
for f in features:
if f in row and not f == "name":
accumulator[f] += row[f]
return accumulator
def findAndRemove(profile, name):
"""Find named row in profile and remove"""
for r in range(len(profile)):
if profile[r]["name"] == name:
row = profile[r]
del profile[r]
return row
return None
def refName(name):
"""Add prefix ref to name"""
return "ref" + name[0].capitalize() + name[1:]
def refFeatures(names):
"""Add prefix ref to features names"""
refNames = []
for name in names:
refNames.append(refName(name))
return refNames
def mergeHeaders(features, skipFirst=True):
"""Duplicate feature names for reference and target profile"""
if skipFirst:
return [features[0]] + refFeatures(features[1:]) + features[1:] + ["% difference"]
return refFeatures(features) + features + ["% difference"]
def addReference(row, reference):
"""Add reference results to results dictionary"""
for k, v in reference.items():
if k == "name":
if k in row:
continue
else:
k = refName(k)
row[k] = v
def mergeRow(reference, profile, diff):
"""Merge reference and target profile results into a single row"""
row = {}
if profile:
row = profile
if reference:
addReference(row, reference)
if diff:
row["% difference"] = diff
return row
def alignData(reference, profile, threshold):
"""Align and merge reference and target profiles"""
alignedData = []
for ref in reference:
prof = findAndRemove(profile, ref["name"])
if prof:
diff = (prof["averageMs"] / ref["averageMs"] - 1) * 100
if abs(diff) >= threshold:
alignedData.append(mergeRow(ref, prof, diff))
else:
alignedData.append(mergeRow(ref, None, None))
for prof in profile:
alignedData.append(mergeRow(None, prof, None))
return alignedData
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--features",
metavar="F[,F]*",
default=defaultFeatures,
help="Comma separated list of features to print. " + featuresDescription,
)
parser.add_argument("--total", action="store_true", help="Add total time row.")
parser.add_argument("--gp", action="store_true", help="Print GNUPlot format.")
parser.add_argument("--no-header", action="store_true", help="Omit the header row.")
parser.add_argument("--threshold", metavar="T", default=0.0, type=float, help="Threshold of percentage difference.")
parser.add_argument("--reference", metavar="R", help="Reference profile file name.")
parser.add_argument("name", metavar="filename", help="Profile file.")
args = parser.parse_args()
global allFeatures
features = args.features.split(",")
for f in features:
if not f in allFeatures:
print("Feature {} not recognized".format(f))
return
count = args.gp and not hasNames(features)
profile = None
reference = None
referenceCount = 0
with open(args.name) as f:
profile = json.load(f)
profileCount = profile[0]["count"]
profile = profile[1:]
if args.reference:
with open(args.reference) as f:
reference = json.load(f)
referenceCount = reference[0]["count"]
reference = reference[1:]
allFeatures = mergeHeaders(allFeatures)
features = mergeHeaders(features, hasNames(features))
if not args.no_header:
if reference:
comment = "#" if args.gp else ""
print(comment + "reference count: {} - profile count: {}".format(referenceCount, profileCount))
pu.printHeader(allFeatures, features, args.gp, count)
if reference:
profile = alignData(reference, profile, args.threshold)
if args.total:
profile.append(totalData(allFeatures, profile))
if reference:
total = profile[len(profile) - 1]
total["% difference"] = (total["averageMs"] / total["refAverageMs"] - 1) * 100
profile = pu.filterData(profile, allFeatures, features)
pu.printCsv(profile, count)
if __name__ == "__main__":
sys.exit(main())
samples/trtexec/tracer.py
Executable
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
Print a trtexec timing trace from a JSON file
Given a JSON file containing a trtexec timing trace,
this program prints the trace in CSV table format.
Each row represents an entry point in the trace.
The columns, as indicated by the header, respresent
one of the metric recorded. The output format can
be optionally converted to a format suitable for
GNUPlot.
"""
import sys
import json
import argparse
import prn_utils as pu
timestamps = ["startInMs", "endInMs", "startComputeMs", "endComputeMs", "startOutMs", "endOutMs"]
intervals = ["inMs", "computeMs", "outMs", "latencyMs", "endToEndMs"]
allMetrics = timestamps + intervals
defaultMetrics = ",".join(allMetrics)
descriptions = [
"start input",
"end input",
"start compute",
"end compute",
"start output",
"end output",
"input",
"compute",
"output",
"latency",
"end to end latency",
]
metricsDescription = pu.combineDescriptions("Possible metrics (all in ms) are:", allMetrics, descriptions)
def skipTrace(trace, start):
"""Skip trace entries until start time"""
for t in range(len(trace)):
if trace[t]["startComputeMs"] >= start:
return trace[t:]
return []
def hasTimestamp(metrics):
"""Check if features have at least one timestamp"""
for timestamp in timestamps:
if timestamp in metrics:
return True
return False
def avgData(data, avg, times):
"""Average trace entries (every avg entries)"""
averaged = []
accumulator = []
r = 0
for row in data:
if r == 0:
for m in row:
accumulator.append(m)
else:
for t, m in enumerate(row[times:], start=times):
accumulator[t] += m
r += 1
if r == avg:
for t in range(times, len(row)):
accumulator[t] /= avg
averaged.append(accumulator)
accumulator = []
r = 0
return averaged
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--metrics",
metavar="M[,M]*",
default=defaultMetrics,
help="Comma separated list of metrics to print. " + metricsDescription,
)
parser.add_argument("--avg", metavar="N", type=int, default=1, help="Print average every N records.")
parser.add_argument(
"--start",
metavar="T",
type=float,
default=0,
help="Start trace at time T (drop records with compute start before T ms).",
)
parser.add_argument("--gp", action="store_true", help="Print GNUPlot format.")
parser.add_argument("--no-header", action="store_true", help="Omit the header row.")
parser.add_argument("name", metavar="filename", help="Trace file.")
args = parser.parse_args()
metrics = args.metrics.split(",")
count = args.gp and (not hasTimestamp(metrics) or len(metrics) == 1)
if not args.no_header:
pu.printHeader(allMetrics, metrics, args.gp, count)
with open(args.name) as f:
trace = json.load(f)
if args.start > 0:
trace = skipTrace(trace, args.start)
trace = pu.filterData(trace, allMetrics, metrics)
if args.avg > 1:
trace = avgData(trace, args.avg, hasTimestamp(metrics))
pu.printCsv(trace, count)
if __name__ == "__main__":
sys.exit(main())
samples/trtexec/trtexec.cpp
+1392
View File
File diff suppressed because it is too large Load Diff
samples/trtexec/trtexec.h
+43
View File
@@ -0,0 +1,43 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#ifndef TRTEXEC_H
#define TRTEXEC_H
#include "sampleEngines.h"
namespace sample
{
//! \brief Reusable trtexec implementation, separated from main() so that
//! custom command-line tools can be built on top of trtexec's engine-building
//! and inference workflow.
//! \param postConfigHook Optional callback invoked after builder configuration, before engine build.
//! \return Exit code (0 for success, non-zero for failure)
[[nodiscard]] int trtexecMain(int argc, char** argv, PostConfigCallback const& postConfigHook = nullptr);
//! \brief Tuning-loop driver invoked when --tuneBuildRoutes / --tuneBuildRouteFile / --continue
//! is present in argv. Enumerates build routes per the tuning expression and search algorithm,
//! and for each iteration fork+execs trtexec with `--setBuildRoute=<route> --tuningResultFile=<json>`
//! injected. Each child runs trtexecMain() unmodified, so any iteration is reproducible by hand
//! using its --setBuildRoute argument.
//! \return Exit code (0 if at least one iteration succeeded, non-zero otherwise).
[[nodiscard]] int32_t runTuningLoop(int32_t argc, char** argv);
} // namespace sample
#endif // TRTEXEC_H
samples/trtexec/trtexec.manifest
+26
View File
@@ -0,0 +1,26 @@
<!--
SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
<description>trtexec with long path support</description>
<application xmlns="urn:schemas-microsoft-com:asm.v3">
<windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
<ws2:longPathAware>true</ws2:longPathAware>
</windowsSettings>
</application>
</assembly>
samples/trtexec/trtexec_main.cpp
+33
View File
@@ -0,0 +1,33 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "sampleUtils.h" // for sample::peekArg
#include "trtexec.h"
//! Tuning vs single-run dispatch. The peek is intentionally minimal — a real
//! parse would discard the original argv before runTuningLoop can re-use it
//! for its child fork+execs. Each branch does its own parseArgs.
int main(int argc, char** argv)
{
if (sample::peekArg(argc, argv, "--tuneBuildRoutes")
|| sample::peekArg(argc, argv, "--tuneBuildRouteFile")
|| sample::peekArg(argc, argv, "--continue"))
{
return sample::runTuningLoop(argc, argv);
}
return sample::trtexecMain(argc, argv);
}