wehub-resource-sync
222 lines
9.4 KiB
Markdown
222 lines
9.4 KiB
Markdown
# “Hello World” For TensorRT Safety
|
||
|
||
|
||
**Table Of Contents**
|
||
- [Description](#description)
|
||
- [How does this sample work?](#how-does-this-sample-work)
|
||
* [TensorRT API layers and ops](#tensorrt-api-layers-and-ops)
|
||
- [Running the sample](#running-the-sample)
|
||
* [Tool command line arguments](#tool-command-line-arguments)
|
||
* [When to use remoteAutoTuningConfig](#when-to-use-remoteautotuningconfig)
|
||
- [Additional resources](#additional-resources)
|
||
- [License](#license)
|
||
- [Changelog](#changelog)
|
||
- [Known issues](#known-issues)
|
||
|
||
## Description
|
||
|
||
This sample, sampleSafeMNIST, consists of two parts; build and infer. The build part of this sample demonstrates how to use the builder `IBuilderConfig::setEngineCapability()` flag for safety. The inference part of this sample demonstrates how to use the safe graph.
|
||
|
||
The build part builds a safe version of a TensorRT engine and saves it into a binary file, then the infer part loads the prebuilt safe engine and performs inference on an input image.
|
||
|
||
## How does this sample work?
|
||
|
||
This sample uses an ONNX model that was trained on the [MNIST dataset](https://github.com/NVIDIA/DIGITS/blob/master/docs/GettingStarted.md).
|
||
|
||
Specifically, this sample:
|
||
- Build (sample_mnist_safe_build):
|
||
- Performs the basic setup and initialization of TensorRT
|
||
- [Imports a trained ONNX model using ONNX parser](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/c-api-docs.html#importing-a-model-using-the-onnx-parser)
|
||
- Preprocesses the input and stores the result in a managed buffer
|
||
- [Builds a safe engine](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/c-api-docs.html#building-an-engine)
|
||
- Infer (sample_mnist_safe_infer):
|
||
- Create a safe graph for setting up tensors and executing inference on a built network.
|
||
|
||
To verify whether the engine is operating correctly, this sample picks a 28x28 image of a digit at random and runs inference on it using the engine it created. The output of the network is a probability distribution on the digit, showing which digit is likely that in the image.
|
||
|
||
### TensorRT API layers and ops
|
||
|
||
In this sample, the following layers are used. For more information about these layers, see the [TensorRT API: Layers](https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Graph/Layers.html) documentation.
|
||
|
||
[Activation layer](https://docs.nvidia.com/deeplearning/tensorrt/operators/docs/Activation.html)
|
||
The Activation layer implements element-wise activation functions. Specifically, this sample uses the Activation layer with the type `kRELU`.
|
||
|
||
[Convolution layer](https://docs.nvidia.com/deeplearning/tensorrt/operators/docs/Convolution.html)
|
||
The Convolution layer computes a 2D (channel, height, and width) convolution, with or without bias.
|
||
|
||
|
||
## Running the sample
|
||
|
||
1. Download the [MNIST dataset](https://github.com/NVIDIA/DIGITS/blob/master/docs/GettingStarted.md) to read images from the ubyte file. The images need to be saved into `.pgm` format and renamed as `<label>.pgm`.
|
||
|
||
2. Put the images into the `data/mnist` directory together with the existing ONNX network `safe_mnist.onnx`.
|
||
|
||
3. Compile the sample by following the build instructions in the [TensorRT README](https://github.com/NVIDIA/TensorRT/). This will build the sample binaries, including `sample_mnist_safe_build` and `sample_mnist_safe_infer`.
|
||
|
||
4. The compile options are summarized in the following table.
|
||
|
||
| Compile Option | Default |Description|
|
||
| ------------------------------- | ------- |---------- |
|
||
|TRT_SAFETY_INFERENCE_ONLY | OFF |When enabled, build the infer part only, skip compiling the builder part.|
|
||
|
||
5. Run the sample to build a TensorRT safe engine.
|
||
```
|
||
|
||
./sample_mnist_safe_build [--datadir=/path/to/data/dir/] [--remoteAutoTuningConfig=<config>] [--cpuOnly]
|
||
|
||
```
|
||
This sample generates `safe_mnist.engine`, which is a binary file that contains the serialized engine data.
|
||
|
||
This sample reads ONNX model to build the network:
|
||
- `safe_mnist.onnx` - The ONNX model that contains the network design.
|
||
|
||
**Note:** By default, this sample expects these files to be in either the `data/samples/mnist/` or `data/mnist/` directories. The list of default directories can be changed by adding one or more paths with `--datadir=/new/path/` as a command line argument.
|
||
|
||
6. Verify that the sample ran successfully. If the sample runs successfully you should see output similar to the following:
|
||
```
|
||
&&&& RUNNING TensorRT.sample_mnist_safe_build # ./sample_mnist_safe_build
|
||
[I] Building a GPU inference engine for MNIST
|
||
[I] [TRT] Detected 1 input and 1 output network tensors.
|
||
&&&& PASSED TensorRT.sample_mnist_safe_build # ./sample_mnist_safe_build
|
||
```
|
||
This output shows that the sample ran successfully; `PASSED`.
|
||
|
||
7. Run the sample to perform inference on the digit:
|
||
`./sample_mnist_safe_infer`
|
||
|
||
**Note:** This sample expects `./sample_mnist_safe_build` has been run to generate a safe engine file. It loads input image from `data/samples/mnist` directory, and walks back 10 directories to locate the image.
|
||
|
||
8. Verify that the sample ran successfully. If the sample runs successfully you should see output similar to the following; ASCII rendering of the input image with digit 3:
|
||
```
|
||
&&&& RUNNING TensorRT.sample_mnist_safe_infer # ./sample_mnist_safe_infer
|
||
[I] Running a GPU inference engine for MNIST
|
||
[I] Input:
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@#-:.-=@@@@@@@@@@@@@@
|
||
@@@@@%= . *@@@@@@@@@@@@@@@@@
|
||
@@@@% .:+%%% *@@@@@@@@@@@@@@
|
||
@@@@+=#@@@@@# @@@@@@@@@@@@@@
|
||
@@@@@@@@@@@% @@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@: *@@@@@@@@@@@@@@
|
||
@@@@@@@@@@- .@@@@@@@@@@@@@@@
|
||
@@@@@@@@@: #@@@@@@@@@@@@@@@@
|
||
@@@@@@@@: +*%#@@@@@@@@@@@@@@
|
||
@@@@@@@% :+*@@@@@@@@@@@@@@@@
|
||
@@@@@@@@#*+--.:: +@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@#=:. +@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@ .@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@#. #@@@@
|
||
@@@@@@@@@@@@@@@@@@@@# @@@@@@
|
||
@@@@@@@@@%@@@@@@@@@@- +@@@@@
|
||
@@@@@@@@#-@@@@@@@@*. =@@@@@@
|
||
@@@@@@@@ .+%%%%+=. =@@@@@@@@
|
||
@@@@@@@@ =@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@*=: :--*@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
|
||
[I] Output:
|
||
[I] Prob 0 0.0000 Class 0:
|
||
[I] Prob 1 0.0000 Class 1:
|
||
[I] Prob 2 0.0000 Class 2:
|
||
[I] Prob 3 1.0000 Class 3: **********
|
||
[I] Prob 4 0.0000 Class 4:
|
||
[I] Prob 5 0.0000 Class 5:
|
||
[I] Prob 6 0.0000 Class 6:
|
||
[I] Prob 7 0.0000 Class 7:
|
||
[I] Prob 8 0.0000 Class 8:
|
||
[I] Prob 9 0.0000 Class 9:
|
||
|
||
&&&& PASSED TensorRT.sample_safe_mnist_infer # ./sample_mnist_safe_infer
|
||
```
|
||
|
||
This output shows that the sample ran successfully; `PASSED`.
|
||
|
||
### Tool command line arguments
|
||
|
||
To see the full list of available options and their descriptions, use the `-h` or `--help` command line option.
|
||
|
||
```bash
|
||
sample_mnist_safe_build --help
|
||
sample_mnist_safe_infer --help
|
||
```
|
||
|
||
### When to use remoteAutoTuningConfig
|
||
|
||
The `--remoteAutoTuningConfig` parameter is designed for **cross-platform development scenarios** where you need to:
|
||
|
||
**Primary Use Case - Cross-Platform Building:**
|
||
- **Build on Host Platform**: Compile and build TensorRT engines on a development machine (e.g., Linux x86_64)
|
||
- **Auto-tune on Target Platform**: Perform kernel auto-tuning on the actual deployment target (e.g., QNX aarch64)
|
||
|
||
Use `--cpuOnly` with `--remoteAutoTuningConfig` to build the engine without a local GPU on the build host:
|
||
```bash
|
||
./sample_mnist_safe_build --remoteAutoTuningConfig=<config> --cpuOnly
|
||
```
|
||
|
||
**Typical Scenarios:**
|
||
- **QNX Development**: Building engines on Linux development machines but deploying on QNX automotive platforms
|
||
|
||
**Important Technical Limitation:**
|
||
- **QNX Safety Devices**: QNX safety platforms do **NOT** support engine building operations. All engine construction must be performed on development platforms (Linux/QNX standard), making remote auto-tuning essential for safety deployments.
|
||
|
||
|
||
## Additional resources
|
||
|
||
The following resources provide a deeper understanding about sampleSafeMNIST.
|
||
|
||
**Dataset**
|
||
- [MNIST dataset](https://github.com/NVIDIA/DIGITS/blob/master/docs/GettingStarted.md)
|
||
|
||
**Documentation**
|
||
- [NVIDIA’s TensorRT Documentation Library](https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/index.html)
|
||
|
||
## License
|
||
|
||
For terms and conditions for use, reproduction, and distribution, see the [TensorRT Software License Agreement](https://docs.nvidia.com/deeplearning/sdk/tensorrt-sla/index.html) documentation.
|
||
|
||
|
||
## Changelog
|
||
|
||
Jun. 2019
|
||
This is the first release of the `README.md` file and sample.
|
||
|
||
Dec. 2019
|
||
Switch the sample to use ONNX model, and update the content of `README.md`.
|
||
|
||
Jun. 2020
|
||
This sample was updated to fit TensorRT API changes since version 6.3. Please see [TensorRT API](http://docs.nvidia.com/deeplearning/sdk/tensorrt-api/index.html).
|
||
|
||
Sep. 2020
|
||
This sample was updated to fit TensorRT API changes since version 6.4.
|
||
|
||
Mar. 2022
|
||
This sample was updated for DriveOS 6.0 and later releases.
|
||
|
||
Jun. 2023
|
||
This sample was updated to remove deprecated APIs of ICudaEngine and IExecutionContext.
|
||
|
||
Jan. 2024
|
||
Update static linking description
|
||
|
||
Feb. 2025
|
||
This sample was updated for TRT 10.x and later releases.
|
||
|
||
Jul. 2025
|
||
This sample was updated for the TRT 10.13.1 safety release.
|
||
|
||
Dec. 2025
|
||
This sample was updated to use the CMake-based build system.
|
||
|
||
Apr. 2026
|
||
This sample was updated to add the `--cpuOnly` build option for remote auto-tuning workflows without requiring a local GPU on the build host.
|
||
|
||
## Known issues
|
||
|
||
There are no known issues in this sample.
|