wehub-resource-sync
281 lines
11 KiB
Markdown
281 lines
11 KiB
Markdown
# Digit Recognition With Dynamic Shapes In TensorRT
|
||
|
||
|
||
**Table Of Contents**
|
||
- [Description](#description)
|
||
- [How does this sample work?](#how-does-this-sample-work)
|
||
* [Creating the preprocessing network](#creating-the-preprocessing-network)
|
||
* [Parsing the ONNX MNIST model](#parsing-the-onnx-mnist-model)
|
||
* [Building engines](#building-engines)
|
||
* [Running inference](#running-inference)
|
||
* [TensorRT API layers and ops](#tensorrt-api-layers-and-ops)
|
||
- [Preparing sample data](#preparing-sample-data)
|
||
- [Running the sample](#running-the-sample)
|
||
* [Sample `--help` options](#sample-help-options)
|
||
- [Additional resources](#additional-resources)
|
||
- [License](#license)
|
||
- [Changelog](#changelog)
|
||
- [Known issues](#known-issues)
|
||
|
||
## Description
|
||
|
||
This sample, sampleDynamicReshape, demonstrates how to use dynamic input dimensions in TensorRT. It creates an engine that takes a dynamically shaped input and resizes it to be consumed by an ONNX MNIST model that expects a fixed size input. For more information, see [Working With Dynamic Shapes](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#work_dynamic_shapes) in the TensorRT Developer Guide.
|
||
|
||
## How does this sample work?
|
||
|
||
This sample creates an engine for resizing an input with dynamic dimensions to a size that an ONNX MNIST model can consume.
|
||
|
||
Specifically, this sample:
|
||
- Creates a network with dynamic input dimensions to act as a preprocessor for the model
|
||
- Parses an ONNX MNIST model to create a second network
|
||
- Builds engines for both networks
|
||
- Runs inference using both engines
|
||
|
||
### Creating the preprocessing network
|
||
|
||
First, create a network with full dims support:
|
||
`auto preprocessorNetwork = std::unique_ptr<nvinfer1::INetworkDefinition>(builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)));`
|
||
|
||
Next, add an input layer that accepts an input with a dynamic shape, followed by a resize layer that will reshape the input to the shape the model expects:
|
||
```
|
||
auto input = preprocessorNetwork->addInput("input", nvinfer1::DataType::kFLOAT, Dims4{-1, 1, -1, -1});
|
||
auto resizeLayer = preprocessorNetwork->addResize(*input);
|
||
resizeLayer->setOutputDimensions(mPredictionInputDims);
|
||
preprocessorNetwork->markOutput(*resizeLayer->getOutput(0));
|
||
```
|
||
|
||
The -1 dimensions denote dimensions that will be supplied at runtime.
|
||
|
||
### Parsing the ONNX MNIST model
|
||
|
||
First, create an empty full-dims network, and parser:
|
||
```
|
||
auto network = std::unique_ptr<nvinfer1::INetworkDefinition>(builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)));
|
||
auto parser = nvonnxparser::createParser(*network, sample::gLogger.getTRTLogger());
|
||
```
|
||
|
||
Next, parse the model file to populate the network:
|
||
```
|
||
parser->parseFromFile(locateFile(mParams.onnxFileName, mParams.dataDirs).c_str(), static_cast<int>(sample::gLogger.getReportableSeverity()));
|
||
```
|
||
|
||
### Building engines
|
||
|
||
When building the preprocessor engine, also provide an optimization profile so that TensorRT knows which input shapes to optimize for:
|
||
```
|
||
auto preprocessorConfig = std::unique_ptr<nvinfer1::IBuilderConfig>(builder->createBuilderConfig());
|
||
auto profile = builder->createOptimizationProfile();
|
||
```
|
||
|
||
`OptProfileSelector::kOPT` specifies the dimensions that the profile will be optimized for, whereas `OptProfileSelector::kMIN` and `OptProfileSelector::kMAX` specify the minimum and maximum dimensions for which the profile will be valid:
|
||
```
|
||
profile->setDimensions(input->getName(), OptProfileSelector::kMIN, Dims4{1, 1, 1, 1});
|
||
profile->setDimensions(input->getName(), OptProfileSelector::kOPT, Dims4{1, 1, 28, 28});
|
||
profile->setDimensions(input->getName(), OptProfileSelector::kMAX, Dims4{1, 1, 56, 56});
|
||
preprocessorConfig->addOptimizationProfile(profile);
|
||
```
|
||
|
||
Run engine build with config:
|
||
```
|
||
auto preprocessorPlan = std::unique_ptr<nvinfer1::IHostMemory>(
|
||
builder->buildSerializedNetwork(*preprocessorNetwork, *preprocessorConfig));
|
||
if (!preprocessorPlan)
|
||
{
|
||
sample::gLogError << "Preprocessor serialized engine build failed." << std::endl;
|
||
return false;
|
||
}
|
||
|
||
mPreprocessorEngine = std::unique_ptr<nvinfer1::ICudaEngine>(
|
||
runtime->deserializeCudaEngine(preprocessorPlan->data(), preprocessorPlan->size()));
|
||
if (!mPreprocessorEngine)
|
||
{
|
||
sample::gLogError << "Preprocessor engine deserialization failed." << std::endl;
|
||
return false;
|
||
}
|
||
```
|
||
|
||
For the MNIST model, attach a Softmax layer to the end of the network, set softmax axis to 1 since network output has shape [1, 10] in full dims mode and replace the existing network output with the Softmax:
|
||
```
|
||
auto softmax = network->addSoftMax(*network->getOutput(0));
|
||
softmax->setAxes(1 << 1);
|
||
network->unmarkOutput(*network->getOutput(0));
|
||
network->markOutput(*softmax->getOutput(0));
|
||
```
|
||
|
||
Finally, build as normal:
|
||
```
|
||
auto predictionPlan = std::unique_ptr<nvinfer1::IHostMemory>(builder->buildSerializedNetwork(*network, *config));
|
||
if (!predictionPlan)
|
||
{
|
||
sample::gLogError << "Prediction serialized engine build failed." << std::endl;
|
||
return false;
|
||
}
|
||
|
||
mPredictionEngine = std::unique_ptr<nvinfer1::ICudaEngine>(
|
||
runtime->deserializeCudaEngine(predictionPlan->data(), predictionPlan->size()));
|
||
if (!mPredictionEngine)
|
||
{
|
||
sample::gLogError << "Prediction engine deserialization failed." << std::endl;
|
||
return false;
|
||
}
|
||
```
|
||
|
||
### Running inference
|
||
|
||
During inference, first copy the input buffer to the device:
|
||
```
|
||
CHECK(cudaMemcpy(mInput.deviceBuffer.data(), mInput.hostBuffer.data(), mInput.hostBuffer.nbBytes(), cudaMemcpyHostToDevice));
|
||
```
|
||
|
||
Since the preprocessor engine accepts dynamic shapes, specify the actual shape of the current input to the execution context:
|
||
`mPreprocessorContext->setInputShape(inputTensorName, inputDims);`, where inputTensorName is the name of the input tensor on binding index 0.
|
||
|
||
Next, run the preprocessor using the `executeV2` function. The example writes the output of the preprocessor engine directly to the input device buffer of the MNIST engine:
|
||
```
|
||
std::vector<void*> preprocessorBindings = {mInput.deviceBuffer.data(), mPredictionInput.data()};
|
||
bool status = mPreprocessorContext->executeV2(preprocessorBindings.data());
|
||
```
|
||
|
||
Then, run the MNIST engine:
|
||
```
|
||
std::vector<void*> predicitonBindings = {mPredictionInput.data(), mOutput.deviceBuffer.data()};
|
||
status = mPredictionContext->executeV2(predicitonBindings.data());
|
||
```
|
||
|
||
Finally, copy the output back to the host:
|
||
```
|
||
CHECK(cudaMemcpy(mOutput.hostBuffer.data(), mOutput.deviceBuffer.data(), mOutput.deviceBuffer.nbBytes(), cudaMemcpyDeviceToHost));
|
||
```
|
||
|
||
### TensorRT API layers and ops
|
||
|
||
In this sample, the following layers are used. For more information about these layers, see the [TensorRT Developer Guide: Layers](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#layers) documentation.
|
||
|
||
[Resize layer](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#resize-layer)
|
||
The IResizeLayer implements the resize operation on an input tensor.
|
||
|
||
## Prerequisites
|
||
1. See [Preparing sample data](../README.md#preparing-sample-data) in the main samples README.
|
||
|
||
## Running the sample
|
||
|
||
1. Compile the sample by following build instructions in [TensorRT README](https://github.com/NVIDIA/TensorRT/).
|
||
|
||
2. Run the sample.
|
||
```bash
|
||
./sample_dynamic_reshape [-h or --help] [-d or --datadir=<path to data directory>] [--useDLACore=<int>]
|
||
```
|
||
|
||
For example:
|
||
```bash
|
||
./sample_dynamic_reshape --datadir $TRT_DATADIR/mnist
|
||
```
|
||
|
||
3. Verify that the sample ran successfully. If the sample runs successfully you should see output similar to the following:
|
||
```
|
||
&&&& RUNNING TensorRT.sample_dynamic_reshape # ./sample_dynamic_reshape
|
||
----------------------------------------------------------------
|
||
Input filename: ../../../../../data/samples/mnist/mnist.onnx
|
||
ONNX IR version: 0.0.3
|
||
Opset version: 8
|
||
Producer name: CNTK
|
||
Producer version: 2.5.1
|
||
Domain: ai.cntk
|
||
Model version: 1
|
||
Doc string:
|
||
----------------------------------------------------------------
|
||
[W] [TRT] onnx2trt_utils.cpp:214: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
|
||
[W] [TRT] onnx2trt_utils.cpp:214: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
|
||
[I] [TRT] Detected 1 inputs and 1 output network tensors.
|
||
[I] [TRT] Detected 1 inputs and 1 output network tensors.
|
||
[I] Profile dimensions in preprocessor engine:
|
||
[I] Minimum = (1, 1, 1, 1)
|
||
[I] Optimum = (1, 1, 28, 28)
|
||
[I] Maximum = (1, 1, 56, 56)
|
||
[I] Input:
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@*. .*@@@@@@@@@@@
|
||
@@@@@@@@@@*. +@@@@@@@@@@
|
||
@@@@@@@@@@. :#+ %@@@@@@@@@
|
||
@@@@@@@@@@.:@@@+ +@@@@@@@@@
|
||
@@@@@@@@@@.:@@@@: +@@@@@@@@@
|
||
@@@@@@@@@@=%@@@@: +@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@# +@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@* +@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@: +@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@: +@@@@@@@@@
|
||
@@@@@@@@@@@@@@@* .@@@@@@@@@@
|
||
@@@@@@@@@@%**%@. *@@@@@@@@@@
|
||
@@@@@@@@%+. .: .@@@@@@@@@@@
|
||
@@@@@@@@= .. :@@@@@@@@@@@
|
||
@@@@@@@@: *@@: :@@@@@@@@@@@
|
||
@@@@@@@% %@* *@@@@@@@@@@
|
||
@@@@@@@% ++ ++ .%@@@@@@@@@
|
||
@@@@@@@@- +@@- +@@@@@@@@@
|
||
@@@@@@@@= :*@@@# .%@@@@@@@@
|
||
@@@@@@@@@+*@@@@@%. %@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||
|
||
[I] Output:
|
||
[I] Prob 0 0.0000 Class 0:
|
||
[I] Prob 1 0.0000 Class 1:
|
||
[I] Prob 2 1.0000 Class 2: **********
|
||
[I] Prob 3 0.0000 Class 3:
|
||
[I] Prob 4 0.0000 Class 4:
|
||
[I] Prob 5 0.0000 Class 5:
|
||
[I] Prob 6 0.0000 Class 6:
|
||
[I] Prob 7 0.0000 Class 7:
|
||
[I] Prob 8 0.0000 Class 8:
|
||
[I] Prob 9 0.0000 Class 9:
|
||
&&&& PASSED TensorRT.sample_dynamic_reshape # ./sample_dynamic_reshape
|
||
```
|
||
|
||
This output shows that the sample ran successfully; `PASSED`.
|
||
|
||
|
||
### Sample `--help` options
|
||
|
||
To see the full list of available options and their descriptions, use the `-h` or `--help` command line option.
|
||
|
||
|
||
# Additional resources
|
||
|
||
The following resources provide a deeper understanding of dynamic shapes.
|
||
|
||
**ONNX**
|
||
- [GitHub: ONNX](https://github.com/onnx/onnx)
|
||
- [GitHub: ONNX-TensorRT open source parser](https://github.com/onnx/onnx-tensorrt)
|
||
|
||
**Models**
|
||
- [MNIST - Handwritten Digit Recognition](https://github.com/onnx/models/tree/main/validated/vision/classification/mnist)
|
||
- [GitHub: ONNX Models](https://github.com/onnx/models)
|
||
|
||
**Documentation**
|
||
- [Introduction To NVIDIA’s TensorRT Samples](https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#samples)
|
||
- [Working With TensorRT Using The Python API](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#python_topics)
|
||
- [NVIDIA’s TensorRT Documentation Library](https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/index.html)
|
||
|
||
# License
|
||
|
||
For terms and conditions for use, reproduction, and distribution, see the [TensorRT Software License Agreement](https://docs.nvidia.com/deeplearning/sdk/tensorrt-sla/index.html) documentation.
|
||
|
||
|
||
# Changelog
|
||
|
||
October 2025
|
||
Migrate to strongly typed APIs.
|
||
|
||
February 2020
|
||
This is the second release of the `README.md` file and sample.
|
||
|
||
|
||
# Known issues
|
||
|
||
There are no known issues in this sample.
|