# Digit Recognition With Dynamic Shapes In TensorRT


**Table Of Contents**
- [Description](#description)
- [How does this sample work?](#how-does-this-sample-work)
    * [Creating the preprocessing network](#creating-the-preprocessing-network)
    * [Parsing the ONNX MNIST model](#parsing-the-onnx-mnist-model)
    * [Building engines](#building-engines)
    * [Running inference](#running-inference)
	* [TensorRT API layers and ops](#tensorrt-api-layers-and-ops)
- [Preparing sample data](#preparing-sample-data)
- [Running the sample](#running-the-sample)
	* [Sample `--help` options](#sample-help-options)
- [Additional resources](#additional-resources)
- [License](#license)
- [Changelog](#changelog)
- [Known issues](#known-issues)

## Description

This sample, sampleDynamicReshape, demonstrates how to use dynamic input dimensions in TensorRT. It creates an engine that takes a dynamically shaped input and resizes it to be consumed by an ONNX MNIST model that expects a fixed size input. For more information, see [Working With Dynamic Shapes](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#work_dynamic_shapes) in the TensorRT Developer Guide.

## How does this sample work?

This sample creates an engine for resizing an input with dynamic dimensions to a size that an ONNX MNIST model can consume.

Specifically, this sample:
-   Creates a network with dynamic input dimensions to act as a preprocessor for the model
-   Parses an ONNX MNIST model to create a second network
-   Builds engines for both networks
-   Runs inference using both engines

### Creating the preprocessing network

First, create a network with full dims support:
`auto preprocessorNetwork = std::unique_ptr<nvinfer1::INetworkDefinition>(builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)));`

Next, add an input layer that accepts an input with a dynamic shape, followed by a resize layer that will reshape the input to the shape the model expects:
```
auto input = preprocessorNetwork->addInput("input", nvinfer1::DataType::kFLOAT, Dims4{-1, 1, -1, -1});
auto resizeLayer = preprocessorNetwork->addResize(*input);
resizeLayer->setOutputDimensions(mPredictionInputDims);
preprocessorNetwork->markOutput(*resizeLayer->getOutput(0));
```

The -1 dimensions denote dimensions that will be supplied at runtime.

### Parsing the ONNX MNIST model

First, create an empty full-dims network, and parser:
```
auto network = std::unique_ptr<nvinfer1::INetworkDefinition>(builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)));
auto parser = nvonnxparser::createParser(*network, sample::gLogger.getTRTLogger());
```

Next, parse the model file to populate the network:
```
parser->parseFromFile(locateFile(mParams.onnxFileName, mParams.dataDirs).c_str(), static_cast<int>(sample::gLogger.getReportableSeverity()));
```

### Building engines

When building the preprocessor engine, also provide an optimization profile so that TensorRT knows which input shapes to optimize for:
```
auto preprocessorConfig = std::unique_ptr<nvinfer1::IBuilderConfig>(builder->createBuilderConfig());
auto profile = builder->createOptimizationProfile();
```

`OptProfileSelector::kOPT` specifies the dimensions that the profile will be optimized for, whereas `OptProfileSelector::kMIN` and `OptProfileSelector::kMAX` specify the minimum and maximum dimensions for which the profile will be valid:
```
profile->setDimensions(input->getName(), OptProfileSelector::kMIN, Dims4{1, 1, 1, 1});
profile->setDimensions(input->getName(), OptProfileSelector::kOPT, Dims4{1, 1, 28, 28});
profile->setDimensions(input->getName(), OptProfileSelector::kMAX, Dims4{1, 1, 56, 56});
preprocessorConfig->addOptimizationProfile(profile);
```

Run engine build with config: 
```
auto preprocessorPlan = std::unique_ptr<nvinfer1::IHostMemory>(
        builder->buildSerializedNetwork(*preprocessorNetwork, *preprocessorConfig));
if (!preprocessorPlan)
{
    sample::gLogError << "Preprocessor serialized engine build failed." << std::endl;
    return false;
}

mPreprocessorEngine = std::unique_ptr<nvinfer1::ICudaEngine>(
    runtime->deserializeCudaEngine(preprocessorPlan->data(), preprocessorPlan->size()));
if (!mPreprocessorEngine)
{
    sample::gLogError << "Preprocessor engine deserialization failed." << std::endl;
    return false;
}
```

For the MNIST model, attach a Softmax layer to the end of the network, set softmax axis to 1 since network output has shape [1, 10] in full dims mode and replace the existing network output with the Softmax:
```
auto softmax = network->addSoftMax(*network->getOutput(0));
softmax->setAxes(1 << 1);
network->unmarkOutput(*network->getOutput(0));
network->markOutput(*softmax->getOutput(0));
```

Finally, build as normal:
```
auto predictionPlan = std::unique_ptr<nvinfer1::IHostMemory>(builder->buildSerializedNetwork(*network, *config));
if (!predictionPlan)
{
    sample::gLogError << "Prediction serialized engine build failed." << std::endl;
    return false;
}

mPredictionEngine = std::unique_ptr<nvinfer1::ICudaEngine>(
    runtime->deserializeCudaEngine(predictionPlan->data(), predictionPlan->size()));
if (!mPredictionEngine)
{
    sample::gLogError << "Prediction engine deserialization failed." << std::endl;
    return false;
}
```

### Running inference

During inference, first copy the input buffer to the device:
```
CHECK(cudaMemcpy(mInput.deviceBuffer.data(), mInput.hostBuffer.data(), mInput.hostBuffer.nbBytes(), cudaMemcpyHostToDevice));
```

Since the preprocessor engine accepts dynamic shapes, specify the actual shape of the current input to the execution context:
`mPreprocessorContext->setInputShape(inputTensorName, inputDims);`, where inputTensorName is the name of the input tensor on binding index 0.

Next, run the preprocessor using the `executeV2` function. The example writes the output of the preprocessor engine directly to the input device buffer of the MNIST engine:
```
std::vector<void*> preprocessorBindings = {mInput.deviceBuffer.data(), mPredictionInput.data()};
bool status = mPreprocessorContext->executeV2(preprocessorBindings.data());
```

Then, run the MNIST engine:
```
std::vector<void*> predicitonBindings = {mPredictionInput.data(), mOutput.deviceBuffer.data()};
status = mPredictionContext->executeV2(predicitonBindings.data());
```

Finally, copy the output back to the host:
```
CHECK(cudaMemcpy(mOutput.hostBuffer.data(), mOutput.deviceBuffer.data(), mOutput.deviceBuffer.nbBytes(), cudaMemcpyDeviceToHost));
```

### TensorRT API layers and ops

In this sample, the following layers are used. For more information about these layers, see the [TensorRT Developer Guide: Layers](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#layers) documentation.

[Resize layer](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#resize-layer)
The IResizeLayer implements the resize operation on an input tensor.

## Prerequisites
1. See [Preparing sample data](../README.md#preparing-sample-data) in the main samples README.

## Running the sample

1. Compile the sample by following build instructions in [TensorRT README](https://github.com/NVIDIA/TensorRT/).

2.  Run the sample.
    ```bash
    ./sample_dynamic_reshape [-h or --help] [-d or --datadir=<path to data directory>] [--useDLACore=<int>]
    ```

    For example:
    ```bash
    ./sample_dynamic_reshape --datadir $TRT_DATADIR/mnist
    ```

3. Verify that the sample ran successfully. If the sample runs successfully you should see output similar to the following:
    ```
  	&&&& RUNNING TensorRT.sample_dynamic_reshape # ./sample_dynamic_reshape
    ----------------------------------------------------------------
    Input filename:   ../../../../../data/samples/mnist/mnist.onnx
    ONNX IR version:  0.0.3
    Opset version:    8
    Producer name:    CNTK
    Producer version: 2.5.1
    Domain:           ai.cntk
    Model version:    1
    Doc string:  
    ----------------------------------------------------------------
    [W] [TRT] onnx2trt_utils.cpp:214: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
    [W] [TRT] onnx2trt_utils.cpp:214: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
    [I] [TRT] Detected 1 inputs and 1 output network tensors.
    [I] [TRT] Detected 1 inputs and 1 output network tensors.
    [I] Profile dimensions in preprocessor engine:
    [I]     Minimum = (1, 1, 1, 1)
    [I]     Optimum = (1, 1, 28, 28)
    [I]     Maximum = (1, 1, 56, 56)
    [I] Input:
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @@@@@@@@@@@*.  .*@@@@@@@@@@@
    @@@@@@@@@@*.     +@@@@@@@@@@
    @@@@@@@@@@. :#+   %@@@@@@@@@
    @@@@@@@@@@.:@@@+  +@@@@@@@@@
    @@@@@@@@@@.:@@@@: +@@@@@@@@@
    @@@@@@@@@@=%@@@@: +@@@@@@@@@
    @@@@@@@@@@@@@@@@# +@@@@@@@@@
    @@@@@@@@@@@@@@@@* +@@@@@@@@@
    @@@@@@@@@@@@@@@@: +@@@@@@@@@
    @@@@@@@@@@@@@@@@: +@@@@@@@@@
    @@@@@@@@@@@@@@@* .@@@@@@@@@@
    @@@@@@@@@@%**%@. *@@@@@@@@@@
    @@@@@@@@%+.  .: .@@@@@@@@@@@
    @@@@@@@@=  ..   :@@@@@@@@@@@
    @@@@@@@@: *@@:  :@@@@@@@@@@@
    @@@@@@@%  %@*    *@@@@@@@@@@
    @@@@@@@%  ++  ++ .%@@@@@@@@@
    @@@@@@@@-    +@@- +@@@@@@@@@
    @@@@@@@@=  :*@@@# .%@@@@@@@@
    @@@@@@@@@+*@@@@@%.  %@@@@@@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@

    [I] Output:
    [I]  Prob 0  0.0000 Class 0: 
    [I]  Prob 1  0.0000 Class 1: 
    [I]  Prob 2  1.0000 Class 2: **********
    [I]  Prob 3  0.0000 Class 3: 
    [I]  Prob 4  0.0000 Class 4: 
    [I]  Prob 5  0.0000 Class 5: 
    [I]  Prob 6  0.0000 Class 6: 
    [I]  Prob 7  0.0000 Class 7: 
    [I]  Prob 8  0.0000 Class 8: 
    [I]  Prob 9  0.0000 Class 9: 
    &&&& PASSED TensorRT.sample_dynamic_reshape # ./sample_dynamic_reshape
    ```

    This output shows that the sample ran successfully; `PASSED`.


### Sample `--help` options

To see the full list of available options and their descriptions, use the `-h` or `--help` command line option.


# Additional resources

The following resources provide a deeper understanding of dynamic shapes.

**ONNX**
- [GitHub: ONNX](https://github.com/onnx/onnx)
- [GitHub: ONNX-TensorRT open source parser](https://github.com/onnx/onnx-tensorrt)

**Models**
- [MNIST - Handwritten Digit Recognition](https://github.com/onnx/models/tree/main/validated/vision/classification/mnist)
- [GitHub: ONNX Models](https://github.com/onnx/models)

**Documentation**
- [Introduction To NVIDIA’s TensorRT Samples](https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#samples)
- [Working With TensorRT Using The Python API](https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#python_topics)
- [NVIDIA’s TensorRT Documentation Library](https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/index.html)

# License

For terms and conditions for use, reproduction, and distribution, see the [TensorRT Software License Agreement](https://docs.nvidia.com/deeplearning/sdk/tensorrt-sla/index.html) documentation.


# Changelog

October 2025
Migrate to strongly typed APIs.

February 2020
This is the second release of the `README.md` file and sample.


# Known issues

There are no known issues in this sample.
