chore: import upstream snapshot with attribution
Docker Image CI / build-ubuntu2004 (push) Waiting to run
Docker Image CI / build-ubuntu2004 (push) Waiting to run
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,144 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Running This Guide:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This guide is presented as a series of Jupyter notebooks covering both Tensorflow and PyTorch using a Python runtime.\n",
|
||||
"\n",
|
||||
"If you would like to run this code yourself, you can do so using the following steps:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__Setup__:\n",
|
||||
"\n",
|
||||
"The basic prerequisites you will need to use TensorRT are:\n",
|
||||
"\n",
|
||||
"- a [supported NVIDIA GPU](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#hardware-precision-matrix) (preferred compute capability 7.0 or higher - which includes INT8 precision support) \n",
|
||||
"- latest [NVIDIA GPU drivers](https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html)\n",
|
||||
"- a [supported CUDA and cuDNN](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#platform-matrix) installation\n",
|
||||
"\n",
|
||||
"You can make sure your GPU environment is properly configured and check which GPU and CUDA version you are using with nvidia-smi:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!nvidia-smi "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For some of the examples you will also need cuda-python, skimage, matplotlib, and onnx:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install cuda-python onnx scikit-image matplotlib"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__PyTorch__:\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We will be using PyTorch to walk through the basic steps of deploying a TensorRT model by [exporting the model in ONNX format](https://pytorch.org/docs/stable/onnx.html).\n",
|
||||
"\n",
|
||||
"You can find PyTorch installation instructions [here](https://pytorch.org/get-started/locally/), or use one of NVIDIA's NGC containers [here](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch). \n",
|
||||
"\n",
|
||||
"You will also need torchvision:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__Using Colab:__\n",
|
||||
"\n",
|
||||
"You can also test these notebooks out using [Google Colab](https://colab.research.google.com/), which includes PyTorch, as well as supported NVIDIA drivers. __Make sure to select a GPU hardware accelerator__ in the runtime options. Just note that TensorRT performance is best on newer gpus, Colab often has trouble with reduced precision inference, and you will have to use an older version of TensorRT."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__TensorRT Support:__\n",
|
||||
"\n",
|
||||
"TensorRT support for NVIDIA GPUs is determined by their __compute capability__. You can check the compute cabapility of your card on the [NVIDIA website](https://developer.nvidia.com/cuda-gpus).\n",
|
||||
"\n",
|
||||
"TensorRT supports different feautures depending on your compute capability. Higher compute capabilities allow additional TensorRT optimizations, like reduced precision inference. You can check which TensorRT features are supported by your compute capability in the [TensorRT documentation](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#hardware-precision-matrix).\n",
|
||||
"\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__Next Steps:__\n",
|
||||
"\n",
|
||||
"Now, start by opening [1. Introduction.ipynb](./1.%20Introduction.ipynb) and proceed through the notebook!\n",
|
||||
"\n",
|
||||
"The notebooks included with this guide are:\n",
|
||||
"- [1. Introduction.ipynb](./1.%20Introduction.ipynb)\n",
|
||||
"- [2. Using PyTorch through ONNX.ipynb](./2.%20Using%20PyTorch%20through%20ONNX.ipynb)\n",
|
||||
"- [3. Understanding TensorRT Runtimes.ipynb](./3.%20Understanding%20TensorRT%20Runtimes.ipynb)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,392 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Getting Started with TensorRT"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"TensorRT is an SDK for optimizing trained deep learning models to enable high-performance inference. TensorRT contains a deep learning inference __optimizer__ for trained deep learning models and an optimized __runtime__ for execution. After you have trained your deep learning model in a framework of your choice, TensorRT enables you to run it with higher throughput and lower latency. \n",
|
||||
"\n",
|
||||
"The TensorRT ecosystem breaks broadly down into two parts:\n",
|
||||
"<br><br><br>\n",
|
||||
"\n",
|
||||
"<br><br><br>\n",
|
||||
"Essentially,\n",
|
||||
"\n",
|
||||
"1. The various paths users can follow to convert their models to optimized TensorRT engines\n",
|
||||
"2. The various runtimes users can target with TensorRT when deploying their optimized TensorRT engines\n",
|
||||
"\n",
|
||||
"If you have a model in PyTorch and want to run inference as efficiently as possible - with low latency, high throughput, and less memory consumption - this guide will help you achieve just that!!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How Do I Use TensorRT:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"TensorRT is a large and flexible project. It can handle a variety of workflows, and which workflow is best for you will depend on your specific use-case and problem setting. Abstractly, the process for deploying a model from a deep learning framework to TensorRT looks like this:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"To help you get there, this guide will help you answer five key questions:\n",
|
||||
"\n",
|
||||
"1. __What format should I save my model in?__\n",
|
||||
"2. __What batch size(s) am I running inference at?__\n",
|
||||
"3. __What precision am I running inference at?__\n",
|
||||
"4. __What TensorRT path am I using to convert my model?__\n",
|
||||
"5. __What runtime am I targeting?__\n",
|
||||
"\n",
|
||||
"This guide will walk you broadly through all of these decision points while giving you an overview of your options at each step.\n",
|
||||
"\n",
|
||||
"We could talk about these points in isolation, but they are best understood in the context of an actual end-to-end workflow. Let's get started on a simple one here, using a TensorRT API wrapper written for this guide. Once you understand the basic workflow, you can dive into the more in depth notebooks on the Torch-TRT and ONNX converters!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Simple TensorRT Demonstration through ONNX:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"There are several ways of approaching TensorRT conversion and deployment. Here, we will take a pretrained ResNet50 model, convert it to an optimized TensorRT engine, and run it in the TensorRT runtime.\n",
|
||||
"\n",
|
||||
"For this simple demonstration we will focus the ONNX path - one of the two main automatic approaches for TensorRT conversion. We will then run the model in the TensorRT Python API using a simplified wrapper written for this guide. Essentially, we will follow this path to convert and deploy our model:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"We will follow the five questions above. For a more in depth discussion, the section following this demonstration will cover options available at these steps in more detail."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__IMPORTANT NOTE:__ Please __shutdown all other notebooks and PyTorch processes__ before running these steps. TensorRT and PyTorch can not be loaded into your Python processes at the same time."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### 1. What format should I save my model in?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The two main automatic conversion paths for TensorRT require different model formats to successfully convert a model. Torch-TRT uses PyTorch saved models and the ONNX path requires models be saved in ONNX. Here, we will use ONNX.\n",
|
||||
"\n",
|
||||
"We are going to use ResNet50 - a basic backbone vision model that can be used for a variety of purposes. For the sake of demonstration, here we will perform classification using a __pretrained ResNet50 ONNX__ model included with the [ONNX model zoo](https://github.com/onnx/models).\n",
|
||||
"\n",
|
||||
"We can download a pretrained ResNet50 from the ONNX model zoo and untar it by doing the following:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!wget https://download.onnxruntime.ai/onnx/models/resnet50.tar.gz -O resnet50.tar.gz\n",
|
||||
"!tar xzf resnet50.tar.gz"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"See how to export ONNX models that will work with this same trtexec command in the [PyTorch through ONNX notebook](./2.%20Using%20PyTorch%20through%20ONNX.ipynb)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### 2. What precision will I use?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Inference typically requires less numeric precision than training. With some care, lower precision can give you faster computation and lower memory consumption without sacrificing any meaningful accuracy. TensorRT supports TF32, FP32, FP16, and INT8 precisions.\n",
|
||||
"\n",
|
||||
"FP32 is the default training precision of most frameworks, so we will start by using FP32 for inference here. Let's create a \"dummy\" batch to work with in order to test our model. TensorRT will use the precision of the input batch throughout the rest of the network by default."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"PRECISION = np.float32\n",
|
||||
"\n",
|
||||
"# The input tensor shape of the ONNX model.\n",
|
||||
"input_shape = (1, 3, 224, 224)\n",
|
||||
"\n",
|
||||
"dummy_input_batch = np.zeros(input_shape, dtype=PRECISION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### 3. What TensorRT path am I using to convert my model?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The ONNX conversion path is one of the most universal and performant paths for automatic TensorRT conversion. It works for PyTorch and many other frameworks. There are several tools to help users convert models from ONNX to a TensorRT engine. \n",
|
||||
"\n",
|
||||
"One common approach is to use trtexec - a command line tool included with TensorRT that can, among other things, convert ONNX models to TensorRT engines and profile them."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!trtexec --onnx=resnet50/model.onnx --saveEngine=resnet_engine_intro.engine "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__Notes on the flags above:__\n",
|
||||
" \n",
|
||||
"Tell trtexec where to find our ONNX model:\n",
|
||||
"\n",
|
||||
" --onnx=resnet50/model.onnx \n",
|
||||
"\n",
|
||||
"Tell trtexec where to save our optimized TensorRT engine:\n",
|
||||
"\n",
|
||||
" --saveEngine=resnet_engine_intro.engine\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### 4. What runtime will I use?\n",
|
||||
"\n",
|
||||
"After we have our TensorRT engine created successfully, we need to decide how to run it with TensorRT.\n",
|
||||
"\n",
|
||||
"There are two types of TensorRT runtimes: a standalone runtime which has C++ and Python bindings, and a native integration into PyTorch. In this section, we will use a simplified wrapper (ONNXClassifierWrapper) which calls the standalone runtime. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!python3 -m pip install \"numpy<2.0\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# If you get an error in this cell, restart your notebook (possibly your whole machine) and do not run anything that imports/uses PyTorch\n",
|
||||
"\n",
|
||||
"from onnx_helper import ONNXClassifierWrapper\n",
|
||||
"trt_model = ONNXClassifierWrapper(\"resnet_engine_intro.engine\", target_dtype = PRECISION)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"__Note__: If this conversion fails, please restart your Jupyter notebook kernel (in menu bar Kernel->Restart Kernel) and run steps 3 to 5 again. If you get an error like 'TypeError: pybind11::init(): factory function returned nullptr' there is likely some dangling process on the GPU - restart your machine and try again.\n",
|
||||
"\n",
|
||||
"We will feed our batch of randomized dummy data into our ONNXClassifierWrapper to run inference on that batch:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Warm up:\n",
|
||||
"trt_model.predict(dummy_input_batch)[:10] # softmax probability predictions for the first 10 classes of the first sample"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can get a rough sense of performance using %%timeit:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%timeit\n",
|
||||
"trt_model.predict(dummy_input_batch)[:10]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Applying TensorRT to Your Model:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This is a simple example applied to a single model, but how should you go about answering these questions for your workload?\n",
|
||||
"\n",
|
||||
"First and foremost, it is a good idea to get an understanding of what your options are, and where you can learn more about them! \n",
|
||||
"\n",
|
||||
"### __Compatible Models:__ MLP/CNN/RNN/Transformer/Embedding/Etc\n",
|
||||
"\n",
|
||||
"TensorRT is compatible with models consisting of [these layers](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#layers-matrix). Using only supported layers ensures optimal performance without having to write any custom plugin code.\n",
|
||||
"\n",
|
||||
"In terms of framework, TensorRT is integrated directly with PyTorch - and most other major deep learning frameworks are supported by first converting to ONNX format.\n",
|
||||
"\n",
|
||||
"### __Conversion Methods:__ ONNX/TensorRT API\n",
|
||||
"\n",
|
||||
"The __ONNX__ path is the most performant and framework-agnostic automatic way of converting models. It's main disadvantage is that it must convert networks completely - if a network has an unsupported layer ONNX can't convert it unless you write a custom plugin.\n",
|
||||
"\n",
|
||||
"You can see an example of how to use TensorRT with ONNX:\n",
|
||||
"- [Here](./2.%20Using%20PyTorch%20through%20ONNX.ipynb) in this guide for PyTorch\n",
|
||||
"\n",
|
||||
"There is the __TensorRT API__. The TensorRT ONNX path automatically convert models to TensorRT engines for you. Sometimes, however, we want to convert something complex, or have the maximum amount of control in how our TensorRT engine is created. This let's us do things like creating custom plugins for layers that TensorRT doesn't support. \n",
|
||||
"\n",
|
||||
"When using this approach, we create TensorRT engine manually operation-by-operation using the TensorRT API's available in Python and C++. This process involves building a network identical in structure to your target network using the TensorRT API, and then loading in the weights directly in proper format. You can find more details on this [in the TensorRT documentation](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#c_topics).\n",
|
||||
"\n",
|
||||
"### __Precision:__ TF32/FP32/FP16/INT8/FP8\n",
|
||||
"\n",
|
||||
"TensorRT feature support - such as precision - for NVIDIA GPUs is determined by their __compute capability__. You can check the compute cabapility of your card on the [NVIDIA website](https://developer.nvidia.com/cuda-gpus).\n",
|
||||
"\n",
|
||||
"TensorRT supports different precisions depending on said compute capability. You can check what features are supported by your compute capability in the [TensorRT documentation](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#hardware-precision-matrix).\n",
|
||||
"\n",
|
||||
"__TF32__ is the default training precision on cards with compute cabapilities 8.0 and higher (e.g. NVIDIA A100 and later) - use when you want to replicate your original model performance as closely as possible on cards with compute capability of 8.0 or higher. \n",
|
||||
"\n",
|
||||
"TF32 is a precision designed to preserve the range of FP32 with the precision of FP16. In practice, this means that TF32 models train faster than FP32 models while still converging to the same accuracy. This feature is only available on newer GPUs.\n",
|
||||
"\n",
|
||||
"__FP32__ is the default training precision on cards with compute cabapilities of less than 8.0 (e.g. pre-NVIDIA A100) - use when you want to replicate your original model performance as closely as possible on cards with compute capability of less than 8.0\n",
|
||||
"\n",
|
||||
"__FP16__ is an inference focused reduced precision. It gives up some accuracy for faster models with lower latency and lower memory footprint. In practice, the accuracy loss is generally negligible in FP16 - so FP16 is a fairly safe bet in most cases for inference. Cards that are focused on deep learning training often have strong FP16 capabilities, making FP16 a great choice for GPUs that are expected to be used for both training and inference.\n",
|
||||
"\n",
|
||||
"__INT8__ is an inference focused reduced precision. It further reduces memory requirements and latency compared to FP16. INT8 has the potential to lose more accuracy than FP16 - but TensorRT provides tools to help you quantize your network's INT8 weights to avoid this as much as possible. INT8 requires the extra step of calibrating how TensorRT should quantize your weights to integers - requiring some sample data. With careful tuning and a good calibration dataset, accuracy loss from INT8 is often minimal. This makes INT8 a great precision for lower-power environments such as those using T4 GPUs or AGX Jetson modules - both of which have strong INT8 capabilities.\n",
|
||||
"\n",
|
||||
"__FP8__ 8-bit floating point type with 1-bit for sign, 4-bits for exponent, 3-bits for mantissa is an inference focused reduced precision. Similar to INT8, it further reduces memory requirements and latency compared to FP16 with potential suffer from precision loss, especially in deep models with large ranges of values. Extra steps of calibration are needed to minimize accuracy loss. The FP8 precision is supported on NVIDIA GPUs with compute cabapilities of 9.0 and higher (e.g. NVIDIA H100 and later). \n",
|
||||
"\n",
|
||||
"### __Runtime:__ Torch-TRT/Python API/C++ API/TRITON\n",
|
||||
"\n",
|
||||
"For a more in depth discussion of these options and how they compare see [this notebook on TensorRT Runtimes!](./3.%20Understanding%20TensorRT%20Runtimes.ipynb)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## What do I do if I run into issues with conversion?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Here are several steps you can try if your model is not converting to TensorRT properly:\n",
|
||||
"\n",
|
||||
"1. Check the logs - if you are using a tool such as trtexec to convert your model, it will tell you which layer is problematic\n",
|
||||
"2. Write a custom plugin - you can find more information on it [here](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#extending).\n",
|
||||
"3. Use alternative implementations of the layers or operations in question in your network definition - for example, it can be easier to use the padding argument in your convolutional layers instead of adding an explicit padding layer to the network. \n",
|
||||
"4. ONNX to TensorRT conversion can be hard to debug, but tools like graph surgeon https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/graphsurgeon/graphsurgeon.html can help you fix specific nodes in your graph as well as pull it apart for analysis or patch specific nodes in your graph\n",
|
||||
"5. Ask on the [NVIDIA developer forums](https://forums.developer.nvidia.com/c/ai-data-science/deep-learning/tensorrt) - we have many active TensorRT experts at NVIDIA who who browse the forums and can help\n",
|
||||
"6. Post an issue on the [TensorRT OSS Github](https://github.com/NVIDIA/TensorRT)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next Steps:\n",
|
||||
"\n",
|
||||
"You have now taken a model saved in ONNX format, converted it to an optimized TensorRT engine, and deployed it using the Python runtime. This is a great first step towards getting better performance out of your deep learning models at inference time!\n",
|
||||
"\n",
|
||||
"Now, you can check out the remaining notebooks in this guide. See:\n",
|
||||
"\n",
|
||||
"- [2. Using PyTorch through ONNX.ipynb](./2.%20Using%20PyTorch%20through%20ONNX.ipynb)\n",
|
||||
"- [3. Understanding TensorRT Runtimes.ipynb](./3.%20Understanding%20TensorRT%20Runtimes.ipynb)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<h4> Profiling </h4>\n",
|
||||
"\n",
|
||||
"This is a great next step for further optimizing and debugging models you are working on productionizing\n",
|
||||
"\n",
|
||||
"You can find it here: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#performance\n",
|
||||
"\n",
|
||||
"<h4> TRT Dev Docs </h4>\n",
|
||||
"\n",
|
||||
"Main documentation page for the ONNX, layer builder, C++, and legacy APIs\n",
|
||||
"\n",
|
||||
"You can find it here: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html\n",
|
||||
"\n",
|
||||
"<h4> TRT OSS GitHub </h4>\n",
|
||||
"\n",
|
||||
"Contains OSS TRT components, sample applications, and plugin examples\n",
|
||||
"\n",
|
||||
"You can find it here: https://github.com/NVIDIA/TensorRT"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,665 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Using PyTorch with TensorRT through ONNX:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"TensorRT is a great way to take a trained PyTorch model and optimize it to run more efficiently during inference on an NVIDIA GPU.\n",
|
||||
"\n",
|
||||
"One approach to convert a PyTorch model to TensorRT is to export a PyTorch model to ONNX (an open format exchange for deep learning models) and then convert into a TensorRT engine. Essentially, we will follow this path to convert and deploy our model:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Both PyTorch and TensorFlow models can be exported to ONNX, as well as many other frameworks. This allows models created using either framework to flow into common downstream pipelines.\n",
|
||||
"\n",
|
||||
"To get started, let's take a well-known computer vision model and follow five key steps to deploy it to the TensorRT Python runtime:\n",
|
||||
"\n",
|
||||
"1. __What format should I save my model in?__\n",
|
||||
"2. __What batch size(s) am I running inference at?__\n",
|
||||
"3. __What precision am I running inference at?__\n",
|
||||
"4. __What TensorRT path am I using to convert my model?__\n",
|
||||
"5. __What runtime am I targeting?__"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. What format should I save my model in?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We are going to use ResNet50, a widely used CNN architecture first described in <a href=https://arxiv.org/abs/1512.03385>this paper</a>.\n",
|
||||
"\n",
|
||||
"Let's start by loading dependencies and downloading the model. We will also move our Resnet model onto the GPU and set it to evaluation mode."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torchvision.models as models\n",
|
||||
"import torch\n",
|
||||
"import torch.onnx\n",
|
||||
"\n",
|
||||
"# load the pretrained model\n",
|
||||
"resnet50 = models.resnet50(pretrained=True, progress=False).eval()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When saving a model to ONNX, PyTorch requires a test batch in proper shape and format. We pick a batch size:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BATCH_SIZE=32\n",
|
||||
"\n",
|
||||
"dummy_input=torch.randn(BATCH_SIZE, 3, 224, 224)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next, we will export the model using the dummy input batch:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# export the model to ONNX\n",
|
||||
"torch.onnx.export(resnet50, dummy_input, \"resnet50_pytorch.onnx\", verbose=False)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Note that we are picking a BATCH_SIZE of 32 in this example."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Now Test with a Real Image:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's try a real image batch! For this example, we will simply repeat one open-source dog image from http://www.dog.ceo:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from skimage import io\n",
|
||||
"from skimage.transform import resize\n",
|
||||
"from matplotlib import pyplot as plt\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"url='https://images.dog.ceo/breeds/retriever-golden/n02099601_3004.jpg'\n",
|
||||
"img = resize(io.imread(url), (224, 224))\n",
|
||||
"img = np.expand_dims(np.array(img, dtype=np.float32), axis=0) # Expand image to have a batch dimension\n",
|
||||
"input_batch = np.array(np.repeat(img, BATCH_SIZE, axis=0), dtype=np.float32) # Repeat across the batch dimension\n",
|
||||
"\n",
|
||||
"input_batch.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.imshow(input_batch[0].astype(np.float32))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"resnet50_gpu = models.resnet50(pretrained=True, progress=False).to(\"cuda\").eval()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We need to move our batch onto GPU and properly format it to shape [32, 3, 224, 224]. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"input_batch_chw = torch.from_numpy(input_batch).transpose(1,3).transpose(2,3)\n",
|
||||
"input_batch_gpu = input_batch_chw.to(\"cuda\")\n",
|
||||
"\n",
|
||||
"input_batch_gpu.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can run a prediction on a batch using .forward():"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with torch.no_grad():\n",
|
||||
" predictions = np.array(resnet50_gpu(input_batch_gpu).cpu())\n",
|
||||
"\n",
|
||||
"predictions.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Verify Baseline Model Performance/Accuracy:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For a baseline, lets time our prediction in FP32:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%timeit\n",
|
||||
"\n",
|
||||
"with torch.no_grad():\n",
|
||||
" preds = np.array(resnet50_gpu(input_batch_gpu).cpu())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can also time FP16 precision performance:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"resnet50_gpu_half = resnet50_gpu.half()\n",
|
||||
"input_half = input_batch_gpu.half()\n",
|
||||
"\n",
|
||||
"with torch.no_grad():\n",
|
||||
" preds = np.array(resnet50_gpu_half(input_half).cpu()) # Warm Up\n",
|
||||
"\n",
|
||||
"preds.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%timeit\n",
|
||||
"\n",
|
||||
"with torch.no_grad():\n",
|
||||
" preds = np.array(resnet50_gpu_half(input_half).cpu())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's also make sure our results are accurate. We will look at the top 5 accuracy on a single image prediction. The image we are using is of a Golden Retriever, which is class 207 in the ImageNet dataset our model was trained on."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"indices = (-predictions[0]).argsort()[:5]\n",
|
||||
"print(\"Class | Likelihood\")\n",
|
||||
"list(zip(indices, predictions[0][indices]))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We have a model exported to ONNX and a baseline to compare against! Let's now take our ONNX model and convert it to a TensorRT inference engine."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. What batch size(s) am I running inference at?\n",
|
||||
"\n",
|
||||
"We are going to run with a fixed batch size of 32 for this example. Note that above we set BATCH_SIZE to 32 when saving our model to ONNX. We need to create another dummy batch of the same size (this time it will need to be in our target precision) to test out our engine.\n",
|
||||
"\n",
|
||||
"First, as before, we will set our BATCH_SIZE to 32. Note that our trtexec command above includes the '--explicitBatch' flag to signal to TensorRT that we will be using a fixed batch size at runtime."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"BATCH_SIZE = 32"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Importantly, by default TensorRT will use the input precision you give the runtime as the default precision for the rest of the network. So before we create our new dummy batch, we also need to choose a precision as in the next section:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. What precision am I running inference at?\n",
|
||||
"\n",
|
||||
"Remember that lower precisions than FP32 tend to run faster. There are two common reduced precision modes - FP16 and INT8. Graphics cards that are designed to do inference well often have an affinity for one of these two types. This guide was developed on an NVIDIA V100, which favors FP16, so we will use that here by default. INT8 is a more complicated process that requires a calibration step.\n",
|
||||
"\n",
|
||||
"__NOTE__: Make sure you use the same precision (USE_FP16) here you saved your model in above!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"USE_FP16 = True\n",
|
||||
"target_dtype = np.float16 if USE_FP16 else np.float32"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
" To create a test batch, we will once again repeat one open-source dog image from http://www.dog.ceo:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from skimage import io\n",
|
||||
"from skimage.transform import resize\n",
|
||||
"from matplotlib import pyplot as plt\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"url='https://images.dog.ceo/breeds/retriever-golden/n02099601_3004.jpg'\n",
|
||||
"img = resize(io.imread(url), (224, 224))\n",
|
||||
"input_batch = np.array(np.repeat(np.expand_dims(np.array(img, dtype=np.float32), axis=0), BATCH_SIZE, axis=0), dtype=np.float32)\n",
|
||||
"\n",
|
||||
"input_batch.shape"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.imshow(input_batch[0].astype(np.float32))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Preprocess Images:\n",
|
||||
"\n",
|
||||
"PyTorch has a normalization that it applies by default in all of its pretrained vision models - we can preprocess our images to match this normalization by the following, making sure our final result is in FP16 precision:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"from torchvision.transforms import Normalize\n",
|
||||
"\n",
|
||||
"def preprocess_image(img):\n",
|
||||
" norm = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
|
||||
" result = norm(torch.from_numpy(img).transpose(0,2).transpose(1,2))\n",
|
||||
" return np.array(result, dtype=np.float16)\n",
|
||||
"\n",
|
||||
"preprocessed_images = np.array([preprocess_image(image) for image in input_batch])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. What TensorRT path am I using to convert my model?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We can use trtexec, a command line tool for working with TensorRT, in order to convert an ONNX model originally from PyTorch to an engine file.\n",
|
||||
"\n",
|
||||
"Let's make sure we have TensorRT installed (this comes with trtexec):"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import tensorrt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"To convert the model we saved in the previous step, we need to point to the ONNX file, give trtexec a name to save the engine as, and last specify that we want to use a fixed batch size instead of a dynamic one."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# step out of Python for a moment to convert the ONNX model to a TRT engine using trtexec\n",
|
||||
"if USE_FP16:\n",
|
||||
" !trtexec --onnx=resnet50_pytorch.onnx --saveEngine=resnet_engine_pytorch.trt --inputIOFormats=fp16:chw --outputIOFormats=fp16:chw --fp16\n",
|
||||
"else:\n",
|
||||
" !trtexec --onnx=resnet50_pytorch.onnx --saveEngine=resnet_engine_pytorch.trt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This will save our model as 'resnet_engine.trt'."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5. What TensorRT runtime am I targeting?\n",
|
||||
"\n",
|
||||
"Now, we have a converted our model to a TensorRT engine. Great! That means we are ready to load it into the native Python TensorRT runtime. This runtime strikes a balance between the ease of use of the high level Python APIs used in frameworks and the fast, low level C++ runtimes available in TensorRT."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"\n",
|
||||
"import tensorrt as trt\n",
|
||||
"from cuda.bindings import runtime as cudart\n",
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"err, = cudart.cudaSetDevice(0)\n",
|
||||
"assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
"f = open(\"resnet_engine_pytorch.trt\", \"rb\")\n",
|
||||
"runtime = trt.Runtime(trt.Logger(trt.Logger.WARNING))\n",
|
||||
"\n",
|
||||
"engine = runtime.deserialize_cuda_engine(f.read())\n",
|
||||
"context = engine.create_execution_context()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Now allocate input and output memory, give TRT pointers (bindings) to it:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"\n",
|
||||
"# need to set input and output precisions to FP16 to fully enable it\n",
|
||||
"output = np.empty([BATCH_SIZE, 1000], dtype = target_dtype)\n",
|
||||
"\n",
|
||||
"# allocate device memory\n",
|
||||
"err, d_input = cudart.cudaMalloc(input_batch.nbytes)\n",
|
||||
"assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"err, d_output = cudart.cudaMalloc(output.nbytes)\n",
|
||||
"assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
"tensor_names = [engine.get_tensor_name(i) for i in range(engine.num_io_tensors)]\n",
|
||||
"assert(len(tensor_names) == 2)\n",
|
||||
"\n",
|
||||
"context.set_tensor_address(tensor_names[0], d_input)\n",
|
||||
"context.set_tensor_address(tensor_names[1], d_output)\n",
|
||||
"\n",
|
||||
"err, stream = cudart.cudaStreamCreate()\n",
|
||||
"assert err == cudart.cudaError_t.cudaSuccess"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Next, set up the prediction function.\n",
|
||||
"\n",
|
||||
"This involves a copy from CPU RAM to GPU VRAM, executing the model, then copying the results back from GPU VRAM to CPU RAM:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def predict(batch): # result gets copied into output\n",
|
||||
" # transfer input data to device\n",
|
||||
" err, = cudart.cudaMemcpyAsync(d_input, batch.ctypes.data, batch.nbytes,\n",
|
||||
" cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
" # execute model\n",
|
||||
" context.execute_async_v3(stream)\n",
|
||||
"\n",
|
||||
" # transfer predictions back\n",
|
||||
" err, = cudart.cudaMemcpyAsync(output.ctypes.data, d_output, output.nbytes,\n",
|
||||
" cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
" # synchronize stream\n",
|
||||
" err, = cudart.cudaStreamSynchronize(stream)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
" return output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Let's time the function!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"Warming up...\")\n",
|
||||
"\n",
|
||||
"pred = predict(preprocessed_images)\n",
|
||||
"\n",
|
||||
"print(\"Done warming up!\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%%timeit\n",
|
||||
"\n",
|
||||
"pred = predict(preprocessed_images)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Finally we should verify our TensorRT output is still accurate."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"indices = (-pred[0]).argsort()[:5]\n",
|
||||
"print(\"Class | Probability (out of 1)\")\n",
|
||||
"list(zip(indices, pred[0][indices]))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"err, = cudart.cudaStreamDestroy(stream)\n",
|
||||
"assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"err, = cudart.cudaFree(d_input)\n",
|
||||
"assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"err, = cudart.cudaFree(d_output)\n",
|
||||
"assert err == cudart.cudaError_t.cudaSuccess"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Look for ImageNet indices 150-275 above, where 207 is the ground truth correct class (Golden Retriever). Compare with the results of the original unoptimized model in the first section!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Next Steps:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"<h4> TRT Dev Docs </h4>\n",
|
||||
"\n",
|
||||
"Main documentation page for the ONNX, layer builder, C++, and legacy APIs\n",
|
||||
"\n",
|
||||
"You can find it here: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html\n",
|
||||
"\n",
|
||||
"<h4> TRT OSS GitHub </h4>\n",
|
||||
"\n",
|
||||
"Contains OSS TRT components, sample applications, and plugin examples\n",
|
||||
"\n",
|
||||
"You can find it here: https://github.com/NVIDIA/TensorRT\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"#### TRT Supported Layers:\n",
|
||||
"\n",
|
||||
"https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#layers-precision-matrix\n",
|
||||
"\n",
|
||||
"#### TRT ONNX Plugin Example:\n",
|
||||
"\n",
|
||||
"https://github.com/NVIDIA/TensorRT/tree/main/samples/sampleOnnxMnistCoordConvAC"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Runtimes: What are my options? How do I choose?\n",
|
||||
"\n",
|
||||
"Remember that TensorRT consists of two main components - __1. A series of parsers and integrations__ to convert your model to an optimized engine and __2. An series of TensorRT runtime APIs__ with several associated tools for deployment.\n",
|
||||
"\n",
|
||||
"In this notebook, we will focus on the latter - various runtime options for TensorRT engines.\n",
|
||||
"\n",
|
||||
"The runtimes have different use cases for running TRT engines. \n",
|
||||
"\n",
|
||||
"### Considerations when picking a runtime:\n",
|
||||
"\n",
|
||||
"Generally speaking, there are a few major considerations when picking a runtime:\n",
|
||||
"- __Framework__ - Some options, like Torch-TRT, are only relevant to PyTorch\n",
|
||||
"- __Time-to-solution__ - Torch-TRT is much more likely to work 'out-of-the-box' if a quick solution is required and ONNX fails\n",
|
||||
"- __Serving needs__ - Torch-TRT can use TorchServe to serve models over Cloud as a simple solution. For other frameworks (or for more advanced features) TRITON is framework agnostic, allows for concurrent model execution or multiple copies within a GPU to reduce latency, and can accept engines created through both the ONNX and Torch-TRT paths\n",
|
||||
"- __Performance__ - Different TensorRT runtimes offer varying levels of performance. For example, Torch-TRT is generally going to be slower than using ONNX or the C++ API directly.\n",
|
||||
"\n",
|
||||
"### Python API:\n",
|
||||
"\n",
|
||||
"__Use this when:__\n",
|
||||
"- You can accept some performance overhead, and\n",
|
||||
"- You are most familiar with Python, or\n",
|
||||
"- You are performing initial debugging and testing with TRT\n",
|
||||
"\n",
|
||||
"__More info:__\n",
|
||||
"\n",
|
||||
" \n",
|
||||
"The [TensorRT Python API](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#perform_inference_python) gives you fine-grained control over the execution of your engine using a Python interface. It makes memory allocation, kernel execution, and copies to and from the GPU explicit - which can make integration into high performance applications easier. It is also great for testing models in a Python environment - such as in a Jupyter notebook.\n",
|
||||
" \n",
|
||||
"The [ONNX notebook for PyTorch](./2.%20Using%20PyTorch%20through%20ONNX.ipynb) is a good example of using TensorRT to get great performance while staying in Python.\n",
|
||||
"\n",
|
||||
"### C++ API: \n",
|
||||
"\n",
|
||||
"__Use this when:__\n",
|
||||
"- You want the least amount of overhead possible to maximize the performance of your models and achieve better latency\n",
|
||||
"- You are not using Torch-TRT (though Torch-TRT graph conversions that only generate a single engine can still be exported to C++)\n",
|
||||
"- You are most familiar with C++\n",
|
||||
"- You want to optimize your inference pipeline as much as possible\n",
|
||||
"\n",
|
||||
"__More info:__\n",
|
||||
"\n",
|
||||
"The [TensorRT C++ API](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#perform_inference_c) gives you fine-grained control over the execution of your engine using a C++ interface. It makes memory allocation, kernel execution, and copies to and from the GPU explicit - which can make integration into high performance C++ applications easier. The C++ API is generally the most performant option for running TensorRT engines, with the least overhead.\n",
|
||||
"\n",
|
||||
"[This NVIDIA Developer blog](https://developer.nvidia.com/blog/speed-up-inference-tensorrt/) is a good example of taking an ONNX model and running it with dynamic batch size support using the C++ API.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Torch-TRT Runtime:\n",
|
||||
" \n",
|
||||
"__Use this when:__\n",
|
||||
" \n",
|
||||
"- You are using Torch-TRT, and\n",
|
||||
"- Your model converts to more than one TensorRT engine\n",
|
||||
"\n",
|
||||
"__More info:__\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Torch-TRT is the standard runtime used with models that were converted in Torch-TRT. It works by taking groups of nodes at once in the PyTorch graph, and replacing them with a singular optimized engine that calls the TensorRT Python API behind the scenes. This optimized engine is in the form of a PyTorch operation - which means that your graph is still in PyTorch and will essentially function like any other PyTorch model.\n",
|
||||
"\n",
|
||||
"If your graph entirely converts to a single Torch-TRT engine, it can be more efficient to export the engine node and run it using one of the other APIs. You can find instructions to do this in the [Torch-TRT documentation](https://pytorch.org/TensorRT/index.html).\n",
|
||||
"\n",
|
||||
"As an example, the Torch-TRT notebooks included with this guide use the Torch-TRT runtime.\n",
|
||||
"\n",
|
||||
"### TRITON Inference Server\n",
|
||||
"\n",
|
||||
"__Use this when:__\n",
|
||||
"- You want to serve your models over HTTP or gRPC\n",
|
||||
"- You want to load balance across multiple models or copies of models across GPUs to minimze latency and make better use of the GPU\n",
|
||||
"- You want to have multiple models running efficiently on a single GPU at the same time\n",
|
||||
"- You want to serve a variety of models converted using a variety of converters and frameworks (including Torch-TRT and ONNX) through a uniform interface\n",
|
||||
"- You need serving support but are using PyTorch, another framework, or the ONNX path in general\n",
|
||||
"\n",
|
||||
"__More info:__\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"TRITON is an open source inference serving software that lets teams deploy trained AI models from any framework (TensorFlow, TensorRT, PyTorch, ONNX Runtime, or a custom framework), from local storage or Google Cloud Platform or AWS S3 on any GPU- or CPU-based infrastructure (cloud, data center, or edge). It is a flexible project with several unique features - such as concurrent model execution of both heterogeneous models and multiple copies of the same model (multiple model copies can reduce latency further) as well as load balancing and model analysis. It is a good option if you need to serve your models over HTTP - such as in a cloud inferencing solution.\n",
|
||||
" \n",
|
||||
"You can find the TRITON home page [here](https://developer.nvidia.com/triton-inference-server), and the documentation [here](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
from tensorflow.python.compiler.tensorrt import trt_convert as tf_trt
|
||||
from tensorflow.python.saved_model import tag_constants
|
||||
import tensorflow as tf
|
||||
import tensorrt as trt
|
||||
|
||||
import numpy as np
|
||||
|
||||
precision_dict = {
|
||||
"FP32": tf_trt.TrtPrecisionMode.FP32,
|
||||
"FP16": tf_trt.TrtPrecisionMode.FP16,
|
||||
"INT8": tf_trt.TrtPrecisionMode.INT8,
|
||||
}
|
||||
|
||||
# For TF-TRT:
|
||||
|
||||
class OptimizedModel():
|
||||
def __init__(self, saved_model_dir = None):
|
||||
self.loaded_model_fn = None
|
||||
|
||||
if not saved_model_dir is None:
|
||||
self.load_model(saved_model_dir)
|
||||
|
||||
|
||||
def predict(self, input_data):
|
||||
if self.loaded_model_fn is None:
|
||||
raise(Exception("Haven't loaded a model"))
|
||||
x = tf.constant(input_data.astype('float32'))
|
||||
labeling = self.loaded_model_fn(x)
|
||||
try:
|
||||
preds = labeling['predictions'].numpy()
|
||||
except:
|
||||
try:
|
||||
preds = labeling['probs'].numpy()
|
||||
except:
|
||||
try:
|
||||
preds = labeling[next(iter(labeling.keys()))]
|
||||
except:
|
||||
raise(Exception("Failed to get predictions from saved model object"))
|
||||
return preds
|
||||
|
||||
def load_model(self, saved_model_dir):
|
||||
saved_model_loaded = tf.saved_model.load(saved_model_dir, tags=[tag_constants.SERVING])
|
||||
wrapper_fp32 = saved_model_loaded.signatures['serving_default']
|
||||
|
||||
self.loaded_model_fn = wrapper_fp32
|
||||
|
||||
class ModelOptimizer():
|
||||
def __init__(self, input_saved_model_dir, calibration_data=None):
|
||||
self.input_saved_model_dir = input_saved_model_dir
|
||||
self.calibration_data = None
|
||||
self.loaded_model = None
|
||||
|
||||
if not calibration_data is None:
|
||||
self.set_calibration_data(calibration_data)
|
||||
|
||||
|
||||
def set_calibration_data(self, calibration_data):
|
||||
|
||||
def calibration_input_fn():
|
||||
yield (tf.constant(calibration_data.astype('float32')), )
|
||||
|
||||
self.calibration_data = calibration_input_fn
|
||||
|
||||
|
||||
def convert(self, output_saved_model_dir, precision="FP32", max_workspace_size_bytes=8000000000, **kwargs):
|
||||
|
||||
if precision == "INT8" and self.calibration_data is None:
|
||||
raise(Exception("No calibration data set!"))
|
||||
|
||||
trt_precision = precision_dict[precision]
|
||||
conversion_params = tf_trt.DEFAULT_TRT_CONVERSION_PARAMS._replace(precision_mode=trt_precision,
|
||||
max_workspace_size_bytes=max_workspace_size_bytes,
|
||||
use_calibration= precision == "INT8")
|
||||
converter = tf_trt.TrtGraphConverterV2(input_saved_model_dir=self.input_saved_model_dir,
|
||||
conversion_params=conversion_params)
|
||||
|
||||
if precision == "INT8":
|
||||
converter.convert(calibration_input_fn=self.calibration_data)
|
||||
else:
|
||||
converter.convert()
|
||||
|
||||
converter.save(output_saved_model_dir=output_saved_model_dir)
|
||||
|
||||
return OptimizedModel(output_saved_model_dir)
|
||||
|
||||
def predict(self, input_data):
|
||||
if self.loaded_model is None:
|
||||
self.load_default_model()
|
||||
|
||||
return self.loaded_model.predict(input_data)
|
||||
|
||||
def load_default_model(self):
|
||||
self.loaded_model = tf.keras.models.load_model('resnet50_saved_model')
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 110 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 110 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 95 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 370 KiB |
@@ -0,0 +1,145 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import numpy as np
|
||||
import tensorrt as trt
|
||||
import weakref
|
||||
|
||||
from cuda.bindings import runtime as cudart
|
||||
|
||||
|
||||
def _cleanup_cuda_resources(d_input, d_output, stream):
|
||||
"""cleanup function for CUDA resources"""
|
||||
if d_input is not None:
|
||||
cudart.cudaFree(d_input)
|
||||
if d_output is not None:
|
||||
cudart.cudaFree(d_output)
|
||||
if stream is not None:
|
||||
cudart.cudaStreamDestroy(stream)
|
||||
|
||||
|
||||
class ONNXClassifierWrapper:
|
||||
def __init__(self, file, target_dtype=np.float32):
|
||||
self.stream = None
|
||||
self.d_input = None
|
||||
self.d_output = None
|
||||
|
||||
self.target_dtype = target_dtype
|
||||
self.num_classes = 1000
|
||||
self.load(file)
|
||||
|
||||
self._finalizer = weakref.finalize(self, _cleanup_cuda_resources, self.d_input, self.d_output, self.stream)
|
||||
|
||||
def load(self, file):
|
||||
with open(file, "rb") as f:
|
||||
self.runtime = trt.Runtime(trt.Logger(trt.Logger.WARNING))
|
||||
self.engine = self.runtime.deserialize_cuda_engine(f.read())
|
||||
self.context = self.engine.create_execution_context()
|
||||
|
||||
def allocate_memory(self, batch):
|
||||
self.output = np.empty(
|
||||
self.num_classes, dtype=self.target_dtype
|
||||
) # Need to set both input and output precisions to FP16 to fully enable FP16
|
||||
|
||||
# allocate device memory
|
||||
err, self.d_input = cudart.cudaMalloc(batch.nbytes)
|
||||
if err != cudart.cudaError_t.cudaSuccess:
|
||||
raise RuntimeError(f"Failed to allocate input memory: {cudart.cudaGetErrorString(err)}")
|
||||
|
||||
err, self.d_output = cudart.cudaMalloc(self.output.nbytes)
|
||||
if err != cudart.cudaError_t.cudaSuccess:
|
||||
raise RuntimeError(f"Failed to allocate output memory: {cudart.cudaGetErrorString(err)}")
|
||||
|
||||
tensor_names = [self.engine.get_tensor_name(i) for i in range(self.engine.num_io_tensors)]
|
||||
assert len(tensor_names) == 2
|
||||
|
||||
self.context.set_tensor_address(tensor_names[0], int(self.d_input))
|
||||
self.context.set_tensor_address(tensor_names[1], int(self.d_output))
|
||||
|
||||
err, self.stream = cudart.cudaStreamCreate()
|
||||
if err != cudart.cudaError_t.cudaSuccess:
|
||||
raise RuntimeError(f"Failed to create stream: {cudart.cudaGetErrorString(err)}")
|
||||
|
||||
# update the finalizer with new resources
|
||||
self._finalizer.detach()
|
||||
self._finalizer = weakref.finalize(self, _cleanup_cuda_resources, self.d_input, self.d_output, self.stream)
|
||||
|
||||
def predict(self, batch): # result gets copied into output
|
||||
if self.stream is None:
|
||||
self.allocate_memory(batch)
|
||||
|
||||
# transfer input data to device
|
||||
err = cudart.cudaMemcpyAsync(
|
||||
self.d_input, batch.ctypes.data, batch.nbytes, cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, self.stream
|
||||
)
|
||||
if err != cudart.cudaError_t.cudaSuccess:
|
||||
raise RuntimeError(f"Failed to copy input to device: {cudart.cudaGetErrorString(err)}")
|
||||
|
||||
# execute model
|
||||
self.context.execute_async_v3(self.stream)
|
||||
|
||||
# transfer predictions back
|
||||
err = cudart.cudaMemcpyAsync(
|
||||
self.output.ctypes.data,
|
||||
self.d_output,
|
||||
self.output.nbytes,
|
||||
cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost,
|
||||
self.stream,
|
||||
)
|
||||
if err != cudart.cudaError_t.cudaSuccess:
|
||||
raise RuntimeError(f"Failed to copy output from device: {cudart.cudaGetErrorString(err)}")
|
||||
|
||||
# synchronize threads
|
||||
err = cudart.cudaStreamSynchronize(self.stream)
|
||||
if err != cudart.cudaError_t.cudaSuccess:
|
||||
raise RuntimeError(f"Failed to synchronize stream: {cudart.cudaGetErrorString(err)}")
|
||||
|
||||
return self.output
|
||||
|
||||
def cleanup(self):
|
||||
"""free allocated CUDA memory and destroy stream"""
|
||||
if hasattr(self, "_finalizer"):
|
||||
self._finalizer()
|
||||
|
||||
self.d_input = None
|
||||
self.d_output = None
|
||||
self.stream = None
|
||||
|
||||
|
||||
def convert_onnx_to_engine(onnx_filename, engine_filename=None, max_workspace_size=1 << 30, fp16_mode=True):
|
||||
logger = trt.Logger(trt.Logger.WARNING)
|
||||
with trt.Builder(logger) as builder, builder.create_network() as network, trt.OnnxParser(
|
||||
network, logger
|
||||
) as parser, builder.create_builder_config() as builder_config:
|
||||
builder_config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, max_workspace_size)
|
||||
if fp16_mode:
|
||||
builder_config.set_flag(trt.BuilderFlag.FP16)
|
||||
|
||||
print("Parsing ONNX file.")
|
||||
with open(onnx_filename, "rb") as model:
|
||||
if not parser.parse(model.read()):
|
||||
for error in range(parser.num_errors):
|
||||
print(parser.get_error(error))
|
||||
|
||||
print("Building TensorRT engine. This may take a few minutes.")
|
||||
serialized_engine = builder.build_serialized_network(network, builder_config)
|
||||
|
||||
if engine_filename:
|
||||
with open(engine_filename, "wb") as f:
|
||||
f.write(serialized_engine)
|
||||
|
||||
return serialized_engine, logger
|
||||
@@ -0,0 +1,76 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
SHELL=/bin/bash -o pipefail
|
||||
TARGET?=$(shell uname -m)
|
||||
LIBDIR?=lib
|
||||
VERBOSE?=0
|
||||
ifeq ($(VERBOSE), 1)
|
||||
AT=
|
||||
else
|
||||
AT=@
|
||||
endif
|
||||
CUDA_TRIPLE=x86_64-linux
|
||||
CUBLAS_TRIPLE=x86_64-linux-gnu
|
||||
DLSW_TRIPLE=x86_64-linux-gnu
|
||||
ifeq ($(TARGET), aarch64)
|
||||
CUDA_TRIPLE=aarch64-linux
|
||||
CUBLAS_TRIPLE=aarch64-linux-gnu
|
||||
DLSW_TRIPLE=aarch64-linux-gnu
|
||||
endif
|
||||
ifeq ($(TARGET), qnx)
|
||||
CUDA_TRIPLE=aarch64-qnx
|
||||
CUBLAS_TRIPLE=aarch64-qnx-gnu
|
||||
DLSW_TRIPLE=aarch64-unknown-nto-qnx
|
||||
endif
|
||||
ifeq ($(TARGET), ppc64le)
|
||||
CUDA_TRIPLE=ppc64le-linux
|
||||
CUBLAS_TRIPLE=ppc64le-linux
|
||||
DLSW_TRIPLE=ppc64le-linux
|
||||
endif
|
||||
ifeq ($(TARGET), android64)
|
||||
DLSW_TRIPLE=aarch64-linux-androideabi
|
||||
CUDA_TRIPLE=$(DLSW_TRIPLE)
|
||||
CUBLAS_TRIPLE=$(DLSW_TRIPLE)
|
||||
endif
|
||||
export TARGET
|
||||
export VERBOSE
|
||||
export LIBDIR
|
||||
export CUDA_TRIPLE
|
||||
export CUBLAS_TRIPLE
|
||||
export DLSW_TRIPLE
|
||||
|
||||
samples = SemanticSegmentation
|
||||
|
||||
.PHONY: all clean help
|
||||
all:
|
||||
$(AT)$(foreach sample,$(samples), $(MAKE) -C $(sample) &&) :
|
||||
|
||||
clean:
|
||||
$(AT)$(foreach sample,$(samples), $(MAKE) clean -C $(sample) &&) :
|
||||
|
||||
help:
|
||||
$(AT)echo "Sample building help menu."
|
||||
$(AT)echo "Samples:"
|
||||
$(AT)$(foreach sample,$(samples), echo -e "\t$(sample)" &&) :
|
||||
$(AT)echo -e "\nCommands:"
|
||||
$(AT)echo -e "\tall - build all samples."
|
||||
$(AT)echo -e "\tclean - clean all samples."
|
||||
$(AT)echo -e "\nVariables:"
|
||||
$(AT)echo -e "\tTARGET - Specify the target to build for."
|
||||
$(AT)echo -e "\tVERBOSE - Specify verbose output."
|
||||
$(AT)echo -e "\tCUDA_INSTALL_DIR - Directory where cuda installs to."
|
||||
@@ -0,0 +1,242 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
.SUFFIXES:
|
||||
CUDA_TRIPLE ?= x86_64-linux
|
||||
CUBLAS_TRIPLE ?= x86_64-linux-gnu
|
||||
DLSW_TRIPLE ?= x86_64-linux-gnu
|
||||
TARGET ?= $(shell uname -m)
|
||||
|
||||
ifeq ($(CUDA_INSTALL_DIR),)
|
||||
CUDA_INSTALL_DIR ?= /usr/local/cuda
|
||||
$(warning CUDA_INSTALL_DIR variable is not specified, using $(CUDA_INSTALL_DIR) by default, use CUDA_INSTALL_DIR=<cuda_directory> to change.)
|
||||
endif
|
||||
|
||||
ifeq ($(CUDNN_INSTALL_DIR),)
|
||||
CUDNN_INSTALL_DIR ?= $(CUDA_INSTALL_DIR)
|
||||
$(warning CUDNN_INSTALL_DIR variable is not specified, using $(CUDA_INSTALL_DIR) by default, use CUDNN_INSTALL_DIR=<cudnn_directory> to change.)
|
||||
endif
|
||||
|
||||
|
||||
ifeq ($(TRT_LIB_DIR),)
|
||||
TRT_LIB_DIR ?= /usr/lib/x86_64-linux-gnu/
|
||||
$(warning TRT_LIB_DIR is not specified, searching $(TRT_LIB_DIR), /usr/lib/x86_64-linux-gnu/ by default, use TRT_LIB_DIR=<trt_lib_directory> to change.)
|
||||
endif
|
||||
|
||||
ifeq ($(TRT_INC_DIR),)
|
||||
TRT_INC_DIR ?= /usr/include/x86_64-linux-gnu/
|
||||
$(warning TRT_INC_DIR is not specified, searching $(TRT_INC_DIR), /usr/include/x86_64-linux-gnu/ by default, use TRT_INC_DIR=<trt_include_directory> to change.)
|
||||
endif
|
||||
|
||||
CUDA_LIBDIR = lib
|
||||
CUDNN_LIBDIR = lib64
|
||||
ifeq ($(TARGET), aarch64)
|
||||
ifeq ($(shell uname -m), aarch64)
|
||||
CUDA_LIBDIR = lib64
|
||||
CC = g++
|
||||
else
|
||||
CC = aarch64-linux-gnu-g++
|
||||
endif
|
||||
CUCC = $(CUDA_INSTALL_DIR)/bin/nvcc -m64 -ccbin $(CC)
|
||||
else ifeq ($(TARGET), x86_64)
|
||||
CUDA_LIBDIR = lib64
|
||||
CC = g++
|
||||
CUCC = $(CUDA_INSTALL_DIR)/bin/nvcc -m64
|
||||
else ifeq ($(TARGET), ppc64le)
|
||||
CUDA_LIBDIR = lib64
|
||||
CC = g++
|
||||
CUCC = $(CUDA_INSTALL_DIR)/bin/nvcc -m64
|
||||
else ifeq ($(TARGET), qnx)
|
||||
CC = ${QNX_HOST}/usr/bin/aarch64-unknown-nto-qnx7.0.0-g++
|
||||
CUCC = $(CUDA_INSTALL_DIR)/bin/nvcc -m64 -ccbin $(CC)
|
||||
else ifeq ($(TARGET), android64)
|
||||
ifeq ($(ANDROID_CC),)
|
||||
$(error ANDROID_CC must be set to the clang compiler to build for android 64bit, for example /path/to/my-toolchain/bin/aarch64-linux-android-clang++)
|
||||
endif
|
||||
CUDA_LIBDIR = lib
|
||||
ANDROID_FLAGS = -DANDROID -D_GLIBCXX_USE_C99=1 -Wno-sign-compare -D__aarch64__ -Wno-strict-aliasing -Werror -pie -fPIE -Wno-unused-command-line-argument
|
||||
COMMON_FLAGS += $(ANDROID_FLAGS)
|
||||
COMMON_LD_FLAGS += $(ANDROID_FLAGS)
|
||||
CC = $(ANDROID_CC)
|
||||
CUCC = $(CUDA_INSTALL_DIR)/bin/nvcc -m64 -ccbin $(CC) --compiler-options="-DANDROID -D_GLIBCXX_USE_C99=1 -Wno-sign-compare"
|
||||
ANDROID = 1
|
||||
else ########
|
||||
$(error Auto-detection of platform failed. Please specify one of the following arguments to make: TARGET=[aarch64|x86_64|qnx|android64])
|
||||
endif
|
||||
|
||||
ifdef VERBOSE
|
||||
AT=
|
||||
else
|
||||
AT=@
|
||||
endif
|
||||
|
||||
AR = ar cr
|
||||
ECHO = @echo
|
||||
|
||||
SHELL = /bin/sh
|
||||
|
||||
ROOT_PATH = .
|
||||
OUTDIR = $(ROOT_PATH)/bin
|
||||
|
||||
define concat
|
||||
$1$2$3$4$5$6$7$8
|
||||
endef
|
||||
|
||||
ifneq ($(USE_QCC),1)
|
||||
# Usage: $(call make-depend,source-file,object-file,depend-file)
|
||||
define make-depend
|
||||
$(AT)$(CC) -MM -MF $3 -MP -MT $2 $(COMMON_FLAGS) $1
|
||||
endef
|
||||
# Usage: $(call make-cuda-depend,source-file,object-file,depend-file,flags)
|
||||
define make-cuda-depend
|
||||
$(AT)$(CUCC) -M -MT $2 $4 $1 > $3
|
||||
endef
|
||||
endif
|
||||
|
||||
CUDART_LIB = -lcudart
|
||||
CUDNN_LIB = -lcudnn
|
||||
CUBLAS_LIB = -lcublas
|
||||
NVINFER_LIB = -lnvinfer
|
||||
NVINFER_PLUGIN_LIB = -lnvinfer_plugin
|
||||
NVONNXPARSERS_LIB = -lnvonnxparser
|
||||
NVRTC_LIB = -lnvrtc
|
||||
PROTO_LIBDIR =
|
||||
STUBS_DIR =
|
||||
|
||||
#########################
|
||||
INCPATHS=
|
||||
LIBPATHS=
|
||||
COMMON_LIBS=
|
||||
|
||||
# Add extra libraries if TRT_STATIC is enabled
|
||||
ifeq ($(TRT_STATIC), 1)
|
||||
COMMON_LIBS += -lculibos -lcublasLt_static
|
||||
endif
|
||||
|
||||
# add cross compile directories
|
||||
ifneq ($(shell uname -m), $(TARGET))
|
||||
INCPATHS += -I"/usr/include/$(DLSW_TRIPLE)" -I"/usr/include/$(CUBLAS_TRIPLE)"
|
||||
LIBPATHS += -L"../lib/stubs" -L"../../lib/stubs" -L"/usr/lib/$(DLSW_TRIPLE)/stubs" -L"/usr/lib/$(DLSW_TRIPLE)" -L"/usr/lib/$(CUBLAS_TRIPLE)/stubs" -L"/usr/lib/$(CUBLAS_TRIPLE)"
|
||||
LIBPATHS += -L"$(CUDA_INSTALL_DIR)/targets/$(CUDA_TRIPLE)/$(CUDA_LIBDIR)/stubs" -L"$(CUDA_INSTALL_DIR)/targets/$(CUDA_TRIPLE)/$(CUDA_LIBDIR)"
|
||||
endif
|
||||
INCPATHS += -I"../common" -I"$(TRT_INC_DIR)" -I"$(CUDA_INSTALL_DIR)/include" -I"$(CUDNN_INSTALL_DIR)/include"
|
||||
LIBPATHS += -L"$(CUDA_INSTALL_DIR)/$(CUDA_LIBDIR)" -Wl,-rpath-link="$(CUDA_INSTALL_DIR)/$(CUDA_LIBDIR)"
|
||||
LIBPATHS += -L"$(CUDNN_INSTALL_DIR)/$(CUDNN_LIBDIR)" -Wl,-rpath-link="$(CUDNN_INSTALL_DIR)/$(CUDNN_LIBDIR)"
|
||||
LIBPATHS += -L"$(TRT_LIB_DIR)" -Wl,-rpath-link="$(TRT_LIB_DIR)" $(STUBS_DIR)
|
||||
|
||||
.SUFFIXES:
|
||||
vpath %.h $(EXTRA_DIRECTORIES)
|
||||
vpath %.cpp $(EXTRA_DIRECTORIES)
|
||||
|
||||
COMMON_FLAGS += -Wall -Wno-deprecated-declarations -std=c++11 $(INCPATHS)
|
||||
ifneq ($(ANDROID),1)
|
||||
COMMON_FLAGS += -D_REENTRANT
|
||||
endif
|
||||
|
||||
ifeq ($(TARGET), qnx)
|
||||
COMMON_FLAGS += -D_POSIX_C_SOURCE=200112L -D_QNX_SOURCE -D_FILE_OFFSET_BITS=64 -fpermissive
|
||||
endif
|
||||
|
||||
COMMON_LD_FLAGS += $(LIBPATHS) -L$(OUTDIR)
|
||||
|
||||
OBJDIR = $(call concat,$(OUTDIR),/chobj)
|
||||
DOBJDIR = $(call concat,$(OUTDIR),/dchobj)
|
||||
|
||||
COMMON_LIBS += $(CUDART_LIB) $(CUBLAS_LIB) $(CUDNN_LIB)
|
||||
|
||||
ifneq ($(TARGET), qnx)
|
||||
ifneq ($(ANDROID), 1)
|
||||
COMMON_LIBS += -lrt -ldl -lpthread
|
||||
endif
|
||||
endif
|
||||
ifeq ($(ANDROID),1)
|
||||
COMMON_LIBS += -lculibos -llog
|
||||
endif
|
||||
|
||||
COMMON_LIBS_FOR_EXECUTABLE := $(filter-out -lcudart_static ,$(COMMON_LIBS))
|
||||
ifeq ($(USE_CUDART_STATIC), 1)
|
||||
COMMON_LIBS_FOR_EXECUTABLE += $(CUDART_LIB)
|
||||
endif
|
||||
|
||||
LIBS = $(NVINFER_LIB) $(NVINFER_PLUGIN_LIB) $(NVONNXPARSERS_LIB) $(COMMON_LIBS_FOR_EXECUTABLE) $(PROTO_LIB) $(EXTRA_LIBS)
|
||||
DLIBS = $(NVINFER_LIB) $(NVINFER_PLUGIN_LIB) $(NVONNXPARSERS_LIB) $(COMMON_LIBS_FOR_EXECUTABLE) $(PROTO_LIB) $(EXTRA_LIBS)
|
||||
|
||||
OBJS = $(patsubst %.cpp, $(OBJDIR)/%.o, $(wildcard *.cpp $(addsuffix /*.cpp, $(EXTRA_DIRECTORIES))))
|
||||
DOBJS = $(patsubst %.cpp, $(DOBJDIR)/%.o, $(wildcard *.cpp $(addsuffix /*.cpp, $(EXTRA_DIRECTORIES))))
|
||||
CUOBJS = $(patsubst %.cu, $(OBJDIR)/%.o, $(wildcard *.cu $(addsuffix /*.cu, $(EXTRA_DIRECTORIES))))
|
||||
CUDOBJS = $(patsubst %.cu, $(DOBJDIR)/%.o, $(wildcard *.cu $(addsuffix /*.cu, $(EXTRA_DIRECTORIES))))
|
||||
|
||||
CFLAGS = $(COMMON_FLAGS)
|
||||
CFLAGSD = $(COMMON_FLAGS) -g
|
||||
LFLAGS = $(COMMON_LD_FLAGS)
|
||||
LFLAGSD = $(COMMON_LD_FLAGS)
|
||||
|
||||
all: debug release
|
||||
|
||||
release : $(OUTDIR)/$(OUTNAME_RELEASE)
|
||||
|
||||
debug : $(OUTDIR)/$(OUTNAME_DEBUG)
|
||||
|
||||
$(OUTDIR)/$(OUTNAME_RELEASE) : $(OBJS) $(CUOBJS)
|
||||
$(ECHO) Linking: $@
|
||||
$(AT)$(CC) -o $@ $^ $(LFLAGS) -Wl,--start-group $(LIBS) -Wl,--end-group
|
||||
# Copy every EXTRA_FILE of this sample to bin dir
|
||||
$(foreach EXTRA_FILE,$(EXTRA_FILES), cp -f $(EXTRA_FILE) $(OUTDIR)/$(EXTRA_FILE); )
|
||||
|
||||
$(OUTDIR)/$(OUTNAME_DEBUG) : $(DOBJS) $(CUDOBJS)
|
||||
$(ECHO) Linking: $@
|
||||
$(AT)$(CC) -o $@ $^ $(LFLAGSD) -Wl,--start-group $(DLIBS) -Wl,--end-group
|
||||
|
||||
$(OBJDIR)/%.o: %.cpp
|
||||
$(AT)if [ ! -d $(OBJDIR) ]; then mkdir -p $(OBJDIR); fi
|
||||
$(foreach XDIR,$(EXTRA_DIRECTORIES), if [ ! -d $(OBJDIR)/$(XDIR) ]; then mkdir -p $(OBJDIR)/$(XDIR); fi;) :
|
||||
$(call make-depend,$<,$@,$(subst .o,.d,$@))
|
||||
$(ECHO) Compiling: $<
|
||||
$(AT)$(CC) $(CFLAGS) -c -o $@ $<
|
||||
|
||||
$(DOBJDIR)/%.o: %.cpp
|
||||
$(AT)if [ ! -d $(DOBJDIR) ]; then mkdir -p $(DOBJDIR); fi
|
||||
$(foreach XDIR,$(EXTRA_DIRECTORIES), if [ ! -d $(OBJDIR)/$(XDIR) ]; then mkdir -p $(DOBJDIR)/$(XDIR); fi;) :
|
||||
$(call make-depend,$<,$@,$(subst .o,.d,$@))
|
||||
$(ECHO) Compiling: $<
|
||||
$(AT)$(CC) $(CFLAGSD) -c -o $@ $<
|
||||
|
||||
######################################################################### CU
|
||||
$(OBJDIR)/%.o: %.cu
|
||||
$(AT)if [ ! -d $(OBJDIR) ]; then mkdir -p $(OBJDIR); fi
|
||||
$(foreach XDIR,$(EXTRA_DIRECTORIES), if [ ! -d $(OBJDIR)/$(XDIR) ]; then mkdir -p $(OBJDIR)/$(XDIR); fi;) :
|
||||
$(call make-cuda-depend,$<,$@,$(subst .o,.d,$@))
|
||||
$(ECHO) Compiling CUDA release: $<
|
||||
$(AT)$(CUCC) $(CUFLAGS) -c -o $@ $<
|
||||
|
||||
$(DOBJDIR)/%.o: %.cu
|
||||
$(AT)if [ ! -d $(DOBJDIR) ]; then mkdir -p $(DOBJDIR); fi
|
||||
$(foreach XDIR,$(EXTRA_DIRECTORIES), if [ ! -d $(DOBJDIR)/$(XDIR) ]; then mkdir -p $(DOBJDIR)/$(XDIR); fi;) :
|
||||
$(call make-cuda-depend,$<,$@,$(subst .o,.d,$@))
|
||||
$(ECHO) Compiling CUDA debug: $<
|
||||
$(AT)$(CUCC) $(CUFLAGSD) -c -o $@ $<
|
||||
|
||||
clean:
|
||||
$(ECHO) Cleaning...
|
||||
$(foreach XDIR,$(EXTRA_DIRECTORIES), if [ -d $(OBJDIR)/$(XDIR) ]; then rm -rf $(OBJDIR)/$(XDIR); fi;) :
|
||||
$(foreach XDIR,$(EXTRA_DIRECTORIES), if [ -d $(DOBJDIR)/$(XDIR) ]; then rm -rf $(DOBJDIR)/$(XDIR); fi;) :
|
||||
$(AT)rm -rf $(OBJDIR) $(DOBJDIR) $(OUTDIR)/$(OUTNAME_RELEASE) $(OUTDIR)/$(OUTNAME_DEBUG)
|
||||
$(foreach EXTRA_FILE,$(EXTRA_FILES), if [ -f $(OUTDIR)/$(EXTRA_FILE) ]; then rm -f $(OUTDIR)/$(EXTRA_FILE); fi;) :
|
||||
|
||||
ifneq "$(MAKECMDGOALS)" "clean"
|
||||
-include $(OBJDIR)/*.d $(DOBJDIR)/*.d
|
||||
endif # ifneq "$(MAKECMDGOALS)" "clean"
|
||||
@@ -0,0 +1,14 @@
|
||||
# QuickStartGuide
|
||||
|
||||
Introductory notebooks in TensorRT QuickStartGuide
|
||||
|
||||
- [Running the QuickStartGuide](IntroNotebooks/0.%20Running%20This%20Guide.ipynb)
|
||||
- [Getting Started with TensorRT](IntroNotebooks/1.%20Introduction.ipynb)
|
||||
- [Using PyTorch through ONNX](IntroNotebooks/2.%20Using%20PyTorch%20through%20ONNX.ipynb)
|
||||
- [Understanding TensorRT Runtimes](IntroNotebooks/3.%20Understanding%20TensorRT%20Runtimes.ipynb)
|
||||
|
||||
Tutorials corresponding to TensorRT QuickStartGuide
|
||||
|
||||
- Semantic Segmentation using TensorRT - [C++ sample and Python notebook](./SemanticSegmentation/)
|
||||
- Optimize with TensorRT, Deploy with Triton - [Walkthrough and Python code](./deploy_to_triton/)
|
||||
- Quantization with TensorRT Model Optimizer - [Stable Diffusion XL (Base/Turbo) and Stable Diffusion 1.5 Quantization with Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer/tree/main/diffusers/quantization)
|
||||
@@ -0,0 +1,5 @@
|
||||
bin/*
|
||||
.ipynb_checkpoints/
|
||||
*.ppm
|
||||
*.onnx
|
||||
*.engine
|
||||
@@ -0,0 +1,22 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
OUTNAME_RELEASE = segmentation_tutorial
|
||||
OUTNAME_DEBUG = segmentation_tutorial_debug
|
||||
EXTRA_DIRECTORIES = ../common
|
||||
MAKEFILE ?= ../Makefile.config
|
||||
include $(MAKEFILE)
|
||||
@@ -0,0 +1,70 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
from PIL import Image
|
||||
from io import BytesIO
|
||||
import requests
|
||||
|
||||
output_image = "input.ppm"
|
||||
|
||||
# Read sample image input and save it in ppm format
|
||||
print("Exporting ppm image {}".format(output_image))
|
||||
response = requests.get("https://pytorch.org/assets/images/deeplab1.png")
|
||||
with Image.open(BytesIO(response.content)) as img:
|
||||
ppm = Image.new("RGB", img.size, (255, 255, 255))
|
||||
ppm.paste(img, mask=img.split()[3])
|
||||
ppm.save(output_image)
|
||||
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torchvision.models.segmentation as segmentation
|
||||
|
||||
output_onnx = "fcn-resnet101.onnx"
|
||||
|
||||
|
||||
# FC-ResNet101 pretrained model from torch-hub extended with argmax layer
|
||||
class FCN_ResNet101(nn.Module):
|
||||
def __init__(self):
|
||||
super(FCN_ResNet101, self).__init__()
|
||||
self.model = segmentation.fcn_resnet101(pretrained=True)
|
||||
|
||||
def forward(self, inputs):
|
||||
x = self.model(inputs)["out"]
|
||||
x = x.argmax(1, keepdims=True)
|
||||
return x
|
||||
|
||||
|
||||
model = FCN_ResNet101()
|
||||
model.eval()
|
||||
|
||||
# Generate input tensor with random values
|
||||
input_tensor = torch.rand(4, 3, 224, 224)
|
||||
|
||||
# Export torch model to ONNX
|
||||
print("Exporting ONNX model {}".format(output_onnx))
|
||||
torch.onnx.export(
|
||||
model,
|
||||
input_tensor,
|
||||
output_onnx,
|
||||
opset_version=12,
|
||||
do_constant_folding=True,
|
||||
input_names=["input"],
|
||||
output_names=["output"],
|
||||
dynamic_axes={"input": {0: "batch", 2: "height", 3: "width"}, "output": {0: "batch", 2: "height", 3: "width"}},
|
||||
verbose=False,
|
||||
)
|
||||
@@ -0,0 +1,193 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
#include <cassert>
|
||||
#include <cfloat>
|
||||
#include <fstream>
|
||||
#include <iostream>
|
||||
#include <memory>
|
||||
#include <sstream>
|
||||
|
||||
#include <cuda_runtime_api.h>
|
||||
#include "NvInfer.h"
|
||||
#include "NvOnnxParser.h"
|
||||
#include "logger.h"
|
||||
#include "util.h"
|
||||
|
||||
constexpr long long operator"" _MiB(long long unsigned val)
|
||||
{
|
||||
return val * (1 << 20);
|
||||
}
|
||||
|
||||
using sample::gLogError;
|
||||
using sample::gLogInfo;
|
||||
|
||||
//!
|
||||
//! \class SampleSegmentation
|
||||
//!
|
||||
//! \brief Implements semantic segmentation using FCN-ResNet101 ONNX model.
|
||||
//!
|
||||
class SampleSegmentation
|
||||
{
|
||||
|
||||
public:
|
||||
SampleSegmentation(const std::string& engineFilename);
|
||||
bool infer(const std::string& input_filename, int32_t width, int32_t height, const std::string& output_filename);
|
||||
|
||||
private:
|
||||
std::string mEngineFilename; //!< Filename of the serialized engine.
|
||||
|
||||
nvinfer1::Dims mInputDims; //!< The dimensions of the input to the network.
|
||||
nvinfer1::Dims mOutputDims; //!< The dimensions of the output to the network.
|
||||
|
||||
std::unique_ptr<nvinfer1::IRuntime> mRuntime; //!< The TensorRT runtime used to run the network
|
||||
std::unique_ptr<nvinfer1::ICudaEngine> mEngine; //!< The TensorRT engine used to run the network
|
||||
};
|
||||
|
||||
SampleSegmentation::SampleSegmentation(const std::string& engineFilename)
|
||||
: mEngineFilename(engineFilename)
|
||||
, mEngine(nullptr)
|
||||
{
|
||||
// De-serialize engine from file
|
||||
std::ifstream engineFile(engineFilename, std::ios::binary);
|
||||
if (engineFile.fail())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
engineFile.seekg(0, std::ifstream::end);
|
||||
auto fsize = engineFile.tellg();
|
||||
engineFile.seekg(0, std::ifstream::beg);
|
||||
|
||||
std::vector<char> engineData(fsize);
|
||||
engineFile.read(engineData.data(), fsize);
|
||||
|
||||
mRuntime.reset(nvinfer1::createInferRuntime(sample::gLogger.getTRTLogger()));
|
||||
mEngine.reset(mRuntime->deserializeCudaEngine(engineData.data(), fsize));
|
||||
assert(mEngine.get() != nullptr);
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief Runs the TensorRT inference.
|
||||
//!
|
||||
//! \details Allocate input and output memory, and executes the engine.
|
||||
//!
|
||||
bool SampleSegmentation::infer(const std::string& input_filename, int32_t width, int32_t height, const std::string& output_filename)
|
||||
{
|
||||
auto context = std::unique_ptr<nvinfer1::IExecutionContext>(mEngine->createExecutionContext());
|
||||
if (!context)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
char const* input_name = "input";
|
||||
assert(mEngine->getTensorDataType(input_name) == nvinfer1::DataType::kFLOAT);
|
||||
auto input_dims = nvinfer1::Dims4{1, /* channels */ 3, height, width};
|
||||
context->setInputShape(input_name, input_dims);
|
||||
auto input_size = util::getMemorySize(input_dims, sizeof(float));
|
||||
|
||||
char const* output_name = "output";
|
||||
assert(mEngine->getTensorDataType(output_name) == nvinfer1::DataType::kINT64);
|
||||
auto output_dims = context->getTensorShape(output_name);
|
||||
auto output_size = util::getMemorySize(output_dims, sizeof(int64_t));
|
||||
|
||||
// Allocate CUDA memory for input and output bindings
|
||||
void* input_mem{nullptr};
|
||||
if (cudaMalloc(&input_mem, input_size) != cudaSuccess)
|
||||
{
|
||||
gLogError << "ERROR: input cuda memory allocation failed, size = " << input_size << " bytes" << std::endl;
|
||||
return false;
|
||||
}
|
||||
void* output_mem{nullptr};
|
||||
if (cudaMalloc(&output_mem, output_size) != cudaSuccess)
|
||||
{
|
||||
gLogError << "ERROR: output cuda memory allocation failed, size = " << output_size << " bytes" << std::endl;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Read image data from file and mean-normalize it
|
||||
const std::vector<float> mean{0.485f, 0.456f, 0.406f};
|
||||
const std::vector<float> stddev{0.229f, 0.224f, 0.225f};
|
||||
auto input_image{util::RGBImageReader(input_filename, input_dims, mean, stddev)};
|
||||
input_image.read();
|
||||
auto input_buffer = input_image.process();
|
||||
cudaStream_t stream;
|
||||
if (cudaStreamCreate(&stream) != cudaSuccess)
|
||||
{
|
||||
gLogError << "ERROR: cuda stream creation failed." << std::endl;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Copy image data to input binding memory
|
||||
if (cudaMemcpyAsync(input_mem, input_buffer.get(), input_size, cudaMemcpyHostToDevice, stream) != cudaSuccess)
|
||||
{
|
||||
gLogError << "ERROR: CUDA memory copy of input failed, size = " << input_size << " bytes" << std::endl;
|
||||
return false;
|
||||
}
|
||||
context->setTensorAddress(input_name, input_mem);
|
||||
context->setTensorAddress(output_name, output_mem);
|
||||
|
||||
// Run TensorRT inference
|
||||
bool status = context->enqueueV3(stream);
|
||||
if (!status)
|
||||
{
|
||||
gLogError << "ERROR: TensorRT inference failed" << std::endl;
|
||||
return false;
|
||||
}
|
||||
|
||||
// Copy predictions from output binding memory
|
||||
auto output_buffer = std::unique_ptr<int64_t>{new int64_t[output_size]};
|
||||
if (cudaMemcpyAsync(output_buffer.get(), output_mem, output_size, cudaMemcpyDeviceToHost, stream) != cudaSuccess)
|
||||
{
|
||||
gLogError << "ERROR: CUDA memory copy of output failed, size = " << output_size << " bytes" << std::endl;
|
||||
return false;
|
||||
}
|
||||
cudaStreamSynchronize(stream);
|
||||
|
||||
// Plot the semantic segmentation predictions of 21 classes in a colormap image and write to file
|
||||
const int num_classes{21};
|
||||
const std::vector<int> palette{(0x1 << 25) - 1, (0x1 << 15) - 1, (0x1 << 21) - 1};
|
||||
auto output_image{util::ArgmaxImageWriter(output_filename, output_dims, palette, num_classes)};
|
||||
int64_t* output_ptr = output_buffer.get();
|
||||
std::vector<int32_t> output_buffer_casted(output_size);
|
||||
for (size_t i = 0; i < output_size; ++i) {
|
||||
output_buffer_casted[i] = static_cast<int32_t>(output_ptr[i]);
|
||||
}
|
||||
output_image.process(output_buffer_casted.data());
|
||||
output_image.write();
|
||||
|
||||
// Free CUDA resources
|
||||
cudaFree(input_mem);
|
||||
cudaFree(output_mem);
|
||||
return true;
|
||||
}
|
||||
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
int32_t width{1282};
|
||||
int32_t height{1026};
|
||||
|
||||
SampleSegmentation sample("fcn-resnet101.engine");
|
||||
|
||||
gLogInfo << "Running TensorRT inference for FCN-ResNet101" << std::endl;
|
||||
if (!sample.infer("input.ppm", width, height, "output.ppm"))
|
||||
{
|
||||
return -1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,341 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#\n",
|
||||
"# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.\n",
|
||||
"#\n",
|
||||
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
|
||||
"# you may not use this file except in compliance with the License.\n",
|
||||
"# You may obtain a copy of the License at\n",
|
||||
"#\n",
|
||||
"# http://www.apache.org/licenses/LICENSE-2.0\n",
|
||||
"#\n",
|
||||
"# Unless required by applicable law or agreed to in writing, software\n",
|
||||
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
|
||||
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
|
||||
"# See the License for the specific language governing permissions and\n",
|
||||
"# limitations under the License.\n",
|
||||
"#"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Check the TensorRT version"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!python3 -c 'import tensorrt; print(\"TensorRT version: {}\".format(tensorrt.__version__))'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Prepare the input image and ONNX model file"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!python3 /workspace/TensorRT/quickstart/SemanticSegmentation/export.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Build TensorRT engine from the ONNX model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!trtexec --onnx=fcn-resnet101.onnx --saveEngine=fcn-resnet101.engine --optShapes=input:1x3x1026x1282 --stronglyTyped"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Import required modules"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"import os\n",
|
||||
"import ctypes\n",
|
||||
"from cuda.bindings import runtime as cudart\n",
|
||||
"import tensorrt as trt\n",
|
||||
"\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"from PIL import Image\n",
|
||||
"\n",
|
||||
"TRT_LOGGER = trt.Logger()\n",
|
||||
"\n",
|
||||
"assert cudart.cudaSetDevice(0) == (cudart.cudaError_t.cudaSuccess,)\n",
|
||||
"\n",
|
||||
"# Filenames of TensorRT plan file and input/output images.\n",
|
||||
"engine_file = \"/workspace/fcn-resnet101.engine\"\n",
|
||||
"input_file = \"/workspace/input.ppm\"\n",
|
||||
"output_file = \"/workspace/output.ppm\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Utilities for input / output processing"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# For torchvision models, input images are loaded in to a range of [0, 1] and\n",
|
||||
"# normalized using mean = [0.485, 0.456, 0.406] and stddev = [0.229, 0.224, 0.225].\n",
|
||||
"def preprocess(image):\n",
|
||||
" # Mean normalization\n",
|
||||
" mean = np.array([0.485, 0.456, 0.406]).astype('float32')\n",
|
||||
" stddev = np.array([0.229, 0.224, 0.225]).astype('float32')\n",
|
||||
" data = (np.asarray(image).astype('float32') / float(255.0) - mean) / stddev\n",
|
||||
" # Switch from HWC to to CHW order\n",
|
||||
" return np.moveaxis(data, 2, 0)\n",
|
||||
"\n",
|
||||
"def postprocess(data):\n",
|
||||
" num_classes = 21\n",
|
||||
" # create a color palette, selecting a color for each class\n",
|
||||
" palette = np.array([2 ** 25 - 1, 2 ** 15 - 1, 2 ** 21 - 1])\n",
|
||||
" colors = np.array([palette*i%255 for i in range(num_classes)]).astype(\"uint8\")\n",
|
||||
" # plot the segmentation predictions for 21 classes in different colors\n",
|
||||
" img = Image.fromarray(data.astype('uint8'), mode='P')\n",
|
||||
" img.putpalette(colors)\n",
|
||||
" return img\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load TensorRT engine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Deserialize the TensorRT engine from specified plan file. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def load_engine(engine_file_path):\n",
|
||||
" assert os.path.exists(engine_file_path)\n",
|
||||
" print(\"Reading engine from file {}\".format(engine_file_path))\n",
|
||||
" with open(engine_file_path, \"rb\") as f, trt.Runtime(TRT_LOGGER) as runtime:\n",
|
||||
" return runtime.deserialize_cuda_engine(f.read())"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Inference pipeline"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Starting with a deserialized engine, TensorRT inference pipeline consists of the following steps:\n",
|
||||
"- Create an execution context and specify input shape (based on the image dimensions for inference).\n",
|
||||
"- Allocate CUDA device memory for input and output.\n",
|
||||
"- Allocate CUDA page-locked host memory to efficiently copy back the output.\n",
|
||||
"- Transfer the processed image data into input memory using asynchronous host-to-device CUDA copy.\n",
|
||||
"- Kickoff the TensorRT inference pipeline using the asynchronous execute API.\n",
|
||||
"- Transfer the segmentation output back into pagelocked host memory using device-to-host CUDA copy.\n",
|
||||
"- Synchronize the stream used for data transfers and inference execution to ensure all operations are completes.\n",
|
||||
"- Finally, write out the segmentation output to an image file for visualization."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def infer(engine, input_file, output_file):\n",
|
||||
" print(\"Reading input image from file {}\".format(input_file))\n",
|
||||
" with Image.open(input_file) as img:\n",
|
||||
" input_image = preprocess(img)\n",
|
||||
" image_width = img.width\n",
|
||||
" image_height = img.height\n",
|
||||
"\n",
|
||||
" with engine.create_execution_context() as context:\n",
|
||||
" input_buffers = {}\n",
|
||||
" input_memories = {}\n",
|
||||
"\n",
|
||||
" # Allocate host and device buffers\n",
|
||||
" tensor_names = [engine.get_tensor_name(i) for i in range(engine.num_io_tensors)]\n",
|
||||
" for tensor in tensor_names:\n",
|
||||
" size = trt.volume(context.get_tensor_shape(tensor))\n",
|
||||
" dtype = trt.nptype(engine.get_tensor_dtype(tensor))\n",
|
||||
"\n",
|
||||
" if engine.get_tensor_mode(tensor) == trt.TensorIOMode.INPUT:\n",
|
||||
" context.set_input_shape(tensor, (1, 3, image_height, image_width))\n",
|
||||
" input_buffers[tensor] = np.ascontiguousarray(input_image)\n",
|
||||
" err, input_memories[tensor] = cudart.cudaMalloc(input_image.nbytes)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
" context.set_tensor_address(tensor, input_memories[tensor])\n",
|
||||
" else:\n",
|
||||
" err, output_buffer_ptr = cudart.cudaMallocHost(size * dtype().itemsize)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
" pointer_type = ctypes.POINTER(np.ctypeslib.as_ctypes_type(dtype))\n",
|
||||
" output_buffer = np.ctypeslib.as_array(ctypes.cast(output_buffer_ptr, pointer_type), (size,))\n",
|
||||
"\n",
|
||||
" err, output_memory = cudart.cudaMalloc(output_buffer.nbytes)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
" context.set_tensor_address(tensor, output_memory)\n",
|
||||
"\n",
|
||||
" err, stream = cudart.cudaStreamCreate()\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
" # Transfer input data to the GPU for all input tensors\n",
|
||||
" for tensor_name, input_buffer in input_buffers.items():\n",
|
||||
" input_memory = input_memories[tensor_name]\n",
|
||||
" err, = cudart.cudaMemcpyAsync(input_memory, input_buffer.ctypes.data, input_buffer.nbytes,\n",
|
||||
" cudart.cudaMemcpyKind.cudaMemcpyHostToDevice, stream)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
" # Run inference\n",
|
||||
" context.execute_async_v3(stream)\n",
|
||||
"\n",
|
||||
" # Transfer prediction output from the GPU.\n",
|
||||
" err, = cudart.cudaMemcpyAsync(output_buffer.ctypes.data, output_memory, output_buffer.nbytes,\n",
|
||||
" cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost, stream)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
" # Synchronize the stream\n",
|
||||
" err, = cudart.cudaStreamSynchronize(stream)\n",
|
||||
" assert err == cudart.cudaError_t.cudaSuccess\n",
|
||||
"\n",
|
||||
" output_d64 = np.array(output_buffer, dtype=np.int64)\n",
|
||||
" np.savetxt('test.out', output_d64.astype(int), fmt='%i', delimiter=' ', newline=' ')\n",
|
||||
"\n",
|
||||
" with postprocess(np.reshape(output_buffer, (image_height, image_width))) as img:\n",
|
||||
" print(\"Writing output image to file {}\".format(output_file))\n",
|
||||
" img.convert('RGB').save(output_file, \"PPM\")\n",
|
||||
"\n",
|
||||
" # cleanup cuda resources for all input tensors\n",
|
||||
" for input_memory in input_memories.values():\n",
|
||||
" cudart.cudaFree(input_memory)\n",
|
||||
" cudart.cudaFree(output_memory)\n",
|
||||
" cudart.cudaFreeHost(output_buffer_ptr)\n",
|
||||
" cudart.cudaStreamDestroy(stream)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Plot input image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.imshow(Image.open(input_file))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Run inference"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"print(\"Running TensorRT inference for FCN-ResNet101\")\n",
|
||||
"with load_engine(engine_file) as engine:\n",
|
||||
" infer(engine, input_file, output_file)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Plot segmentation output"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"plt.imshow(Image.open(output_file))"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.12"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
#include "logger.h"
|
||||
#include "logging.h"
|
||||
|
||||
namespace sample
|
||||
{
|
||||
Logger gLogger{Logger::Severity::kINFO};
|
||||
LogStreamConsumer gLogVerbose{LOG_VERBOSE(gLogger)};
|
||||
LogStreamConsumer gLogInfo{LOG_INFO(gLogger)};
|
||||
LogStreamConsumer gLogWarning{LOG_WARN(gLogger)};
|
||||
LogStreamConsumer gLogError{LOG_ERROR(gLogger)};
|
||||
LogStreamConsumer gLogFatal{LOG_FATAL(gLogger)};
|
||||
|
||||
void setReportableSeverity(Logger::Severity severity)
|
||||
{
|
||||
gLogger.setReportableSeverity(severity);
|
||||
gLogVerbose.setReportableSeverity(severity);
|
||||
gLogInfo.setReportableSeverity(severity);
|
||||
gLogWarning.setReportableSeverity(severity);
|
||||
gLogError.setReportableSeverity(severity);
|
||||
gLogFatal.setReportableSeverity(severity);
|
||||
}
|
||||
} // namespace sample
|
||||
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
#ifndef TENSORRT_SAMPLE_COMMON_LOGGER_H
|
||||
#define TENSORRT_SAMPLE_COMMON_LOGGER_H
|
||||
|
||||
#include "logging.h"
|
||||
|
||||
namespace sample
|
||||
{
|
||||
extern Logger gLogger;
|
||||
extern LogStreamConsumer gLogVerbose;
|
||||
extern LogStreamConsumer gLogInfo;
|
||||
extern LogStreamConsumer gLogWarning;
|
||||
extern LogStreamConsumer gLogError;
|
||||
extern LogStreamConsumer gLogFatal;
|
||||
|
||||
void setReportableSeverity(Logger::Severity severity);
|
||||
} // namespace sample
|
||||
|
||||
#endif // TENSORRT_SAMPLE_COMMON_LOGGER_H
|
||||
@@ -0,0 +1,514 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
#ifndef TENSORRT_SAMPLE_COMMON_LOGGING_H
|
||||
#define TENSORRT_SAMPLE_COMMON_LOGGING_H
|
||||
|
||||
#include "NvInferRuntime.h"
|
||||
#include "NvInferRuntimeCommon.h"
|
||||
#include <cassert>
|
||||
#include <ctime>
|
||||
#include <iomanip>
|
||||
#include <iostream>
|
||||
#include <ostream>
|
||||
#include <sstream>
|
||||
#include <string>
|
||||
|
||||
namespace sample
|
||||
{
|
||||
|
||||
using Severity = nvinfer1::ILogger::Severity;
|
||||
|
||||
class LogStreamConsumerBuffer : public std::stringbuf
|
||||
{
|
||||
public:
|
||||
LogStreamConsumerBuffer(std::ostream& stream, const std::string& prefix, bool shouldLog)
|
||||
: mOutput(stream)
|
||||
, mPrefix(prefix)
|
||||
, mShouldLog(shouldLog)
|
||||
{
|
||||
}
|
||||
|
||||
LogStreamConsumerBuffer(LogStreamConsumerBuffer&& other)
|
||||
: mOutput(other.mOutput)
|
||||
, mPrefix(other.mPrefix)
|
||||
, mShouldLog(other.mShouldLog)
|
||||
{
|
||||
}
|
||||
|
||||
~LogStreamConsumerBuffer()
|
||||
{
|
||||
// std::streambuf::pbase() gives a pointer to the beginning of the buffered part of the output sequence
|
||||
// std::streambuf::pptr() gives a pointer to the current position of the output sequence
|
||||
// if the pointer to the beginning is not equal to the pointer to the current position,
|
||||
// call putOutput() to log the output to the stream
|
||||
if (pbase() != pptr())
|
||||
{
|
||||
putOutput();
|
||||
}
|
||||
}
|
||||
|
||||
// synchronizes the stream buffer and returns 0 on success
|
||||
// synchronizing the stream buffer consists of inserting the buffer contents into the stream,
|
||||
// resetting the buffer and flushing the stream
|
||||
virtual int sync()
|
||||
{
|
||||
putOutput();
|
||||
return 0;
|
||||
}
|
||||
|
||||
void putOutput()
|
||||
{
|
||||
if (mShouldLog)
|
||||
{
|
||||
// prepend timestamp
|
||||
std::time_t timestamp = std::time(nullptr);
|
||||
tm* tm_local = std::localtime(×tamp);
|
||||
std::cout << "[";
|
||||
std::cout << std::setw(2) << std::setfill('0') << 1 + tm_local->tm_mon << "/";
|
||||
std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_mday << "/";
|
||||
std::cout << std::setw(4) << std::setfill('0') << 1900 + tm_local->tm_year << "-";
|
||||
std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_hour << ":";
|
||||
std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_min << ":";
|
||||
std::cout << std::setw(2) << std::setfill('0') << tm_local->tm_sec << "] ";
|
||||
// std::stringbuf::str() gets the string contents of the buffer
|
||||
// insert the buffer contents pre-appended by the appropriate prefix into the stream
|
||||
mOutput << mPrefix << str();
|
||||
// set the buffer to empty
|
||||
str("");
|
||||
// flush the stream
|
||||
mOutput.flush();
|
||||
}
|
||||
}
|
||||
|
||||
void setShouldLog(bool shouldLog)
|
||||
{
|
||||
mShouldLog = shouldLog;
|
||||
}
|
||||
|
||||
private:
|
||||
std::ostream& mOutput;
|
||||
std::string mPrefix;
|
||||
bool mShouldLog;
|
||||
};
|
||||
|
||||
//!
|
||||
//! \class LogStreamConsumerBase
|
||||
//! \brief Convenience object used to initialize LogStreamConsumerBuffer before std::ostream in LogStreamConsumer
|
||||
//!
|
||||
class LogStreamConsumerBase
|
||||
{
|
||||
public:
|
||||
LogStreamConsumerBase(std::ostream& stream, const std::string& prefix, bool shouldLog)
|
||||
: mBuffer(stream, prefix, shouldLog)
|
||||
{
|
||||
}
|
||||
|
||||
protected:
|
||||
LogStreamConsumerBuffer mBuffer;
|
||||
};
|
||||
|
||||
//!
|
||||
//! \class LogStreamConsumer
|
||||
//! \brief Convenience object used to facilitate use of C++ stream syntax when logging messages.
|
||||
//! Order of base classes is LogStreamConsumerBase and then std::ostream.
|
||||
//! This is because the LogStreamConsumerBase class is used to initialize the LogStreamConsumerBuffer member field
|
||||
//! in LogStreamConsumer and then the address of the buffer is passed to std::ostream.
|
||||
//! This is necessary to prevent the address of an uninitialized buffer from being passed to std::ostream.
|
||||
//! Please do not change the order of the parent classes.
|
||||
//!
|
||||
class LogStreamConsumer : protected LogStreamConsumerBase, public std::ostream
|
||||
{
|
||||
public:
|
||||
//! \brief Creates a LogStreamConsumer which logs messages with level severity.
|
||||
//! Reportable severity determines if the messages are severe enough to be logged.
|
||||
LogStreamConsumer(Severity reportableSeverity, Severity severity)
|
||||
: LogStreamConsumerBase(severityOstream(severity), severityPrefix(severity), severity <= reportableSeverity)
|
||||
, std::ostream(&mBuffer) // links the stream buffer with the stream
|
||||
, mShouldLog(severity <= reportableSeverity)
|
||||
, mSeverity(severity)
|
||||
{
|
||||
}
|
||||
|
||||
LogStreamConsumer(LogStreamConsumer&& other)
|
||||
: LogStreamConsumerBase(severityOstream(other.mSeverity), severityPrefix(other.mSeverity), other.mShouldLog)
|
||||
, std::ostream(&mBuffer) // links the stream buffer with the stream
|
||||
, mShouldLog(other.mShouldLog)
|
||||
, mSeverity(other.mSeverity)
|
||||
{
|
||||
}
|
||||
|
||||
void setReportableSeverity(Severity reportableSeverity)
|
||||
{
|
||||
mShouldLog = mSeverity <= reportableSeverity;
|
||||
mBuffer.setShouldLog(mShouldLog);
|
||||
}
|
||||
|
||||
private:
|
||||
static std::ostream& severityOstream(Severity severity)
|
||||
{
|
||||
return severity >= Severity::kINFO ? std::cout : std::cerr;
|
||||
}
|
||||
|
||||
static std::string severityPrefix(Severity severity)
|
||||
{
|
||||
switch (severity)
|
||||
{
|
||||
case Severity::kINTERNAL_ERROR: return "[F] ";
|
||||
case Severity::kERROR: return "[E] ";
|
||||
case Severity::kWARNING: return "[W] ";
|
||||
case Severity::kINFO: return "[I] ";
|
||||
case Severity::kVERBOSE: return "[V] ";
|
||||
default: assert(0); return "";
|
||||
}
|
||||
}
|
||||
|
||||
bool mShouldLog;
|
||||
Severity mSeverity;
|
||||
};
|
||||
|
||||
//! \class Logger
|
||||
//!
|
||||
//! \brief Class which manages logging of TensorRT tools and samples
|
||||
//!
|
||||
//! \details This class provides a common interface for TensorRT tools and samples to log information to the console,
|
||||
//! and supports logging two types of messages:
|
||||
//!
|
||||
//! - Debugging messages with an associated severity (info, warning, error, or internal error/fatal)
|
||||
//! - Test pass/fail messages
|
||||
//!
|
||||
//! The advantage of having all samples use this class for logging as opposed to emitting directly to stdout/stderr is
|
||||
//! that the logic for controlling the verbosity and formatting of sample output is centralized in one location.
|
||||
//!
|
||||
//! In the future, this class could be extended to support dumping test results to a file in some standard format
|
||||
//! (for example, JUnit XML), and providing additional metadata (e.g. timing the duration of a test run).
|
||||
//!
|
||||
//! TODO: For backwards compatibility with existing samples, this class inherits directly from the nvinfer1::ILogger
|
||||
//! interface, which is problematic since there isn't a clean separation between messages coming from the TensorRT
|
||||
//! library and messages coming from the sample.
|
||||
//!
|
||||
//! In the future (once all samples are updated to use Logger::getTRTLogger() to access the ILogger) we can refactor the
|
||||
//! class to eliminate the inheritance and instead make the nvinfer1::ILogger implementation a member of the Logger
|
||||
//! object.
|
||||
|
||||
class Logger : public nvinfer1::ILogger
|
||||
{
|
||||
public:
|
||||
Logger(Severity severity = Severity::kWARNING)
|
||||
: mReportableSeverity(severity)
|
||||
{
|
||||
}
|
||||
|
||||
//!
|
||||
//! \enum TestResult
|
||||
//! \brief Represents the state of a given test
|
||||
//!
|
||||
enum class TestResult
|
||||
{
|
||||
kRUNNING, //!< The test is running
|
||||
kPASSED, //!< The test passed
|
||||
kFAILED, //!< The test failed
|
||||
kWAIVED //!< The test was waived
|
||||
};
|
||||
|
||||
//!
|
||||
//! \brief Forward-compatible method for retrieving the nvinfer::ILogger associated with this Logger
|
||||
//! \return The nvinfer1::ILogger associated with this Logger
|
||||
//!
|
||||
//! TODO Once all samples are updated to use this method to register the logger with TensorRT,
|
||||
//! we can eliminate the inheritance of Logger from ILogger
|
||||
//!
|
||||
nvinfer1::ILogger& getTRTLogger()
|
||||
{
|
||||
return *this;
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief Implementation of the nvinfer1::ILogger::log() virtual method
|
||||
//!
|
||||
//! Note samples should not be calling this function directly; it will eventually go away once we eliminate the
|
||||
//! inheritance from nvinfer1::ILogger
|
||||
//!
|
||||
void log(Severity severity, const char* msg) noexcept override
|
||||
{
|
||||
LogStreamConsumer(mReportableSeverity, severity) << "[TRT] " << std::string(msg) << std::endl;
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief Method for controlling the verbosity of logging output
|
||||
//!
|
||||
//! \param severity The logger will only emit messages that have severity of this level or higher.
|
||||
//!
|
||||
void setReportableSeverity(Severity severity)
|
||||
{
|
||||
mReportableSeverity = severity;
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief Opaque handle that holds logging information for a particular test
|
||||
//!
|
||||
//! This object is an opaque handle to information used by the Logger to print test results.
|
||||
//! The sample must call Logger::defineTest() in order to obtain a TestAtom that can be used
|
||||
//! with Logger::reportTest{Start,End}().
|
||||
//!
|
||||
class TestAtom
|
||||
{
|
||||
public:
|
||||
TestAtom(TestAtom&&) = default;
|
||||
|
||||
private:
|
||||
friend class Logger;
|
||||
|
||||
TestAtom(bool started, const std::string& name, const std::string& cmdline)
|
||||
: mStarted(started)
|
||||
, mName(name)
|
||||
, mCmdline(cmdline)
|
||||
{
|
||||
}
|
||||
|
||||
bool mStarted;
|
||||
std::string mName;
|
||||
std::string mCmdline;
|
||||
};
|
||||
|
||||
//!
|
||||
//! \brief Define a test for logging
|
||||
//!
|
||||
//! \param[in] name The name of the test. This should be a string starting with
|
||||
//! "TensorRT" and containing dot-separated strings containing
|
||||
//! the characters [A-Za-z0-9_].
|
||||
//! For example, "TensorRT.sample_googlenet"
|
||||
//! \param[in] cmdline The command line used to reproduce the test
|
||||
//
|
||||
//! \return a TestAtom that can be used in Logger::reportTest{Start,End}().
|
||||
//!
|
||||
static TestAtom defineTest(const std::string& name, const std::string& cmdline)
|
||||
{
|
||||
return TestAtom(false, name, cmdline);
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief A convenience overloaded version of defineTest() that accepts an array of command-line arguments
|
||||
//! as input
|
||||
//!
|
||||
//! \param[in] name The name of the test
|
||||
//! \param[in] argc The number of command-line arguments
|
||||
//! \param[in] argv The array of command-line arguments (given as C strings)
|
||||
//!
|
||||
//! \return a TestAtom that can be used in Logger::reportTest{Start,End}().
|
||||
static TestAtom defineTest(const std::string& name, int argc, char const* const* argv)
|
||||
{
|
||||
auto cmdline = genCmdlineString(argc, argv);
|
||||
return defineTest(name, cmdline);
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief Report that a test has started.
|
||||
//!
|
||||
//! \pre reportTestStart() has not been called yet for the given testAtom
|
||||
//!
|
||||
//! \param[in] testAtom The handle to the test that has started
|
||||
//!
|
||||
static void reportTestStart(TestAtom& testAtom)
|
||||
{
|
||||
reportTestResult(testAtom, TestResult::kRUNNING);
|
||||
assert(!testAtom.mStarted);
|
||||
testAtom.mStarted = true;
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief Report that a test has ended.
|
||||
//!
|
||||
//! \pre reportTestStart() has been called for the given testAtom
|
||||
//!
|
||||
//! \param[in] testAtom The handle to the test that has ended
|
||||
//! \param[in] result The result of the test. Should be one of TestResult::kPASSED,
|
||||
//! TestResult::kFAILED, TestResult::kWAIVED
|
||||
//!
|
||||
static void reportTestEnd(const TestAtom& testAtom, TestResult result)
|
||||
{
|
||||
assert(result != TestResult::kRUNNING);
|
||||
assert(testAtom.mStarted);
|
||||
reportTestResult(testAtom, result);
|
||||
}
|
||||
|
||||
static int reportPass(const TestAtom& testAtom)
|
||||
{
|
||||
reportTestEnd(testAtom, TestResult::kPASSED);
|
||||
return EXIT_SUCCESS;
|
||||
}
|
||||
|
||||
static int reportFail(const TestAtom& testAtom)
|
||||
{
|
||||
reportTestEnd(testAtom, TestResult::kFAILED);
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
|
||||
static int reportWaive(const TestAtom& testAtom)
|
||||
{
|
||||
reportTestEnd(testAtom, TestResult::kWAIVED);
|
||||
return EXIT_SUCCESS;
|
||||
}
|
||||
|
||||
static int reportTest(const TestAtom& testAtom, bool pass)
|
||||
{
|
||||
return pass ? reportPass(testAtom) : reportFail(testAtom);
|
||||
}
|
||||
|
||||
Severity getReportableSeverity() const
|
||||
{
|
||||
return mReportableSeverity;
|
||||
}
|
||||
|
||||
private:
|
||||
//!
|
||||
//! \brief returns an appropriate string for prefixing a log message with the given severity
|
||||
//!
|
||||
static const char* severityPrefix(Severity severity)
|
||||
{
|
||||
switch (severity)
|
||||
{
|
||||
case Severity::kINTERNAL_ERROR: return "[F] ";
|
||||
case Severity::kERROR: return "[E] ";
|
||||
case Severity::kWARNING: return "[W] ";
|
||||
case Severity::kINFO: return "[I] ";
|
||||
case Severity::kVERBOSE: return "[V] ";
|
||||
default: assert(0); return "";
|
||||
}
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief returns an appropriate string for prefixing a test result message with the given result
|
||||
//!
|
||||
static const char* testResultString(TestResult result)
|
||||
{
|
||||
switch (result)
|
||||
{
|
||||
case TestResult::kRUNNING: return "RUNNING";
|
||||
case TestResult::kPASSED: return "PASSED";
|
||||
case TestResult::kFAILED: return "FAILED";
|
||||
case TestResult::kWAIVED: return "WAIVED";
|
||||
default: assert(0); return "";
|
||||
}
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief returns an appropriate output stream (cout or cerr) to use with the given severity
|
||||
//!
|
||||
static std::ostream& severityOstream(Severity severity)
|
||||
{
|
||||
return severity >= Severity::kINFO ? std::cout : std::cerr;
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief method that implements logging test results
|
||||
//!
|
||||
static void reportTestResult(const TestAtom& testAtom, TestResult result)
|
||||
{
|
||||
severityOstream(Severity::kINFO) << "&&&& " << testResultString(result) << " " << testAtom.mName << " # "
|
||||
<< testAtom.mCmdline << std::endl;
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief generate a command line string from the given (argc, argv) values
|
||||
//!
|
||||
static std::string genCmdlineString(int argc, char const* const* argv)
|
||||
{
|
||||
std::stringstream ss;
|
||||
for (int i = 0; i < argc; i++)
|
||||
{
|
||||
if (i > 0)
|
||||
{
|
||||
ss << " ";
|
||||
}
|
||||
ss << argv[i];
|
||||
}
|
||||
return ss.str();
|
||||
}
|
||||
|
||||
Severity mReportableSeverity;
|
||||
};
|
||||
|
||||
namespace
|
||||
{
|
||||
|
||||
//!
|
||||
//! \brief produces a LogStreamConsumer object that can be used to log messages of severity kVERBOSE
|
||||
//!
|
||||
//! Example usage:
|
||||
//!
|
||||
//! LOG_VERBOSE(logger) << "hello world" << std::endl;
|
||||
//!
|
||||
inline LogStreamConsumer LOG_VERBOSE(const Logger& logger)
|
||||
{
|
||||
return LogStreamConsumer(logger.getReportableSeverity(), Severity::kVERBOSE);
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief produces a LogStreamConsumer object that can be used to log messages of severity kINFO
|
||||
//!
|
||||
//! Example usage:
|
||||
//!
|
||||
//! LOG_INFO(logger) << "hello world" << std::endl;
|
||||
//!
|
||||
inline LogStreamConsumer LOG_INFO(const Logger& logger)
|
||||
{
|
||||
return LogStreamConsumer(logger.getReportableSeverity(), Severity::kINFO);
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief produces a LogStreamConsumer object that can be used to log messages of severity kWARNING
|
||||
//!
|
||||
//! Example usage:
|
||||
//!
|
||||
//! LOG_WARN(logger) << "hello world" << std::endl;
|
||||
//!
|
||||
inline LogStreamConsumer LOG_WARN(const Logger& logger)
|
||||
{
|
||||
return LogStreamConsumer(logger.getReportableSeverity(), Severity::kWARNING);
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief produces a LogStreamConsumer object that can be used to log messages of severity kERROR
|
||||
//!
|
||||
//! Example usage:
|
||||
//!
|
||||
//! LOG_ERROR(logger) << "hello world" << std::endl;
|
||||
//!
|
||||
inline LogStreamConsumer LOG_ERROR(const Logger& logger)
|
||||
{
|
||||
return LogStreamConsumer(logger.getReportableSeverity(), Severity::kERROR);
|
||||
}
|
||||
|
||||
//!
|
||||
//! \brief produces a LogStreamConsumer object that can be used to log messages of severity kINTERNAL_ERROR
|
||||
//! ("fatal" severity)
|
||||
//!
|
||||
//! Example usage:
|
||||
//!
|
||||
//! LOG_FATAL(logger) << "hello world" << std::endl;
|
||||
//!
|
||||
inline LogStreamConsumer LOG_FATAL(const Logger& logger)
|
||||
{
|
||||
return LogStreamConsumer(logger.getReportableSeverity(), Severity::kINTERNAL_ERROR);
|
||||
}
|
||||
|
||||
} // anonymous namespace
|
||||
|
||||
} // namespace sample
|
||||
|
||||
#endif // TENSORRT_SAMPLE_COMMON_LOGGING_H
|
||||
@@ -0,0 +1,139 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
#include <algorithm>
|
||||
#include <cassert>
|
||||
#include <fstream>
|
||||
#include <iostream>
|
||||
#include <memory>
|
||||
#include <numeric>
|
||||
#include <string>
|
||||
|
||||
#include "NvInfer.h"
|
||||
#include "util.h"
|
||||
|
||||
namespace util
|
||||
{
|
||||
|
||||
size_t getMemorySize(const nvinfer1::Dims& dims, const int32_t elem_size)
|
||||
{
|
||||
return std::accumulate(dims.d, dims.d + dims.nbDims, 1, std::multiplies<int64_t>()) * elem_size;
|
||||
}
|
||||
|
||||
ImageBase::ImageBase(const std::string& filename, const nvinfer1::Dims& dims)
|
||||
: mDims(dims)
|
||||
{
|
||||
assert(4 == mDims.nbDims);
|
||||
assert(1 == mDims.d[0]);
|
||||
mPPM.filename = filename;
|
||||
}
|
||||
|
||||
size_t ImageBase::volume() const
|
||||
{
|
||||
return mDims.d[3] /* w */ * mDims.d[2] /* h */ * 3;
|
||||
}
|
||||
|
||||
void ImageBase::read()
|
||||
{
|
||||
std::ifstream infile(mPPM.filename, std::ifstream::binary);
|
||||
if (!infile.is_open())
|
||||
{
|
||||
std::cerr << "ERROR: cannot open PPM image file: " << mPPM.filename << std::endl;
|
||||
}
|
||||
infile >> mPPM.magic >> mPPM.w >> mPPM.h >> mPPM.max;
|
||||
|
||||
infile.seekg(1, infile.cur);
|
||||
mPPM.buffer.resize(volume());
|
||||
infile.read(reinterpret_cast<char*>(mPPM.buffer.data()), volume());
|
||||
infile.close();
|
||||
}
|
||||
|
||||
void ImageBase::write()
|
||||
{
|
||||
std::ofstream outfile(mPPM.filename, std::ofstream::binary);
|
||||
if (!outfile.is_open())
|
||||
{
|
||||
std::cerr << "ERROR: cannot open PPM image file: " << mPPM.filename << std::endl;
|
||||
}
|
||||
outfile << mPPM.magic << " " << mPPM.w << " " << mPPM.h << " " << mPPM.max << std::endl;
|
||||
outfile.write(reinterpret_cast<char*>(mPPM.buffer.data()), volume());
|
||||
outfile.close();
|
||||
}
|
||||
|
||||
RGBImageReader::RGBImageReader(const std::string& filename, const nvinfer1::Dims& dims, const std::vector<float>& mean, const std::vector<float>& std)
|
||||
: ImageBase(filename, dims)
|
||||
, mMean(mean)
|
||||
, mStd(std)
|
||||
{
|
||||
}
|
||||
|
||||
std::unique_ptr<float> RGBImageReader::process() const
|
||||
{
|
||||
const int C = mDims.d[1];
|
||||
const int H = mDims.d[2];
|
||||
const int W = mDims.d[3];
|
||||
auto buffer = std::unique_ptr<float>{new float[volume()]};
|
||||
|
||||
if (mPPM.h == H && mPPM.w == W)
|
||||
{
|
||||
for (int c = 0; c < C; c++)
|
||||
{
|
||||
for (int j = 0, HW = H * W; j < HW; ++j)
|
||||
{
|
||||
buffer.get()[c * HW + j] = (static_cast<float>(mPPM.buffer[j * C + c])/mPPM.max - mMean[c]) / mStd[c];
|
||||
}
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
assert(!"Specified dimensions don't match PPM image");
|
||||
}
|
||||
|
||||
return buffer;
|
||||
}
|
||||
|
||||
ArgmaxImageWriter::ArgmaxImageWriter(const std::string& filename, const nvinfer1::Dims& dims, const std::vector<int>& palette, const int num_classes)
|
||||
: ImageBase(filename, dims)
|
||||
, mNumClasses(num_classes)
|
||||
, mPalette(palette)
|
||||
{
|
||||
}
|
||||
|
||||
void ArgmaxImageWriter::process(const int* buffer)
|
||||
{
|
||||
mPPM.magic = "P6";
|
||||
mPPM.w = mDims.d[3];
|
||||
mPPM.h = mDims.d[2];
|
||||
mPPM.max = 255;
|
||||
mPPM.buffer.resize(volume());
|
||||
std::vector<std::vector<int>> colors;
|
||||
for (auto i = 0, max = mPPM.max; i < mNumClasses; i++)
|
||||
{
|
||||
std::vector<int> c{mPalette};
|
||||
std::transform(c.begin(), c.end(), c.begin(), [i, max](int p){return (p*i) % max;});
|
||||
colors.push_back(c);
|
||||
}
|
||||
for (int j = 0, HW = mPPM.h * mPPM.w; j < HW; ++j)
|
||||
{
|
||||
auto clsid{static_cast<uint8_t>(buffer[j])};
|
||||
mPPM.buffer.data()[j*3] = colors[clsid][0];
|
||||
mPPM.buffer.data()[j*3+1] = colors[clsid][1];
|
||||
mPPM.buffer.data()[j*3+2] = colors[clsid][2];
|
||||
}
|
||||
}
|
||||
|
||||
}; // namespace util
|
||||
@@ -0,0 +1,77 @@
|
||||
/*
|
||||
* SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
#ifndef TENSORRT_SAMPLE_COMMON_UTILS_H
|
||||
#define TENSORRT_SAMPLE_COMMON_UTILS_H
|
||||
#include <memory>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "NvInfer.h"
|
||||
|
||||
namespace util
|
||||
{
|
||||
|
||||
size_t getMemorySize(const nvinfer1::Dims& dims, const int32_t elem_size);
|
||||
|
||||
struct PPM
|
||||
{
|
||||
std::string filename;
|
||||
std::string magic;
|
||||
int c;
|
||||
int h;
|
||||
int w;
|
||||
int max;
|
||||
std::vector<uint8_t> buffer;
|
||||
};
|
||||
|
||||
class ImageBase
|
||||
{
|
||||
public:
|
||||
ImageBase(const std::string& filename, const nvinfer1::Dims& dims);
|
||||
virtual ~ImageBase() {}
|
||||
virtual size_t volume() const;
|
||||
void read();
|
||||
void write();
|
||||
protected:
|
||||
nvinfer1::Dims mDims;
|
||||
PPM mPPM;
|
||||
};
|
||||
|
||||
class RGBImageReader : public ImageBase
|
||||
{
|
||||
public:
|
||||
RGBImageReader(const std::string& filename, const nvinfer1::Dims& dims, const std::vector<float>& mean, const std::vector<float>& std);
|
||||
std::unique_ptr<float> process() const;
|
||||
private:
|
||||
std::vector<float> mMean;
|
||||
std::vector<float> mStd;
|
||||
};
|
||||
|
||||
class ArgmaxImageWriter : public ImageBase
|
||||
{
|
||||
public:
|
||||
ArgmaxImageWriter(const std::string& filename, const nvinfer1::Dims& dims, const std::vector<int>& palette, const int num_classes);
|
||||
void process(const int* buffer);
|
||||
private:
|
||||
int mNumClasses;
|
||||
std::vector<int> mPalette;
|
||||
};
|
||||
|
||||
}; // namespace util
|
||||
|
||||
#endif // TENSORRT_SAMPLE_COMMON_UTILS_H
|
||||
@@ -0,0 +1,129 @@
|
||||
# TensorRT to Triton
|
||||
|
||||
This quick start guide showcases how to deploy a simple ResNet model accelerated by using TensorRT on Triton Inference Server. Optimization and deployment go hand in hand in a discussion about Machine Learning infrastructure. For a TensorRT user, network level optimization to get the maximum performance would already be an area of expertize.
|
||||
|
||||
However, serving this optimized model comes with its own set of considerations and challenges, such as building infrastructure to support concurrent model executions, supporting clients over HTTP, gRPC and more.
|
||||
|
||||
The [Triton Inference Server](https://github.com/triton-inference-server/server) solves the aforementioned and more. Let's discuss step-by-step, the process of optimizing a model with Torch-TensorRT, deploying it on Triton Inference Server, and building a client to query the model.
|
||||
|
||||
|
||||
## Step 1: Optimize your model with TensorRT
|
||||
|
||||
If you are unfamiliar with TensorRT, please refer to this [video](https://youtu.be/rK-jxPPY9V4). The first step in this pipeline is to accelerate your model with TensorRT. For the purposes of this demonstration, we are going to assume that you have your trained model in ONNX format.
|
||||
|
||||
(Optional) If you don't have an ONNX model handy and just want to follow along, feel free to use this script:
|
||||
```
|
||||
# <xx.xx> is the yy:mm for the publishing tag for NVIDIA's TensorRT
|
||||
# container; eg. 24.07
|
||||
# Check https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch for latest releases
|
||||
|
||||
docker run -it --gpus all -v /path/to/this/folder:/resnet50_eg nvcr.io/nvidia/pytorch:<xx.xx>-py3
|
||||
|
||||
python export_resnet_to_onnx.py
|
||||
exit
|
||||
```
|
||||
|
||||
You may need to create an account and get the API key from [here](https://ngc.nvidia.com/setup/). Sign up and login with your key (follow the instructions [here](https://ngc.nvidia.com/setup/api-key) after signing up).
|
||||
|
||||
Now that we have an ONNX model, we can use TensorRT to optimize your model. These optimizations are stored in the form of a TensorRT Engine, also known as a TensorRT plan file.
|
||||
|
||||
While there are several ways of installing TensorRT, the easiest way is to simply get our pre-built docker container.
|
||||
|
||||
```
|
||||
docker run -it --gpus all -v /path/to/this/folder:/trt_optimize nvcr.io/nvidia/tensorrt:<xx:yy>-py3
|
||||
```
|
||||
There are several ways to build a TensorRT Engine; for this demonstration, we will simply use the `trtexec` [CLI Tool](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#trtexec).
|
||||
|
||||
```
|
||||
trtexec --onnx=resnet50.onnx \
|
||||
--saveEngine=model.plan \
|
||||
--useCudaGraph
|
||||
```
|
||||
|
||||
Before we proceed to the next step, it is important that we know the names of the "input" and "output" layers of your network, as these would be required by Triton. One easy way is to use `polygraphy` which comes packaged with the TensorRT container. If you want to learn more about Polygraphy and its usage, visit [this](https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy) repository. You can checkout a plethora of [examples](https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy/examples/cli/inspect) demonstrating the utility of Polygraphy to inspect models.
|
||||
|
||||
```
|
||||
polygraphy inspect model model.plan
|
||||
```
|
||||
A section of the output mode looks like this:
|
||||
```
|
||||
[I] ==== TensorRT Engine ====
|
||||
Name: Unnamed Network 0 | Explicit Batch Engine
|
||||
|
||||
---- 1 Engine Input(s) ----
|
||||
{input [dtype=float32, shape=(1, 3, 224, 224)]}
|
||||
|
||||
---- 1 Engine Output(s) ----
|
||||
{output [dtype=float32, shape=(1, 1000)]}
|
||||
|
||||
---- Memory ----
|
||||
Device Memory: 8228864 bytes
|
||||
|
||||
---- 1 Profile(s) (2 Tensor(s) Each) ----
|
||||
- Profile: 0
|
||||
Tensor: input (Input), Index: 0 | Shapes: min=(1, 3, 224, 224), opt=(1, 3, 224, 224), max=(1, 3, 224, 224)
|
||||
Tensor: output (Output), Index: 1 | Shape: (1, 1000)
|
||||
|
||||
---- 87 Layer(s) ----
|
||||
```
|
||||
|
||||
With this, we are ready to proceed to the next step; setting up the Triton Inference Server.
|
||||
|
||||
## Step 2: Set Up Triton Inference Server
|
||||
|
||||
If you are new to the Triton Inference Server and want to learn more, we highly recommend to check out our [Github Repository](https://github.com/triton-inference-server).
|
||||
|
||||
To use Triton, we need to make a model repository. A model repository, as the name suggested, is a repository of the models the Inference server hosts. While Triton can serve models from multiple repositories, in this example, we will discuss the simplest possible form of the model repository. To use Triton, we need to make a model repository. The structure of the repository should look something like this:
|
||||
```
|
||||
model_repository
|
||||
|
|
||||
+-- resnet50
|
||||
|
|
||||
+-- config.pbxt
|
||||
+-- 1
|
||||
|
|
||||
+-- model.plan
|
||||
```
|
||||
|
||||
There are two files that Triton requires to serve the model: the model itself and a model configuration file which is typically provided in `config.pbtxt`. We provide a sample of a `config.pbtxt`, which you can use for this specific example. The `config.pbtxt` file is used to describe the exact model configuration with details like the names and shapes of the input and output layer(s), datatypes, scheduling and batching details and more. If you are new to Triton, we highly encourage you to check out this [section of our documentation](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md) for more.
|
||||
|
||||
Once you have the model repository setup, it is time to launch the triton server. You can do that with the docker command below.
|
||||
```
|
||||
# Make sure that the TensorRT version in the Triton container
|
||||
# and TensorRT version in the environment used to optimize the model
|
||||
# are the same. <xx.yy> as 24.07 will work in this example
|
||||
|
||||
|
||||
docker run --gpus all --rm -p 8000:8000 -p 8001:8001 -p 8002:8002 -v /full/path/to/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models
|
||||
```
|
||||
|
||||
## Step 3: Using a Triton Client to Query the Server
|
||||
|
||||
Before proceeding, make sure to have a sample image on hand. If you don't have one, download an example image to test inference. In this section, we will be going over a very basic client. For a variety of more fleshed out examples, refer to the [Triton Client Repository](https://github.com/triton-inference-server/client/tree/main/src/python/examples).
|
||||
|
||||
```
|
||||
wget -O img1.jpg "https://www.hakaimagazine.com/wp-content/uploads/header-gulf-birds.jpg"
|
||||
```
|
||||
|
||||
Install dependencies.
|
||||
```
|
||||
pip install torchvision
|
||||
pip install attrdict
|
||||
pip install nvidia-pyindex
|
||||
pip install tritonclient[all]
|
||||
```
|
||||
|
||||
Building a client requires three basic points.
|
||||
* Firstly, we setup a connection with the Triton Inference Server.
|
||||
* Secondly, we specify the names of the input and output layer(s) of our model.
|
||||
* Lastly, we send an inference request to the Triton Inference Server.
|
||||
|
||||
You can find the corresponding functions of the same in the sample client.
|
||||
```
|
||||
python3 triton_client.py
|
||||
```
|
||||
The output of the same should look like below:
|
||||
```
|
||||
[b'12.477911:90' b'11.527293:92' b'9.662313:14' b'8.411008:136' b'8.220069:11']
|
||||
```
|
||||
The output format here is `<confidence_score>:<classification_index>`. To learn how to map these to the label names and more, refer to our [documentation](https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_classification.md).
|
||||
@@ -0,0 +1,36 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
name: "resnet50"
|
||||
platform: "tensorrt_plan"
|
||||
max_batch_size : 0
|
||||
input [
|
||||
{
|
||||
name: "input"
|
||||
data_type: TYPE_FP32
|
||||
dims: [ 3, 224, 224 ]
|
||||
reshape { shape: [ 1, 3, 224, 224 ] }
|
||||
}
|
||||
]
|
||||
output [
|
||||
{
|
||||
name: "output"
|
||||
data_type: TYPE_FP32
|
||||
dims: [ 1, 1000 ,1, 1]
|
||||
reshape { shape: [ 1, 1000 ] }
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,34 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import torch
|
||||
import torchvision.models as models
|
||||
|
||||
torch.hub._validate_not_a_forked_repo=lambda a,b,c: True
|
||||
|
||||
# load model; We are going to use a pretrained resnet model
|
||||
model = models.resnet50(pretrained=True).eval()
|
||||
x = torch.randn(1, 3, 224, 224, requires_grad=True)
|
||||
|
||||
# Export the model
|
||||
torch.onnx.export(model, # model being run
|
||||
x, # model input (or a tuple for multiple inputs)
|
||||
"resnet50.onnx", # where to save the model (can be a file or file-like object)
|
||||
export_params=True, # store the trained parameter weights inside the model file
|
||||
input_names = ['input'], # the model's input names
|
||||
output_names = ['output'], # the model's output names
|
||||
)
|
||||
@@ -0,0 +1,49 @@
|
||||
#
|
||||
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
import numpy as np
|
||||
from torchvision import transforms
|
||||
from PIL import Image
|
||||
import tritonclient.http as httpclient
|
||||
from tritonclient.utils import triton_to_np_dtype
|
||||
|
||||
def rn50_preprocess(img_path="img1.jpg"):
|
||||
img = Image.open(img_path)
|
||||
preprocess = transforms.Compose([
|
||||
transforms.Resize(256),
|
||||
transforms.CenterCrop(224),
|
||||
transforms.ToTensor(),
|
||||
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
|
||||
])
|
||||
return preprocess(img).numpy()
|
||||
|
||||
transformed_img = rn50_preprocess()
|
||||
|
||||
# Setup a connection with the Triton Inference Server.
|
||||
triton_client = httpclient.InferenceServerClient(url="localhost:8000")
|
||||
|
||||
# Specify the names of the input and output layer(s) of our model.
|
||||
test_input = httpclient.InferInput("input", transformed_img.shape, datatype="FP32")
|
||||
test_input.set_data_from_numpy(transformed_img, binary_data=True)
|
||||
|
||||
test_output = httpclient.InferRequestedOutput("output", binary_data=True, class_count=1000)
|
||||
|
||||
# Querying the server
|
||||
results = triton_client.infer(model_name="resnet50", inputs=[test_input], outputs=[test_output])
|
||||
test_output_fin = results.as_numpy('output')
|
||||
|
||||
print(test_output_fin[:5])
|
||||
Reference in New Issue
Block a user