wehub-resource-sync
258 lines
9.9 KiB
Markdown
258 lines
9.9 KiB
Markdown
# Polygraphy Python API
|
|
|
|
## Table of Contents
|
|
|
|
- [Introduction](#introduction)
|
|
- [Backends](#backends)
|
|
- [Loaders](#loaders)
|
|
- [Runners](#runners)
|
|
- [Writing A Custom Runner](#writing-a-custom-runner)
|
|
- [Comparator](#comparator)
|
|
- [Data Loaders](#data-loaders)
|
|
- [Logger](#logger)
|
|
- [Putting It All Together](#putting-it-all-together)
|
|
- [Enabling PyTorch](#enabling-pytorch)
|
|
- [Examples](#examples)
|
|
- [Python API Reference Documentation](#python-api-reference-documentation)
|
|
- [Building Python API Documentation Locally](#building-python-api-documentation-locally)
|
|
- [Deprecation Policy](#deprecation-policy)
|
|
|
|
## Introduction
|
|
|
|
The Polygraphy API consists broadly of two major components:
|
|
[`Backend`s](#backends) and the [`Comparator`](#comparator).
|
|
|
|
**NOTE:** To help you get started with the API, you can use the [`run`](./tools/run/) tool
|
|
with the `--gen-script` option to auto-generate template scripts that use the Polygraphy API.
|
|
|
|
> :warning: Any APIs not documented in the [API reference documentation](#python-api-reference-documentation)
|
|
should be considered internal only and do not adhere to the [deprecation policy](#deprecation-policy)
|
|
for public APIs. Thus, they may be modified or removed at any time without warning.
|
|
Avoid using undocumented APIs!
|
|
|
|
|
|
## Backends
|
|
|
|
A Polygraphy backend provides an interface for a deep learning framework.
|
|
Backends are comprised of two components: Loaders and Runners.
|
|
|
|
|
|
### Loaders
|
|
|
|
A `Loader` is a functor or callable that loads or operates on models in some way.
|
|
|
|
Existing `Loader`s can be composed for more advanced behaviors.
|
|
For example, we can implement a conversion like `ONNX -> TensorRT Network -> TensorRT Engine`:
|
|
|
|
```python
|
|
from polygraphy.backend.trt import EngineFromNetwork, NetworkFromOnnxPath
|
|
|
|
build_engine = EngineFromNetwork(NetworkFromOnnxPath("/path/to/model.onnx"))
|
|
```
|
|
|
|
`build_engine` is a callable that will build a TensorRT engine.
|
|
|
|
Polygraphy also provides immediately evaluated functional variants of each Loader.
|
|
These use the same names, except `snake_case` instead of `PascalCase`, and expose the same APIs.
|
|
Using the functional loaders, the conversion above would be:
|
|
|
|
```python
|
|
from polygraphy.backend.trt import engine_from_network, network_from_onnx_path
|
|
|
|
engine = engine_from_network(network_from_onnx_path("/path/to/model.onnx"))
|
|
```
|
|
|
|
`engine` is a TensorRT engine as opposed to a callable.
|
|
|
|
|
|
### Runners
|
|
|
|
A `Runner` uses a loader to load a model and can then run inference (see [`BaseRunner`](./backend/base/runner.py)).
|
|
|
|
**IMPORTANT:** Runners may reuse their output buffers. Thus, if you need to save outputs from multiple inferences, you should
|
|
make a copy of the outputs with `copy.deepcopy(outputs)`.
|
|
|
|
To use a runner, you just need to activate it, and then call `infer()`.
|
|
Note that activating a runner can be very expensive, so you should minimize the
|
|
number of times you activate a runner - ideally do not do this more than once.
|
|
|
|
It is recommended to use a context manager to activate and deactivate the
|
|
runner rather than calling the functions manually:
|
|
|
|
```python
|
|
from polygraphy.backend.trt import TrtRunner
|
|
|
|
with TrtRunner(build_engine) as runner:
|
|
outputs = runner.infer(feed_dict={"input0": input_data})
|
|
```
|
|
|
|
#### Writing A Custom Runner
|
|
|
|
Generally, you do not need to write custom runners unless you want to support a new backend.
|
|
|
|
In case you do, in the simplest case, you only need to implement two functions:
|
|
- `infer_impl`: Accepts a dictionary of numpy buffers, runs inference, and finally returns a dictionary containing the outputs.
|
|
- `get_input_metadata_impl`: Returns a [`TensorMetadata`](./common/struct.py) mapping input names to their shapes and data types.
|
|
You may use `None`, negative numbers, or strings to indicate dynamic dimensions.
|
|
|
|
For more advanced runners, where some setup is required, you may also need to implement the `activate_impl()` and `deactivate_impl()` functions.
|
|
|
|
For example, in the `TrtRunner`, engines are created in `activate_impl()` and destroyed in `deactivate_impl()`.
|
|
Importantly, the GPU is *not used at all* until these functions have been called (notice, for example,
|
|
that in the `TrtRunner`, the CUDA runtime library is only loaded in the `activate_impl()` function).
|
|
This allows the `Comparator` to optionally provide each runner with exclusive access to the GPU, to prevent any interference between runners.
|
|
|
|
|
|
## Comparator
|
|
|
|
The `Comparator` is used to run inference for runners, and then compare accuracy (see [Comparator.py](./comparator/comparator.py)).
|
|
This process is divided into two phases:
|
|
|
|
1. Running inference:
|
|
|
|
```python
|
|
run_results = Comparator.run(runners)
|
|
```
|
|
|
|
This function accepts a list of runners and returns a `RunResults` object (see [Comparator.py](./comparator/comparator.py))
|
|
containing the inference outputs of each run.
|
|
It also accepts an optional `data_loader` argument to control the input data. If not provided, it will use the
|
|
default data loader. `Comparator.run()` continues until inputs from the data loader are exhausted.
|
|
|
|
2. Comparing results:
|
|
|
|
```python
|
|
Comparator.compare_accuracy(run_results)
|
|
```
|
|
|
|
This function accepts the results returned by `Comparator.run` and compares them between runners.
|
|
|
|
**IMPORTANT:** The Comparator is designed for scenarios where you need to compare a small number
|
|
of inputs across multiple runners. It is **not** a good idea to use it
|
|
to validate a model with an entire dataset! Instead, runners should be used directly for such
|
|
cases (see the [example](../examples/api/02_validating_on_a_dataset)).
|
|
|
|
|
|
### Data Loaders
|
|
|
|
A data loader is used by the `Comparator` to load input data to feed to each runner
|
|
(for example, see Polygraphy's [default data loader](./comparator/data_loader.py)).
|
|
|
|
A data loader can be any generator or iterable that yields
|
|
a dictionary of input buffers. In the simplest case, this can just be a `list` of `dict`s.
|
|
|
|
In case you don't know details about the inputs ahead of time, you can access the `input_metadata`
|
|
property in your data loader, which will be set to an `TensorMetadata` instance by the Comparator.
|
|
|
|
**NOTE:** Polygraphy provides a default `DataLoader` class that uses numpy to generate random input buffers.
|
|
The input data can be bounded via parameters to the constructor.
|
|
|
|
|
|
## Logger
|
|
|
|
Polygraphy also includes a global logger which can control the verbosity not only of messages emitted by Polygraphy,
|
|
but also of those emitted by underlying frameworks, like TensorRT. For example, the `EXTRA_VERBOSE` verbosity corresponds
|
|
to TensorRT's `kVERBOSE` logging mode.
|
|
|
|
To set the verbosity of the logger, use:
|
|
```py
|
|
G_LOGGER.module_severity = severity
|
|
```
|
|
|
|
For example:
|
|
```py
|
|
G_LOGGER.module_severity = G_LOGGER.EXTRA_VERBOSE
|
|
```
|
|
|
|
By default logs are emitted to the stdout/stderr stream. If you would like to have them stored in a file, set the `log_file` property to your log file:
|
|
```py
|
|
G_LOGGER.log_file = "your_log_file.log"
|
|
```
|
|
|
|
If you would like to have logs being emitted to python `logging` module, set the following flag:
|
|
```py
|
|
G_LOGGER.use_python_logging_system = True
|
|
```
|
|
|
|
After that, you can define your logging configuration using the `logging` module.
|
|
|
|
## Putting It All Together
|
|
|
|
Now that you know the basic components of Polygraphy, let's take a look at how they fit together.
|
|
|
|
In this example, we will write a script that:
|
|
1. Builds a TensorRT engine from an ONNX model
|
|
2. Bounds input values in the range `[0, 2]`
|
|
3. Runs inference using ONNX-Runtime and TensorRT
|
|
4. Compares the results and checks that they match
|
|
|
|
```python
|
|
from polygraphy.backend.onnxrt import OnnxrtRunner, SessionFromOnnx
|
|
from polygraphy.backend.trt import TrtRunner, EngineFromNetwork, NetworkFromOnnxPath
|
|
from polygraphy.comparator import Comparator, DataLoader
|
|
|
|
model_path = "/path/to/model.onnx"
|
|
|
|
build_onnxrt_session = SessionFromOnnx(model_path)
|
|
build_engine = EngineFromNetwork(NetworkFromOnnxPath(model_path))
|
|
|
|
runners = [
|
|
OnnxrtRunner(build_onnxrt_session),
|
|
TrtRunner(build_engine),
|
|
]
|
|
|
|
data_loader = DataLoader(val_range=(0, 2))
|
|
|
|
run_results = Comparator.run(runners, data_loader=data_loader)
|
|
assert bool(Comparator.compare_accuracy(run_results))
|
|
```
|
|
|
|
## Enabling PyTorch
|
|
|
|
In order to enable PyTorch, you need to provide three things to the `PytRunner`:
|
|
1. A model loader: In the simplest case, this can be a callable that returns a `torch.nn.Module`.
|
|
|
|
2. `input_metadata`: A `TensorMetadata` describing the inputs of the model. This maps input names to their shapes and data types. As with other runners, `None` may be used to indicate dynamic dimensions.
|
|
|
|
**NOTE:** Other runners are able to automatically determine input metadata by inspecting the model definition, but because of the way PyTorch is implemented, it is difficult to write a generic function to determine model inputs from a `torch.nn.Module`.
|
|
|
|
3. `output_names`: A list of output names. This is used by the `Comparator` to match `PytRunner` outputs to those of other runners.
|
|
|
|
|
|
## Examples
|
|
|
|
You can find complete code examples that use the Polygraphy Python API [here](../examples/api).
|
|
|
|
|
|
## Python API Reference Documentation
|
|
|
|
For more details, see the [Polygraphy Python API reference documentation](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/polygraphy/index.html).
|
|
|
|
### Building Python API Documentation Locally
|
|
|
|
To build the API documentation, first install required packages:
|
|
|
|
```bash
|
|
python -m pip install -r docs/requirements.txt
|
|
```
|
|
|
|
and then use the `make` target to build docs:
|
|
|
|
```bash
|
|
make docs
|
|
```
|
|
|
|
The HTML documentation will be generated under `build/docs`
|
|
To view the docs, open `build/docs/index.html` in a browser or HTML viewer.
|
|
|
|
## Deprecation Policy
|
|
|
|
When removing or modifying public APIs in breaking ways, Polygraphy follows the following procedure:
|
|
|
|
1. The API is marked to be removed in some future version of Polygraphy (let's call it version `N`).
|
|
|
|
2. The API continues to work as normal until *at least* version `N`, but will issue deprecation warnings when used.
|
|
These warnings typically suggest which new APIs should be used instead.
|
|
|
|
3. The API may be removed in any Polygraphy version >= `N`.
|