wehub/nvidia--tensorrt
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Docker Image CI / build-ubuntu2004 (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:36:55 +08:00
commit c8a779b1bb
1887 changed files with 3245738 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/Polygraphy/.gitignore
+75
View File
@@ -0,0 +1,75 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Sphinx documentation
docs/_build/
# IPython
profile_default/
ipython_config.py
# pyenv
.python-version
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Crash dumps
core
# IDEs
.vscode
tools/Polygraphy/CHANGELOG.md
+1971
View File
File diff suppressed because it is too large Load Diff
tools/Polygraphy/CONTRIBUTING.md
+118
View File
@@ -0,0 +1,118 @@
# Contributing to Polygraphy
## Table of Contents
- [Contributing](#contributing)
- [Deprecation Scheme](#deprecation-scheme)
- [Design Principles](#design-principles)
## Contributing
0. *Optional, but recommended:* Read the [Design Principles](#design-principles) section in this document.
1. Create a separate branch for your feature or bug fix.
You may want to create the branch on your own fork of Polygraphy.
2. Make your changes and add corresponding tests.
The structure of the `tests` directory closely mirrors that of the main source directory (`polygraphy`),
so in general, for every source file you change, you'll need to modify the corresponding test file.
If you need to deprecate a public API, make sure to follow the [deprecation scheme](#deprecation-scheme).
If your changes are user-visible, make sure to update [CHANGELOG.md](CHANGELOG.md).
3. Run Tests:
- Install prerequisite packages with:
- `python3 -m pip install -r tests/requirements.txt --index-url https://pypi.ngc.nvidia.com --extra-index-url https://pypi.org/simple`
- `python3 -m pip install -r docs/requirements.txt --index-url https://pypi.ngc.nvidia.com --extra-index-url https://pypi.org/simple`
- Install TensorRT. If you don't already have it installed, there are two options:
1. Install the Python package:
```
python3 -m pip install tensorrt
```
2. Install it manually following the instructions in the [installation guide](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing).
- Run tests with: `make test`
4. Commit, push, and submit a merge request.
## Deprecation Scheme
### Marking Classes And Functions Deprecated
To indicate that a class or function is deprecated, you can decorate it
with the `deprecate()` decorator defined in `exporter.py`. For example:
```python
@mod.deprecate(remove_in="0.25.0", use_instead="NewClass")
class OldClass:
...
```
When the decorated type is used, a `DeprecationWarning` will be issued.
### Renaming Existing Classes And Functions
In some cases, it may be necessary to rename a function, class, or module.
In those cases, we can export the old name as a deprecated alias to preserve backwards compatibility.
- For a class or function, annotate the replacement with the `export_deprecated_alias` decorator.
For example:
```python
@mod.export_deprecated_alias("Old", remove_in="0.25.0")
class New:
...
```
- For modules, invoke the decorator manually within the module file.
For example:
```python
mod.export_deprecated_alias("old_mod_name", remove_in="0.25.0")(sys.modules[__name__])
```
### Adding Tests
When you deprecate an API, be sure to add a test into `tests/test_deprecated_aliases.py`
for the deprecated type.
The tests there will automatically fail if the deprecated type is not removed in the version
specified in `remove_in`.
## Design Principles
### Amazing Error Messages
Error messages should ideally tell the user how to fix the error, or, failing that,
should try to make the cause of the error as obvious as possible. An overly verbose error
is better than a cryptic one.
### Simple But Flexible
The API should be as simple as possible, with plug-and-play modular components.
Loader composition is an example of this - users can freely intermix Polygraphy's
loaders with backend APIs. See [example 03](examples/api/03_interoperating_with_tensorrt/).
### None Means Default
Universally using `None` to indicate default value has some advantages:
- Makes it easier to write wrappers - instead of trying to match the default
values of the function being wrapped, users can just use `None` .
- Can help prevent surprises caused by default value behavior in Python, as explained in
the [comment for default()](./polygraphy/util/util.py)
### Descriptive Loader Names
- Loaders that convert from a source format to some target format should
follow the naming convention: `<Target>From<Source>`, e.g. `OnnxFromTfGraph`, `NetworkFromOnnxBytes`
- Loaders that do not affect the format of their source should follow the naming convention:
`<Verb><Source>`, e.g. `ModifyOutputs`, `SaveEngine`
- For all other loaders, make sure the name is concise, but descriptive, e.g. `LoadPlugins`,
`CreateConfig`
tools/Polygraphy/LICENSE.txt
+191
View File
@@ -0,0 +1,191 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
Copyright 2020 NVIDIA Corporation
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
tools/Polygraphy/Makefile
+65
View File
@@ -0,0 +1,65 @@
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
.ONESHELL: # Run all rules in the same shell to make it easier to run tests with environment variables
.PHONY: test_venv test leak_check clean build install docs
NPROC ?= 8
BUILD_DIR := build
TESTS_DIR := tests
RUN_ALL_TESTS ?= 0
EXTRA_PYTEST_OPTS ?=
PYTEST_OPTS := $(EXTRA_PYTEST_OPTS) -v --durations=15 --failed-first --new-first --script-launch-mode=subprocess
EXTRA_PYTEST_MARKERS :=
ifeq ($(RUN_ALL_TESTS),0)
EXTRA_PYTEST_MARKERS += and not slow
endif
# Tests also check that docs can build
test: docs install
export PYTHONPATH=$(CURDIR):$${PYTHONPATH}
export POLYGRAPHY_INTERNAL_CORRECTNESS_CHECKS=1
export CUDA_MODULE_LOADING=LAZY
# Some tests need to be run serially - we annotate those with a `serial` marker.
python3 -m pytest $(TESTS_DIR) -m "serial $(EXTRA_PYTEST_MARKERS)" $(PYTEST_OPTS) && \
python3 -m pytest $(TESTS_DIR) -n $(NPROC) --dist=loadscope -m "not serial $(EXTRA_PYTEST_MARKERS)" $(PYTEST_OPTS)
leak_check:
export PYTHONPATH=$(CURDIR):$${PYTHONPATH}
export POLYGRAPHY_INTERNAL_CORRECTNESS_CHECKS=1
export PYTHONMALLOC=malloc
valgrind --leak-check=full python3 -m pytest $(TESTS_DIR) -v --durations=5 2>&1 | tee leak-check.log
clean:
rm -rf dist/ $(BUILD_DIR)/ polygraphy.egg-info/
build: clean
python3 setup.py bdist_wheel
install_deps: build
- python3 -m pip install colored wheel
install: install_deps
- python3 -m pip install --force-reinstall $(CURDIR)/dist/*.whl
docs: build
mkdir -p $(BUILD_DIR)/docs
python3 `which sphinx-build` docs $(BUILD_DIR)/docs/ -j $(NPROC) -W
tools/Polygraphy/README.md
+168
View File
@@ -0,0 +1,168 @@
# Polygraphy: A Deep Learning Inference Prototyping and Debugging Toolkit
## Table of Contents
- [Introduction](#introduction)
- [Installation](#installation)
- [Command-line Toolkit](#command-line-toolkit)
- [Python API](#python-api)
- [Examples](#examples)
- [How-To Guides](#how-to-guides)
- [Contributing](#contributing)
## Introduction
Polygraphy is a toolkit designed to assist in running and debugging deep learning models
in various frameworks. It includes a [Python API](./polygraphy) and
[a command-line interface (CLI)](./polygraphy/tools) built using this API.
Among other things, Polygraphy lets you:
- Run inference among multiple backends, like TensorRT and ONNX-Runtime, and compare results
(example: [API](examples/api/01_comparing_frameworks/), [CLI](examples/cli/run/01_comparing_frameworks/))
- Convert models to various formats, e.g. TensorRT engines with post-training quantization
(example: [API](examples/api/04_int8_calibration_in_tensorrt/), [CLI](examples/cli/convert/01_int8_calibration_in_tensorrt/))
- View information about various types of models
(example: [CLI](examples/cli/inspect/))
- Modify ONNX models on the command-line:
- Extract subgraphs (example: [CLI](examples/cli/surgeon/01_isolating_subgraphs/))
- Simplify and sanitize (example: [CLI](examples/cli/surgeon/02_folding_constants/))
- Isolate faulty tactics in TensorRT
(example: [CLI](examples/cli/debug/01_debugging_flaky_trt_tactics/))
## Installation
**IMPORTANT**: **Polygraphy supports only Python 3.6 and later.**
**Before following the instructions below, please ensure you are using a supported version of Python.**
### Installing Prebuilt Wheels
```bash
python -m pip install colored polygraphy --extra-index-url https://pypi.ngc.nvidia.com
```
**NOTE:** *On Linux, the command-line toolkit is usually installed to `${HOME}/.local/bin` by default.*
*Make sure to add this directory to your `PATH` environment variable.*
### Building From Source
#### Using Make Targets (Linux)
```bash
make install
```
#### Using Powershell Script (Windows)
Make sure you are allowed to execute scripts on your system then run:
```ps
.\install.ps1
```
#### Building Manually
1. Install prerequisites:
```
python -m pip install wheel
```
2. Build a wheel:
```
python setup.py bdist_wheel
```
3. Install the wheel manually from **outside** the repository:
On Linux, run:
```
python -m pip install Polygraphy/dist/polygraphy-*-py2.py3-none-any.whl
```
On Windows, using Powershell, run:
```ps
$wheel_path = gci -Name Polygraphy\dist
python -m pip install Polygraphy\dist\$wheel_path
```
**NOTE:** *It is strongly recommended to install the `colored` module for colored output*
*from Polygraphy, as this can greatly improve readability:*
```
python -m pip install colored
```
### Installing Dependencies
Polygraphy has no hard-dependencies on other Python packages. However, much of the functionality included
does require other Python packages.
#### Automatically Installing Dependencies
It's non-trivial to determine all the packages that will be required ahead of time,
since it depends on exactly what functionality is being used.
To make this easier, Polygraphy can optionally automatically install or upgrade dependencies at runtime, as they are needed.
To enable this behavior, set the `POLYGRAPHY_AUTOINSTALL_DEPS` environment variable to `1` or
`polygraphy.config.AUTOINSTALL_DEPS = True` using the Python API.
**NOTE**: *By default, dependencies will be installed using the current interpreter, and may overwrite existing*
*packages. The default installation command, which is `python -m pip install`, can be overriden by setting*
*the `POLYGRAPHY_INSTALL_CMD` environment variable, or setting `polygraphy.config.INSTALL_CMD` using the Python API.*
If you'd like Polygraphy to prompt you before automatically installing or
upgrading pacakges, set the `POLYGRAPHY_ASK_BEFORE_INSTALL` environment variable to `1`
or `polygraphy.config.ASK_BEFORE_INSTALL = True` using the Python API.
#### Installing Manually
Each `backend` directory includes a `requirements.txt` file that specifies the minimum set of packages
it depends on. This does not necessarily include all packages required for all the functionality provided
by the backend, but does serve as a good starting point.
You can install the requirements for whichever backends you're interested in with:
```bash
python -m pip install -r polygraphy/backend/<name>/requirements.txt
```
If additional packages are required, warnings or errors will be logged.
You can install the additional packages manually with:
```bash
python -m pip install <package_name>
```
## Command-line Toolkit
For details on the various tools included in the Polygraphy toolkit,
see the [CLI User Guide](./polygraphy/tools).
### Python API
For more information on the Polygraphy Python API, including a high-level overview and the
Python API reference documentation, see the [API directory](./polygraphy).
## Examples
For examples of both the CLI and Python API, see the [examples directory](./examples).
## How-To Guides
For how-to guides, see the [how-to guides directory](./how-to).
## Contributing
For information on how you can contribute to this project, see [CONTRIBUTING.md](./CONTRIBUTING.md)
tools/Polygraphy/bin/polygraphy
Executable
+32
View File
@@ -0,0 +1,32 @@
#!/usr/bin/env python3
#
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
A wrapper around ``polygraphy.tools.main()``. This is primarily for development purposes
as it will inject the Polygraphy source path into PYTHONPATH.
"""
import os
import sys
G_SCRIPT_FILE = os.path.realpath(__file__)
G_ROOT_DIR = os.path.join(os.path.dirname(G_SCRIPT_FILE), os.pardir)
sys.path.insert(0, G_ROOT_DIR)
from polygraphy import tools
if __name__ == "__main__":
sys.exit(tools.main())
tools/Polygraphy/docs/_static/img/nvlogo_black.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 KiB

tools/Polygraphy/docs/_static/img/nvlogo_white.png
Vendored
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.3 KiB

tools/Polygraphy/docs/_static/style.css
Vendored
+3
View File
@@ -0,0 +1,3 @@
.wy-nav-content {
max-width: 1100px !important;
}
tools/Polygraphy/docs/_templates/page.html
Vendored
+37
View File
@@ -0,0 +1,37 @@
{% extends '!page.html' %}
{% block extrahead %}
<script src="//assets.adobedtm.com/5d4962a43b79/c1061d2c5e7b/launch-191c2462b890.min.js"></script>
{% endblock %}
{% block footer %}
{{ super() }}
<div class="footer">
<p>
Copyright © 2025 NVIDIA Corporation
</p>
<p>
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/" target="_blank" rel="noopener"
data-cms-ai="0">Privacy Policy</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-center/" target="_blank" rel="noopener"
data-cms-ai="0">Manage My Privacy</a> |
<a class="Link" href="https://www.nvidia.com/en-us/preferences/start/" target="_blank" rel="noopener"
data-cms-ai="0">Do Not Sell or Share My Data</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/terms-of-service/" target="_blank"
rel="noopener" data-cms-ai="0">Terms of Service</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/accessibility/" target="_blank" rel="noopener"
data-cms-ai="0">Accessibility</a> |
<a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/company-policies/" target="_blank"
rel="noopener" data-cms-ai="0">Corporate Policies</a> |
<a class="Link" href="https://www.nvidia.com/en-us/product-security/" target="_blank" rel="noopener"
data-cms-ai="0">Product Security</a> |
<a class="Link" href="https://www.nvidia.com/en-us/contact/" target="_blank" rel="noopener"
data-cms-ai="0">Contact</a>
</p>
</div>
<script type="text/javascript">_satellite.pageBottom();</script>
<script type="text/javascript">document.getElementsByClassName("bottom-of-page")[0].innerHTML = "";</script>
{% endblock %}
tools/Polygraphy/docs/backend/base/loader.rst
+7
View File
@@ -0,0 +1,7 @@
============
Loaders
============
Module: ``polygraphy.backend.base``
.. automodule:: polygraphy.backend.base.loader
tools/Polygraphy/docs/backend/base/runner.rst
+7
View File
@@ -0,0 +1,7 @@
============
Runners
============
Module: ``polygraphy.backend.base``
.. automodule:: polygraphy.backend.base.runner
tools/Polygraphy/docs/backend/base/toc.rst
+11
View File
@@ -0,0 +1,11 @@
===============
Base Interface
===============
The base interface for all loaders and runners.
Module: ``polygraphy.backend.base``
.. toctree::
loader
runner
tools/Polygraphy/docs/backend/common/loader.rst
+8
View File
@@ -0,0 +1,8 @@
============
Loaders
============
Module: ``polygraphy.backend.common``
.. automodule:: polygraphy.backend.common.loader
:inherited-members:
tools/Polygraphy/docs/backend/common/toc.rst
+8
View File
@@ -0,0 +1,8 @@
==========
Common
==========
Module: ``polygraphy.backend.common``
.. toctree::
loader
tools/Polygraphy/docs/backend/onnx/loader.rst
+8
View File
@@ -0,0 +1,8 @@
============
Loaders
============
Module: ``polygraphy.backend.onnx``
.. automodule:: polygraphy.backend.onnx.loader
:inherited-members:
tools/Polygraphy/docs/backend/onnx/toc.rst
+8
View File
@@ -0,0 +1,8 @@
====
ONNX
====
Module: ``polygraphy.backend.onnx``
.. toctree::
loader
tools/Polygraphy/docs/backend/onnxrt/loader.rst
+8
View File
@@ -0,0 +1,8 @@
============
Loaders
============
Module: ``polygraphy.backend.onnxrt``
.. automodule:: polygraphy.backend.onnxrt.loader
:inherited-members:
tools/Polygraphy/docs/backend/onnxrt/runner.rst
+8
View File
@@ -0,0 +1,8 @@
============
Runners
============
Module: ``polygraphy.backend.onnxrt``
.. automodule:: polygraphy.backend.onnxrt.runner
:inherited-members:
tools/Polygraphy/docs/backend/onnxrt/toc.rst
+9
View File
@@ -0,0 +1,9 @@
============
ONNX-Runtime
============
Module: ``polygraphy.backend.onnxrt``
.. toctree::
loader
runner
tools/Polygraphy/docs/backend/pluginref/runner.rst
+8
View File
@@ -0,0 +1,8 @@
============
Runners
============
Module: ``polygraphy.backend.pluginref``
.. automodule:: polygraphy.backend.pluginref.runner
:inherited-members:
tools/Polygraphy/docs/backend/pluginref/toc.rst
+8
View File
@@ -0,0 +1,8 @@
================
Plugin Reference
================
Module: ``polygraphy.backend.pluginref``
.. toctree::
runner
tools/Polygraphy/docs/backend/tf/loader.rst
+8
View File
@@ -0,0 +1,8 @@
============
Loaders
============
Module: ``polygraphy.backend.tf``
.. automodule:: polygraphy.backend.tf.loader
:inherited-members:
tools/Polygraphy/docs/backend/tf/runner.rst
+8
View File
@@ -0,0 +1,8 @@
============
Runners
============
Module: ``polygraphy.backend.tf``
.. automodule:: polygraphy.backend.tf.runner
:inherited-members:
tools/Polygraphy/docs/backend/tf/toc.rst
+9
View File
@@ -0,0 +1,9 @@
==============
TensorFlow 1.X
==============
Module: ``polygraphy.backend.tf``
.. toctree::
loader
runner
tools/Polygraphy/docs/backend/toc.rst
+14
View File
@@ -0,0 +1,14 @@
========
Backends
========
Module: ``polygraphy.backend``
.. toctree::
base/toc
common/toc
onnx/toc
onnxrt/toc
pluginref/toc
tf/toc
trt/toc
tools/Polygraphy/docs/backend/trt/algorithm_selector.rst
+11
View File
@@ -0,0 +1,11 @@
==================
Algorithm Selector
==================
Module: ``polygraphy.backend.trt``
.. warning::
Algorithm Selector is deprecated in TensorRT 10.8. Please use editable mode in ITimingCache instead.
See https://github.com/NVIDIA/TensorRT/tree/release/10.8/samples/sampleEditableTimingCache.
.. automodule:: polygraphy.backend.trt.algorithm_selector
tools/Polygraphy/docs/backend/trt/calibrator.rst
+7
View File
@@ -0,0 +1,7 @@
============
Calibrator
============
Module: ``polygraphy.backend.trt``
.. automodule:: polygraphy.backend.trt.calibrator
tools/Polygraphy/docs/backend/trt/config.rst
+8
View File
@@ -0,0 +1,8 @@
============
Config
============
Module: ``polygraphy.backend.trt``
.. automodule:: polygraphy.backend.trt.config
:inherited-members:
tools/Polygraphy/docs/backend/trt/loader.rst
+9
View File
@@ -0,0 +1,9 @@
============
Loaders
============
Module: ``polygraphy.backend.trt``
.. automodule:: polygraphy.backend.trt.loader
:inherited-members:
:exclude-members: polygraphy.backend.trt.loader.BaseNetworkFromOnnx
tools/Polygraphy/docs/backend/trt/profile.rst
+7
View File
@@ -0,0 +1,7 @@
====================
Optimization Profile
====================
Module: ``polygraphy.backend.trt``
.. automodule:: polygraphy.backend.trt.profile
tools/Polygraphy/docs/backend/trt/runner.rst
+8
View File
@@ -0,0 +1,8 @@
============
Runners
============
Module: ``polygraphy.backend.trt``
.. automodule:: polygraphy.backend.trt.runner
:inherited-members:
tools/Polygraphy/docs/backend/trt/toc.rst
+14
View File
@@ -0,0 +1,14 @@
=========
TensorRT
=========
Module: ``polygraphy.backend.trt``
.. toctree::
algorithm_selector
calibrator
config
loader
profile
runner
util
tools/Polygraphy/docs/backend/trt/util.rst
+7
View File
@@ -0,0 +1,7 @@
============
Utilities
============
Module: ``polygraphy.backend.trt``
.. automodule:: polygraphy.backend.trt.util
tools/Polygraphy/docs/common/data_structures.rst
+7
View File
@@ -0,0 +1,7 @@
===============
Data Structures
===============
Module: ``polygraphy.common``
.. automodule:: polygraphy.common.struct
tools/Polygraphy/docs/common/toc.rst
+8
View File
@@ -0,0 +1,8 @@
============
Common
============
Module: ``polygraphy.common``
.. toctree::
data_structures
tools/Polygraphy/docs/comparator/comparator.rst
+7
View File
@@ -0,0 +1,7 @@
==========
Comparator
==========
Module: ``polygraphy.comparator``
.. automodule:: polygraphy.comparator.comparator
tools/Polygraphy/docs/comparator/compare_func.rst
+7
View File
@@ -0,0 +1,7 @@
====================
Comparison Functions
====================
Module: ``polygraphy.comparator``
.. automodule:: polygraphy.comparator.compare
tools/Polygraphy/docs/comparator/data_loader.rst
+7
View File
@@ -0,0 +1,7 @@
===========
Data Loader
===========
Module: ``polygraphy.comparator``
.. automodule:: polygraphy.comparator.data_loader
tools/Polygraphy/docs/comparator/data_structures.rst
+7
View File
@@ -0,0 +1,7 @@
================
Data Structures
================
Module: ``polygraphy.comparator``
.. automodule:: polygraphy.comparator.struct
tools/Polygraphy/docs/comparator/postprocess_func.rst
+7
View File
@@ -0,0 +1,7 @@
=========================
Postprocessing Functions
=========================
Module: ``polygraphy.comparator``
.. automodule:: polygraphy.comparator.postprocess
tools/Polygraphy/docs/comparator/toc.rst
+12
View File
@@ -0,0 +1,12 @@
==================
Comparing Results
==================
Module: ``polygraphy.comparator``
.. toctree::
comparator
data_structures
compare_func
postprocess_func
data_loader
tools/Polygraphy/docs/conf.py
+128
View File
@@ -0,0 +1,128 @@
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import sys
import os
ROOT_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), os.path.pardir)
sys.path.insert(0, ROOT_DIR)
import polygraphy
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.intersphinx",
"sphinx.ext.autosummary",
"sphinx.ext.napoleon",
"sphinx.ext.mathjax",
"sphinx.ext.viewcode",
"myst_parser",
]
# Want to be able to generate docs with no dependencies installed
autodoc_mock_imports = [
"tensorrt",
"onnx",
"numpy",
"tensorflow",
"onnx_graphsurgeon",
"onnxruntime",
"tf2onnx",
]
autodoc_default_options = {
"members": True,
"show-inheritance": True,
"exclude-members": "activate_impl, deactivate_impl, get_input_metadata_impl, BaseNetworkFromOnnx, Encoder, Decoder, add_json_methods, constantmethod",
"special-members": "__call__, __getitem__, __bool__, __enter__, __exit__",
}
autodoc_member_order = "bysource"
autodoc_inherit_docstrings = True
add_module_names = False
autosummary_generate = True
source_suffix = [".rst"]
# The master toctree document.
master_doc = "index"
# General information about the project.
project = "Polygraphy"
copyright = "2024 NVIDIA Corporation"
author = "NVIDIA"
version = polygraphy.__version__
# The full version, including alpha/beta/rc tags.
release = version
# Style
pygments_style = "colorful"
html_theme = "furo"
html_title = f"{project}<br>{version}"
# Use the TRT theme and NVIDIA logo
html_static_path = ["_static"]
# Hide source link
html_show_sourcelink = False
# Output file base name for HTML help builder.
htmlhelp_basename = "PolygraphyDoc"
# Template files to extend default Sphinx templates.
# See https://www.sphinx-doc.org/en/master/templating.html for details.
templates_path = ["_templates"]
# For constructor arguments to show up in Sphinx generated doc
autoclass_content = "both"
html_theme_options = {
"light_logo": os.path.join("img", "nvlogo_black.png"),
"dark_logo": os.path.join("img", "nvlogo_white.png"),
"light_css_variables": {
"color-api-pre-name": "#4e9a06",
"color-api-name": "#4e9a06",
"color-api-background": "#e8e8e8",
},
"dark_css_variables": {
"color-api-background": "#303030",
},
"footer_icons": [
{
"name": "GitHub",
"url": "https://github.com/NVIDIA/TensorRT",
"html": """
<svg stroke="currentColor" fill="currentColor" stroke-width="0" viewBox="0 0 16 16">
<path fill-rule="evenodd" d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0 0 16 8c0-4.42-3.58-8-8-8z"></path>
</svg>
""",
"class": "",
},
],
}
# Allows us to override the default page width in the Sphinx theme.
def setup(app):
app.add_css_file("style.css")
LATEX_BUILDER = "sphinx.builders.latex"
if LATEX_BUILDER in app.config.extensions:
app.config.extensions.remove(LATEX_BUILDER)
tools/Polygraphy/docs/config/toc.rst
+7
View File
@@ -0,0 +1,7 @@
=====================
Global Configuration
=====================
Module: ``polygraphy.config``
.. automodule:: polygraphy.config
tools/Polygraphy/docs/constants/toc.rst
+7
View File
@@ -0,0 +1,7 @@
================
Constants
================
Module: ``polygraphy.constants``
.. automodule:: polygraphy.constants
tools/Polygraphy/docs/cuda/toc.rst
+8
View File
@@ -0,0 +1,8 @@
==============
CUDA Wrapper
==============
Module: ``polygraphy.cuda``
.. automodule:: polygraphy.cuda.cuda
:inherited-members:
tools/Polygraphy/docs/datatype/toc.rst
+7
View File
@@ -0,0 +1,7 @@
================
Data Types
================
Module: ``polygraphy.datatype``
.. automodule:: polygraphy.datatype.datatype
tools/Polygraphy/docs/exception/toc.rst
+7
View File
@@ -0,0 +1,7 @@
==========
Exceptions
==========
Module: ``polygraphy.exception``
.. automodule:: polygraphy.exception.exception
tools/Polygraphy/docs/func/toc.rst
+7
View File
@@ -0,0 +1,7 @@
================
Function Helpers
================
Module: ``polygraphy.func``
.. automodule:: polygraphy.func.func
tools/Polygraphy/docs/index.rst
+49
View File
@@ -0,0 +1,49 @@
==========
Polygraphy
==========
This page includes the Python API reference documentation for Polygraphy. Polygraphy is a toolkit
designed to assist in running and debugging deep learning models in various frameworks.
For installation instructions, examples, and information about the CLI tools,
see `the GitHub repository <https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy>`_ instead.
For a conceptual overview of the Python API,
see `this page <https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy/polygraphy>`_.
.. warning::
Any APIs not documented here should be considered internal only and do not adhere to the
deprecation policy for public APIs. Thus, they may be modified or removed at any time without warning.
Avoid using undocumented APIs!
.. toctree::
:hidden:
self
.. toctree::
:caption: API Reference: Main
:maxdepth: 4
backend/toc
comparator/toc
.. toctree::
:caption: API Reference: Miscellaneous
:maxdepth: 4
common/toc
config/toc
constants/toc
cuda/toc
datatype/toc
exception/toc
func/toc
json/toc
logger/toc
.. toctree::
:caption: API Reference: Development
:maxdepth: 4
tool/toc
tools/Polygraphy/docs/json/toc.rst
+7
View File
@@ -0,0 +1,7 @@
================
JSON Utilities
================
Module: ``polygraphy.json``
.. automodule:: polygraphy.json.serde
tools/Polygraphy/docs/logger/toc.rst
+7
View File
@@ -0,0 +1,7 @@
================
Logger
================
Module: ``polygraphy.logger``
.. automodule:: polygraphy.logger.logger
tools/Polygraphy/docs/requirements.txt
+8
View File
@@ -0,0 +1,8 @@
docutils==0.16; python_version<"3.7"
docutils==0.18.1; python_version>="3.7"
sphinx==4.4.0; python_version<"3.7"
sphinx==7.1.2; python_version>="3.7"
furo==2022.4.7; python_version<"3.7"
furo==2024.8.6; python_version>="3.7"
myst-parser==0.16.1; python_version<"3.7"
myst-parser==3.0.1; python_version>="3.7"
tools/Polygraphy/docs/tool/args/backend/onnx/loader.rst
+7
View File
@@ -0,0 +1,7 @@
============
Loaders
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.onnx.loader
tools/Polygraphy/docs/tool/args/backend/onnx/toc.rst
+8
View File
@@ -0,0 +1,8 @@
==========
ONNX
==========
Module: ``polygraphy.tools.args``
.. toctree::
loader
tools/Polygraphy/docs/tool/args/backend/onnxrt/loader.rst
+7
View File
@@ -0,0 +1,7 @@
============
Loaders
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.onnxrt.loader
tools/Polygraphy/docs/tool/args/backend/onnxrt/runner.rst
+7
View File
@@ -0,0 +1,7 @@
============
Runners
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.onnxrt.runner
tools/Polygraphy/docs/tool/args/backend/onnxrt/toc.rst
+9
View File
@@ -0,0 +1,9 @@
=============
ONNX-Runtime
=============
Module: ``polygraphy.tools.args``
.. toctree::
loader
runner
tools/Polygraphy/docs/tool/args/backend/pluginref/runner.rst
+7
View File
@@ -0,0 +1,7 @@
============
Runners
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.pluginref.runner
tools/Polygraphy/docs/tool/args/backend/pluginref/toc.rst
+8
View File
@@ -0,0 +1,8 @@
================
Plugin Reference
================
Module: ``polygraphy.tools.args``
.. toctree::
runner
tools/Polygraphy/docs/tool/args/backend/tf/loader.rst
+9
View File
@@ -0,0 +1,9 @@
============
Loaders
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.tf.loader
.. automodule:: polygraphy.tools.args.backend.tf.config
tools/Polygraphy/docs/tool/args/backend/tf/runner.rst
+7
View File
@@ -0,0 +1,7 @@
============
Runners
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.tf.runner
tools/Polygraphy/docs/tool/args/backend/tf/toc.rst
+9
View File
@@ -0,0 +1,9 @@
==========
TensorFlow
==========
Module: ``polygraphy.tools.args``
.. toctree::
loader
runner
tools/Polygraphy/docs/tool/args/backend/toc.rst
+12
View File
@@ -0,0 +1,12 @@
========
Backends
========
Module: ``polygraphy.tools.args``
.. toctree::
onnx/toc
onnxrt/toc
pluginref/toc
tf/toc
trt/toc
tools/Polygraphy/docs/tool/args/backend/trt/loader.rst
+9
View File
@@ -0,0 +1,9 @@
============
Loaders
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.trt.loader
.. automodule:: polygraphy.tools.args.backend.trt.config
tools/Polygraphy/docs/tool/args/backend/trt/runner.rst
+7
View File
@@ -0,0 +1,7 @@
============
Runners
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.backend.trt.runner
tools/Polygraphy/docs/tool/args/backend/trt/toc.rst
+9
View File
@@ -0,0 +1,9 @@
=========
TensorRT
=========
Module: ``polygraphy.tools.args``
.. toctree::
loader
runner
tools/Polygraphy/docs/tool/args/base.rst
+9
View File
@@ -0,0 +1,9 @@
=========================
Base Interface
=========================
The base interface for all argument groups.
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.base
tools/Polygraphy/docs/tool/args/comparator/comparator.rst
+7
View File
@@ -0,0 +1,7 @@
============
Comparator
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.comparator.comparator
tools/Polygraphy/docs/tool/args/comparator/compare.rst
+7
View File
@@ -0,0 +1,7 @@
====================
Comparison Functions
====================
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.comparator.compare
tools/Polygraphy/docs/tool/args/comparator/data_loader.rst
+7
View File
@@ -0,0 +1,7 @@
============
Data Loader
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.comparator.data_loader
tools/Polygraphy/docs/tool/args/comparator/postprocess.rst
+7
View File
@@ -0,0 +1,7 @@
=========================
Postprocessing Functions
=========================
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.comparator.postprocess
tools/Polygraphy/docs/tool/args/comparator/toc.rst
+11
View File
@@ -0,0 +1,11 @@
=============
Comparator
=============
Module: ``polygraphy.tools.args``
.. toctree::
comparator
compare
data_loader
postprocess
tools/Polygraphy/docs/tool/args/logger/toc.rst
+7
View File
@@ -0,0 +1,7 @@
============
Logger
============
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.logger.logger
tools/Polygraphy/docs/tool/args/model.rst
+7
View File
@@ -0,0 +1,7 @@
=========================
Model
=========================
Module: ``polygraphy.tools.args``
.. automodule:: polygraphy.tools.args.model
tools/Polygraphy/docs/tool/args/toc.rst
+17
View File
@@ -0,0 +1,17 @@
============================
Command-line Argument Groups
============================
Command-line argument groups bundle arguments and related functionality together into reusable components
and are used throughout the Polygraphy command-line toolkit.
Argument groups are used to add new command-line tools to Polygraphy or extend existing tools with new functionality.
Module: ``polygraphy.tools.args``
.. toctree::
base
backend/toc
comparator/toc
logger/toc
model
tools/Polygraphy/docs/tool/script.rst
+7
View File
@@ -0,0 +1,7 @@
=========================
Script Interface
=========================
Module: ``polygraphy.tools.script``
.. automodule:: polygraphy.tools.script
tools/Polygraphy/docs/tool/toc.rst
+9
View File
@@ -0,0 +1,9 @@
============================
Command-line Tool APIs
============================
Module: ``polygraphy.tools``
.. toctree::
args/toc
script
tools/Polygraphy/examples/README.md
+5
View File
@@ -0,0 +1,5 @@
# Examples
This directory includes various examples covering the Polygraphy [CLI](./cli), [Python API](./api), and [development practices](./dev).
The paths used in each example assume that the example is being run from within that example's directory.
tools/Polygraphy/examples/api/00_inference_with_tensorrt/README.md
+48
View File
@@ -0,0 +1,48 @@
# Converting To TensorRT And Running Inference
## Introduction
Polygraphy includes a high-level Python API that can convert models
and run inference with various backends. For an overview of the Polygraphy
Python API, see [here](../../../polygraphy/).
In this example, we'll look at how you can leverage the API to easily convert an ONNX
model to TensorRT and run inference with FP16 precision enabled. We'll then save the
engine to a file and see how you can load it again and run inference.
## Running The Example
1. Install prerequisites
* Ensure that TensorRT is installed
* Install other dependencies with `python3 -m pip install -r requirements.txt`
2. **[Optional]** Inspect the model before running the example:
```bash
polygraphy inspect model identity.onnx
```
3. Run the script that builds and runs the engine:
```bash
python3 build_and_run.py
```
4. **[Optional]** Inspect the TensorRT engine built by the example:
```bash
polygraphy inspect model identity.engine
```
5. Run the script that loads the previously built engine, then runs it:
```bash
python3 load_and_run.py
```
## Further Reading
For more details on the Polygraphy Python API, see the
[Polygraphy API reference](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/polygraphy/index.html).
tools/Polygraphy/examples/api/00_inference_with_tensorrt/build_and_run.py
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
This script builds and runs a TensorRT engine with FP16 precision enabled
starting from an ONNX identity model.
"""
import numpy as np
from polygraphy.backend.trt import (
CreateConfig,
EngineFromNetwork,
NetworkFromOnnxPath,
SaveEngine,
TrtRunner,
)
def main():
# We can compose multiple lazy loaders together to get the desired conversion.
# In this case, we want ONNX -> TensorRT Network -> TensorRT engine (w/ fp16).
#
# NOTE: `build_engine` is a *callable* that returns an engine, not the engine itself.
# To get the engine directly, you can use the immediately evaluated functional API.
# See examples/api/06_immediate_eval_api for details.
build_engine = EngineFromNetwork(
NetworkFromOnnxPath("identity.onnx"), config=CreateConfig(fp16=True)
) # Note that config is an optional argument.
# To reuse the engine elsewhere, we can serialize and save it to a file.
# The `SaveEngine` lazy loader will return the TensorRT engine when called,
# which allows us to chain it together with other loaders.
build_engine = SaveEngine(build_engine, path="identity.engine")
# Once our loader is ready, inference is simply a matter of constructing a runner,
# activating it with a context manager (i.e. `with TrtRunner(...)`) and calling `infer()`.
#
# NOTE: You can use the activate() function instead of a context manager, but you will need to make sure to
# deactivate() to avoid a memory leak. For that reason, a context manager is the safer option.
with TrtRunner(build_engine) as runner:
inp_data = np.ones(shape=(1, 1, 2, 2), dtype=np.float32)
# NOTE: The runner owns the output buffers and is free to reuse them between `infer()` calls.
# Thus, if you want to store results from multiple inferences, you should use `copy.deepcopy()`.
outputs = runner.infer(feed_dict={"x": inp_data})
assert np.array_equal(outputs["y"], inp_data) # It's an identity model!
print("Inference succeeded!")
if __name__ == "__main__":
main()
tools/Polygraphy/examples/api/00_inference_with_tensorrt/identity.onnx
+15
View File
@@ -0,0 +1,15 @@
 backend-test:[

xy"Identity
test_identityZ
x




b
y




tools/Polygraphy/examples/api/00_inference_with_tensorrt/load_and_run.py
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
This script loads the TensorRT engine built by `build_and_run.py` and runs inference.
"""
import numpy as np
from polygraphy.backend.common import BytesFromPath
from polygraphy.backend.trt import EngineFromBytes, TrtRunner
def main():
# Just as we did when building, we can compose multiple loaders together
# to achieve the behavior we want. Specifically, we want to load a serialized
# engine from a file, then deserialize it into a TensorRT engine.
load_engine = EngineFromBytes(BytesFromPath("identity.engine"))
# Inference remains virtually exactly the same as before:
with TrtRunner(load_engine) as runner:
inp_data = np.ones(shape=(1, 1, 2, 2), dtype=np.float32)
# NOTE: The runner owns the output buffers and is free to reuse them between `infer()` calls.
# Thus, if you want to store results from multiple inferences, you should use `copy.deepcopy()`.
outputs = runner.infer(feed_dict={"x": inp_data})
assert np.array_equal(outputs["y"], inp_data) # It's an identity model!
print("Inference succeeded!")
if __name__ == "__main__":
main()
tools/Polygraphy/examples/api/00_inference_with_tensorrt/requirements.txt
+1
View File
@@ -0,0 +1 @@
numpy
tools/Polygraphy/examples/api/01_comparing_frameworks/README.md
+45
View File
@@ -0,0 +1,45 @@
# Comparing Frameworks
## Introduction
One of the core features of Polygraphy is comparison of model outputs across multiple
different backends. This makes it possible to check the accuracy of one backend with
respect to another.
In this example, we'll look at how you can use the Polygraphy API to run inference
with synthetic input data using ONNX-Runtime and TensorRT, and then compare the results
using two different comparison methods:
1. A simple comparison using absolute tolerance
2. A more comprehensive comparison using distance metrics (L2 distance, cosine similarity, and PSNR)
## Running The Example
1. Install prerequisites
* Ensure that TensorRT is installed
* Install other dependencies with `python3 -m pip install -r requirements.txt`
2. Run the example
```bash
python3 example.py
```
3. **[Optional]** Inspect the inference outputs from the example:
```bash
polygraphy inspect data inference_results.json
```
## Comparison Methods
The example demonstrates two approaches for comparing outputs:
- **Simple Comparison**: Uses absolute tolerance to determine if outputs match within a specified threshold.
- **Distance Metrics**: Performs a more comprehensive comparison using multiple metrics including:
- L2 distance (Euclidean distance)
- Cosine similarity (measures the angle between vectors)
- PSNR (Peak Signal-to-Noise Ratio, useful for comparing image-like data)
These comparison methods help validate that frameworks produce equivalent results within acceptable margins.
tools/Polygraphy/examples/api/01_comparing_frameworks/example.py
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
This script runs an identity model with ONNX-Runtime and TensorRT,
then compares outputs.
"""
from polygraphy.backend.onnxrt import OnnxrtRunner, SessionFromOnnx
from polygraphy.backend.trt import EngineFromNetwork, NetworkFromOnnxPath, TrtRunner
from polygraphy.comparator import Comparator, CompareFunc
def main():
# The OnnxrtRunner requires an ONNX-RT session.
# We can use the SessionFromOnnx lazy loader to construct one easily:
build_onnxrt_session = SessionFromOnnx("identity.onnx")
# The TrtRunner requires a TensorRT engine.
# To create one from the ONNX model, we can chain a couple lazy loaders together:
build_engine = EngineFromNetwork(NetworkFromOnnxPath("identity.onnx"))
runners = [
TrtRunner(build_engine),
OnnxrtRunner(build_onnxrt_session),
]
# `Comparator.run()` will run each runner separately using synthetic input data and
# return a `RunResults` instance. See `polygraphy/comparator/struct.py` for details.
#
# TIP: To use custom input data, you can set the `data_loader` parameter in `Comparator.run()``
# to a generator or iterable that yields `Dict[str, np.ndarray]`.
run_results = Comparator.run(runners)
# `Comparator.compare_accuracy()` checks that outputs match between runners.
#
# TIP: The `compare_func` parameter can be used to control how outputs are compared (see API reference for details).
# The default comparison function is created by `CompareFunc.simple()`, but we can construct it
# explicitly if we want to change the default parameters, such as tolerance.
assert bool(
Comparator.compare_accuracy(
run_results, compare_func=CompareFunc.simple(atol=1e-8)
)
)
# Use distance metrics comparison for more comprehensive evaluation
assert bool(
Comparator.compare_accuracy(
run_results,
compare_func=CompareFunc.distance_metrics(
l2_tolerance=1e-5, # Maximum allowed L2 norm (Euclidean distance)
cosine_similarity_threshold=0.99, # Minimum cosine similarity (angular similarity)
)
)
)
print("All outputs matched using distance metrics (L2 norm, Cosine Similarity)")
# Use quality metrics for signal quality evaluation
assert bool(
Comparator.compare_accuracy(
run_results,
compare_func=CompareFunc.quality_metrics(
psnr_tolerance=50.0, # Minimum Peak Signal-to-Noise Ratio in dB
snr_tolerance=25.0 # Minimum Signal-to-Noise Ratio in dB
)
)
)
print("All outputs matched using quality metrics (PSNR, SNR)")
# We can use `RunResults.save()` method to save the inference results to a JSON file.
# This can be useful if you want to generate and compare results separately.
run_results.save("inference_results.json")
if __name__ == "__main__":
main()
tools/Polygraphy/examples/api/01_comparing_frameworks/identity.onnx
+15
View File
@@ -0,0 +1,15 @@
 backend-test:[

xy"Identity
test_identityZ
x




b
y




tools/Polygraphy/examples/api/01_comparing_frameworks/requirements.txt
+2
View File
@@ -0,0 +1,2 @@
onnx
onnxruntime
tools/Polygraphy/examples/api/02_validating_on_a_dataset/README.md
+34
View File
@@ -0,0 +1,34 @@
# Validating Accuracy On A Real Dataset
## Introduction
The `Comparator` provided by Polygraphy can be useful for comparing a small number of
results across multiple runners, but is not well suited for validating a single runner
with a real dataset that includes labels or golden values - especially if the dataset is large.
In such cases, it is recommended to use a runner directly instead.
*NOTE: It is possible to provide custom input data to `Comparator.run()` using the `data_loader`*
*parameter. This may be a viable option when using a smaller dataset.*
In this example, we use a `TrtRunner` directly to validate an identity model on
a trivial dataset. Unlike using the `Comparator`, using a runner gives you complete
freedom as to how you load your input data, as well as how you validate the results.
Since all runners provide the same interface, you can freely drop-in other runners
without touching the rest of your validation code. For example, in this case, validating
the model using ONNX-Runtime would require changing just 2 lines; this is left as an
exercise for the reader.
## Running The Example
1. Install prerequisites
* Ensure that TensorRT is installed
* Install other dependencies with `python3 -m pip install -r requirements.txt`
2. Run the example
```bash
python3 example.py
```
tools/Polygraphy/examples/api/02_validating_on_a_dataset/example.py
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
This script uses the Polygraphy Runner API to validate the outputs
of an identity model using a trivial dataset.
"""
import numpy as np
from polygraphy.backend.trt import EngineFromNetwork, NetworkFromOnnxPath, TrtRunner
# Pretend that this is a very large dataset.
REAL_DATASET = [
np.ones((1, 1, 2, 2), dtype=np.float32),
np.zeros((1, 1, 2, 2), dtype=np.float32),
np.ones((1, 1, 2, 2), dtype=np.float32),
np.zeros((1, 1, 2, 2), dtype=np.float32),
] # Definitely real data
# For an identity network, the golden output values are the same as the input values.
# Though such a network appears useless at first glance, it can be very useful in some cases (like here!).
EXPECTED_OUTPUTS = REAL_DATASET
def main():
build_engine = EngineFromNetwork(NetworkFromOnnxPath("identity.onnx"))
with TrtRunner(build_engine) as runner:
for data, golden in zip(REAL_DATASET, EXPECTED_OUTPUTS):
# NOTE: The runner owns the output buffers and is free to reuse them between `infer()` calls.
# Thus, if you want to store results from multiple inferences, you should use `copy.deepcopy()`.
outputs = runner.infer(feed_dict={"x": data})
assert np.array_equal(outputs["y"], golden)
print("Validation succeeded!")
if __name__ == "__main__":
main()
tools/Polygraphy/examples/api/02_validating_on_a_dataset/identity.onnx
+15
View File
@@ -0,0 +1,15 @@
 backend-test:[

xy"Identity
test_identityZ
x




b
y




tools/Polygraphy/examples/api/02_validating_on_a_dataset/requirements.txt
+1
View File
@@ -0,0 +1 @@
numpy
tools/Polygraphy/examples/api/03_interoperating_with_tensorrt/README.md
+41
View File
@@ -0,0 +1,41 @@
# Interoperating With TensorRT
## Introduction
A key feature of Polygraphy is complete interoperability with TensorRT, as well as
with other backends. Since Polygraphy does not hide the underlying backend APIs,
it is possible to freely switch between using the Polygraphy API and a backend API,
such as TensorRT.
In this example, we'll look at how you can retain access to the advanced functionality
provided by a backend without giving up the conveniences provided by Polygraphy - the
best of both worlds.
Polygraphy provides an `extend` decorator which can be used to easily extend existing
Polygraphy loaders. This can be useful in many scenarios, but for this example,
we will focus on cases where you may want to:
- Modify the TensorRT network prior to building the engine
- Use a TensorRT builder flag not currently supported by Polygraphy
## Running The Example
1. Install prerequisites
* Ensure that TensorRT is installed
* Install other dependencies with `python3 -m pip install -r requirements.txt`
2. **[Optional]** Inspect the TensorRT network generated by `load_network()`.
This will invoke `load_network()` from within the script and display the
generated TensorRT network, which should be named `"MyIdentity"`:
```bash
polygraphy inspect model example.py --trt-network-func load_network --show layers attrs weights
```
3. Run the example:
```bash
python3 example.py
```
tools/Polygraphy/examples/api/03_interoperating_with_tensorrt/example.py
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
This script demonstrates how to use Polygraphy in conjunction with APIs
provided by a backend. Specifically, in this case, we use TensorRT APIs
to print the network name and enable FP16 mode.
"""
import numpy as np
import tensorrt as trt
from polygraphy import func
from polygraphy.backend.trt import (
CreateConfig,
EngineFromNetwork,
NetworkFromOnnxPath,
TrtRunner,
)
# TIP: The immediately evaluated functional API makes it very easy to interoperate
# with backends like TensorRT. For details, see example 06 (`examples/api/06_immediate_eval_api`).
# We can use the `extend` decorator to easily extend lazy loaders provided by Polygraphy
# The parameters our decorated function takes should match the return values of the loader we are extending.
# For `NetworkFromOnnxPath`, we can see from the API documentation that it returns a TensorRT
# builder, network and parser. That is what our function will receive.
@func.extend(NetworkFromOnnxPath("identity.onnx"))
def load_network(builder, network, parser):
# Here we can modify the network. For this example, we'll just set the network name.
network.name = "MyIdentity"
print(f"Network name: {network.name}")
# Notice that we don't need to return anything - `extend()` takes care of that for us!
# In case a builder configuration option is missing from Polygraphy, we can easily set it using TensorRT APIs.
# Our function will receive a TensorRT IBuilderConfig since that's what `CreateConfig` returns.
@func.extend(CreateConfig())
def load_config(config):
# Polygraphy supports the fp16 flag, but in case it didn't, we could do this:
config.set_flag(trt.BuilderFlag.FP16)
def main():
# Since we have no further need of TensorRT APIs, we can come back to regular Polygraphy.
#
# NOTE: Since we're using lazy loaders, we provide the functions as arguments - we do *not* call them ourselves.
build_engine = EngineFromNetwork(load_network, config=load_config)
with TrtRunner(build_engine) as runner:
inp_data = np.ones(shape=(1, 1, 2, 2), dtype=np.float32)
# NOTE: The runner owns the output buffers and is free to reuse them between `infer()` calls.
# Thus, if you want to store results from multiple inferences, you should use `copy.deepcopy()`.
outputs = runner.infer({"x": inp_data})
assert np.array_equal(outputs["y"], inp_data) # It's an identity model!
print("Inference succeeded!")
if __name__ == "__main__":
main()
tools/Polygraphy/examples/api/03_interoperating_with_tensorrt/identity.onnx
+15
View File
@@ -0,0 +1,15 @@
 backend-test:[

xy"Identity
test_identityZ
x




b
y




tools/Polygraphy/examples/api/03_interoperating_with_tensorrt/requirements.txt
+1
View File
@@ -0,0 +1 @@
numpy
tools/Polygraphy/examples/api/04_int8_calibration_in_tensorrt/README.md
+46
View File
@@ -0,0 +1,46 @@
# Int8 Calibration In TensorRT
## Introduction
Int8 calibration in TensorRT involves providing a representative set of input data
to TensorRT as part of the engine building process. The
[calibration API](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/python-api/infer/Int8/Calibrator.html)
included in TensorRT requires the user to handle copying input data to the GPU and
manage the calibration cache generated by TensorRT.
While the TensorRT API provides a higher degree of control, we can greatly simplify the
process for many common use-cases. For that purpose, Polygraphy provides a calibrator, which
can be used either with Polygraphy or directly with TensorRT. In the latter
case, the Polygraphy calibrator behaves exactly like a normal TensorRT int8 calibrator.
In this example, we'll look at how you can use Polygraphy's calibrator to calibrate a network
with (fake) calibration data, and how you can manage the calibration cache with just a single
parameter.
## Running The Example
1. Install prerequisites
* Ensure that TensorRT is installed
* Install other dependencies with `python3 -m pip install -r requirements.txt`
2. Run the example:
```bash
python3 example.py
```
3. The first time you run the example, it will create a calibration cache
called `identity-calib.cache`. If you run the example again, you should see that
it now uses the cache instead of running calibration again:
```bash
python3 example.py
```
## Further Reading
For more information on how int8 calibration works in TensorRT, see the
[developer guide](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#optimizing_int8_c)
tools/Polygraphy/examples/api/04_int8_calibration_in_tensorrt/example.py
+73
View File
@@ -0,0 +1,73 @@
#!/usr/bin/env python3
#
# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
This script demonstrates how to use the Calibrator API provided by Polygraphy
to calibrate a TensorRT engine to run in INT8 precision.
"""
import numpy as np
from polygraphy.backend.trt import (
Calibrator,
CreateConfig,
EngineFromNetwork,
NetworkFromOnnxPath,
TrtRunner,
)
from polygraphy.logger import G_LOGGER
# The data loader argument to `Calibrator` can be any iterable or generator that yields `feed_dict`s.
# A `feed_dict` is just a mapping of input names to corresponding inputs.
def calib_data():
for _ in range(4):
# TIP: If your calibration data is already on the GPU, you can instead provide GPU pointers
# (as `int`s), Polygraphy `DeviceView`s, or PyTorch tensors instead of NumPy arrays.
#
# For details on `DeviceView`, see `polygraphy/cuda/cuda.py`.
yield {"x": np.ones(shape=(1, 1, 2, 2), dtype=np.float32)} # Totally real data
def main():
# We can provide a path or file-like object if we want to cache calibration data.
# This lets us avoid running calibration the next time we build the engine.
#
# TIP: You can use this calibrator with TensorRT APIs directly (e.g. config.int8_calibrator).
# You don't have to use it with Polygraphy loaders if you don't want to.
calibrator = Calibrator(data_loader=calib_data(), cache="identity-calib.cache")
# We must enable int8 mode in addition to providing the calibrator.
build_engine = EngineFromNetwork(
NetworkFromOnnxPath("identity.onnx"),
config=CreateConfig(int8=True, calibrator=calibrator),
)
# When we activate our runner, it will calibrate and build the engine. If we want to
# see the logging output from TensorRT, we can temporarily increase logging verbosity:
with G_LOGGER.verbosity(G_LOGGER.VERBOSE), TrtRunner(build_engine) as runner:
# Finally, we can test out our int8 TensorRT engine with some dummy input data:
inp_data = np.ones(shape=(1, 1, 2, 2), dtype=np.float32)
# NOTE: The runner owns the output buffers and is free to reuse them between `infer()` calls.
# Thus, if you want to store results from multiple inferences, you should use `copy.deepcopy()`.
outputs = runner.infer({"x": inp_data})
assert np.array_equal(outputs["y"], inp_data) # It's an identity model!
if __name__ == "__main__":
main()
tools/Polygraphy/examples/api/04_int8_calibration_in_tensorrt/identity.onnx
+15
View File
@@ -0,0 +1,15 @@
 backend-test:[

xy"Identity
test_identityZ
x




b
y





Some files were not shown because too many files have changed in this diff Show More