The Polygraphy command-line toolkit includes several tools covering a wide variety of prototyping and debugging use-cases. This guide provides a broad overview of the capabilities included.

For more information about a specific tool, see the README in the corresponding directory here.

All the tools provided by Polygraphy can be invoked using the polygraphy binary: polygraphy.

For usage information on a specific tool, you can refer to the help output with: polygraphy <subtool> -h.

NOTE: Some of the tools included are still experimental. Any tool labeled [EXPERIMENTAL] may be subject to backwards-incompatible changes, or even complete removal at any point in time.

Common Use-Cases

Inspecting A Model

It is often useful to inspect various aspects of a model, such as layer names, attributes, or overall structure. The inspect model subtool aims to provide this functionality in a way that is conducive to programmatic analysis; that is, it provides a human-readable text representation of the model (and it's grep-friendly!).

For more details, refer to the examples, which, among other things, show how to inspect:

You can find the complete listing of inspect examples here.

Converting A Model To TensorRT

The convert tool can be used to convert various types of models to other formats; for example, converting ONNX models to TensorRT.

For more information, refer to the examples, which, among other things, show how to:

You can find the complete listing of convert examples here.

Linting An ONNX Model

Validating an ONNX model can be more than checking if it passes ONNX specifications. For example, it could be data-dependant, or depend on the underlying runtime. Moreover, an ONNX model can be broken in multiple places that may only be uncovered iteratively.

The subtool check lint validates the model for specific use-cases (dynamic shape, custom data, custom weights etc.). When model is broken, it also attempts to catch all independent exceptions/warnings in one go.

For more details, refer to check example 01.

Sanitizing An ONNX Model

Sometimes, TensorRT may be unable to import an ONNX model. In these cases, it often helps to sanitize the ONNX model, removing excess nodes and folding constants. The surgeon sanitize subtool provides this capability using ONNX-GraphSurgeon and ONNX-Runtime.

For more details, refer to surgeon example 02.

You can find the complete listing of surgeon examples here.

Comparing A Model Between Frameworks

The run tool can run inference and compare outputs generated by one or more frameworks. This can be used to check the accuracy of a framework for a particular model and debug when the accuracy is poor.

Moreover, it can be used to generate Python scripts that do the same.

For more information, refer to the examples, which, among other things, show how to:

You can find the complete listing of run examples here.

Modifying Input Shapes In An ONNX Model

The best way to modify input shapes in an ONNX model is to re-export the model with the desired input shapes. When this is not possible, you can use the surgeon sanitize tool to do this.

For more information, refer to surgeon example 03.

Advanced Topics

Deterministic Engine Builds

Because of the way TensorRT works, the engine building process is non-deterministic. Even when using the same model with the same builder configuration, engines can vary slightly.

To make engine building reproducible, all Polygraphy CLI tools that involve building TensorRT engines accept --save-tactics and --load-tactics options, which allow you to save the tactics selected for an engine and reload them during subsequent builds.

For more information, refer to convert example 02.

Defining A Custom TensorRT Network Or Builder Configuration

Many of the command-line tools involve creating TensorRT networks. Most of the time, networks are created by parsing a model from a framework (generally in ONNX format). However, it is also possible to define the TensorRT network manually using a Python script.

This is useful if you want to modify the network in some way using the TensorRT Python API; For example, setting layer precisions, or per-layer device preferences.

Similarly, it is also possible to provide a custom builder configuration.

This is useful in cases where Polygraphy may not yet support certain TensorRT builder configuration options.

For more information, refer to run example 04.

Extracting A Subgraph Of An ONNX Model

When debugging ONNX models, it can be useful to extract subgraphs to look at them in isolation. The surgeon extract tool allows you to do so.

For more information, refer to surgeon example 01.

Debugging Intermittent TensorRT Failures

Since the TensorRT optimizer is inherently non-deterministic, rebuilding an engine may result in different tactics being selected. If a particular tactic is faulty, this may manifest as an intermittent failure. The debug build tools helps find faulty tactics in such cases and reproduce failures reliably.

For more information, refer to debug example 01.

Reducing Failing ONNX Models

When investigating bugs involving ONNX models, it can be useful to reduce the model to a minimum failing subgraph. This helps us pinpoint the issue and makes further debugging much easier. The debug reduce tools helps us automate this process.

For more information, refer to debug example 02.

Examples

For examples, see the corresponding subdirectory in examples/cli

Adding New Tools

You can add a new tool by adding a new file in this directory, creating a class that extends the Tool base class, and adding the new tool to the registry.

For details on developing tools, see this example.

README.md

Polygraphy Command-line Toolkit User Guide

Table of Contents

Introduction