wehub-resource-sync
1972 lines
95 KiB
Markdown
1972 lines
95 KiB
Markdown
# Polygraphy Change Log
|
|
|
|
Dates are in YYYY-MM-DD format.
|
|
|
|
## v0.49.27
|
|
### Added
|
|
- Added `polygraphy template shard-hints` to generate hints file for `polygraphy multi-device shard`.
|
|
- Added support for inserting transposes to `polygraphy multi-device shard` when the sequence length dimension of sharded tensors is not 0.
|
|
|
|
### Changed
|
|
- Changed hints file format for `polygraphy multi-device shard` to support new additions.
|
|
- Changed `polygraphy multi-device shard` example and README to reflect new additions.
|
|
|
|
### Fixed
|
|
- Fixed issue when `polygraphy multi-device shard` would exceed python recursive depth limit on large models.
|
|
|
|
|
|
## v0.49.26 (2025-07-16)
|
|
### Added
|
|
- Added support for dumping unfused tensors when running TensorRT using `--mark-unfused-tensors-as-debug-tensors`.
|
|
int4, fp4, fp8, and bfloat16 tensors are not supported and will be skipped.
|
|
- Added support for combining tensor statistics from the tensor JSON file with the output of the engine inspector.
|
|
|
|
### Changed
|
|
- Relaxed `onnxconverter_common` version requirements to allow newer versions to be installed.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `--save-heatmaps` and `--save-error-metrics-plot` would not handle slashes in output names well.
|
|
- Fixed a bug where `--save-error-metrics-plot` would not work if the minimum error was 0.
|
|
|
|
|
|
## v0.49.25 (2025-06-11)
|
|
### Added
|
|
- Added `multi-device shard` tool for converting SD models to MD.
|
|
|
|
|
|
## v0.49.24 (2025-05-27)
|
|
### Fixed
|
|
- Fixed various version checks to account for TRT-RTX versions.
|
|
|
|
|
|
## v0.49.23 (2025-05-22)
|
|
### Fixed
|
|
- Fixed various issues with loading TRT-RTX.
|
|
|
|
|
|
## v0.49.22 (2025-05-09)
|
|
### Changed
|
|
- Updated TensorRT loaders to handle some missing attributes more gracefully.
|
|
|
|
|
|
## v0.49.21 (2025-05-08)
|
|
### Added
|
|
- Added new comparison functions to `CompareFunc`
|
|
|
|
### Changed
|
|
- Updated `Comparator` to avoid making deep copies of outputs where possible.
|
|
|
|
|
|
## v0.49.20 (2025-03-04)
|
|
### Fixed
|
|
- Fixed a bug where `colored` did not work for *older* versions of the `colored` package.
|
|
|
|
|
|
## v0.49.19 (2025-03-04)
|
|
### Added
|
|
- Added support for int4 types in various ONNX tooling (e.g. `surgeon extract`).
|
|
|
|
### Fixed
|
|
- Fixed a bug where attempting to print layer information with `inspect model` would sometimes crash for
|
|
engines built with TensorRT 10.0 and newer.
|
|
- Fixed a bug where logging colors did not work correctly with newer versions of the `colored` package.
|
|
- Fixed a bug where `SessionFromOnnx` would try to use UNIX only `os` APIs on other platforms.
|
|
- Fixed a bug where some Polygraphy commands would crash with Python 3.13.
|
|
|
|
|
|
## v0.49.18 (2025-01-07)
|
|
### Added
|
|
- Added `BufferFromEngine` that serializes an engine without making an extra copy of the serialized engine.
|
|
|
|
### Changed
|
|
- Changed `SaveEngine` to use `BufferFromEngine` instead of `BytesFromEngine` to save memory.
|
|
|
|
|
|
## v0.49.17 (2024-12-06)
|
|
### Added
|
|
- Added TensorRT 10.8 tiling optimization api
|
|
|
|
|
|
## v0.49.16 (2024-11-21)
|
|
### Added
|
|
- Print a message to tell users that TF32 can be enabled to improve perf with
|
|
minor accuracy differences.
|
|
|
|
### Fixed
|
|
- Fixed a bug where the `inspect model` would not display the data type
|
|
correctly for `bfloat16` tensors in TensorRT engines.
|
|
- Fixed a bug where `FormattedArray` would not work correctly for sub-byte types.
|
|
|
|
|
|
## v0.49.15 (2024-11-13)
|
|
### Changed
|
|
- Deprecated the classes and functions in algorithm_selector.py as we deprecated Algorithm Selector in TensorRT 10.8.
|
|
|
|
|
|
## v0.49.14 (2024-09-10)
|
|
### Added
|
|
- Added `DataType.FLOAT4` for 4-bit floats (E2M1).
|
|
|
|
|
|
## v0.49.13 (2024-07-15)
|
|
### Added
|
|
- Added option to emit logs using python `logging` module.
|
|
|
|
|
|
## v0.49.12 (2024-05-28)
|
|
### Added
|
|
- Added `runtime_platform` to `CreateConfig` for TensorRT and corresponding `--runtime-platform` command-line option.
|
|
|
|
|
|
## v0.49.11 (2024-05-09)
|
|
### Added
|
|
- Added TensorRT 10.1 weight streaming V2 APIs.
|
|
- Added TensorRT 10.1 runtime device memory V2 APIs.
|
|
|
|
### Changed
|
|
- Changed the meaning of the `TrtRunner`'s weight streaming budget argument.
|
|
|
|
|
|
## v0.49.10 (2024-04-19)
|
|
### Added
|
|
- Added an `EngineFromPath` loader to deserialize an engine directly from disk. This will save CPU memory when weight streaming is enabled.
|
|
|
|
### Fixed
|
|
- Fixed a memory leak in `TrtRunner` caused by creating a new output allocator per inference.
|
|
- Fixed a bug where the `Calibrator` would not force non-index inputs to FP32; this is required by TensorRT.
|
|
|
|
|
|
## v0.49.9 (2024-03-19)
|
|
### Added
|
|
- Added `run_opts` argument to `tools.main` to allow calling polygraphy tools from within other Python scripts.
|
|
|
|
### Changed
|
|
- Updated weight streaming flag to accept a percentage.
|
|
|
|
### Fixed
|
|
- Fixed a bug where a large amount of mismatch between two runner outputs would generate an out of memory error.
|
|
|
|
|
|
## v0.49.8 (2024-02-16)
|
|
### Added
|
|
- Added `--mark-debug` command-line option.
|
|
- Initializes trt plugin library by default when using the TensorRT ONNX parser
|
|
|
|
|
|
## v0.49.7 (2024-02-02)
|
|
### Added
|
|
- Added `plugin match` subtool that finds opportunities for plugin substitution in an ONNX Model and prepares an intermediate file to be used for actual substitution
|
|
- Added `plugin list` subtool that lists opportunities for plugin substitution, without preparing an intermediate file.
|
|
- Added support for building engines with the refittable weights stripped.
|
|
Setting the `strip_plan` parameter of `CreateConfig` or passing in the `--strip-plan` flag
|
|
enables building engines with the refittable weights stripped.
|
|
- Added `plugin replace` subtool that replaces subgraphs with plugins, based on an intermediate file (config.yaml)
|
|
- Added `polygraphy surgeon weight-strip` to strip the initializers of selective nodes in an ONNX model
|
|
- Added `polygraphy surgeon weight-reconstruct` to read a weightless ONNX model and fill the empty initializers with proxy tensors
|
|
- Added '--weight-streaming` and `--weight-streaming-budget` APIs to control TRT weight streaming
|
|
|
|
|
|
## v0.49.6 (2024-01-18)
|
|
### Fixed
|
|
- Fixed a bug where `explicit_batch` would be provided by default on TRT 10.0, where it has been removed.
|
|
|
|
|
|
## v0.49.5 (2024-01-16)
|
|
### Added
|
|
- Added an `allocation_strategy` to `TrtRunner` and corresponding `--allocation-strategy` argument to CLI tools.
|
|
|
|
### Fixed
|
|
- Fixed a bug where the reference count of the TensorRT engine would not be decremented correctly in `TrtRunner`.
|
|
|
|
|
|
## v0.49.4 (2023-12-20)
|
|
### Fixed
|
|
- Fixed a bug where the comparator would modify the original output tensor in some cases instead of operating on a copy.
|
|
- Fixed a bug where `is_installed()` for lazily imported modules would not work if the package name differed from the module name.
|
|
|
|
|
|
## v0.49.3 (2023-11-30)
|
|
### Changed
|
|
- Improved error messages in the default data loader for invalid backend modules.
|
|
|
|
|
|
## v0.49.2 (2023-11-27)
|
|
### Added
|
|
- Added `DataType.INT4` for 4-bit signed integers.
|
|
|
|
### Changed
|
|
- Removed internal usage of several deprecated TensorRT APIs.
|
|
|
|
### Fixed
|
|
- Fixed a bug in the default data loader where scalars would not be generated correctly.
|
|
|
|
|
|
## v0.49.1 (2023-10-03)
|
|
### Added
|
|
- Added `--profiling-verbosity` command-line option.
|
|
- Added a `progress_monitor` parameter to `CreateConfig`.
|
|
- Added a `data_loader_backend_module` parameter to `DataLoader` and corresponding `--data-loader-backend-module` argument to CLI tools to
|
|
choose between generating `numpy.ndarray` and `torch.tensor` in the default dataloader.
|
|
|
|
### Fixed
|
|
- Fixed a bug where warnings would be issued for unsupported versions of `torch` even if
|
|
`torch` was not being used.
|
|
|
|
|
|
## v0.49.0 (2023-07-28)
|
|
### Added
|
|
- Added `check lint` subtool that validates ONNX Models and generates human-readable console output and
|
|
a JSON report detailing unused or invalid nodes as well as model errors.
|
|
- Added a new `inspect sparsity` subtool that can check whether the weights in a model are sparse.
|
|
- Added a `disable_compilation_cache` parameter to `CreateConfig` and corresponding `--disable-compilation-cache` argument
|
|
to CLI tools.
|
|
- Added a `"quantile"` mode to `CompareFunc.simple()`'s `check_error_stat` parameter.
|
|
- Added an `error_quantile` parameter to `CompareFunc.simple()` and corresponding `--error-quantile` argument to
|
|
CLI tools to specify the error quantile when `check_error_stat="quantile"`.
|
|
|
|
### Changed
|
|
- Improved error messages when deserialization fails due to a missing module.
|
|
- Changed `SaveOnnx` to use `external_data_path=""` when the model exceeds the protobuf size limit and no external data path is provided.
|
|
This prevents scenarios where a long running command like `surgeon sanitize --fold-constants` would finally
|
|
complete only to fail when attempting to save the final model. With the new behavior, the model will be saved successfully
|
|
with external data in a default location.
|
|
|
|
|
|
## v0.48.1 (2023-07-05)
|
|
### Changed
|
|
- Updated default `DataLoader` to show better errors when it can't generate inputs due to data types unsupported by NumPy.
|
|
In such cases, you must provide a custom data loader.
|
|
|
|
### Fixed
|
|
- Fixed a bug where older versions of ONNX would cause failures due to missing data types.
|
|
- Fixed a bug where top-K implementation would not work for PyTorch FP16 tensors on CPU.
|
|
|
|
|
|
## v0.48.0 (2023-06-13)
|
|
### Added
|
|
- Added a `quantization_flags` parameter to `CreateConfig` and corresponding `--quantization-flags` argument
|
|
to CLI tools to enable setting TensorRT builder quantization flags.
|
|
- Added a `error_on_timing_cache_miss` parameter to `CreateConfig` and corresponding `--error-on-timing-cache-miss` argument
|
|
to CLI tools.
|
|
- Added a `bf16` option to the TensorRT `CreateConfig` loader and corresponding `--bf16` argument to CLI tools.
|
|
- Added a common `DataType` class which can convert between data type classes of various other frameworks,
|
|
like NumPy, PyTorch, and TensorRT.
|
|
- Added support for PyTorch tensors in `TrtRunner` and `Calibrator`. See [the example](examples/api/09_working_with_pytorch_tensors/) for details.
|
|
- Added support for PyTorch tensors in `OnnxrtRunner`.
|
|
- Added a `strongly_typed` option to TensorRT network loaders and a corresponding `--strongly-typed` argument to CLI tools.
|
|
- Added `polygraphy surgeon prune` to prune a model to be sparse. Note that this will *not* retain the accuracy of the model and should hence be used only for functional testing.
|
|
- Added a `--toposort` option to `surgeon sanitize` to topologically sort nodes.
|
|
|
|
### Changed
|
|
- `TensorMetadata` will now automatically convert data types to Polygraphy's `DataType` class.
|
|
The data types can be converted to corresponding NumPy types using the `numpy()` method.
|
|
This affects any interface that returns instances of this class. For example, the `Calibrator`
|
|
sets input metadata on the provided data loader in the form of a `TensorMetadata` instance.
|
|
*NOTE: For compatibility, the runner method, `get_input_metadata` will retain its previous behavior of using NumPy types.*
|
|
*In a future version of Polygraphy, it will be updated to return only Polygraphy `DataType`s.*
|
|
|
|
### Fixed
|
|
- Fixed a bug where the `DataLoader` would not generate inputs correctly if a scalar was provided
|
|
in the input metadata. Note that since scalars have fixed shapes, they do need to be specified
|
|
via the `input_metadata` argument, so a workaround on older versions is to simply omit scalars.
|
|
|
|
### Removed
|
|
- Removed the TensorRT Legacy runner which supported UFF and Caffe models.
|
|
- Removed support for TensorRT versions older than 8.5.
|
|
- Removed `max_workspace_size` option in TensorRT's `CreateConfig` and corresponding `--workspace` argument from CLI tools.
|
|
- Removed `strict_types` option in TensorRT's `CreateConfig` and corresponding `--strict-types` argument from CLI tools.
|
|
- Removed `debug diff-tactics` alias for `inspect diff-tactics`. `diff-tactics` is now available only under the `inspect` tool.
|
|
|
|
|
|
## v0.47.1 (2023-03-29)
|
|
### Changed
|
|
- Updated `TrtOnnxFlagArgs` to automatically enable `NATIVE_INSTANCENORM` when either hardware or
|
|
version compatibility is enabled in the builder configuration.
|
|
- Downgraded errors for extra layers/tensors in `SetLayerPrecisions`, `SetTensorDatatypes`, and `SetTensorFormats` to warnings.
|
|
|
|
|
|
## v0.47.0 (2023-03-28)
|
|
### Added
|
|
- Added experimental support for error heatmaps. These can be visualized and/or saved with the
|
|
`--show-heatmaps`/`--save-heatmaps` command-line options or `show_heatmaps`/`save_heatmaps`
|
|
arguments to `CompareFunc.simple`.
|
|
- Added experimental `--show-error-metrics-plot/--save-error-metrics-plot` command-line options and
|
|
corresponding `show_error_metrics_plot`/`save_error_metrics_plot` arguments to `CompareFunc.simple`.
|
|
These allow you to generate plots of error vs. magnitude.
|
|
- Added `--version-compatible` flag for building version-compatible engines.
|
|
Note that for building version compatible engines for ONNX models, `--onnx-flags native_instancenorm` must also be provided.
|
|
- Added `TrtSaveEngineBytesArgs` and `TrtLoadEngineBytesArgs` to allow for avoiding engine deserialization until necessary.
|
|
- Added an `exclude_lean_runtime` parameter to `CreateConfig` and corresponding `--exclude-lean-runtime` CLI option.
|
|
- Added a `runtime` parameter to `EngineFromBytes` and `EngineFromNetwork` to enable deserializing plans with a custom runtime.
|
|
- Added a `LoadRuntime` TensorRT loader that can be used to load a runtime from a path and a corresponding `--load-runtime` CLI option.
|
|
|
|
### Changed
|
|
- Updated Polygraphy to warn when it detects unsupported TensorRT and NumPy version combinations.
|
|
- `TrtSaveEngineArgs` and `TrtLoadEngineArgs` now depend on `TrtSaveEngineBytesArgs` and `TrtLoadEngineBytesArgs` respectively.
|
|
Additionally, all command-line options have been migrated to the latter argument groups.
|
|
|
|
### Fixed
|
|
- Fixed a bug in `debug precision` where Polygraphy would attempt to set the layer precision for layers producing
|
|
non-activation outputs, which is an error in TensorRT.
|
|
|
|
|
|
## v0.46.2 (2023-02-28)
|
|
### Fixed
|
|
- Fixed minor formatting issues in help text
|
|
|
|
|
|
## v0.46.1 (2023-02-27)
|
|
### Changed
|
|
- `CompareFunc.simple` will now add a small epsilon when computing relative error to avoid Inf/NaNs.
|
|
|
|
|
|
## v0.46.0 (2023-02-10)
|
|
### Changed
|
|
- `polygraphy run` will now print warnings when command-line options are provided for comparison
|
|
functions types other than the current one specified by `--compare-func`.
|
|
- Added a `TensorInfo` class to the TensorRT backend to track information from `IAlgorithmIOInfo`.
|
|
The `Algorithm` class now keeps `TensorInfo`s instead of tuples.
|
|
- Changed the format of tactic replay files to include more information about tensor formats where possible.
|
|
*NOTE: This means that tactic replay files generated with previous versions of Polygraphy are not compatible*
|
|
*with this version!*
|
|
|
|
### Fixed
|
|
- Fixed a bug where the `--trt-legacy` runner would not work with `--input-shapes` specified.
|
|
- Fixed a bug where `debug reduce` would not work correctly for models where a node had multiple outputs
|
|
which were also graph outputs. See the comment in [`reduce.py`](./polygraphy/tools/debug/subtool/reduce.py) for details.
|
|
|
|
|
|
## v0.45.3 (2023-01-25)
|
|
### Changed
|
|
- Updated comparison functions so that the output array is now displayed *in addition* to the histogram
|
|
rather than instead of it.
|
|
|
|
|
|
## v0.45.2 (2023-01-25)
|
|
### Changed
|
|
- Updated comparison functions to display the entire output array instead of a histogram if it is small enough.
|
|
|
|
|
|
## v0.45.1 (2023-01-19)
|
|
### Added
|
|
- Added `max_aux_streams` to `CreateConfig` for TensorRT and corresponding `--max-aux-streams` command-line option.
|
|
- Added support for HWC I/O formats in `TrtRunner` for TensorRT 8.6+.
|
|
|
|
|
|
## v0.45.0 (2023-01-12)
|
|
### Added
|
|
- Added an `-n/--num-items` option to `inspect data` to control how many elements of an array are shown.
|
|
- Added a `--line-width` option to `inspect data` to control how many characters are displayed per line when showing an array.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `CreateConfig` was not included in the API documentation.
|
|
- Reimplemented calibrator restriction so inputs always use `OPT` shape as TensorRT does not currently support
|
|
using other shapes during calibration.
|
|
|
|
## Removed
|
|
- Removed various deprecated APIs:
|
|
- `input_metadata` parameter in `Calibrator.reset()`.
|
|
Now, `Calibrator.set_input_metadata()` implements this functionality and generally does not need to be called manually.
|
|
- `version` parameter in `mod.lazy_import()`.
|
|
The version can now be specified as part of the package name. For example, `tensorrt>=8.0`.
|
|
- `--timing-cache` CLI argument.
|
|
This is replaced by `--load-timing-cache` and `--save-timing-cache`.
|
|
- Legacy CLI shape syntax. See the [CHANGELOG entry for v0.32.0](#v0320-2021-08-10) for details.
|
|
|
|
|
|
## v0.44.2 (2022-12-16)
|
|
### Added
|
|
- Added `fp8` parameter to `CreateConfig` for TensorRT and corresponding `--fp8` command-line option.
|
|
- Added `hardware_compatibility_level` to `CreateConfig` for TensorRT and corresponding `--hardware-compatibility-level` command-line option.
|
|
|
|
### Fixed
|
|
- Fixed a bug where the calibrator would not accept inputs with a shape other than the `OPT` shape set in the profile.
|
|
|
|
|
|
## v0.44.1 (2022-12-06)
|
|
### Fixed
|
|
- Fixed a bug where paths on Windows including drive letters would not be parsed correctly.
|
|
|
|
|
|
## v0.44.0 (2022-11-30)
|
|
### Added
|
|
- Added an experimental `PostprocessConfig` loader to edit the TensorRT `IBuilderConfig`s generated by Polygraphy
|
|
and a corresponding CLI option, `--trt-config-postprocess-script`.
|
|
|
|
### Changed
|
|
- Updated `TrtRunner` and associated utilities to use the TensorRT v3 inference APIs exclusively when available.
|
|
Previous changes only used the v3 APIs in the `infer()` method.
|
|
- Updated all arguments that take a script to allow the function name to be specified with the script argument instead
|
|
of with a separate option. For example, you can now write: `--data-loader-script my_script.py:my_func` instead of
|
|
`--data-loader-script my_script.py --data-loader-func-name my_func`.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `TrtRunner.get_input_metadata()` would crash in older versions of TensorRT when no optimization profile was set.
|
|
Now, an error message is emitted instead.
|
|
- Fixed an issue where immediately evaluated loaders would display a generic function name in error messages
|
|
instead of the name of the loader being called.
|
|
- Fixed a bug where setting `preview_features` in `CreateConfig` would not disable default preview features
|
|
even when they were not part of the provided preview feature list.
|
|
|
|
|
|
## v0.43.1 (2022-10-12)
|
|
### Fixed
|
|
- Fixed a bug due to API breakages in `importlib_metadata`.
|
|
|
|
|
|
## v0.43.0 (2022-10-06)
|
|
### Added
|
|
- Added a new `template onnx-gs` subtool that allows you to generate template files for
|
|
Python scripts to modify an ONNX model using ONNX-GraphSurgeon.
|
|
- Added a new `PostprocessNetwork` loader and corresponding `--trt-network-postprocess-script` command-line flag
|
|
to allow for postprocessing a loaded TensorRT network using a user-defined function.
|
|
|
|
### Changed
|
|
- Updated the [working with run results example](./examples/api/08_working_with_run_results_and_saved_inputs_manually/)
|
|
to demonstrate how to work with saved input data as well.
|
|
- Updated CLI subtools to report duration of the command by default.
|
|
- Improved error messages in the `ExtractSubgraph` loader and consequently `surgeon extract` when the extracted
|
|
subgraph is invalid due to tensors without producers not marked as graph inputs.
|
|
- Updated the [adding precision constraints run example](./examples/cli/run/08_adding_precision_constraints/)
|
|
to demonstrate how to use the `--trt-postprocess-network` flag to apply precision constraints to a parsed TensorRT
|
|
network.
|
|
|
|
### Fixed
|
|
- Fixed a bug where specifying the same runner multiple times would have no effect.
|
|
- Fixed a bug where `Script` would close `stdout` after writing to it.
|
|
- Fixed a bug where an API change in `onnxmltools` caused a breakage in the `ConvertToFp16` loader.
|
|
- Fixed a bug where `debug precision` would not respect the layer precisions set by `--layer-precisions`.
|
|
|
|
|
|
## 0.42.2 (2022-09-22)
|
|
### Added
|
|
- Added a [how-to guide](./how-to/use_debug_subtools_effectively.md) on using `debug` subtools effectively.
|
|
- Added a [how-to guide](./how-to/use_debug_reduce_effectively.md) on using `debug reduce` effectively
|
|
and refactored the [`debug reduce` example README](./examples/cli/debug/02_reducing_failing_onnx_models/)
|
|
to omit information that's now captured in the how-to guide.
|
|
- Added a [how-to guide](./how-to/use_custom_input_data.md) on using custom inference input data.
|
|
|
|
### Changed
|
|
- Moved the `diff-tactics` tool to `inspect`. `debug diff-tactics` is now an alias for `inspect diff-tactics`
|
|
to preserve backwards compatibility.
|
|
- Updated command-line parsing logic to handle `inf`, `-inf`, and `nan`. These can now be specified to
|
|
options that expect floating point numbers, such as `--val-range`.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `Profile.fill_defaults` would not work correctly for scalar shape-tensor
|
|
network inputs.
|
|
- Fixed a bug where `lazy_import` would repeatedly call `importlib.import_module`.
|
|
|
|
|
|
## v0.42.1 (2022-09-07)
|
|
### Changed
|
|
- Updated Polygraphy's TensorRT algorithm selectors to keep track of strides in addition to
|
|
tensor formats and data types.
|
|
|
|
### Fixed
|
|
- Fixed a bug where marking output tensors in a TensorRT network containing layers with omitted
|
|
optional inputs would raise an exception.
|
|
|
|
|
|
## v0.42.0 (2022-09-01)
|
|
### Added
|
|
- Added a `SetTensorDatatypes` loader and corresponding `--tensor-datatypes` command-line flag
|
|
to allow for setting per-tensor data types in TensorRT networks.
|
|
- Added a `direct_io` parameter to `CreateConfig` and corresponding `--direct-io` command-line flag.
|
|
- Added a `SetTensorFormats` loader and corresponding `--tensor-formats` command-line flag
|
|
to allow for setting per-tensor `allowed_formats` in TensorRT networks.
|
|
- Added an *experimental*, *untested* API, `FormattedArray` to allow `TrtRunner` to support
|
|
vectorized formats. This API will be documented, tested, and improved in a future release.
|
|
Until then, use with caution!
|
|
|
|
### Changed
|
|
- Updated `InferShapes` to use ONNX-Runtime's shape inference utilities where possible,
|
|
as they are more performant and memory efficient than the ONNX shape inference utilities.
|
|
This new behavior, which is on by default, can be disabled by setting `allow_onnxruntime=False`,
|
|
or the `--no-onnxruntime-shape-inference` CLI option.
|
|
|
|
### Fixed
|
|
- Fixed a bug in `inspect model` where dimensions of ONNX tensors without `dim_value` or `dim_param`
|
|
set would be shown as `0`s instead of `-1`s.
|
|
|
|
### Removed
|
|
- Removed the `explicit_precision` parameter from TensorRT network loaders.
|
|
|
|
|
|
## v0.41.1 (2022-08-25)
|
|
### Changed
|
|
- Updated `surgeon extract` and `debug reduce` to no longer attempt to retrieve values for all intermediate
|
|
tensors when running fallback shape inference. Instead, only the minimum required tensors
|
|
are retrieved. This greatly reduces the memory requirements for large models where fallback shape
|
|
inference is required.
|
|
|
|
### Fixed
|
|
- Fixed a bug where the logger would display incorrect file paths and line numbers when `line_info`
|
|
was enabled.
|
|
|
|
|
|
## v0.41.0 (2022-08-24)
|
|
### Added
|
|
- Added a [new example](./examples/api/08_working_with_run_results_and_saved_inputs_manually/) showing how to work with `RunResults` using the API.
|
|
- Added a `size_threshold` option in `FoldConstants` and corersponding `--fold-size-threshold` CLI option,
|
|
which allows for skipping constant folding on operations which would generate constants larger than
|
|
the provided threshold.
|
|
- Added support for data-dependent shapes with TensorRT 8.5 by updating `TrtRunner`
|
|
to use the new `execute_async_v3` and related APIs when possible.
|
|
|
|
### Changed
|
|
- Updated the `Calibrator` with a `set_input_metadata()` method and deprecated the `input_metadata`
|
|
parameter in `reset()`.
|
|
- Updated `CreateConfig` to no longer implicitly `reset()` the calibrator.
|
|
Resetting is only required if the calibrator is used across multiple different networks.
|
|
- Updated `EngineFromNetwork` and related loaders to set input metadata on Polygraphy calibrators.
|
|
This way, metadata is set even when using a TensorRT `IBuilderConfig` created outside Polygraphy.
|
|
|
|
### Removed
|
|
- Removed support for ONNX-GraphSurgeon `0.3.20` and older in `FoldConstants`.
|
|
- Removed support for TensorRT 6.0 and earlier.
|
|
|
|
|
|
## v0.40.4 (2022-08-17)
|
|
### Changed
|
|
- Updated the logger to perform less processing on messages to speed up logging.
|
|
|
|
### Fixed
|
|
- Fixed an issue where `ctrl-C` would be caught and ignored by TensorRT's logger.
|
|
Polygraphy will now generate a `SIGTERM` signal when a `KeyboardInterrupt` is triggered.
|
|
|
|
|
|
## v0.40.3 (2022-08-17)
|
|
### Added
|
|
- Added a `SetLayerPrecisions` loader to set layer compute precisions in TensorRT networks
|
|
and corresponding `--layer-precisions` CLI option.
|
|
- Added an `ignore_external_data` option to `OnnxFromPath` and corresponding `--ignore-external-data` option
|
|
so that it's still possible to manipulate the model to some degree if external weights are missing.
|
|
|
|
### Changed
|
|
- Updated Polygraphy's TensorRT logger implementation for TensorRT 8.0 and newer to redirect to
|
|
the Polygraphy global logger. This enables Polygraphy logging features like colored output
|
|
and redirection to a log file.
|
|
|
|
|
|
## v0.40.2 (2022-08-12)
|
|
### Fixed
|
|
- Fixed a bug where the context manager created by `G_LOGGER.verbosity()` would not correctly
|
|
revert the logging verbosity to its original value.
|
|
|
|
|
|
## v0.40.1 (2022-08-08)
|
|
### Added
|
|
- Added per-path logging verbosity settings and a corresponding `--verbosity` command-line option.
|
|
- Added an `infinities_compare_equal` parameter to `CompareFunc.simple()` as well as corresponding
|
|
`--infinities-compare-equal` command-line flag, to allow matching infinite
|
|
values in outputs to have an absdiff of 0 for the purpose of comparison.
|
|
|
|
### Changed
|
|
- Updated the `max_workspace_size` option in `CreateConfig` to match the default behavior of TensorRT
|
|
instead of defaulting to 16 MiB.
|
|
|
|
|
|
## v0.40.0 (2022-07-29)
|
|
### Added
|
|
- Added a TensorRT `engine_capability` parameter to `CreateConfig()` and
|
|
corresponding `--engine-capability` option to `polygraphy convert`.
|
|
|
|
### Changed
|
|
- Updated `CompareFunc.simple()` to cast unsigned and low-precision arrays to signed, higher precisions.
|
|
This prevents underflows/overflows during difference computations.
|
|
- Updated `copy_to` and `copy_from` in `cuda.DeviceView` and `cuda.DeviceArray` to no longer implicitly resize
|
|
the host and device buffers.
|
|
- Updated `debug reduce` to automatically remove unused graph inputs to create more minimal models.
|
|
- Updated `func.extend()` to provide a mechanism to accept the input parameters of the extended
|
|
function as parameters to the decorated function (see the docstring for details).
|
|
|
|
### Removed
|
|
- Removed various deprecated APIs:
|
|
- `fold_constant` parameter in `OnnxFromTfGraph` (removed in the underlying library, `tf2onnx`)
|
|
- `obey_precision_constraints` in `backend.trt.CreateConfig` (replaced by `precision_constraints`)
|
|
- `basic_compare_func` in `CompareFunc` (replaced by `CompareFunc.simple`)
|
|
- `topk_func` in `PostprocessFunc` (replaced by `PostprocessFunc.top_k`)
|
|
- `axis` parameter in `top_k` (`axis` can be specified as part of the `k` parameter)
|
|
- `--mode` option in `inspect model` (replaced by `--show`)
|
|
|
|
|
|
## v0.39.0 (2022-07-01)
|
|
### Added
|
|
- Added a `preview_features` parameter to `CreateConfig` and corresponding `--preview-features` argument
|
|
to CLI tools to enable TensorRT preview features.
|
|
|
|
|
|
## v0.38.1 (2022-06-22)
|
|
|
|
### Added
|
|
- Added support for building refittable engines in Polygraphy.
|
|
Setting the `refittable` parameter of `CreateConfig` or passing in the `--refittable` flag
|
|
enables building refittable engines.
|
|
|
|
### Changed
|
|
- Passing in a nonexistent file to Polygraphy's `--load-timing-cache` option or the
|
|
`load_timing_cache` parameter of `CreateConfig` is no longer a fatal error.
|
|
Polygraphy will now warn the user and fall back to using an empty timing cache.
|
|
|
|
### Fixed
|
|
- Fixed broken links in documentation.
|
|
|
|
|
|
## v0.38.0 (2022-05-24)
|
|
### Added
|
|
- Added an `optimization_profile` parameter to the constructor of `TrtRunner` to allow for setting an optimization profile index
|
|
whenver the runner is activated.
|
|
- Added an `--optimization-profile` argument to CLI tools to allow for setting the optimization profile to use for inference.
|
|
- Rewrote examples on [comparing frameworks](./examples/cli/run/01_comparing_frameworks) and
|
|
[comparing across runs](./examples/cli/run/02_comparing_across_runs) to provide more detailed use cases and tips.
|
|
- Added new examples:
|
|
- An [example](./examples/cli/run/07_checking_nan_inf) on how to use the `run` subtool's `--validate` flag to
|
|
check for intermediate NaN/infinity outputs,
|
|
- An [example](./examples/cli/run/08_adding_precision_constraints) on how to selectively constrain
|
|
layer precisions in a model using Polygraphy network scripts,
|
|
- An [example](./examples/cli/convert/04_converting_models_to_fp16) on how to use the `convert` subtool's `--fp-to-fp16` flag
|
|
to convert an ONNX model to fp16.
|
|
- Added an extensibility interface for `polygraphy run` along with a [new example](./examples/dev/02_extending_polygraphy_run/)
|
|
demonstrating how to write an extension module.
|
|
- Added `--load-debug-replay` and `--save-debug-replay` options to various `debug` subtools.
|
|
This allows you to save your progress when debugging and resume from that point later.
|
|
- Added a [how-to guide](./how-to/work_with_reduced_precision.md) on working with reduced precision optimizations using Polygraphy.
|
|
|
|
### Changed
|
|
- The `fold_constant` parameter in `OnnxFromTfGraph` has been deprecated since the corresponding parameter in `tf2onnx` was removed,
|
|
and will be removed in a future release.
|
|
- The `explicit_precision` parameter to `CreateNetwork` has been deprecated and will be removed in a future release.
|
|
- Updated Polygraphy wheels to use `entry_points` instead of `scripts`, which should improve cross-platform support.
|
|
- Updated `debug` subtools to support an interactive mode. When no `--check` command is provided, the tools will now interactively
|
|
prompt you during each iteration to determine whether it passed or failed.
|
|
|
|
### Removed
|
|
- Removed support for Python 3.5 and older.
|
|
|
|
|
|
## v0.37.3 (2022-05-04)
|
|
### Added
|
|
- Added a new example for `inspect capability`.
|
|
|
|
### Changed
|
|
- Updated `polygraphy debug` subtools which use `--check` commands to log the duration of each iteration of the command.
|
|
- Updated the CUDA wrapper to search more paths on Linux for the CUDA runtime library.
|
|
|
|
|
|
## v0.37.2 (2022-04-20)
|
|
### Changed
|
|
- Updated the calibrator to check the data type and shape of data provided by the data loader in cases where input metadata
|
|
is available.
|
|
|
|
|
|
## v0.37.1 (2022-04-18)
|
|
### Added
|
|
- Added support for a top-K axis argument in `--postprocess`.
|
|
- Extended `PostprocessFunc.topk_func()` to support per-output axes.
|
|
|
|
|
|
## v0.37.0 (2022-04-18)
|
|
### Added
|
|
- Added an `precision_constraints` argument to `CreateConfig` and corresponding `--precision-constraints` CLI argument.
|
|
- Added a generic `--postprocess` CLI option which can apply different post-processing functions to different output tensors.
|
|
- Added an experimental `--compare-func-script` option to allow for custom comparison functions in the CLI.
|
|
- Added a new `indices` comparison function and corresponding entry to `--compare-func` to work with outputs including indices,
|
|
e.g. class indices in image classification.
|
|
|
|
### Changed
|
|
- Deprecated `obey_precision_constraints` option in `CreateConfig` and corresponding `--obey-precision-constraints` CLI argument.
|
|
- Made `--obey-precision-constraints`, `--precision-constraints`, and `--strict-types` mutually exclusive options.
|
|
|
|
|
|
## v0.36.4 (2022-04-12)
|
|
### Fixed
|
|
- Fixed `CompareFunc.simple()` so that `NaNs` in the output always result in mismatches.
|
|
|
|
|
|
## v0.36.3 (2022-04-07)
|
|
### Added
|
|
- Added DLA information to build configuration summary.
|
|
- Added warnings when the `--load-inputs` or `--data-loader-script` options are used together with options only applicable to the default data loader.
|
|
|
|
### Changed
|
|
- Updated the included `Calibrator` to no longer require manual activation before use with TensorRT.
|
|
|
|
|
|
## v0.36.2 (2022-03-31)
|
|
### Fixed
|
|
- Fixed a bug in `debug reduce` where models with multiple branches would not be correctly reduced when using custom input data.
|
|
|
|
|
|
## v0.36.1 (2022-03-25)
|
|
### Added
|
|
- Updates `inspect model` to show ONNX graph docstrings when `--show attrs` is set.
|
|
|
|
### Fixed
|
|
- Fixed a bug in the `TrtLegacyRunner` where attempting to use Polygraphy's calibrator could sometimes result in a crash.
|
|
|
|
|
|
## v0.36.0 (2022-02-24)
|
|
### Added
|
|
- Added an `ASK_BEFORE_INSTALL` option to `polygraphy.config` and corresponding environment variable, `POLYGRAPHY_ASK_BEFORE_INSTALL`.
|
|
This option will cause Polygraphy to prompt before automatically installing any packages.
|
|
Only relevant if `POLYGRAPHY_AUTOINSTALL_DEPS` is set.
|
|
- Added `--load-timing-cache`/`--save-timing-cache` options to be more consistent with other options.
|
|
The `--timing-cache` argument will be deprecated in favor of these.
|
|
|
|
### Changed
|
|
- Updated `EngineBytesFromNetwork` to append to existing timing caches instead of overwriting them
|
|
Additionally, the write operation is now atomic so that multiple processes can safely write to the same cache.
|
|
- Updated `CreateConfig` to read from timing caches atomically.
|
|
|
|
### Fixed
|
|
- Fixed a bug in `surgeon insert` where tensors connected to multiple consumers would not be correctly replaced.
|
|
|
|
|
|
## v0.35.2 (2022-02-03)
|
|
### Fixed
|
|
- Fixed a bug in `inspect model` where tensors appearing more than once in a node's inputs or outputs would only be displayed once.
|
|
- Fixed a bug in `inspect model` where ONNX opset would not be displayed correctly for models using multiple opsets.
|
|
- Fixed an edge case where Polygraphy's lazy importer would not ignore modules that are installed but cannot be imported.
|
|
Now, modules that cannot be imported for any reason are treated as though they are not available.
|
|
|
|
|
|
## v0.35.1 (2022-01-14)
|
|
### Added
|
|
- Added a `providers` parameter to `SessionFromOnnx` to specify execution providers for ONNX-Runtime and a
|
|
corresponding `--providers` argument to CLI tools.
|
|
|
|
### Changed
|
|
- `CompareFunc.simple()` will now correctly display the minimum required tolerances when using `elemwise` mode.
|
|
Note that in elementwise comparison mode, each element of the output is compared against both tolerances, and
|
|
only counted as a mismatch if both are exceeded. Hence, the minimum required tolerances apply if only one type
|
|
of tolerance is being used. When both absolute/relative tolerance are set, the requirements may be lower.
|
|
|
|
|
|
## v0.35.0 (2022-01-06)
|
|
### Added
|
|
- Added a `memory_pool_limits` parameter to `CreateConfig`.
|
|
- Added a `--pool-limit`/`--memory-pool-limit` argument to command-line tools.
|
|
|
|
### Changed
|
|
- Changed the default base calibrator class to `IInt8EntropyCalibrator2` since it works across both GPU and DLA.
|
|
To preserve the old behavior, specify `--calibration-base-class=IInt8MinMaxCalibrator` on the command-line
|
|
or specify the `BaseClass` argument in `Calibrator` in the Python API.
|
|
- Deprecated `--workspace` command-line option and `max_workspace_size` parameter in `CreateConfig`.
|
|
Use `--pool-limit` and `memory_pool_limits` respectively instead.
|
|
|
|
### Removed
|
|
- Removed deprecated module `polygraphy.util.serde`. Use `polygraphy.json` instead.
|
|
- Removed `--tactic-replay` command-line option. Use `--load-tactics`/`--save-tactics` instead.
|
|
|
|
|
|
## v0.34.2 (2021-11-29)
|
|
### Added
|
|
- Added support for `MeanVarianceNormalization` to `PluginRefRunner`.
|
|
|
|
|
|
## v0.34.1 (2021-11-24)
|
|
### Added
|
|
- Added a `profiling_verbosity` parameter to `CreateConfig()`.
|
|
- Added support for displaying layer-level engine information in `inspect model` for newer versions of TensorRT.
|
|
|
|
|
|
## v0.34.0 (2021-11-22)
|
|
### Added
|
|
- Added a new `add()` API to `RunResults` to make it easier to create custom output data.
|
|
Added [a new example](./examples/cli/run/06_comparing_with_custom_output_data/) to demonstrate how to use this API.
|
|
|
|
### Changed
|
|
- Deprecated `--mode` option in `inspect model`; a new `--show` option has been introduced
|
|
which can be used to individually control what is displayed.
|
|
- Command-line tools will now use `IInt8EntropyCalibrator2` for calbration if DLA and int8 mode are enabled
|
|
since the default does not work with DLA.
|
|
|
|
### Removed
|
|
- Removed several deprecated submodules of `polygraphy.common`: `constants`, `cuda`, `exception`, `func`.
|
|
These can now be found under the top-level `polygraphy` module instead.
|
|
|
|
|
|
## v0.33.2 (2021-10-21)
|
|
### Changed
|
|
- Improved the help messages of various subtools, including `run`, `debug build`, and `debug reduce`.
|
|
- Added a default value for `--artifacts-dir` in `debug` subtools.
|
|
|
|
### Fixed
|
|
- Fixed a bug in `surgeon insert` where data types of graph output tensors would not be preserved.
|
|
- Fixed broken links in various READMEs.
|
|
|
|
|
|
## v0.33.1 (2021-10-08)
|
|
### Added
|
|
- Added an `OnnxFromBytes` loader that can deserialize ONNX models.
|
|
- Added an `obey_precision_constraints` argument to `CreateConfig` and corresponding `--obey-precision-constraints` CLI argument.
|
|
|
|
### Changed
|
|
- Deprecated `strict_types` option in `CreateConfig` and corresponding `--strict-types` CLI argument.
|
|
|
|
|
|
## v0.33.0 (2021-09-16)
|
|
### Added
|
|
- Added various examples, a [CLI User Guide](polygraphy/tools/) and [directory for how-to guides](./how-to).
|
|
- Added an experimental `template trt-config` tool to generate template scripts that create TensorRT builder configurations.
|
|
- Added `--hide-fail-output` to make `debug` subtools suppress output from failed iterations.
|
|
- Added experimental support for DLA.
|
|
- Added a `data to-input` tool that can combine inputs/outputs created by `--save-inputs`/`--save-outputs`.
|
|
The resulting file is compatible with `--load-inputs`.
|
|
|
|
### Changed
|
|
- Updated `debug` subtools to show captured output on failed iterations.
|
|
- The logger will now emit all `CRITICAL` messages to `stderr` instead of `stdout`.
|
|
- Renamed `CompareFunc.basic_compare_func` to `CompareFunc.simple`. The old name is preserved as an alias.
|
|
- The `--good` and `--bad` arguments in `diff-tactics` can now also accept single files instead of directories.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `debug reduce` would crash when ONNX models included `Constant` nodes whose outputs
|
|
needed to be marked as model outputs.
|
|
|
|
|
|
## v0.32.0 (2021-08-10)
|
|
### Added
|
|
- Added support for `K`, `M`, and `G` suffixes to CLI arguments that expect a number of bytes (e.g. `--workspace`).
|
|
These correspond to `KiB`, `MiB`, and `GiB` respectively.
|
|
For example, `--workspace=16M` is equivalent to `--workspace=16777216`.
|
|
- Added a `copy_outputs_to_host` parameter in `TrtRunner.infer()`, which, when set to `False`, will cause the runner
|
|
to return `DeviceView`s instead of NumPy arrays for inference outputs. This allows us to avoid a
|
|
device-to-host and host-to-device copy if we want outputs to remain on the device.
|
|
- Added a `view()` method to `DeviceArray`s to create read-only `DeviceView`s over their data.
|
|
- Added a `PluginRefRunner` which provides CPU reference implementations for TensorRT plugins
|
|
and a corresponding `--pluginref` runner option in `polygraphy run`.
|
|
|
|
### Changed
|
|
- Marked old shape syntax (`<name>,dim0xdim1x...xdimN,<dtype>`) as deprecated since it leads to ambiguity when
|
|
parsing shapes including named dynamic dimensions.
|
|
|
|
For example, compare:
|
|
```
|
|
--input-shapes input0,xxyxz
|
|
```
|
|
|
|
and:
|
|
```
|
|
--input-shapes input0:[x,y,z]
|
|
```
|
|
|
|
For now, the old syntax continues to work for shapes without named dimensions,
|
|
but it will be removed in a future version of Polygraphy.
|
|
|
|
The newer syntax, which was originally introduced in Polygraphy 0.25.0,
|
|
uses the list syntax already present in other parts of Polygraphy.
|
|
For example, `--val-range [0,1]` in `run` and `--attrs axes=[0,1]` in `surgeon insert` use the same syntax.
|
|
- Made several performance improvements in the Polygraphy CUDA wrapper.
|
|
- Added a loud warning when the deprecated `--int-min`/`--int-max` or `--float-min`/`--float-max` options are used.
|
|
These are superseded by `--val-range` which allows you to specify data ranges on a per-input basis.
|
|
|
|
### Removed
|
|
- Removed various deprecated aliases: `ModifyOnnx`, `SessionFromOnnxBytes`, `ModifyNetwork`, `ModifyGraph`
|
|
- Removed the `to-json` tool which was used to convert Pickled data generated by Polygraphy 0.26.1 and older to JSON.
|
|
Polygraphy 0.27.0 and later only support reading and writing data in JSON format.
|
|
- Removed deprecated legacy submodule `polygraphy.util.misc` which was just an alias for `polygraphy.util`.
|
|
|
|
|
|
## v0.31.1 (2021-07-16)
|
|
### Changed
|
|
- Improved the quality of several examples and added information on how to load serialized TensorRT engines
|
|
as well as how to use custom input data.
|
|
|
|
|
|
## v0.31.0 (2021-07-02)
|
|
### Added
|
|
- Added `inspect capability` subtool that will partition a ONNX graph into supported and unsupported subgraphs for usage within TensorRT.
|
|
- Added Windows support to the CUDA wrapper in `polygraphy/cuda/cuda.py`.
|
|
|
|
### Changed
|
|
- `SaveOnnx` will now create parent directories if they do not already exist.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `ExtractSubgraph` would modify the original graph instead of creating a new graph.
|
|
|
|
|
|
## v0.30.3 (2021-06-25)
|
|
### Fixed
|
|
- Fixed various typos, added more details to some tool READMEs.
|
|
|
|
|
|
## v0.30.2 (2021-06-15)
|
|
### Changed
|
|
- Added `polygraphy.config` as a top-level import so that it no longer needs to be imported separately
|
|
(i.e. `from polygraphy import config`).
|
|
|
|
### Fixed
|
|
- Fixed a bug where `surgeon sanitize` would not re-run shape inference after overriding model input
|
|
shapes, causing constant folding to be sub-optimal.
|
|
|
|
|
|
## v0.30.1 (2021-06-07)
|
|
### Changed
|
|
- CLI tools will no longer print long stack traces on user error.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `surgeon` subtools would not work with ONNX models without an `.onnx` extension.
|
|
- Fixed a bug where `surgeon insert` would not correctly run shape inference if the inserted node replaced
|
|
the graph outputs.
|
|
- Fixed a bug where `POLYGRAPHY_AUTOINSTALL_DEPS` would not work correctly for nested modules,
|
|
e.g. `mod.lazy_import("onnx.shape_inference")`.
|
|
|
|
|
|
## v0.30.0 (2021-05-26)
|
|
### Added
|
|
- Added an `--ignore-fail-code` option to `debug` subtools to ignore certain types of failures.
|
|
- Added a highly experimental `OnnxLikeFromNetwork` loader that can generate a file using the ONNX
|
|
format based on a TensorRT network. The resulting model is **not** valid ONNX, but is useful for visualization.
|
|
- Added a `onnx-like-trt-network` type in `convert` to generate ONNX-like models from TensorRT networks.
|
|
- Added support for custom installation commands during dependency autoinstall.
|
|
This can be configured using `config.INSTALL_CMD` or by setting the `POLYGRAPHY_INSTALL_CMD` environment variable.
|
|
- Added support for loading external data in `InferShapes`.
|
|
- Added a `--no-per-pass-shape-inference` argument to `surgeon sanitize` to disable shape inference
|
|
between constant-folding passes.
|
|
- Added a `--external-data-size-threshold` CLI option for saving external data for ONNX models.
|
|
- Added a `--no-save-all-tensors-to-one-file` CLI option to avoid saving ONNX external data to a single file.
|
|
|
|
### Changed
|
|
- Improved logic for auto-permuting tensors in `basic_compare_func`. The new logic can handle
|
|
an arbitrary number of dimensions. For example, if two tensors with shapes `(1, 3, 45, 45, 45)`
|
|
and `(1, 45, 45, 45, 3)` are being compared, `basic_compare_func` will now guess that the latter
|
|
should be transposed using a permutation of `(0, 4, 1, 2, 3)` to match the former.
|
|
- Improved display of `Profile` in logging messages.
|
|
- Updated NumPy array encoding to use `base64`. In some cases, this can reduce file sizes by a factor of 4.
|
|
- Updated `debug precision` default direction to `forward` as this typically leads to better results.
|
|
- Added a `--no-strict-types` flag to `debug precision` in case strict types needs to be disabled for any reason.
|
|
- `FoldConstants` will no longer run shape inference if shape folding is disabled.
|
|
- `InferShapes` will now automatically write large models to the disk to work around the 2 GiB protobuf size limitation.
|
|
The threshold can be configured using the `save_to_disk_threshold_bytes` parameter.
|
|
|
|
### Fixed
|
|
- Fixed a bug in `inspect model` where engine output bindings would all be printed on one line.
|
|
- Fixed a bug where using `set_profile` in the `TrtRunner` would sometimes cause input shape
|
|
checks in `infer` to fail even when shapes were valid.
|
|
- Fixed a bug in `inspect model` where engine output bindings would display the wrong shapes
|
|
for profiles after the first.
|
|
- Fixed a bug where `debug precision` would incorrectly mark constant layer outputs and non-execution tensors
|
|
to run in higher precision.
|
|
- Fixed a bug where `debug precision` would crash if engine building failed. It now continues to the next iteration,
|
|
counting the previous one as a failure.
|
|
- Fixed a bug where `InferShapes` would require `--external-data-dir` to be set even if the external data
|
|
were in the same directory as the model.
|
|
- Fixed a bug where `--data-loader-script` would not provide data in the `run` tool if int8 calibration were enabled in TensorRT.
|
|
|
|
|
|
## v0.29.2 (2021-04-30)
|
|
### Added
|
|
- Added a `--log-file` option to CLI tools to store logging output to a file.
|
|
- Added an `--iteration-info` argument to `debug` subtools so that `--check` commands can get information
|
|
about the current iteration.
|
|
- Added an experimental `debug repeat` subtool, which is more generic than the existing `debug` subtools.
|
|
|
|
|
|
## v0.29.1 (2021-04-28)
|
|
### Changed
|
|
- Swapping NumPy arrays to the disk is now disabled by default. It can be re-enabled by setting the
|
|
`POLYGRAPHY_ARRAY_SWAP_THRESHOLD_MB` environment variable.
|
|
|
|
|
|
## v0.29.0 (2021-04-28)
|
|
### Added
|
|
- Added support for per-output `check_error_stat`, which allows different metrics to be checked for different outputs.
|
|
|
|
### Changed
|
|
- Moved JSON utilities into a separate `polygraphy.json` submodule. For backwards compatibility, they remain accessible
|
|
via `polygraphy.util` as well.
|
|
- The `max` value for `check_error_stat` in `basic_compare_func` now only checks the maximum absolute/relative tolerances.
|
|
The previous behavior of checking the values element-wise is preserved in the `elemwise` option, which is now the default.
|
|
|
|
### Fixed
|
|
- Fixed a bug where data loader would not cast value ranges provided for `bool` input types, which could lead
|
|
to generating out-of-bound values.
|
|
|
|
|
|
## v0.28.7 (2021-04-26)
|
|
### Fixed
|
|
- Fixed a bug where NumPy arrays smaller than 8 MiB would be serialized to disk unnecessarily.
|
|
|
|
|
|
## v0.28.6 (2021-04-23)
|
|
### Added
|
|
- Added a `check_error_stat` option in `basic_compare_func` and corresponding `--check-error-stat` CLI option to control
|
|
which statistic (e.g. mean, median, max) of the error is used to determine whether outputs matched.
|
|
|
|
### Changed
|
|
- A histogram of output/error values will now be displayed at INFO severity on comparison failures.
|
|
Otherwise, it is displayed at VERBOSE severity.
|
|
|
|
### Fixed
|
|
- Fixed a bug where histogram display would wrap to subsequent lines.
|
|
|
|
|
|
## v0.28.5 (2021-04-23)
|
|
### Added
|
|
- Added more information about absolute/relative difference in `basic_compare_func`. For example,
|
|
it will now print a histogram of the distribution of the outputs and errors.
|
|
|
|
|
|
## v0.28.4 (2021-04-23)
|
|
### Added
|
|
- Added mean absolute/relative error to `OutputCompareResult`, which is returned by `Comparator.compare_accuracy`.
|
|
This makes it easier to programmatically access this information.
|
|
|
|
### Changed
|
|
- Several improvements to the quality of error messages and warnings.
|
|
|
|
|
|
## v0.28.3 (2021-04-22)
|
|
### Fixed
|
|
- Fixed a bug where `basic_compare_func` and `DataLoader` would issue warnings when default
|
|
tolerances/value ranges were used.
|
|
|
|
|
|
## v0.28.2 (2021-04-22)
|
|
### Fixed
|
|
- Fixed a bug where command-line tools would fail if a `--timing-cache` argument was provided
|
|
but the file did not exist.
|
|
|
|
|
|
## v0.28.1 (2021-04-22)
|
|
### Fixed
|
|
- `basic_compare_func` will now issue warnings if `atol`/`rtol` contain invalid keys.
|
|
- `DataLoader` will now issue warnings if `val_range` contains invalid keys.
|
|
|
|
|
|
## v0.28.0 (2021-04-20)
|
|
### Added
|
|
- Added a `tactic_sources` parameter in `CreateConfig` to control TensorRT's tactic sources.
|
|
- Added a `--tactic-sources` argument to CLI tools.
|
|
- Added a `DeviceView` class in the `cuda` submodule to represent views of GPU memory.
|
|
`DeviceArray` is now a subclass of `DeviceView`.
|
|
- Added support for accepting `DeviceView`s or device pointers in the `Calibrator`. This means that you can now
|
|
run calibration using data already on the GPU.
|
|
- Added support for `DeviceView`s in `TrtRunner.infer()`. Note that `DeviceView`s cannot be used for input
|
|
shape-tensors, which must be allocated on the host.
|
|
- Added support for using `trt.IInt8Calibrator` as the `BaseClass` of `Calibrator`.
|
|
- Exposed some lower level functions like `malloc`, `free`, and `memcpy` in the Polygraphy CUDA wrapper.
|
|
- Added a `set_profile()` method to `TrtRunner` to control the active optimization profile.
|
|
- Added `-q`/`--quiet` option to CLI tools. This can be used to suppress logging output without eliminating
|
|
all output like `--silent` does.
|
|
- Added a `to_trt()` method to `Profile` to convert it to a TensorRT `IOptimizationProfile`.
|
|
- Added `--force-fallback-shape-inference` option to `debug reduce`.
|
|
- Added `--fail-regex` option to `debug reduce` to distinguish among different types of falures based on command output.
|
|
|
|
### Changed
|
|
- Changed `TRT_LOGGER` to `get_trt_logger()` to make it work properly with lazy imports.
|
|
- Further improved lazy imports such that no modules are required in order to import Polygraphy modules.
|
|
Using functionality from Polygraphy modules still requires dependencies.
|
|
- Various submodules have been restructured. The old import structure is preserved for backwards compatibility.
|
|
- Added `Profile.fill_defaults()` which makes it possible to automatically fill a TensorRT optimization profile
|
|
with sane default values.
|
|
- It is now possible to provide TensorRT optimization profile shapes for a subset of the network inputs. In such
|
|
cases, the rest of the profile will be populated automatically with `Profile.fill_defaults()`
|
|
- `surgeon extract` will no longer run shape inference unless it is required, e.g. if `auto` is specified for one of
|
|
the shape/data type arguments.
|
|
- ONNX shape inference will now be skipped when `--force-fallback-shape-inference` is enabled in `surgeon extract/sanitize`.
|
|
- `debug reduce` will now freeze intermediate shapes in the model if `--model-input-shapes` is provided.
|
|
- `IterationResult`s now store `LazyArray` rather than `np.ndarray`.
|
|
The public interface for `IterationResult` will automatically pack or unpack `np.ndarray`s into/from `LazyArray`,
|
|
so the change is completely transparent.
|
|
This can significantly reduce memory usage for tools like `debug reduce` and `run`.
|
|
|
|
### Fixed
|
|
- Attempting to load a non-existent file will now cause a friendly error message to be displayed rather than crashing.
|
|
- `surgeon sanitize` will no longer override shapes other than those specified in `--override-input-shapes`.
|
|
|
|
### Removed
|
|
- Removed optional `symbol` parameter from `lazy_import`.
|
|
|
|
|
|
## v0.27.0 (2021-04-06)
|
|
### Changed
|
|
- For security reasons, all serialization/deserialization code in Polygraphy has been updated to use JSON
|
|
instead of `pickle`. Use the included `to-json` tool to convert data serialized with `pickle` to JSON
|
|
format.
|
|
- Split `TacticReplayer` into separate `TacticRecorder` and `TacticReplayer` classes. This provides more
|
|
fine-grained control over whether to record or replay tactics.
|
|
- Deprecated `--tactic-replay` in favor of `--save-tactics` and `--load-tactics`.
|
|
- Changed `check_finite` parameter in `Comparator.validate()` to `check_inf`, since it checks whether
|
|
values are non-finite rather than the opposite.
|
|
|
|
### Fixed
|
|
- Polygraphy will now validate command-line arguments so that code-injection is not possible.
|
|
- `debug diff-tactics` will now work correctly when replay files are in nested directories.
|
|
|
|
|
|
## v0.26.1 (2021-04-01)
|
|
### Added
|
|
- Added `--force-fallback-shape-inference` option to `surgeon sanitize` in case ONNX shape inference doesn't work
|
|
well enough to allow for folding.
|
|
- Added a `--calibration-base-class` option to allow changing base class for the TensorRT int8 calibrator.
|
|
|
|
### Fixed
|
|
- `FoldConstants` will no longer fail if a constant folding pass fails. Set `error_ok=False` to disable this behavior.
|
|
|
|
|
|
## v0.26.0 (2021-03-30)
|
|
### Added
|
|
- Added support for saving/loading ONNX models with externally stored weights.
|
|
- Added support for automatically installing dependencies as they are needed. This behavior can be enabled by
|
|
setting the `POLYGRAPHY_AUTOINSTALL_DEPS` environment variable to `1`.
|
|
When auto-install is enabled, Polygraphy can also automatically upgrade existing packages if a newer version is requested.
|
|
- Added `error_ok` option in `InferShapes`, which can be set to `False` to make the loader raise an error when shape
|
|
inference fails.
|
|
|
|
### Changed
|
|
- `val_range` in the DataLoader now falls back to the default range if no range is specified for an input.
|
|
- `atol` and `rtol` in `CompareFunc.basic_compare_func` now fall back to the default tolerance values
|
|
if no tolerance is specified for an output.
|
|
- Folding shapes is now optional in `FoldConstants`.
|
|
- `surgeon sanitize` now includes a `--no-fold-shapes` option to disable shape folding.
|
|
|
|
### Fixed
|
|
- Fixed a bug in `surgeon insert` where input tensors would be disconnected from all their consumers.
|
|
Previously, in a model with branches, if one entire branch was replaced by `surgeon insert`, the other branch
|
|
would be invalidated. This is no longer the case.
|
|
- `run` will now attempt to avoid introducing a dependency on the `onnx` Python package when using an ONNX
|
|
model when `--trt` is the only specified runner.
|
|
- When `--force-fallback-shape-inference` is set in `surgeon extract`, it will now correctly ignore shapes
|
|
already inferred in the model.
|
|
- ONNX loaders will no longer make a copy of the model unnecessarily. If a copy is desired, the `copy` parameter can
|
|
be set to `True` for loaders that may modify the model.
|
|
- `InferShapes`/`infer_shapes` will now work with ONNX models larger than 2 GiB if a path to the model is provided
|
|
instead of an `onnx.ModelProto`
|
|
- Fixed a bug where `FoldConstants` would not count nodes within subgraphs.
|
|
|
|
### Removed
|
|
- Removed `OnnxTfRunner` and associated CLI options.
|
|
|
|
|
|
## v0.25.1 (2021-03-15)
|
|
### Added
|
|
- Added `--partitioning` flag to `surgeon sanitize` to control how ONNX-GraphSurgeon partitions the graph during
|
|
constant folding.
|
|
- Added `--cleanup` flag to `surgeon sanitize` to remove dead layers in ONNX models.
|
|
|
|
### Changed
|
|
- `ExtractSubgraph` loader will now fallback to using shapes/dtypes defined in the model when none are specified.
|
|
|
|
### Fixed
|
|
- `surgeon sanitize` no longer runs inference when the `--override-input-shapes` option is set. Instead, intermediate shapes are cleared.
|
|
- `surgeon extract` will no longer override shapes or data types already set in the model when running fallback shape inference.
|
|
|
|
|
|
## v0.25.0 (2021-03-01)
|
|
### Added
|
|
- Added support for list attributes in `surgeon insert`.
|
|
- Added `val_range` parameter to data loader, which is more generic than `int_range`/`float_range`, which are now deprecated.
|
|
- Added support for per-input data ranges to `val_range` parameter.
|
|
- Added `--val-range` CLI option to set input ranges on the command-line.
|
|
- Added `:` as a valid separator for various options and `[dim0,...,dimN]` as valid syntax for shapes.
|
|
For example, you can now optionally use:
|
|
```bash
|
|
--inputs input0:[3,4]:int64 input1:[4,64,64]:float32
|
|
```
|
|
instead of:
|
|
```bash
|
|
--inputs input0,3x4,int64 input1,4x64x64,float32
|
|
```
|
|
The new and old styles cannot be mixed.
|
|
- Added support for specifying per-output top-k values to CLI tools.
|
|
- Added `--trt-config-script` argument, which allows CLI tools to accept scripts that define functions that create TensorRT
|
|
builder configurations.
|
|
- Added `--data-loader-script` argument, which allows CLI tools to accept scripts that define data loaders.
|
|
- Added a new example for the `convert` CLI tool, which shows how to use a custom data loader for int8 calibration on the command-line.
|
|
|
|
### Fixed
|
|
- Fixed a bug where `debug reduce` would remove branches even if they were required to reproduce failures.
|
|
|
|
|
|
## v0.24.2 (2021-02-25)
|
|
### Added
|
|
- Added support for string input types in `OnnxrtRunner`.
|
|
- Added `reduce` subtool to `debug` which can reduce failing ONNX models to the smallest possible failing subgraph.
|
|
|
|
### Fixed
|
|
- ONNX loaders will no longer modify the original model provided, but instead make a shallow copy.
|
|
|
|
|
|
## v0.24.1 (2021-02-22)
|
|
### Added
|
|
- Added an example to `dev/` showing how to write new command-line tools.
|
|
|
|
### Fixed
|
|
- Verbose TensorRT network logging will no longer fail to show attributes for layers on older versions of TensorRT.
|
|
|
|
|
|
## v0.24.0 (2021-02-19)
|
|
### Added
|
|
- `convert` can now automatically determine the output model type based on the file extension.
|
|
- Added immediately evaluated functional variants for all loaders exported by Polygraphy.
|
|
The variants use the same name as the loaders, except `snake_case` instead of `PascalCase`.
|
|
See [the example](./examples/api/06_immediate_eval_api) for details.
|
|
|
|
### Changed
|
|
- Polygraphy no longer has `numpy` as an install requirement.
|
|
Note however that most, but not all, APIs and CLI tools in Polygraphy still do require `numpy`.
|
|
|
|
### Removed
|
|
- Removed `func.invoke()` since immediately evaluated functions now supersede it.
|
|
|
|
### Fixed
|
|
- Fixed a bug where some `debug` subtools would write engines to the wrong path.
|
|
|
|
|
|
## v0.23.4 (2021-02-15)
|
|
### Added
|
|
- Added `FoldConstants` loader for ONNX models.
|
|
- Added `ExtractSubgraph` loader for ONNX models.
|
|
|
|
### Changed
|
|
- Moved `--fp-to-fp16` option to `convert`.
|
|
|
|
|
|
## v0.23.3 (2021-02-13)
|
|
### Added
|
|
- Added `ConvertToFp16` as a separate loader for ONNX models.
|
|
- Added `InferShapes` loader for ONNX models.
|
|
|
|
### Changed
|
|
- `surgeon sanitize` will now run shape inference by default.
|
|
- `Modify<X>` loaders have been renamed to `Modify<X>Outputs` to better reflect their purpose.
|
|
- `surgeon sanitize` can now run multiple passes of constant folding to deal with nodes that may not be folded
|
|
after the first pass (for example, `Shape` nodes in cases where ONNX shape inference does not complete).
|
|
|
|
|
|
## v0.23.2 (2021-02-11)
|
|
### Added
|
|
- Added an experimental `debug` subtool, which includes `build` and `diff-tactics` (formerly part of `flaky`)
|
|
and `precision` (formerly a separate tool).
|
|
|
|
### Changed
|
|
- `flaky diff-tactics` will now only show layers that have potentially bad tactics.
|
|
To view an entire tactic replay, use `inspect tactics`
|
|
- `flaky repeat` will now only log command `stderr` output with `ERROR` severity if the command failed.
|
|
Otherwise, `stderr` output is logged with `WARNING` severity.
|
|
- `TacticReplayer` can now accept a `TacticReplayData` instance directly.
|
|
- `TacticReplayData` can now be constructed manually instead of relying on TensorRT types.
|
|
|
|
### Removed
|
|
- `flaky` and `precision` tools have been removed and replaced by the `debug` subtool, which includes the
|
|
functionality of both.
|
|
|
|
|
|
## v0.23.1 (2021-02-05)
|
|
### Added
|
|
- Added a `POLYGRAPHY_INTERNAL_CORRECTNESS_CHECKS` environment variable to enable internal correctness checks at runtime.
|
|
By default, these checks are disabled. A failure in such a check typically indicates a bug in Polygraphy.
|
|
- Added context managers for CUDA helper classes. This helps ensure they are correctly freed.
|
|
- Added `sparse_weights` parameter to `CreateConfig`, which enables TensorRT optimizations related to sparse weights.
|
|
- Added a `--sparse-weights` option to various CLI tools.
|
|
|
|
### Fixed
|
|
- Added checks for cases where paths provided to `BytesFromPath` did not exist.
|
|
|
|
|
|
## v0.23.0 (2021-02-02)
|
|
### Added
|
|
- Added `__enter__`/`__exit__` to `Calibrator` so that device buffers can be reliably freed after calibration using a context manager.
|
|
- Added `fp_to_fp16` parameter to `ModifyOutputs` which will use `onnxmltools` to convert float tensors in the model to 16-bit floats.
|
|
- Added `--fp-to-fp16` CLI argument to various tools.
|
|
- Added support for `float`, `int`, and `str` attributes to `surgeon insert`.
|
|
- Added `InvokeFromScript` loader, which can import and invoke a function from a Python script.
|
|
- Added support for loading TensorRT networks from Python scripts to various CLI tools.
|
|
CLI tools can now accept a Python script in place of a model file.
|
|
The script should define a `load_network` function that takes no arguments and returns a TensorRT builder, network, and optionally parser.
|
|
See [the example](examples/cli/run/04_defining_a_tensorrt_network_or_config_manually/) for details.
|
|
- Added an experimental `template` tool that can generate template files.
|
|
- Added a `trt-network` subtool that can generate a template script for defining TensorRT networks using the network API.
|
|
- Added a `SaveBytes` loader to facilitate writing bytes to a file between loaders.
|
|
- Added an experimental `flaky` tool that can help debug flaky failures.
|
|
- Added `repeat` subtool, which will run a command repeatedly and sort artifacts into `good` and `bad` directories.
|
|
- Added `diff-tactics` subtool, which compares known-good and known-bad tactic replay files to determine which tactics may be the source of error.
|
|
|
|
### Changed
|
|
- `EngineFromNetwork` and `CreateConfig` no longer use the global timing cache by default.
|
|
- Changed `--timing-cache` default in CLI tools to `None`.
|
|
- Changed `timing_cache` parameter to `load_timing_cache` and `save_timing_cache` in `CreateConfig` and `EngineFromNetwork` respectively.
|
|
- Runners will now raise errors in `infer` if the provided input data types or shapes do not match expected types and shapes.
|
|
This behavior can be disabled by setting `check_inputs=False`.
|
|
- Changed `--toposort` default to off in `surgeon` tools as ONNX models are typically topologically sorted.
|
|
- The logger will now log messages with `WARNING` or greater severity to `sys.stderr` instead of `sys.stdout`
|
|
|
|
### Removed
|
|
- Removed `CNTKRunner` and `--cntk` CLI option.
|
|
- Removed experimental `--network-api` flag in CLI tools. This is superseded by the `template trt-network` subtool.
|
|
|
|
### Fixed
|
|
- Fixed memory leaks in `EngineFromNetwork`, `EngineFromBytes`, and `TrtRunner`.
|
|
|
|
|
|
## v0.22.0 (2021-01-20)
|
|
|
|
### Added
|
|
- Added support for timing caches in `EngineFromNetwork` and `CreateConfig`.
|
|
The former can generate caches, while the latter can load them, resulting in much faster engine builds.
|
|
By default, Polygraphy will use a global timing cache in the temporary directory.
|
|
- Added a `--timing-cache` option to various CLI tools.
|
|
- Added an `EngineBytesFromNetwork` TensorRT loader to provide serialized engines directly.
|
|
- Added a `BytesFromEngine` TensorRT loader to provide a means of in-memory engine serialization.
|
|
- Added an experimental `convert` subtool, which can convert models to various other formats.
|
|
- Added an `algorithm_selector` parameter to `CreateConfig` to allow the user to override TensorRT's tactic choices.
|
|
- Added a `TacticReplayer` algorithm selector to allow for recording and replaying tactics in the TensorRT builder.
|
|
This makes it possible to make the TensorRT builder behave deterministically.
|
|
- Added an experimental `--tactic-replay` option to various CLI tools to make it possible to record to and replay from tactic replay files.
|
|
- Added an experimental `inspect` subtool, `tactics` which can display tactic replay files in a human readable format.
|
|
|
|
### Changed
|
|
- The `surgeon sanitize` subtool can now also modify model outputs.
|
|
- `surgeon insert` will now preserve graph input and output names.
|
|
|
|
### Fixed
|
|
- Fixed a bug where the CUDA wrapper could not allocate buffers larger than 3GiB.
|
|
|
|
|
|
## v0.21.1 (2021-01-12)
|
|
|
|
### Added
|
|
- `TrtRunner` can now optionally accept a `context` directly instead of an `engine`.
|
|
- `basic_compare_func` will now show mismatched indices in addition to mismatched values.
|
|
|
|
|
|
## v0.21.0 (2020-11-30)
|
|
### Added
|
|
- Added an experimental `surgeon` subtool, `insert`, which can insert new nodes into an ONNX model.
|
|
- Added an experimental `surgeon` subtool, `sanitize`, which can remove unused nodes and fold constants in an ONNX model.
|
|
- Added `--load-inputs` and `--save-inputs` to provide a mechanism to supply custom input data on the command line.
|
|
- Added `func.invoke()`, a function that calls a provided callable.
|
|
This can be useful to make it more obvious that a loader is being immediately evaluated.
|
|
For example: `EngineFromNetwork(...)()` vs. `func.invoke(EngineFromNetwork(...))`
|
|
- Added per-output tolerance support in `basic_compare_func`.
|
|
- Added per-output tolerance support to the `--atol` and `--rtol` command-line options.
|
|
|
|
### Changed
|
|
- Renamed `inspect results` to `inspect data` since it can now also be used to inspect input data, not just results.
|
|
- `Comparator.compare_accuracy` now supports comparing a single runner against itself.
|
|
|
|
### Removed
|
|
- Removed experimental surgeon subtool `prepare` and `operate` as they were difficult to maintain and not very useful.
|
|
|
|
### Fixed
|
|
- Fixed a memory leak due to `IBuilderConfig` not being properly freed in the `EngineFromNetwork` loader.
|
|
- Fixed memory leaks on exceptions in TensorRT loaders.
|
|
- Fixed a bug in `inspect model` where `dim_param`s in ONNX models would show up as `-1`.
|
|
|
|
|
|
## v0.20.13 (2020-10-08)
|
|
### Added
|
|
- Shape values in `TensorMetadata` can now be strings to indicate dynamic dimensions.
|
|
- `TRT_LOGGER` is now exported under `polygraphy.backend.trt`
|
|
|
|
### Fixed
|
|
- Fixed a bug in `surgeon extract` where ONNX models using `dim_param` would be rejected.
|
|
|
|
|
|
## v0.20.12 (2020-10-01)
|
|
### Added
|
|
- Added missing copyright headers
|
|
|
|
|
|
## v0.20.11 (2020-09-25)
|
|
### Added
|
|
- Added an `--input-shapes` alias for the `--inputs` option in `run` to better reflect its purpose.
|
|
|
|
### Fixed
|
|
- `inspect model` will no longer show `dtype`/`shape` as `None` if the information is not present in the model. Instead, these are now omitted.
|
|
|
|
|
|
## v0.20.10 (2020-09-23)
|
|
### Fixed
|
|
- Fixed a bug where boolean outputs would cause a crash in `basic_compare_func`
|
|
|
|
|
|
## v0.20.9 (2020-09-22)
|
|
### Fixed
|
|
- Fixed a bug where `TrtRunner` would use the wrong shapes for empty tensor outputs .
|
|
|
|
|
|
## v0.20.8 (2020-09-22)
|
|
### Fixed
|
|
- Fixed a bug where the `Calibrator` would not re-check the cache when `reset()`
|
|
|
|
|
|
## v0.20.7 (2020-09-22)
|
|
### Added
|
|
- Added `-v`/`--version` flag to `polygraphy`
|
|
|
|
### Changed
|
|
- Cleaned up unnecessary logging output, and fixed formatting.
|
|
|
|
|
|
## v0.20.6 (2020-09-18)
|
|
### Added
|
|
- Added new modes to `inspect model`, to control whether to show weights in the model.
|
|
- Added `-s`/`--show-values` option to `inspect results` to display output values.
|
|
- Added an experimental `--top-k` flag to `run`, which will apply a Top-K before comparing outputs.
|
|
- Added `exclude_outputs` to `ModifyOutputs` and `ModifyNetworkOutputs`
|
|
- Added an experimental `--onnx-exclude-outputs` and `--trt-exclude-outputs` to selectively unmark outputs.
|
|
|
|
|
|
## v0.20.5 (2020-09-16)
|
|
### Fixed
|
|
- Fixed a bug in `inspect model` for ONNX models containing nodes with Tensor attributes.
|
|
- Fixed a bug where `DeviceArray.copy_from` would segfault in rare cases.
|
|
|
|
|
|
## v0.20.4 (2020-09-14)
|
|
### Fixed
|
|
- General cleanup and addition of missing docstrings.
|
|
|
|
|
|
## v0.20.3 (2020-09-11)
|
|
### Fixed
|
|
- Fixed a bug where `DataLoader` would use a shape provided by the user even for static shapes in the model.
|
|
- Fixed a bug where `DataLoader` would incorrectly report certain tensors as shape tensors.
|
|
- Fixed a bug where the `DataLoaderCache` would stop checking the cache after the first miss.
|
|
|
|
|
|
## v0.20.2 (2020-09-11)
|
|
### Added
|
|
- Added an `extend` decorator, which makes it easier to extend existing loaders.
|
|
- Added more API examples.
|
|
- `Comparator.compare_accuracy` will now display an accuracy summary after processing all iterations.
|
|
- Added a `CreateNetwork` loader to create new TensorRT networks
|
|
- Added experimental `--network-api` option that works with `--gen` to allow manually defining a TensorRT network.
|
|
|
|
### Changed
|
|
- `Calibrator` can now accept a file-like object for `cache` instead of just a file path.
|
|
|
|
### Fixed
|
|
- Fixed various errors in API documentation.
|
|
- `EngineFromBytes` will now call `trt.init_libnvinfer_plugins` before attempting to deserialize the engine.
|
|
|
|
|
|
## v0.20.1 (2020-09-09)
|
|
### Added
|
|
- Added HTML docs for the Python API
|
|
|
|
### Fixed
|
|
- Fixed a bug where the data loader would not support cases where `int_min` == `int_max` when bounding input data
|
|
- Fixed a bug where OnnxrtRunner would report incorrect metadata for ONNX models using `dim_param` for dynamic dimensions.
|
|
|
|
|
|
## v0.20.0 (2020-09-08)
|
|
### Added
|
|
- `CreateConfig` now accepts a `strict_types` argument.
|
|
- Added a new `polygraphy` binary, which includes several tools
|
|
- Added an experimental new tool: `precision`, which can be used to figure out what layers to run in higher precision in TensorRT to achieve the desired accuracy.
|
|
- Added `bisect` subtool that does binary search
|
|
- Added `linear` subtool that does a linear search
|
|
- Added `worst-first` subtool that marks the layers that introduce the most error first.
|
|
- Added a new tool: `inspect` to inspect supported files
|
|
- Added `model` which displays information about models.
|
|
- Added `results` which displays information about saved `RunResults`
|
|
- Added back `subprocess_polling_interval` to `Comparator.run()`, as this is still required in certain rare cases.
|
|
- Optimization passes are now optional in `OnnxFromTfGraph`, and can be disabled by setting `optimize=False` in the constructor.
|
|
- Runners now include an `is_active` property, which indicates whether the runner is currently activated.
|
|
- Added an experimental new tool: `surgeon`, which can be used to modify ONNX models more easily than using ONNX-GraphSurgeon.
|
|
- Added `prepare` and `operate` which can be used to modify an ONNX model using a JSON configuration.
|
|
- Added `extract` which can extract ONNX subgraphs with a single command.
|
|
- Added `--onnx-outputs` and `--trt-outputs` to set outputs in the corresponding loaders
|
|
- Added a passthrough loader, `LoadPlugins`, that can wrap any other loader, and load plugins
|
|
|
|
### Changed
|
|
- `EngineFromNetwork` will no longer free the builder, network and parser if they are provided directly (as opposed to via a loader).
|
|
- `TrtRunner` will no longer free the engine if it is provided directly (as opposed to via a loader).
|
|
- All file saving arguments now take file paths instead of directories. This makes it easier to know exactly where each file is being written.
|
|
- `compare_func` in `Comparator.compare_accuracy` now accepts a function that returns anything convertible to a boolean, rather than requiring a boolean.
|
|
- `basic_compare_func` now will return information about required tolerances after `Comparator.compare_accuracy`.
|
|
- `Calibrator` can now be configured to inherit from a different TensorRT calibrator base class.
|
|
- ONNX GraphSurgeon is no longer required to mark outputs in ONNX models.
|
|
- `TrtLegacyRunner` no longer depends on `pycuda`
|
|
- `TrtRunner` will now only reset context shapes if the shapes changed. This should improve performance.
|
|
- `DataLoader` now takes `int_range` and `float_range` parameters, so min/max can be provided more concisely.
|
|
- All `Loaders` and `Runner` were renamed to better reflect their purpose, and to improve readability.
|
|
- Renamed `warm_up_runs` to `warm_up`.
|
|
- `Calibrator`'s `data_loader` parameter now accepts any generator or iterable instead of requiring a special type.
|
|
- `Comparator.run`'s `data_loader` parameter now accepts any generator or iterable instead of requiring a special type.
|
|
- The included `DataLoader` can now be used as an iterable, and its iteration length can be controlled via the `iterations` parameter.
|
|
- Renamed `--input-shape` to `--inputs`
|
|
- Renamed `--min-shape`/`--opt-shape`/`--max-shape` to `--trt-min-shapes`/`--trt-opt-shapes`/`--trt-max-shapes`
|
|
- `DataLoader` now accepts an `input_metadata` parameter which can be used to override shapes and data types.
|
|
- Split off `layerwise` and `outputs` functionality into separate `Modify` loaders.
|
|
- Split off artifact saving functionality into separate `Save` loaders.
|
|
- Renamed `--read` options to `--load`, and `--write` to `--save`
|
|
- Renamed `--read-outputs`/`--write-outputs` to `--load-results`/`--save-results`
|
|
- `Calibrator` no longer requires `input_metadata` to be set if the data loader does not need it
|
|
- `TfRunner` now uses a `CreateConfig` loader to supply configuration.
|
|
- `TfRunner` and `OnnxrtRunner` now take a `BuildSession`, so that custom sessions can be used.
|
|
|
|
### Removed
|
|
- Removed iteration arguments from `Comparator.run()` and `Calibrator`. Instead these now iterate the provided data loader until it runs out of data.
|
|
- Removed `--load-engine` option from `polygraphy`. Engines can now be provided as models directly, e.g. `polygraphy run example.engine --trt`
|
|
- `polygraphy_exec` and `polygraphy_gen` were removed. They are superseded by the `run` subtool of `polygraphy`.
|
|
- `--layerwise` and `layerwise` options have been removed. Layerwise behavior is now possible with `outputs=constants.MARK_ALL` or `--<framework>-outputs mark all`
|
|
|
|
### Fixed
|
|
- Fixed bugs in `Comparator.validate` that would cause it not to correctly display non-finite values.
|
|
- `Calibrator` will now warn if a cache exists but is empty
|
|
- `DataLoader` will now used a fixed seed value unless otherwise specified. This ensures consistent run-to-run behavior.
|
|
- The default `find_output_func` will no longer compare outputs whose names don't match if there is another output that does match.
|
|
- Fixed a bug where custom names provided to runners would still be suffixed with a timestamp.
|
|
- Fixed a bug where regular TensorRT calibrators could not be used with `CreateConfig`
|
|
- The missing subtool warning will no longer be displayed if that subtool is not being used.
|
|
|
|
|
|
## v0.17.0 (2020-07-20)
|
|
### Added
|
|
- `basic_compare_func` now accepts a `find_output_func` parameter, allowing users to control which outputs are compared between results.
|
|
- The `--load-outputs` argument can now accept multiple different files. Outputs from each of these will be read in order.
|
|
- Added an implicit batch ONNX network loader for the legacy TensorRT runner. This will not work with recent versions of the parser.
|
|
- Added `RunResults` class which replaces the `OrderedDict` that `Comparator.run` previously returned (structure is unchanged).
|
|
|
|
### Changed
|
|
- `layerwise` mode will no longer mark constants as outputs.
|
|
- The default `compare_func` in `Comparator.compare_accuracy` will now always iterate over the output names in the first `IterationResult` and attempt to find them in the second. The order of the `IterationResult`s provided to this function can be modified either by setting `comparisons` in `Comparator.compare_accuracy`, or changing the order of runners in `Comparator.run`
|
|
- Improves `polygraphy_gen` output formatting
|
|
- Renamed `RunResult` to `IterationResult` to better reflect its purpose.
|
|
- Default runner names now include timestamps to disambiguate when saving and loading multiple runners.
|
|
- `graphsurgeon` is no longer a dependency of Polygraphy
|
|
|
|
### Fixed
|
|
- Logger settings in `polygraphy_exec`/`polygraphy_gen` are now set prior to any logging output.
|
|
- Comparator will no longer attempt to decompress all `bytes` objects sent over the queue when using subprocesses
|
|
|
|
|
|
## v0.16.0 (2020-06-11)
|
|
### Added
|
|
- Added `OnnxExtWeightsNetworkLoader` to support loading ONNX models with externally stored weights into TensorRT.
|
|
- Added a `TensorMetadata` class to replace dictionaries that were used across Polygraphy.
|
|
- Added `CaffeNetworkLoader` for the `TrtLegacyRunner`
|
|
|
|
### Changed
|
|
- `polygraphy_exec` and `polygraphy_gen` will no longer use subprocesses by default. To revert to the old behavior, the `--use-subprocess` flag must now be explicitly provided.
|
|
- `SerializedEngineLoader` now accepts a `buffer_loader`, so that a function that loads a serialized engine may be provided instead of the serialized engine itself.
|
|
- Default opset for `OnnxFromTfGraph` has been updated to `11`
|
|
|
|
### Fixed
|
|
- `polygraphy_exec` and `polygraphy_gen` now correctly handle cases where no model file is provided
|
|
|
|
|
|
## v0.15.0 (2020-05-05)
|
|
### Added
|
|
- Added a `PolygraphyException` class to serve as a base class for exceptions raised by Polygraphy.
|
|
|
|
### Changed
|
|
- `ConfigLoader` now accepts a list of `Profile`s to support multiple optimization profiles.
|
|
- Changed the format of CLI shapes arguments to enable specifying multiple profiles.
|
|
- Moves `outputs` argument from TfRunner to the tensorflow loaders.
|
|
|
|
|
|
## v0.14.1 (2020-04-17)
|
|
### Added
|
|
- Polygraphy now includes a thin `ctypes` wrapper around the CUDA runtime library, accessible in `util/cuda.py`
|
|
|
|
### Changed
|
|
- `TrtRunner` no longer depends on `pycuda`, and instead uses the included CUDA wrapper.
|
|
- Loader parameters may now be loaders themselves, or the result of invoking a loader.
|
|
- Improves the quality of Comparator messages when there are mismatches
|
|
- `basic_compare_func` will now preserve output ordering in the results.
|
|
- Makes `EngineFromNetwork` compatible with TensorRT 7.0
|
|
|
|
|
|
## v0.14.0 (2020-04-09)
|
|
### Added
|
|
- Restructures ONNX Runner, and adds layerwise functionality (using ONNX-GraphSurgeon).
|
|
- Added `--timestamp` and `--line-info` options to `polygraphy_exec` to enable logging of timestamp and line numbers respectively.
|
|
- Added `--no-letter` option to disable severity letter prefixes in log messages
|
|
- Added `register_callback` to Logger, which registers a callback that will be called whenever the severity changes.
|
|
- Added `Logger.verbosity()` which returns a context manager that can be used to temporarily change logging severity.
|
|
- Added new variants to `--model-type` in `polygraphy_exec`: `keras`, `ckpt`, renamed `tf` to `frozen`
|
|
- Added `ConfigLoader` which can be passed to `EngineFromNetwork` to customize the build configuration prior to building.
|
|
|
|
### Changed
|
|
- The logger no longer displays timestamps and line numbers. These can be enabled by setting the `timestamp`/`line_info` properties respectively to `True`.
|
|
- Logger now relies on the `colored` module to provide colored output
|
|
- `polygraphy_exec` now runs runners in the order in which they were specified.
|
|
- Greatly shortens import paths by removing `_runner` suffixes and shortening framework names (e.g. `tensorflow_runner` -> `tf`)
|
|
- `runners` submodule has been renamed to `backend`
|
|
- `TrtRunner` has been renamed to `TrtLegacyRunner`
|
|
- `TrtRunnerV2` has been renamed to `TrtRunner`
|
|
- `polygraphy_gen` is now at parity with `polygraphy_exec`
|
|
|
|
### Removed
|
|
- Removed `--tftrt` as a separate runner in `polygraphy_exec` - instead it is now an option for the `--tf` runner.
|
|
- Removed `--tftrt-gpu-memory-fraction` and renamed `--tf-gpu-memory-fraction` to `--gpu-memory-fraction` in `polygraphy_exec`
|
|
- Removed `--tfonnx`, and instead adds this functionality in `--onnxrt` when using a TensorFlow model in `polygraphy_exec`
|
|
- Removed `Experimental` argument section in `polygraphy_exec`. All functionality has now been integrated into non-experimental arguments.
|
|
- Removed `preprocess_network` argument from `EngineFromNetwork`. This functionality can be achieved by wrapping the network loaders instead.
|
|
|
|
|
|
## v0.13.4 (2020-03-25)
|
|
### Changed
|
|
- `Comparator.run` will now forcefully terminate the subprocess if it does not exit on its own.
|
|
|
|
|
|
## v0.13.3 (2020-03-20)
|
|
### Added
|
|
- Added TF32 support to legacy TrtLegacyRunner.
|
|
|
|
|
|
## v0.13.2 (2020-03-20)
|
|
### Changed
|
|
- Various improvements to automatic shape matching for cases where shapes between runners do not match exactly.
|
|
- Changed `BaseRunner` so that runners can now implement `activate()`/`deactivate()` instead of `__enter__()`/`__exit__()`
|
|
- `polygraphy_exec` now defaults to running just a single iteration of inference.
|
|
|
|
### Removed
|
|
- The `--accuracy` flag has been removed from `polygraphy_exec`, as this is now the default behavior.
|
|
|
|
### Fixed
|
|
- TensorRT runners now use the same builder to build the network and engine, instead of using a separate builder for each.
|
|
|
|
|
|
## v0.13.1 (2020-03-17)
|
|
### Fixed
|
|
- Fixes a bug in `try_match_shape`
|
|
|
|
|
|
## v0.13.0 (2020-03-17)
|
|
### Added
|
|
- Added a `tf32` parameter as well as `--tf32` flag for TensorRT.
|
|
- Added support for `dim_param` in ONNX.
|
|
|
|
### Changed
|
|
- `fp16_mode` and `int8_mode` parameters have been renamed to `fp16` and `int8` respectively.
|
|
- `polygraphy_exec` will now use the runtime shapes specified rather than always using `OPT` shapes from the TensorRT profile.
|
|
- Improves shape matching logic in `DataLoaderCache`
|
|
|
|
|
|
## v0.12.0 (2020-03-06)
|
|
### Added
|
|
- Added a `start_index` and `end_index` to `Comparator.run` to make it easy to skip over inputs from the data loader.
|
|
- Added `CompareFunc` to provide built-in comparison functions.
|
|
- Added `PostprocessFunc` to provide built-in post-processing functions.
|
|
- `Comparator.compare_accuracy` now returns an `AccuracyResult` object, which contains much more information about the results of the comparisons.
|
|
- Added `percentage()` function to `AccuracyResult` to provide an easy way to figure out the percentage of passed iterations.
|
|
|
|
### Changed
|
|
- Replaces `RunInfo` with `IterationResult`. The latter only stores information about a single iteration for a single runner.
|
|
- `compare_func` in `Comparator.compare_accuracy` is now a `Callable(IterationResult, IterationResult) -> Dict[str, bool]`
|
|
- `warm_up_runs` now defaults to `0`, and `end_index` to `1`
|
|
- Ordering of outputs in a single iteration is now preserved in `CompareFunc.basic_compare_func`
|
|
- `use_subprocess` now defaults to `False` in `Comparator.run()` (still defaults to `True` in `polygraphy_exec`).
|
|
- `Calibrator` now takes a `start_index` and `end_index` argument instead of `max_items`.
|
|
|
|
### Removed
|
|
- Removed `Comparator.compare` function since `Comparator.compare_accuracy` includes all of its functionality.
|
|
- `iterations` in `Comparator.run` has been removed and replaced by `start_index` and `end_index`
|
|
- Removed `subprocess_polling_interval` argument, as `Comparator` can now properly detect when the subprocess terminates.
|
|
|
|
### Fixed
|
|
- `Comparator.run()` will no longer hang if there is a segfault in the subprocess.
|
|
|
|
|
|
## v0.11.3 (2020-02-25)
|
|
### Added
|
|
- Added `--int-min`, `--int-max`, `--float-min`, and `--float-max` arguments to `polygraphy_exec`
|
|
- Added `--explicit-precision` option to `polygraphy_exec` to enable QAT models in TRT.
|
|
- Added empty tensor support. Empty tensors are tensors whose shapes contain one or more 0s.
|
|
|
|
### Changed
|
|
- When `--load-outputs` or `--save-outputs` is specified to `polygraphy_exec`, `seed` will default to `1` to ensure consistent inputs across runs.
|
|
|
|
|
|
## v0.11.2 (2020-02-11)
|
|
### Added
|
|
- Added a `--calibration-cache` option to `polygraphy_exec` to enable supplying a calibration cache
|
|
- Added a `--no-color` option to disable color logging.
|
|
|
|
|
|
## v0.11.1 (2020-02-11)
|
|
### Added
|
|
- Added `GraphOptimizerLoader` for freezing TensorFlow graphs and `--freeze-graph` option to `polygraphy_exec`.
|
|
- Added `--load-outputs` and `--save-outputs` to `polygraphy_exec` for comparing across executions.
|
|
- Added `KerasLoader` for loading models stored in `hdf5` format.
|
|
- Added constant folding pass to `GraphOptimizerLoader` for TensorFlow graphs.
|
|
|
|
### Changed
|
|
- Updates `Calibrator` so that it will now use the opt dimension of a profile for networks with dynamic shapes.
|
|
- Updates Legacy TensorRT runner to use `Loaders` for easier UFF debugging.
|
|
|
|
### Fixed
|
|
- `Calibrator` will no longer allocate buffers if a calibration cache was provided.
|
|
|
|
|
|
## v0.11.0 (2020-01-28)
|
|
### Added
|
|
- Added generation of ONNX code to `polygraphy_gen`
|
|
- Added default implementations of some `BaseRunner` methods.
|
|
- Added `last_inference_time()` to `BaseRunner` so that `infer()` now only needs to return outputs.
|
|
- Added `Calibrator` for int8 calibration, along with additional parameters to `EngineFromNetwork`
|
|
|
|
### Changed
|
|
- Better warnings for user-defined implementations of various APIs.
|
|
- `DataLoaderCache` will now warn loudly when a set of inputs needs to be regenerated.
|
|
- Cleans up `Comparator` `run()` function.
|
|
- Moves most `save_*` options into loaders rather than runners.
|
|
- Changed `BaseDataLoader.next()` to take index as an argument. This way, inputs can be reliably repeated across runners.
|
|
- Moves all `layerwise` parameters into loaders rather than runners.
|
|
- `Loader`s are now interchangeable with Python `Callable`s
|
|
- `DataLoader`s are now interchangeable with Python `Callable`s
|
|
|
|
### Fixed
|
|
- `DataLoader` no longer generates all `True` values for boolean types.
|
|
- Various bug fixes in `polygraphy_gen`
|
|
- `DataLoaderCache` is now sent over the queue when runners are run in subprocesses. This resolves an issue where the cache was not being updated correctly.
|
|
- `Comparator` now updates runners correctly when using a subprocess.
|
|
|
|
|
|
## v0.10.6 (2019-12-11)
|
|
### Added
|
|
- Added `--no-fold-constant` option to prevent `OnnxFromTfGraph` from doing constant folding in the TensorFlow graph.
|
|
- Added experimental `polygraphy_gen` script that enables generation of template Python scripts for running Polygraphy.
|
|
|
|
### Fixed
|
|
- Bug fix for cases where TensorFlow nodes with no outputs are recognized as graph outputs by `GraphSurgeon`.
|
|
|
|
|
|
## v0.10.5 (2019-12-9)
|
|
### Added
|
|
- Added `name` parameter to `CheckpointLoader` in case the checkpoint does not include a `checkpoint` file.
|
|
|
|
### Changed
|
|
- `TFTRTLoader` now accepts any kind of TensorFlow Graph loader
|
|
|
|
### Fixed
|
|
- Bug fix in `TrtRunner` `Buffers` so that no-op reshapes (no reallocation) are handled correctly.
|
|
|
|
|
|
## v0.10.4 (2019-12-4)
|
|
### Added
|
|
- Added `check_inf`, `check_nan`, and `fail_fast` options to `Comparator.validate()`
|
|
|
|
### Changed
|
|
- Cleans up `Buffers` implementation for `TrtRunner` - eliminates an unnecessary copy that was happening on the host input.
|
|
- Improved logic for matching output names in `util.find_in_dict()`
|
|
|
|
|
|
## v0.10.3 (2019-11-18)
|
|
### Changed
|
|
- `TrtRunner` will no longer call `context`'s shape setting functions on non-dynamic inputs.
|
|
|
|
### Fixed
|
|
- Bug fix for volume computation for scalars.
|
|
- Updates `DataLoader` to handle scalars correctly, adds several tests.
|
|
|
|
|
|
## v0.10.2 (2019-11-11)
|
|
### Added
|
|
- Added various utility functions as static members of `TrtRunner`, e.g. `create_network` function to simplify TensorRT's network flags.
|
|
|
|
### Changed
|
|
- `EngineFromNetwork` will now mark network outputs when `layerwise=True`
|
|
|
|
|
|
## v0.10.1 (2019-10-31)
|
|
### Added
|
|
- Added support for `bool` outputs in `Comparator`
|
|
|
|
|
|
## v0.10.0 (2019-10-28)
|
|
### Changed
|
|
- Replaces `OnnxEngineLoader` with `OnnxNetworkLoader` and `EngineFromNetwork`. This allows for more flexibility in building engines from TensorRT networks.
|
|
|
|
|
|
## v0.9.8 (2019-10-24)
|
|
### Added
|
|
- Added `allow_growth` option to TfRunner to work around `CUDNN_STATUS_INTERNAL_ERROR`. When `allow_growth` is enabled, the error disappears.
|
|
|
|
### Changed
|
|
- `DataLoaderCache` will now attempt to permute inputs in cases where shapes do not match exactly (e.g. NCHW vs NHWC inputs).
|
|
|
|
### Fixed
|
|
- Fixes a bug in `polygraphy_exec` which caused it to ignore user-defined profiles.
|
|
|
|
|
|
## v0.9.7 (2019-10-18)
|
|
### Added
|
|
- Added support for many more ONNX data types.
|
|
- Added support for `int8` and explicit precision mode in `TrtRunner`
|
|
- Added `preprocess_network` parameter to `OnnxEngineLoader` so that the network can be modified before it is used for building.
|
|
|
|
### Changed
|
|
- `TrtRunner` will now attempt to generate sane default shapes in cases with dynamic shapes where no profiles are provided.
|
|
|
|
|
|
## v0.9.6 (2019-10-15)
|
|
### Changed
|
|
- `DataLoader` no longer overrides static shapes in the model, but issues a warning if an override is requested.
|
|
- `DataLoader` now accepts shape tensor inputs in its `default_shapes` parameter.
|
|
|
|
|
|
## v0.9.5 (2019-10-9)
|
|
### Added
|
|
- Added timestamps to logging output.
|
|
|
|
### Fixed
|
|
- `Comparator` can now catch segfaults in runners properly.
|
|
|
|
|
|
## v0.9.4 (2019-10-7)
|
|
### Added
|
|
- Added options for `DataLoader` to be able to specify input bounds
|
|
- Added smarter matching for input metadata in the `DataLoaderCache`
|
|
|
|
### Changed
|
|
- Default `subprocess_polling_interval` is now 30 seconds.
|
|
- `Comparator` now attempts to partially match output names when no exact matches are found.
|
|
|
|
|
|
## v0.9.3 (2019-10-1)
|
|
### Added
|
|
- Added `subprocess_timeout` parameter to `Comparator.run` to prevent hangs when a subprocess does not terminate.
|
|
- Added `subprocess_polling_interval` parameter to allow the process to be polled so that failing processes can be terminated before the full `subprocess_timeout`.
|
|
|
|
|
|
## v0.9.2 (2019-10-1)
|
|
### Changed
|
|
- If ONNX checker fails due to the IR version of the model being too new, Polygraphy now ignores the error and continues.
|
|
|
|
|
|
## v0.9.1 (2019-10-1)
|
|
### Changed
|
|
- `OnnxEngineLoader` now accepts an `onnx_loader` for better flexibility in loading models.
|
|
- `polygraphy_exec` now supports running TF models in TRT via the tf2onnx converter.
|
|
- Legacy `TrtLegacyRunner` now only supports UFF models.
|
|
|
|
|
|
## v0.9.0 (2019-09-30)
|
|
### Added
|
|
- Added `BaseModelLoader` that can be used to load models. This allows for reuse of existing runners with different import paths. For example, `OnnxrtRunner` can be used with `OnnxFromTfGraph` in order to run a TensorFlow frozen graph via ONNX-Runtime.
|
|
- Implements `ModelLoader`s for `TfRunner`, including a frozen model loader, checkpoint loader, and TF-TRT loader.
|
|
|
|
### Changed
|
|
- `OnnxFromTfGraph` now accepts a TensorFlow ModelLoader to support a wider variety of input formats.
|
|
- Updates legacy `TrtLegacyRunner` to use `get_input_metadata` API, so it is usable for UFF models.
|
|
|
|
|
|
## v0.8.1 (2019-09-26)
|
|
### Changed
|
|
- Comparator will now look at the union of all outputs from all runners when checking for common outputs.
|
|
- `TrtRunner` will no longer mark layers within the loop body as network outputs in `layerwise` mode.
|
|
- `DataLoaderCache` now falls back to reusing inputs based on order if names do not match exactly.
|
|
- `DataLoader` now accepts a `default_shapes` parameter to override dynamic shapes.
|
|
|
|
|
|
## v0.8.0 (2019-09-18)
|
|
### Added
|
|
- Added `get_input_metadata` API to BaseRunner. Overhauls runners so they no longer need to handle dynamic input shapes individually.
|
|
- Added `DataLoader` class which can be used to feed data to the Comparator.
|
|
- Added `DataLoaderCache` so that the data loader does not have to load inputs multiple times for each runner.
|
|
|
|
### Changed
|
|
- `Comparator.compare_accuracy` now fails if no outputs were compared.
|
|
|
|
### Removed
|
|
- Removed support for implicit batch ONNX models in `TrtLegacyRunner`. You should use `TrtRunner` for ONNX models instead.
|
|
|
|
|
|
## v0.7.1 (2019-08-29)
|
|
### Removed
|
|
- Removed `python2` support.
|
|
|
|
### Fixed
|
|
- Bug fixes for TensorFlow Graphs
|
|
- Bug fixes for `polygraphy_exec` when using legacy `TrtLegacyRunner`
|
|
- Bug fixes for `TrtRunner` for cases with multiple outputs
|
|
|
|
|
|
## v0.7.0 (2019-07-30)
|
|
#### Added
|
|
- Added support for compression during communication between the runner subprocesses and the main `Comparator` process. This is because `Pipe`s and `Queue`s can only send objects smaller than 2GB.
|
|
- Added timeouts to reduce the possibility of hangs in runners.
|
|
- Added `--fail-fast` option to `polygraphy_exec` and corresponding `fail_fast` option to `Comparator.compare()`. Useful for determining the first layer at which two models diverge.
|
|
- Added `TrtRunner` that can be used to run TRT networks with dynamic shapes. Currently only supports ONNX.
|
|
|
|
### Changed
|
|
- Runners no longer need to specify inputs up front - they can now be specified after `__enter__` is called. This greatly simplifies much of the logic in several runners.
|
|
- `RunInfo` no longer contains data about the inputs used.
|
|
- `TFOnnxrtRunner` now accepts an opset option when converting graphs to ONNX.
|
|
|
|
|
|
## v0.6.0 (2019-07-17)
|
|
- All runner files are now suffixed with `_runner` to disambiguate them from system packages.
|
|
- Fixes an issue that prevent EXTRA_VERBOSE logging output from TRT from being displayed.
|
|
- Added a `--uff-order` option in case the automatically determined order is wrong.
|
|
- Added an experimental `--build-only` option to `polygraphy_exec`
|
|
- Comparator will now attempt to permute outputs with mismatched shapes when `check_shapes` is disabled.
|
|
- Lowers the default GPU memory fraction, as TensorFlow has OOM issues when it is set too high.
|
|
- Added `TFOnnxrtRunner` and `--tfonnx` option to `polygraphy_exec`
|
|
- Added `OnnxrtRunner` and moves `TFOnnxrtRunner` into `onnx_runner.py`.
|
|
- Added `--save-onnx` option for `OnnxrtRunner`
|
|
- Changed `--onnx` `polygraphy_exec` option to `onnxtf` to disambiguate from `--onnxrt`
|
|
- Added `CNTKRunner` and `--cntk` option to `polygraphy_exec`
|
|
- Changed default shape value to 1. This is the value that is set when no input dimension is specified.
|
|
- Added support for loading TF checkpoints.
|
|
- Added support for overriding automatically determined outputs in the TF and TF-TRT runners. Added `--tf-outputs` argument to `polygraphy_exec`
|
|
- Fixes input shape mismatches between ONNX-RT and TF.
|
|
- Added `--plugins` option to `polygraphy_exec` for loading TRT plugins.
|
|
|
|
## v0.5.0 (2019-04-02)
|
|
|
|
- Added a function in comparator to perform output validation, and a corresponding flag in `polygraphy_exec`.
|
|
- Runners now use OrderedDict for outputs, meaning that the ordering of the outputs will match the order of the layers in the network in most cases.
|
|
- Improved TensorFlow output tensor deduction by excluding certain ops that cannot behave like outputs in TensorFlow.
|
|
- Version information is now logged at INFO logging severity.
|
|
- Removed prepare_inputs/prepare_outputs functions. Instead, runners now do timing on their own in the infer function.
|
|
- Changed runner inputs to use dictionaries that map input names to their numpy buffers.
|
|
- `polygraphy_exec` will no longer fail if the extension for the model file is unrecognized.
|
|
- Added `fp16_mode` option to TfRunner for TF-TRT.
|
|
|
|
## v0.4.0 (2019-02-28)
|
|
|
|
- Added an option to limit TensorFlow GPU memory usage
|
|
- Added an option to specify minimum segment size to TF-TRT.
|
|
- Added an option to write out engine(s) from the TF-TRT graph.
|
|
- `polygraphy_exec` now exits when unknown arguments are encountered
|
|
- Improves timestamps to be human-readable instead of using seconds from epoch.
|
|
- Added support for dynamic ops in TF-TRT
|
|
- Added an option to write out tensorboard visualizations.
|
|
- Added an option for enabling XLA in the TensorFlow runner.
|
|
|
|
## v0.3.0 (2019-02-19)
|
|
|
|
- Added nicer error messages on failed TF-TRT imports
|
|
- If a TensorFlow graph specifies a dynamic shape, Polygraphy now automatically populates it with concrete values.
|
|
- Added argument groups and moves some unstable arguments to Experimental section.
|
|
- Polygraphy will now refuse to write artifacts to the disk if a file already exists wherever it can detect such cases.
|
|
- `polygraphy_exec` now emits warnings when unknown command line parameters are used.
|
|
- Added capability to write out TensorFlow timelines.
|
|
- Changed --save* options to accept directory names instead, and the resulting files are timestamped and named based on the runner name.
|
|
|
|
## v0.2.0 (2019-02-04)
|
|
|
|
- Changed command-line parameters to use dashes instead of underscore.
|
|
- Modifies TrtLegacyRunner to pass along input order to UFF, instead of permuting the order to CHW.
|
|
- Comparator now prints runner output in the same order in which they were specified.
|
|
- Added per-inference-inputs command-line arguments for running multiple comparisons.
|
|
- Seed is now displayed correctly during Comparator.run().
|
|
- User-friendly Comparator output - now suggests command-line flags to get what you were looking for.
|
|
- Added layerwise comparison support for TrtLegacyRunner and TfRunner.
|
|
|
|
## v0.1.2 (2019-01-15)
|
|
|
|
- Renamed to TRT Polygraphy.
|
|
|
|
## v0.1.1 (2019-01-15)
|
|
|
|
- Overhauled README.md
|
|
- Modified project structure - created runners, comparator, and logger submodules.
|
|
- polygraphy_exec now uses batch size specified by model if none is specified by the user.
|
|
- Added framework dependencies to setup.py
|
|
- TrtLegacyRunner now displays ONNX parsing errors and exits early on parsing failures.
|
|
|
|
## v0.1.0 (2019-01-11)
|
|
|
|
- Initial integration
|