chore: import upstream snapshot with attribution
Docker Image CI / build-ubuntu2004 (push) Has been cancelled
Docker Image CI / build-ubuntu2004 (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,276 @@
|
||||
---
|
||||
name: trt-cpp-runtime-quickstart
|
||||
description: >-
|
||||
Load and run a TensorRT engine (.plan / .engine) from C++ using the
|
||||
TensorRT 11 / 10.x **modern Runtime API**, avoiding the deprecated TRT
|
||||
8.x binding-index APIs that older guidance still promotes. Use whenever
|
||||
the user asks about loading
|
||||
or running a TensorRT .plan/.engine from C++, even on "minimal example"
|
||||
requests — without this skill the default reply uses deprecated
|
||||
enqueueV2-style code. Also use when the user hits "Engine plan file
|
||||
is generated on an incompatible device", deserializeCudaEngine returns
|
||||
nullptr, gets an enqueueV2 / IStreamReader deprecation warning, or
|
||||
wants to stream a .plan via IStreamReaderV2. Triggers: TensorRT C++
|
||||
inference, load TensorRT plan C++, run .plan from C++, IRuntime
|
||||
example, deserializeCudaEngine, enqueueV3, enqueueV2 deprecated,
|
||||
setTensorAddress, getBindingIndex, IStreamReaderV2, libnvinfer C++.
|
||||
NOT for building engines (`trt-onnx-quickstart`), Python deploy,
|
||||
plugins, multi-GPU.
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: NVIDIA Corporation
|
||||
version: "1.0"
|
||||
tags:
|
||||
- tensorrt
|
||||
- cpp
|
||||
- inference
|
||||
- deployment
|
||||
- runtime
|
||||
---
|
||||
|
||||
# TensorRT C++ Runtime Deploy
|
||||
|
||||
Load a serialized TensorRT engine from disk and run inference from C++ using only the modern Runtime API. Produces a minimal, copy-pasteable deploy harness that drops next to any `.plan` / `.engine` file and extends to production.
|
||||
|
||||
Reference samples to open before writing new code:
|
||||
|
||||
- `quickstart/SemanticSegmentation/tutorial-runtime.cpp` — cleanest minimal load-and-run example. Mirrors Steps 1–7 below.
|
||||
- `samples/sampleOnnxMNIST/sampleOnnxMNIST.cpp` — end-to-end sample that also builds the engine; the runtime portion shows realistic I/O wiring.
|
||||
- Public headers: `include/NvInferRuntime.h` — read `IRuntime`, `ICudaEngine`, `IExecutionContext`, `IStreamReaderV2`.
|
||||
|
||||
## When to Use
|
||||
|
||||
| Situation | Use this skill? |
|
||||
|-------------------------------------------------------------------------------------------|-----------------|
|
||||
| You have a `.plan`/`.engine` and need to run it from a C++ binary | Yes |
|
||||
| You need a minimal harness that uses `enqueueV3` + `setTensorAddress` | Yes |
|
||||
| You want to load an engine from a `std::istream` or large file via `IStreamReaderV2` | Yes |
|
||||
| You need to wire dynamic shapes (`setInputShape`) before inference | Yes |
|
||||
| You are *building* / optimizing the engine (calibration, INT8, sparsity, builder configs) | No - use trtexec or `IBuilder` directly |
|
||||
| You are deploying in Python | No - use `tensorrt` Python bindings |
|
||||
| You are writing a plugin (`IPluginV3`) or custom layer | No - separate plugin skill |
|
||||
| You need multi-GPU, MPS, MIG, or process-level orchestration | No - out of scope |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **TensorRT installed.** Verify `NvInferRuntime.h` is on the include path
|
||||
and `libnvinfer.so` is on the link path. On a TRT dev container these are
|
||||
in `/usr/include/x86_64-linux-gnu/` and `/usr/lib/x86_64-linux-gnu/` (or
|
||||
`/opt/tensorrt/...` for tarball installs).
|
||||
2. **CUDA toolkit available.** `cuda_runtime_api.h` and `libcudart.so` must
|
||||
be reachable; `nvcc --version` should match the CUDA version the engine
|
||||
was built against.
|
||||
3. **A serialized engine.** A `.plan`/`.engine` file built **on the same
|
||||
major TRT version and the same GPU architecture (compute capability) you
|
||||
will deploy on**. Engines are not portable across major TRT versions or
|
||||
across SMs unless the builder was given `--hardwareCompatibilityLevel`.
|
||||
4. **The engine's I/O tensor names.** Inspect with:
|
||||
```bash
|
||||
trtexec --loadEngine=model.plan --verbose 2>&1 | grep -E 'Input|Output'
|
||||
```
|
||||
5. A C++17 compiler (`g++ >= 9` or `clang++ >= 10`).
|
||||
|
||||
## Step 1: Create the IRuntime
|
||||
|
||||
The runtime owns engine deserialization and must outlive every
|
||||
`ICudaEngine` it creates. Construct one per process for typical deployments.
|
||||
|
||||
```cpp
|
||||
class Logger : public nvinfer1::ILogger {
|
||||
public:
|
||||
void log(Severity severity, char const* msg) noexcept override {
|
||||
if (severity <= Severity::kWARNING) {
|
||||
std::cerr << msg << std::endl;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
Logger gLogger;
|
||||
std::unique_ptr<nvinfer1::IRuntime> runtime{
|
||||
nvinfer1::createInferRuntime(gLogger)};
|
||||
if (!runtime) throw std::runtime_error("createInferRuntime failed");
|
||||
```
|
||||
|
||||
A custom logger is mandatory - TensorRT does not log internally. Keep it
|
||||
process-global so deserialization warnings (version skew, calibrator
|
||||
mismatch) are not lost.
|
||||
|
||||
## Step 2: Read the Plan into Memory
|
||||
|
||||
For small/medium engines (< ~1 GiB) read the whole file into a
|
||||
`std::vector<char>` and hand the pointer to
|
||||
`IRuntime::deserializeCudaEngine(blob, size)`. This is what the
|
||||
`SemanticSegmentation` tutorial does and the simplest correct path:
|
||||
|
||||
```cpp
|
||||
std::ifstream f(planPath, std::ios::binary);
|
||||
if (!f) throw std::runtime_error("cannot open " + planPath);
|
||||
f.seekg(0, std::ios::end);
|
||||
auto size = static_cast<size_t>(f.tellg());
|
||||
f.seekg(0, std::ios::beg);
|
||||
std::vector<char> blob(size);
|
||||
if (!f.read(blob.data(), size))
|
||||
throw std::runtime_error("short read on " + planPath);
|
||||
```
|
||||
|
||||
For very large engines, or when the bytes live behind a stream (HTTP,
|
||||
mmap'd archive, encrypted store), implement an `IStreamReaderV2` - see
|
||||
Step 3.
|
||||
|
||||
## Step 3 (optional): Use IStreamReaderV2 for Streaming Loads
|
||||
|
||||
`IStreamReader` (v1) is **deprecated in TensorRT 11.0**. Always use
|
||||
`IStreamReaderV2`: it reads into both host and device memory and is the
|
||||
only stream-reader form guaranteed for new code. Subclass and implement
|
||||
`read(...)` and `seek(...)`:
|
||||
|
||||
```cpp
|
||||
class FileStreamReader : public nvinfer1::IStreamReaderV2 {
|
||||
public:
|
||||
explicit FileStreamReader(std::string const& path)
|
||||
: mFile(path, std::ios::binary) {
|
||||
if (!mFile) throw std::runtime_error("open failed: " + path);
|
||||
}
|
||||
int64_t read(void* dst, int64_t n,
|
||||
cudaStream_t /*stream*/) noexcept override {
|
||||
mFile.read(static_cast<char*>(dst), n);
|
||||
return mFile.gcount();
|
||||
}
|
||||
bool seek(int64_t off, nvinfer1::SeekPosition where) noexcept override {
|
||||
auto dir = (where == nvinfer1::SeekPosition::kSET) ? std::ios::beg
|
||||
: (where == nvinfer1::SeekPosition::kCUR) ? std::ios::cur
|
||||
: std::ios::end;
|
||||
mFile.clear();
|
||||
mFile.seekg(off, dir);
|
||||
return static_cast<bool>(mFile);
|
||||
}
|
||||
private:
|
||||
std::ifstream mFile;
|
||||
};
|
||||
|
||||
FileStreamReader rd{planPath};
|
||||
std::unique_ptr<nvinfer1::ICudaEngine> engine{
|
||||
runtime->deserializeCudaEngine(rd)};
|
||||
```
|
||||
|
||||
## Step 4: Deserialize and Create an Execution Context
|
||||
|
||||
`ICudaEngine` is thread-safe for read-only queries; `IExecutionContext`
|
||||
is **not** - allocate one per inference thread.
|
||||
|
||||
```cpp
|
||||
std::unique_ptr<nvinfer1::ICudaEngine> engine{
|
||||
runtime->deserializeCudaEngine(blob.data(), blob.size())};
|
||||
if (!engine) throw std::runtime_error("deserializeCudaEngine failed");
|
||||
|
||||
std::unique_ptr<nvinfer1::IExecutionContext> ctx{
|
||||
engine->createExecutionContext()};
|
||||
if (!ctx) throw std::runtime_error("createExecutionContext failed");
|
||||
```
|
||||
|
||||
## Step 5: Wire Tensors with setTensorAddress
|
||||
|
||||
Enumerate I/O tensors via `getNbIOTensors()` + `getIOTensorName(i)`. Use
|
||||
`getTensorIOMode`, `getTensorDataType`, and `getTensorShape` to size and
|
||||
allocate buffers. **Set every tensor address before `enqueueV3`** - the
|
||||
modern API has no implicit binding-index map.
|
||||
|
||||
```cpp
|
||||
for (int i = 0; i < engine->getNbIOTensors(); ++i) {
|
||||
char const* name = engine->getIOTensorName(i);
|
||||
auto mode = engine->getTensorIOMode(name);
|
||||
auto shape = engine->getTensorShape(name); // -1 = dynamic dim
|
||||
if (mode == nvinfer1::TensorIOMode::kINPUT && hasDynamic(shape)) {
|
||||
// Fill in concrete shape, e.g. batch=1
|
||||
shape.d[0] = 1;
|
||||
ctx->setInputShape(name, shape);
|
||||
}
|
||||
}
|
||||
// After setInputShape on all dynamic inputs, query output shapes.
|
||||
for (int i = 0; i < engine->getNbIOTensors(); ++i) {
|
||||
char const* name = engine->getIOTensorName(i);
|
||||
auto bytes = elementCount(ctx->getTensorShape(name))
|
||||
* dtypeSize(engine->getTensorDataType(name));
|
||||
void* dev = nullptr;
|
||||
cudaMalloc(&dev, bytes);
|
||||
ctx->setTensorAddress(name, dev);
|
||||
}
|
||||
```
|
||||
|
||||
Always call `setInputShape` for dynamic inputs **before** querying output
|
||||
shapes - the latter depends on the former.
|
||||
|
||||
## Step 6: Run enqueueV3
|
||||
|
||||
`enqueueV3(stream)` is the only non-deprecated enqueue API;
|
||||
`enqueueV2`/`execute*` are gone in modern flows.
|
||||
|
||||
```cpp
|
||||
cudaStream_t stream{};
|
||||
cudaStreamCreate(&stream);
|
||||
|
||||
cudaMemcpyAsync(devInput, hostInput, inBytes,
|
||||
cudaMemcpyHostToDevice, stream);
|
||||
if (!ctx->enqueueV3(stream))
|
||||
throw std::runtime_error("enqueueV3 failed");
|
||||
cudaMemcpyAsync(hostOutput, devOutput, outBytes,
|
||||
cudaMemcpyDeviceToHost, stream);
|
||||
cudaStreamSynchronize(stream);
|
||||
```
|
||||
|
||||
If you reuse buffers across iterations, skip the per-call
|
||||
`setTensorAddress` - addresses persist on the context until overwritten.
|
||||
|
||||
## Step 7: Shutdown Order
|
||||
|
||||
Destroy in reverse construction order: contexts -> engines -> runtime,
|
||||
then free CUDA memory and destroy the stream. With `std::unique_ptr` this
|
||||
is automatic as long as the context is declared *after* the engine, and
|
||||
the engine *after* the runtime. Free `cudaMalloc` allocations explicitly
|
||||
(RAII wrapper recommended).
|
||||
|
||||
## Build
|
||||
|
||||
Wire the steps above into your application's build system. For a standalone smoke test, a minimal build is:
|
||||
|
||||
```bash
|
||||
g++ -std=c++17 runtime.cpp -o run -lnvinfer -lcudart # adjust CUDA/TRT include + lib paths
|
||||
./run model.plan
|
||||
```
|
||||
|
||||
## Common Errors
|
||||
|
||||
| Symptom | Likely cause |
|
||||
|----------------------------------------------------------------------|------------------------------------------------------------------------------|
|
||||
| `deserializeCudaEngine` returns `nullptr`, log says "version tag" | Engine built on a different TRT major version. Rebuild on the deploy version |
|
||||
| `nullptr` with "engine plan file is generated on an incompatible device" | SM mismatch. Rebuild on the target SM or use `--hardwareCompatibilityLevel` |
|
||||
| `enqueueV3` returns false, log mentions "Tensor X has no address" | Forgot `setTensorAddress` for one of the I/O tensors |
|
||||
| `enqueueV3` false, "shape" in message | Forgot `setInputShape` for a dynamic input, or supplied an out-of-profile shape |
|
||||
| `cudaErrorIllegalAddress` on H->D / D->H copy | Mismatched element count / dtype between host buffer and engine tensor |
|
||||
| Process crashes inside TRT during destruction | Wrong destruction order - context outlived engine, or engine outlived runtime |
|
||||
| `cudaErrorMemoryAllocation` during context creation | Workspace too big for the device; rebuild with smaller workspace |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Do not use `IStreamReader` v1.** Deprecated in TRT 11.0. Use
|
||||
`IStreamReaderV2` (note `cudaStream_t` parameter on `read`).
|
||||
- **Do not use `enqueueV2` / `execute` / binding indices.** These are
|
||||
legacy paths; the only stable modern path is name-based
|
||||
`setTensorAddress` + `enqueueV3`.
|
||||
- **One `IExecutionContext` per thread.** Sharing contexts across threads
|
||||
is undefined behavior; sharing the engine is fine.
|
||||
- **Stream lifetime.** The CUDA stream passed to `enqueueV3` must outlive
|
||||
the inference. Destroying it while work is in flight crashes or corrupts
|
||||
output.
|
||||
- **Async vs sync copies.** Mixing synchronous `cudaMemcpy` with
|
||||
`enqueueV3` on a stream serializes the GPU; always pair `enqueueV3`
|
||||
with `cudaMemcpyAsync` on the same stream.
|
||||
- **Engine portability.** A `.plan` is tied to (TRT major version, GPU SM,
|
||||
CUDA major version). Never check engines into a repo without recording
|
||||
these three facts.
|
||||
- **Logger lifetime.** The logger passed to `createInferRuntime` must
|
||||
outlive the runtime; a stack-local logger in `main` is fine, a function-
|
||||
scope local is a use-after-free.
|
||||
- **Refit / weight streaming.** Engines built with refit or weight
|
||||
streaming enabled need extra setup calls (`setWeightStreamingBudgetV2`,
|
||||
`IRefitter`); out of scope here.
|
||||
@@ -0,0 +1,343 @@
|
||||
---
|
||||
name: trt-onnx-quickstart
|
||||
description: >
|
||||
Build and verify a TensorRT engine from a Hugging Face model ID or ONNX
|
||||
file, with numerical parity checked against ONNX Runtime. Use when the
|
||||
user imports a non-LLM model to TensorRT,
|
||||
needs a verified engine from ONNX, hits trtexec "unsupported operator",
|
||||
must verify the engine matches ONNX numerically, debugs a polygraphy
|
||||
parity failure (large max abs diff at FP16), or configures multi-input
|
||||
dynamic shapes. Triggers: convert ONNX to TensorRT, Hugging Face to
|
||||
TensorRT, trtexec onnx, trtexec unsupported operator, optimum-cli
|
||||
export, polygraphy parity check, polygraphy run --trt --onnxrt, parity
|
||||
check failed, max abs diff, verify engine matches ONNX, --minShapes,
|
||||
dynamic shapes trtexec, multi-input shape profile, FP16 engine, INT64
|
||||
warning. Adjacent skills: `trt-torch-quickstart` (PyTorch frontend),
|
||||
`trt-cpp-runtime-quickstart` (C++ engine load). LLM token generation
|
||||
belongs in TensorRT-LLM, not here.
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: NVIDIA Corporation
|
||||
version: "1.0"
|
||||
tags:
|
||||
- onnx
|
||||
- import
|
||||
- huggingface
|
||||
- quickstart
|
||||
- fp16
|
||||
---
|
||||
|
||||
# TensorRT ONNX Quickstart
|
||||
|
||||
Take a developer from "I have a Hugging Face model ID" or "an ONNX file" to "a TensorRT engine whose outputs match the source model within tolerance." Follows Path 1 (ONNX → TensorRT) of the [Import Workflows Guide](https://github.com/NVIDIA/TensorRT/blob/main/documents/import_workflows.md), specialized for the most common starting point: a Hugging Face Hub model.
|
||||
|
||||
## When to Use
|
||||
|
||||
| Scenario | Use this skill? |
|
||||
|----------|-----------------|
|
||||
| Has a Hugging Face model ID (`google-bert/bert-base-uncased`) and wants TRT-accelerated inference | Yes |
|
||||
| Has an `.onnx` file and wants a `.plan` engine with verified parity | Yes |
|
||||
| Ran `trtexec --onnx=...` and hit a warning/error they don't recognize | Yes |
|
||||
| Wants LLM token generation (Llama, Mistral, Qwen text generation) | **No** — route to [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) |
|
||||
| Has a PyTorch model and wants to stay in PyTorch | No — use `trt-torch-quickstart` |
|
||||
| Already has a `.plan` engine and wants to run inference from C++ | No — use `trt-cpp-runtime-quickstart` |
|
||||
| Is migrating a weakly-typed network to strongly-typed | No — use `trt-strong-typing-migration` |
|
||||
| Has a full diffusion pipeline (SD, FLUX) | Partial — must be component-split first; this skill imports one component at a time |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Verify each before continuing. Failures here surface as confusing errors later, so fix them before running the import.
|
||||
|
||||
1. **NVIDIA GPU + driver matching TensorRT 11.x.** TRT 11 requires CUDA 13.x. Confirm:
|
||||
```bash
|
||||
nvidia-smi # driver + CUDA runtime version
|
||||
```
|
||||
2. **Python ≥ 3.10.** TRT 11 dropped 3.9 and earlier.
|
||||
3. **TensorRT 11 installed — including the `trtexec` CLI**, this skill's primary build tool. **The pip wheel does not ship `trtexec`.** Pick one:
|
||||
|
||||
- **NGC container (recommended for first-time users):** `nvcr.io/nvidia/tensorrt:<tag>` — bundles `trtexec`, Python bindings, libraries, and most common dependencies. Run with `--gpus all`.
|
||||
- **`.tar.gz` or `.deb` from [TensorRT downloads](https://developer.nvidia.com/tensorrt-download):** installs `trtexec` to a system path.
|
||||
|
||||
The pip path (`pip install --extra-index-url https://pypi.nvidia.com tensorrt-cu13`) provides Python bindings but not `trtexec`, so it is **insufficient on its own**. For pip-only, route the developer to the alternative Python-builder flow in `import_workflows.md` Path 1 Option B; this skill assumes `trtexec` is available.
|
||||
|
||||
Verification:
|
||||
```bash
|
||||
python3 -c "import tensorrt; print(tensorrt.__version__)" # expect 11.x.y.z
|
||||
trtexec --help | head -5 # MUST work; if not, install is incomplete
|
||||
```
|
||||
4. **Polygraphy and `optimum-onnx` installed.**
|
||||
```bash
|
||||
pip install polygraphy onnx onnxruntime onnx_graphsurgeon onnxsim 'optimum-onnx'
|
||||
# If the model ID starts with `sentence-transformers/`, also:
|
||||
pip install sentence-transformers
|
||||
```
|
||||
`onnx_graphsurgeon` is required by `polygraphy surgeon sanitize` (Step 3) — Polygraphy does not bundle it. `onnxruntime` is required by `optimum-cli export` (Step 2, to fix dynamic axes) and Polygraphy's `--onnxrt` parity check (Step 5); stock NGC containers do not always ship it.
|
||||
Note: `optimum-onnx` is the active maintained package — the older `optimum` no longer includes ONNX integration. Do **not** use `optimum-nvidia` (last release 2025-01, unmaintained).
|
||||
|
||||
## Step 1 — Choose a model and confirm it's supported
|
||||
|
||||
Consult the [Supported Models](https://github.com/NVIDIA/TensorRT/blob/main/documents/supported_models.md) matrix. If listed, the dtype column gives a known-good starting precision. If not, it is still expected to work — file an issue if it doesn't.
|
||||
|
||||
If the model is a diffusion pipeline, **stop here** — TRT cannot ingest the whole pipeline object. Split it into components (text encoder, UNet/DiT, VAE) and import each separately. See `supported_models.md` for validated splits, and consider whether `trt-export-rewrite` is a better fit.
|
||||
|
||||
This skill's worked example uses `google-bert/bert-base-uncased` from the encoder-NLP section.
|
||||
|
||||
## Step 2 — Export to ONNX
|
||||
|
||||
The `optimum-cli export onnx` command's `--task` flag selects which head of the model to export. The wrong task yields a valid graph but the wrong output — and Step 5 parity validation won't catch it, because reference and engine are both wrong.
|
||||
|
||||
Pick the task from this table based on what the developer wants to do with the model:
|
||||
|
||||
| Developer's intent | `--task` | Typical model families |
|
||||
|--------------------|----------|------------------------|
|
||||
| Get hidden-state embeddings or pooled output | `feature-extraction` | BERT, RoBERTa, sentence-transformers, CLIP |
|
||||
| Classify images into N classes | `image-classification` | ResNet, ViT, MobileNet, EfficientNet |
|
||||
| Classify text into N labels | `text-classification` | BERT for sentiment, DistilBERT, etc. |
|
||||
| Detect / segment objects | `object-detection`, `image-segmentation` | DETR, Mask R-CNN |
|
||||
| Transcribe speech | `automatic-speech-recognition` | Whisper, Wav2Vec2 |
|
||||
| Generate captions / VQA | `image-to-text`, `visual-question-answering` | BLIP, LLaVA |
|
||||
| Translation / summarization | `text2text-generation` | T5, BART |
|
||||
|
||||
For an unknown model, run `optimum-cli export onnx --help` or check the [Optimum supported tasks table](https://huggingface.co/docs/optimum/main/en/exporters/onnx/overview). If unsure, surface the choice to the developer rather than guessing.
|
||||
|
||||
### Sentence-transformers models need an extra install
|
||||
|
||||
If the model ID starts with `sentence-transformers/` (e.g. `all-MiniLM-L6-v2`, `bge-base-en-v1.5`), install `sentence-transformers` *before* exporting:
|
||||
|
||||
```bash
|
||||
pip install sentence-transformers
|
||||
```
|
||||
|
||||
Without it, `optimum-cli` falls back to plain `transformers` and emits raw token-level `last_hidden_state` instead of the pooled sentence embedding the model is designed to produce. The exporter prints `library name was inferred as sentence_transformers, which is not installed. Falling back to transformers.` — treat that warning as a hard error and install before retrying.
|
||||
|
||||
```bash
|
||||
# Worked example: BERT for embedding extraction
|
||||
optimum-cli export onnx \
|
||||
--model google-bert/bert-base-uncased \
|
||||
--task feature-extraction \
|
||||
bert_onnx/
|
||||
```
|
||||
|
||||
### Expected warnings (not errors)
|
||||
|
||||
The exporter often prints a "max diff between reference and ONNX exported model is not within the set tolerance 1e-05" line (with `max diff` around `1e-5` to `1e-4`). This is its own self-check at a strict tolerance — **the export still succeeded.** Step 5's Polygraphy parity check uses `--atol 1e-2 --rtol 1e-2 --check-error-stat mean`, the correct tolerance for FP16 inference. Don't surface this warning as a failure.
|
||||
|
||||
### Authentication for gated models
|
||||
|
||||
LLaMA, FLUX, and some Mistral and Qwen variants are gated behind HF agreements and require `huggingface-cli login` before download. If `optimum-cli` exits with a 401, run `huggingface-cli login` with a token from `huggingface.co/settings/tokens` and retry.
|
||||
|
||||
### If the export fails
|
||||
|
||||
The most common cause is a HF model using patterns `torch.export` / `torch.onnx.export` cannot trace — complex-number arithmetic, data-dependent control flow, non-tensor forward arguments, output dataclasses. **Do not patch the upstream library on disk.** The correct workflow is agentic monkey-patching at runtime: route the developer to the `trt-export-rewrite` skill, which drives it with five concrete patterns from the Qwen-Image case study.
|
||||
|
||||
## Step 3 — Sanitize the ONNX
|
||||
|
||||
ONNX exporters often produce graphs with constant-folding opportunities, spurious dynamic axes, or shape-inference gaps. Sanitizing cleans these up, but it is a **quality pass, not a correctness gate** — TRT parses many un-sanitized exports fine. Both stages below are therefore optional and **allowed to fail**: each falls through to its input, and Step 3 always leaves a `model.clean.onnx` for the rest of the workflow (worst case, a copy of the exported model). Run:
|
||||
|
||||
```bash
|
||||
# Seed model.clean.onnx with the raw export, then update it in-place
|
||||
cp bert_onnx/model.onnx bert_onnx/model.clean.onnx
|
||||
|
||||
# (1) onnxsim — graph simplification / constant folding.
|
||||
python3 -m onnxsim bert_onnx/model.clean.onnx bert_onnx/model.simplified.onnx \
|
||||
&& cp bert_onnx/model.simplified.onnx bert_onnx/model.clean.onnx
|
||||
|
||||
# (2) polygraphy surgeon — fold constants, prune dangling nodes.
|
||||
polygraphy surgeon sanitize bert_onnx/model.clean.onnx \
|
||||
-o bert_onnx/model.surgeon.onnx --fold-constants \
|
||||
&& cp bert_onnx/model.surgeon.onnx bert_onnx/model.clean.onnx
|
||||
```
|
||||
|
||||
## Step 3.5 — Inspect the ONNX inputs (parameterize the rest of the workflow)
|
||||
|
||||
**Every step from here uses input names and ranks that depend on the model.** Read them out of the sanitized ONNX before building. Do not assume `input_ids` / `attention_mask` — those are BERT-family-specific.
|
||||
|
||||
```bash
|
||||
polygraphy inspect model bert_onnx/model.clean.onnx
|
||||
```
|
||||
|
||||
Example output for `bert-base-uncased` (verified against `optimum-onnx` 0.1.0, opset 18):
|
||||
|
||||
```
|
||||
---- 3 Graph Input(s) ----
|
||||
{input_ids [dtype=int64, shape=('batch_size', 'sequence_length')],
|
||||
attention_mask [dtype=int64, shape=('batch_size', 'sequence_length')],
|
||||
token_type_ids [dtype=int64, shape=('batch_size', 'sequence_length')]}
|
||||
|
||||
---- 1 Graph Output(s) ----
|
||||
{last_hidden_state [dtype=float32, shape=('batch_size', 'sequence_length', 768)]}
|
||||
```
|
||||
|
||||
Inputs and outputs vary per model and per `--task`. Inspect, don't assume.
|
||||
|
||||
**Build the shape strings from this output.** Every input with a dynamic (named) axis needs a `--minShapes / --optShapes / --maxShapes` entry; fixed axes are reproduced literally. Use representative values — `--opt` is what the engine optimizes for; `--min` and `--max` define the supported range.
|
||||
|
||||
If any input reports an unknown-rank axis (printed as `?`), shape inference failed; reroute through `trt-export-rewrite` to fix the export.
|
||||
|
||||
## Step 4 — Bake FP16 into the model (TRT 11) or set the build flag (TRT 10)
|
||||
|
||||
Start with FP16 — the safe default for encoder and vision models on modern GPUs. (Use BF16 on Blackwell/Hopper if the developer reports persistent FP16 accuracy issues. Sentence-embedding models sometimes prefer FP32 over lossy FP16 — surface the option.)
|
||||
|
||||
**TRT 11 removed the weak-typing builder flags** (`--fp16`, `--int8`, `--bf16`, `BuilderFlag::kFP16`, etc. — see [the migration guide](https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x.html) and the sibling `trt-strong-typing-migration` SKILL). In TRT 11 you bake precision **into the model** before building, and `trtexec` honors the dtypes it finds via **ModelOpt AutoCast**.
|
||||
|
||||
Detect the version first:
|
||||
|
||||
```bash
|
||||
TRT_MAJOR=$(python3 -c "import tensorrt as t; print(t.__version__.split('.')[0])")
|
||||
echo "TensorRT major version: $TRT_MAJOR"
|
||||
```
|
||||
|
||||
### TRT 11: AutoCast the model, then build
|
||||
|
||||
```bash
|
||||
pip install nvidia-modelopt # one-time
|
||||
|
||||
# Convert FP32 ONNX to mixed FP16 (ModelOpt picks accuracy-sensitive ops to keep FP32):
|
||||
python3 -m modelopt.onnx.autocast \
|
||||
--onnx_path=<MODEL>.clean.onnx \
|
||||
--output_path=<MODEL>.fp16.onnx
|
||||
|
||||
# Build the engine. NO --fp16 flag — strong typing is the default and the model already carries FP16 dtypes.
|
||||
trtexec \
|
||||
--onnx=<MODEL>.fp16.onnx \
|
||||
--saveEngine=<MODEL>.fp16.plan \
|
||||
--memPoolSize=workspace:4096 \
|
||||
--minShapes=<INPUT1>:<MINSHAPE>,... \
|
||||
--optShapes=<INPUT1>:<OPTSHAPE>,... \
|
||||
--maxShapes=<INPUT1>:<MAXSHAPE>,...
|
||||
```
|
||||
|
||||
### TRT 10: pass the legacy --fp16 flag
|
||||
|
||||
```bash
|
||||
trtexec \
|
||||
--onnx=<MODEL>.clean.onnx \
|
||||
--saveEngine=<MODEL>.fp16.plan \
|
||||
--memPoolSize=workspace:4096 \
|
||||
--fp16 \
|
||||
--minShapes=<INPUT1>:<MINSHAPE>,... \
|
||||
--optShapes=<INPUT1>:<OPTSHAPE>,... \
|
||||
--maxShapes=<INPUT1>:<MAXSHAPE>,...
|
||||
```
|
||||
|
||||
Worked example for BERT (three inputs, all dynamic on batch and sequence axes), TRT 11 path:
|
||||
|
||||
```bash
|
||||
python3 -m modelopt.onnx.autocast \
|
||||
--onnx_path=bert_onnx/model.clean.onnx \
|
||||
--output_path=bert_onnx/model.fp16.onnx
|
||||
trtexec \
|
||||
--onnx=bert_onnx/model.fp16.onnx \
|
||||
--saveEngine=bert.fp16.plan \
|
||||
--memPoolSize=workspace:4096 \
|
||||
--minShapes=input_ids:1x32,attention_mask:1x32,token_type_ids:1x32 \
|
||||
--optShapes=input_ids:8x128,attention_mask:8x128,token_type_ids:8x128 \
|
||||
--maxShapes=input_ids:16x512,attention_mask:16x512,token_type_ids:16x512
|
||||
```
|
||||
|
||||
Worked example for ResNet-50 (one input, dynamic on batch axis only), TRT 11 path:
|
||||
|
||||
```bash
|
||||
python3 -m modelopt.onnx.autocast \
|
||||
--onnx_path=resnet50_onnx/model.clean.onnx \
|
||||
--output_path=resnet50_onnx/model.fp16.onnx
|
||||
trtexec \
|
||||
--onnx=resnet50_onnx/model.fp16.onnx \
|
||||
--saveEngine=resnet50.fp16.plan \
|
||||
--memPoolSize=workspace:4096 \
|
||||
--minShapes=pixel_values:1x3x224x224 \
|
||||
--optShapes=pixel_values:16x3x224x224 \
|
||||
--maxShapes=pixel_values:32x3x224x224
|
||||
```
|
||||
|
||||
For purely static-shape models, omit all three shape flags.
|
||||
|
||||
### Common errors at this step
|
||||
|
||||
- **"Engine plan file is generated on an incompatible device"** — plans are not portable across compute capabilities. Rebuild on the deployment GPU.
|
||||
- **OOM during build** — lower `--memPoolSize=workspace:N`. If still failing, try `--tacticSources=-CUBLAS_LT` to disable expensive tactic sources.
|
||||
- **Unsupported operator** — `trtexec` names the op. Route to `trt-unsupported-op` for triage (decompose / plugin / switch frontend / file bug).
|
||||
- **INT64 warnings** — TRT casts to INT32. Usually safe; if values exceed INT32 range, rerun Step 3's `polygraphy surgeon sanitize --fold-constants`.
|
||||
- **"Input <name> has missing shape information"** — you missed an input in the `--minShapes / --optShapes / --maxShapes` flags. Compare against Step 3.5's output and add the missing entry.
|
||||
|
||||
## Step 5 — Verify numerical parity
|
||||
|
||||
This is the step most developers skip — and the one that most often surfaces a silent correctness bug.
|
||||
|
||||
Construct the parity command from the same input names used in Step 4. Beyond input shapes, three knobs matter; get them right or the result is misleading:
|
||||
|
||||
1. **`--val-range`** for any input with constrained semantics. Polygraphy fills with `[0, 1)` floats by default. For BERT-family `input_ids` you must constrain to vocabulary range, set `attention_mask` to all-ones, and `token_type_ids` to all-zeros — otherwise the model sees garbage and FP16 noise amplifies through the attention softmax.
|
||||
2. **`--check-error-stat mean`** for transformer outputs. FP16 commonly has a few outlier positions with large abs diff; the mean across all positions is the meaningful metric for downstream use (embedding similarity, classification). Default `max` is too pessimistic for FP16 transformers and produces false failures.
|
||||
3. **`--seed`** for repeatability across runs and CI environments.
|
||||
|
||||
Worked example for BERT — run polygraphy against the **same ONNX file you fed to `trtexec`** (the AutoCast-converted `.fp16.onnx` on TRT 11; the `.clean.onnx` on TRT 10):
|
||||
|
||||
```bash
|
||||
# TRT 11: polygraphy reads the model's dtypes (already FP16 from AutoCast)
|
||||
polygraphy run bert_onnx/model.fp16.onnx \
|
||||
--trt --onnxrt \
|
||||
--atol 1e-2 --rtol 1e-2 \
|
||||
--check-error-stat mean \
|
||||
--input-shapes input_ids:[8,128] attention_mask:[8,128] token_type_ids:[8,128] \
|
||||
--val-range input_ids:[0,30000] attention_mask:[1,1] token_type_ids:[0,0] \
|
||||
--seed 42
|
||||
|
||||
# TRT 10: pass --fp16 explicitly (the polygraphy flag maps to BuilderFlag::kFP16)
|
||||
polygraphy run bert_onnx/model.clean.onnx \
|
||||
--trt --onnxrt \
|
||||
--atol 1e-2 --rtol 1e-2 \
|
||||
--check-error-stat mean \
|
||||
--input-shapes input_ids:[8,128] attention_mask:[8,128] token_type_ids:[8,128] \
|
||||
--val-range input_ids:[0,30000] attention_mask:[1,1] token_type_ids:[0,0] \
|
||||
--seed 42 \
|
||||
--fp16
|
||||
```
|
||||
|
||||
Expected output: `Pass Rate: 100.0%` with `mean_absdiff ≤ 1e-3` per output. `Pass Rate: 0.0%` with `max_absdiff` near 1e-2 but `mean_absdiff` well below tolerance is the outlier pattern above — use `--check-error-stat mean`.
|
||||
|
||||
Worked example for ResNet-50 (same FP16 recipe; TRT-11 path shown — for TRT 10 add `--fp16` and read from `model.clean.onnx`):
|
||||
|
||||
```bash
|
||||
polygraphy run resnet50_onnx/model.fp16.onnx \
|
||||
--trt --onnxrt \
|
||||
--atol 1e-2 --rtol 1e-2 \
|
||||
--check-error-stat mean \
|
||||
--input-shapes pixel_values:[16,3,224,224] \
|
||||
--val-range pixel_values:[0,1] \
|
||||
--seed 42
|
||||
```
|
||||
|
||||
Polygraphy runs the model through ONNX Runtime (reference) and TensorRT (engine) on the seeded inputs and compares outputs per the chosen error stat.
|
||||
|
||||
### If parity fails
|
||||
|
||||
- **Diverges only at FP16, not FP32**: a few layers are losing precision. Try `--strongly-typed` with an explicit FP32 cast on the offending subgraph (Polygraphy names the layer). For LLM-style models, prefer BF16 over FP16 on Hopper/Blackwell.
|
||||
- **Diverges at all precisions**: the export is wrong. Re-run Steps 2–3 fresh and check `polygraphy inspect model` for shape-inference issues. If the model uses patterns covered by `trt-export-rewrite`, reroute.
|
||||
- **Diverges only for specific input shapes**: dynamic-shape profile is too narrow. Widen `--minShapes` / `--maxShapes` and rebuild.
|
||||
|
||||
## Step 6 — Run the engine
|
||||
|
||||
```bash
|
||||
trtexec --loadEngine=bert.fp16.plan \
|
||||
--shapes=input_ids:8x128,attention_mask:8x128,token_type_ids:8x128 \
|
||||
--verbose
|
||||
```
|
||||
|
||||
This confirms the saved `.plan` deserializes and runs (Step 5's parity used its own engine, not this file) and reports throughput and per-iteration latency. For production, see the developer guide on `IExecutionContext` and stream-aware execution; the `.plan` is portable to any compatible GPU + TRT runtime.
|
||||
|
||||
## Success criteria
|
||||
|
||||
The skill is complete when all of the following are true:
|
||||
|
||||
- `bert.fp16.plan` exists and `trtexec --loadEngine=...` runs without error.
|
||||
- `polygraphy run --trt --onnxrt --atol 1e-2 --rtol 1e-2 --check-error-stat mean` reports `PASSED` across all configured input shapes.
|
||||
- The developer can articulate which precision was used and why.
|
||||
|
||||
If any are not true, hand control back with the specific failing diagnostic — do not declare success.
|
||||
|
||||
## References
|
||||
|
||||
- [Import Workflows Guide (Path 1: ONNX → TensorRT)](https://github.com/NVIDIA/TensorRT/blob/main/documents/import_workflows.md#path-1-onnx--tensorrt)
|
||||
- [Supported Models matrix](https://github.com/NVIDIA/TensorRT/blob/main/documents/supported_models.md)
|
||||
- [Polygraphy documentation](https://docs.nvidia.com/deeplearning/tensorrt/polygraphy/index.html)
|
||||
- [`samples/sampleOnnxMNIST/`](https://github.com/NVIDIA/TensorRT/tree/main/samples/sampleOnnxMNIST) for the C++ equivalent of the build flow
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
name: trt-perf-analysis
|
||||
description: Validate and analyze TensorRT performance data from paired layer-info JSON and profile/latency JSON files. Use when asked to inspect TensorRT, TRT, torch-tensorrt, or ONNX-TensorRT perf reports, verify that layer/profile JSON files are valid and from the same model, infer basic model information, find likely fusion or latency optimization opportunities, and produce a concise Markdown performance report or structured JSON data.
|
||||
---
|
||||
|
||||
# TRT Perf Analysis
|
||||
|
||||
## Workflow
|
||||
|
||||
Use `scripts/run.sh` on Unix-like systems or `scripts\run.cmd` on Windows for Python scripts. These wrappers do best-effort Python 3.8+ discovery; set `SKILL_PYTHON` to a Python executable to override discovery. Replace placeholders with platform-native paths.
|
||||
|
||||
1. Choose the input scope.
|
||||
Use one folder that directly contains `layers_*.json` and/or `profile_*.json`. Do not call the packager on a parent folder that only contains component subfolders. For model suites with separate encoder, transformer, decoder, VAE, or similar components, handle each component folder separately.
|
||||
|
||||
2. Infer model identity and components.
|
||||
Prefer an explicit model name from the user prompt and pass it with `--model-name`. Otherwise rely on the analyzer/packager to inspect likely config files, the input directory name, and layer/profile filenames. Keep the name empty when confidence is low. Serialized JSON records the inferred model identity at the top level and in each successful backend's `model` object.
|
||||
|
||||
3. Run deterministic analysis.
|
||||
Run `scripts/analyze_trt_perf.py` first when you need an integrity gate before interpretation. The script uses only Python built-in modules, extracts structured analysis data, and serializes it as JSON. This JSON is the authoritative analyzer output for validation, packaging, and AI diagnosis.
|
||||
|
||||
4. Check validation before interpreting.
|
||||
The analyzer exits `0` when it can emit structured validation data, even if one or more backends fail validation. Read the generated JSON `validation` object to decide which backend reports are usable. If the analyzer exits nonzero, stop because it could not emit structured data. If the schema validator exits nonzero, stop because the generated JSON contract is invalid. Do not continue into performance interpretation for a backend unless its validation status is `passed`, its analysis mode is `layer_profile`, the layer graph is a DAG, and the layer/profile names match. For `layer_only` backends, graph and layer inspection are available but latency/performance interpretation is not.
|
||||
|
||||
5. Understand the model, then write AI diagnosis.
|
||||
Use the generated JSON as grounding material. Build a clear picture of the model or component scope, backend differences, graph structure, timing distribution, hot layers, fusion clues, and caveats before writing conclusions. The browser `Summary` panel is generated from `analyze-data.json`; do not copy that deterministic summary into `analyze.md`.
|
||||
|
||||
Keep the final AI analysis in two sections:
|
||||
|
||||
1. `## Summary` - short, high-confidence facts and the most important optimization leads.
|
||||
2. `## Details` - interpretation, cross-backend comparisons, hypotheses, ranked experiments, and human judgment that is not a mechanical restatement of the analyzer output.
|
||||
|
||||
Before sending the final response, write the final AI analysis into `analyze.md` for each report folder you created or updated. Match each file to that report folder's scope. For a single model with multiple backend JSON pairs, one `analyze.md` may compare those backends. For multiple model components packaged as multiple report folders, write distinct component-focused `analyze.md` files; do not copy a suite-level analysis into every component report.
|
||||
|
||||
`analyze.md` should not include validation checklists, complete timing tables, top-layer tables, path dumps, or other content already rendered by the report app from `analyze-data.json`. Briefly cite specific metrics or layer names only when they support a new conclusion or next experiment.
|
||||
|
||||
6. Package the report.
|
||||
Run `scripts/package_report.py` through the shared runner. Each invocation creates exactly one final report folder from one input folder that directly contains `layers_*.json` or `profile_*.json` files. The packager writes `analyze-data.json`, validates it against the schema, and copies the browser report template while preserving the generated JSON. Determine the output parent from the user's active workspace, not from a subdirectory of this skill. If the only known directory is the skill directory, omit `--output-parent`; the packaging script will fall back to the system temp directory.
|
||||
|
||||
When producing multiple reports for one user request, prefer explicit `--report-dir` values so report folder names include the model/component identity instead of relying on auto-numbered collision suffixes. If a suite-level conclusion is useful, write it as a separate overview Markdown file in the output parent, not as every component report's `analyze.md`.
|
||||
|
||||
7. Preview only the final report.
|
||||
Serve the completed report folder, not the skill directory, when the user wants a browser preview. The Python `visualize_layer_info.py` script remains available for standalone HTML layer-info debugging, but it is not part of the report packaging flow or report app dependency chain.
|
||||
|
||||
### Common Commands
|
||||
|
||||
Use `<runner>` as `<skill-dir>/scripts/run.sh` on Unix-like systems or `<skill-dir>\scripts\run.cmd` on Windows.
|
||||
|
||||
```bash
|
||||
# Validate or inspect inputs. Folder mode discovers matching backends; repeat --data for explicit backends.
|
||||
<runner> <skill-dir>/scripts/analyze_trt_perf.py <model-folder> --output <analysis.json>
|
||||
<runner> <skill-dir>/scripts/analyze_trt_perf.py --data <layers-a.json> <profile-a.json> --data <layers-b.json> --output <analysis.json>
|
||||
|
||||
# Create a report. Use --report-dir for component-specific reports and --analyze-md after writing final AI analysis.
|
||||
<runner> <skill-dir>/scripts/package_report.py <model-folder> --output-parent <workspace-folder> [--model-name <model>] [--analyze-md <final-analysis.md>]
|
||||
<runner> <skill-dir>/scripts/package_report.py --analyze-data <report-folder>/analyze-data.json [--analyze-md <final-analysis.md>]
|
||||
|
||||
# Optional preview.
|
||||
<runner> -m http.server 8765 --bind 127.0.0.1 --directory <report-folder>
|
||||
```
|
||||
|
||||
Then open `http://127.0.0.1:8765/`.
|
||||
|
||||
## Inputs
|
||||
|
||||
Expect one layer-info JSON and one profile/latency JSON per engine/report. A folder may contain multiple pairs such as:
|
||||
|
||||
- `layers_torch-trt-aot.json` with `profile_torch-trt-aot.json`
|
||||
- `layers_onnx-tensorrt.json` with `profile_onnx-tensorrt.json`
|
||||
|
||||
Layer records are expected to include `Name`, `LayerType`, `Inputs`, and `Outputs`. Profile records are expected to include `name`, `timeMs`, `averageMs`, `medianMs`, and `percentage`; a first record like `{ "count": 20 }` is allowed.
|
||||
|
||||
Infer a per-layer `Type` for report output. Default to the raw `LayerType`, then apply narrow special-case overrides when the layer name, tactic, or metadata makes the specialization clear. Known special cases:
|
||||
|
||||
- `kgen_mha`: a raw `LayerType` of `kgen` that is actually an MHA/FMHA-style attention layer.
|
||||
- `misc`: raw `LayerType` values `reshape`, `shape_call`, `signal`, and `wait` are grouped into this category.
|
||||
|
||||
## Model Categories
|
||||
|
||||
Use this starter category list and revise as evidence warrants:
|
||||
|
||||
- Transformer encoder / sentence embedding
|
||||
- Transformer decoder / LLM
|
||||
- Vision transformer
|
||||
- CNN / convolutional vision model
|
||||
- Diffusion / U-Net style model
|
||||
- Recommender / ranking model
|
||||
- Classical ML or feature pipeline
|
||||
- Unknown DL model
|
||||
|
||||
Report category only when the evidence is strong. Good evidence includes Hugging Face config fields, names like `encoder.layer`, attention layers, input names such as `input_ids`, or repeated convolution/GEMM patterns. Say "unknown" instead of over-claiming.
|
||||
|
||||
## Analysis Heuristics
|
||||
|
||||
Prioritize issues by likely runtime impact and confidence:
|
||||
|
||||
- Hot layers: individual layers with high `percentage` or `averageMs`.
|
||||
- Layer-type concentration: high aggregate time in `kgen`, shape, cast, gather, reshape, or pointwise/reduction layers may indicate fusion or dynamic-shape overhead.
|
||||
- Fusion clues: for transformer encoders, check whether Q/K/V matmuls are fused per block, whether attention is an obvious fused SDPA/FMHA layer, and whether layer-norm/residual/GELU patterns are fused into larger kernels.
|
||||
- Engine partitioning: one engine is usually ideal. Use adjacent `.trt`/`.engine` files, `_subgraph`, and `StreamId` as hints; state when this is only inferred.
|
||||
- GEMM tactics: GEMM layers without Tensor Core or xMMA-like tactic names may deserve inspection.
|
||||
- Many tiny kernels: if no single layer dominates but many small kernels add up, rank launch/fusion overhead above isolated micro-hotspots.
|
||||
|
||||
Keep facts and hypotheses separate. For each optimization opportunity, include evidence from the JSON and a next check or experiment.
|
||||
@@ -0,0 +1,86 @@
|
||||
Third Party Notices
|
||||
===================
|
||||
|
||||
This report includes or loads third-party open source software.
|
||||
These notices are provided for attribution and license-tracking purposes only
|
||||
and do not modify the license terms of the third-party projects.
|
||||
|
||||
|
||||
Netron
|
||||
------
|
||||
|
||||
Project: Netron
|
||||
Homepage: https://github.com/lutzroeder/netron
|
||||
Version included in this report: 9.1.1
|
||||
License: MIT
|
||||
License URL: https://github.com/lutzroeder/netron/blob/main/LICENSE
|
||||
Copyright: Copyright (c) Lutz Roeder
|
||||
|
||||
Notes:
|
||||
- The Netron files are embedded as a trimmed report-specific viewer.
|
||||
- The embedded files have been modified for use in this TensorRT performance
|
||||
report, including report-specific loading, iframe integration, and layer
|
||||
selection behavior.
|
||||
|
||||
|
||||
Dagre
|
||||
-----
|
||||
|
||||
Project: Dagre
|
||||
Homepage: https://github.com/dagrejs/dagre
|
||||
License: MIT
|
||||
License URL: https://github.com/dagrejs/dagre/blob/master/LICENSE
|
||||
Copyright: Copyright (c) 2012-2014 Chris Pettitt
|
||||
|
||||
Notes:
|
||||
- The bundled graph layout code is included through the report-specific Netron
|
||||
viewer bundle.
|
||||
|
||||
|
||||
Graphlib
|
||||
--------
|
||||
|
||||
Project: Graphlib
|
||||
Homepage: https://github.com/dagrejs/graphlib
|
||||
License: MIT
|
||||
Package metadata URL: https://github.com/dagrejs/graphlib/blob/master/package.json
|
||||
Author: Chris Pettitt
|
||||
|
||||
Notes:
|
||||
- Dagre depends on Graphlib. The bundled layout code in dev/report/netron/dagre.js may
|
||||
include or derive from Graphlib functionality.
|
||||
|
||||
|
||||
DOMPurify
|
||||
---------
|
||||
|
||||
Project: DOMPurify
|
||||
Homepage: https://github.com/cure53/DOMPurify
|
||||
Version included in this report: 3.4.9
|
||||
License: MPL-2.0 OR Apache-2.0
|
||||
License option used for this notice: Apache-2.0
|
||||
License URL: https://github.com/cure53/DOMPurify/blob/3.4.9/LICENSE
|
||||
Package metadata URL: https://github.com/cure53/DOMPurify/blob/3.4.9/package.json
|
||||
Author: Dr.-Ing. Mario Heiderich, Cure53
|
||||
|
||||
Notes:
|
||||
- DOMPurify is used at runtime to sanitize rendered Markdown content.
|
||||
|
||||
|
||||
Marked
|
||||
------
|
||||
|
||||
Project: marked
|
||||
Homepage: https://github.com/markedjs/marked
|
||||
Version included in this report: 18.0.5
|
||||
License: MIT
|
||||
License URL: https://github.com/markedjs/marked/blob/v18.0.5/LICENSE.md
|
||||
Package metadata URL: https://github.com/markedjs/marked/blob/v18.0.5/package.json
|
||||
Copyright:
|
||||
- Copyright (c) 2018-2026, MarkedJS
|
||||
- Copyright (c) 2011-2018, Christopher Jeffrey
|
||||
|
||||
Notes:
|
||||
- marked is used at runtime to render Markdown content.
|
||||
- The marked license file also includes attribution and license terms for
|
||||
Markdown by John Gruber. See the upstream license URL above.
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,11 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
|
||||
<title>TRT donut T v3</title>
|
||||
<rect width="32" height="32" rx="6" fill="#f5f7fb"/>
|
||||
<g transform="rotate(-90 16 16)" fill="none" stroke-width="6" stroke-linecap="butt">
|
||||
<circle cx="16" cy="16" r="11" pathLength="100" stroke="#2563eb" stroke-dasharray="50 50" stroke-dashoffset="0"/>
|
||||
<circle cx="16" cy="16" r="11" pathLength="100" stroke="#16a34a" stroke-dasharray="22 78" stroke-dashoffset="-50"/>
|
||||
<circle cx="16" cy="16" r="11" pathLength="100" stroke="#d97706" stroke-dasharray="28 72" stroke-dashoffset="-72"/>
|
||||
</g>
|
||||
<circle cx="16" cy="16" r="6.5" fill="#f5f7fb"/>
|
||||
<path d="M11 11h10v3h-3.5v8h-3v-8H11Z" fill="#1d2433"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 716 B |
@@ -0,0 +1,183 @@
|
||||
<!doctype html>
|
||||
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
|
||||
<!-- SPDX-License-Identifier: Apache-2.0 -->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>TRT Performance Report</title>
|
||||
<link rel="icon" type="image/svg+xml" href="favicon-donut-t-v3.svg">
|
||||
<link rel="stylesheet" href="styles.css">
|
||||
</head>
|
||||
<body>
|
||||
<div class="page">
|
||||
<header class="topbar" id="topbar">
|
||||
<div class="title-block">
|
||||
<p class="eyebrow">TensorRT Analyze Report</p>
|
||||
<div class="title-line">
|
||||
<h1 id="report-title">TRT Performance Report</h1>
|
||||
<span class="backend-chip" id="backend-chip">Backend</span>
|
||||
</div>
|
||||
<p class="subtitle" id="report-subtitle"></p>
|
||||
<div class="pinned-summary" id="pinned-summary" aria-label="Pinned basic report information"></div>
|
||||
</div>
|
||||
<div class="status-stack">
|
||||
<div class="status-pill" id="status-pill">
|
||||
<span class="status-dot"></span>
|
||||
<span id="status-text">Loading data</span>
|
||||
</div>
|
||||
<div class="graph-status" id="graph-status">Loading graph</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<nav class="result-tabs" id="result-tabs" aria-label="Analysis results"></nav>
|
||||
|
||||
<section class="metrics" id="metrics" aria-label="Performance metrics"></section>
|
||||
|
||||
<details class="panel summary-panel analyze-panel" id="analyze-panel" aria-labelledby="analyze-title" open hidden>
|
||||
<summary class="summary-panel-header" aria-label="Toggle AI-generated analysis">
|
||||
<span class="details-toggle" aria-hidden="true"></span>
|
||||
<span class="summary-panel-title" id="analyze-title">Analysis (generated by AI)</span>
|
||||
<button class="panel-help-button" type="button" aria-label="Show help for Analysis generated by AI" aria-expanded="false" aria-controls="panel-help-popover" data-help-title="Analysis (generated by AI)" data-help="Generated by AI from the validated report data.">?</button>
|
||||
</summary>
|
||||
<div class="markdown-content" id="analyze-content"></div>
|
||||
</details>
|
||||
|
||||
<details class="panel summary-panel" aria-labelledby="summary-title">
|
||||
<summary class="summary-panel-header" aria-label="Toggle script-generated analysis">
|
||||
<span class="details-toggle" aria-hidden="true"></span>
|
||||
<span class="summary-panel-title" id="summary-title">Analysis (generated by script)</span>
|
||||
<button class="panel-help-button" type="button" aria-label="Show help for Analysis generated by script" aria-expanded="false" aria-controls="panel-help-popover" data-help-title="Analysis (generated by script)" data-help="Generated by script from the validated report data.">?</button>
|
||||
</summary>
|
||||
<div class="markdown-content" id="summary-content"></div>
|
||||
</details>
|
||||
|
||||
<details class="panel details-panel" aria-labelledby="findings-title">
|
||||
<summary class="details-summary" aria-label="Toggle optimization leads and positive signals">
|
||||
<span class="details-toggle" aria-hidden="true"></span>
|
||||
<span class="details-panel-title" id="findings-title">Optimization Leads & Positive Signals</span>
|
||||
<button class="panel-help-button" type="button" aria-label="Show help for Optimization Leads and Positive Signals" aria-expanded="false" aria-controls="panel-help-popover" data-help-title="Optimization Leads & Positive Signals" data-help="Generated by script from the validated report data.">?</button>
|
||||
</summary>
|
||||
<div class="details-grid">
|
||||
<article class="list-card">
|
||||
<h2>Optimization Leads</h2>
|
||||
<ul class="finding-list" id="issues"></ul>
|
||||
</article>
|
||||
<article class="list-card">
|
||||
<h2>Positive Signals</h2>
|
||||
<ul class="finding-list" id="positives"></ul>
|
||||
<p class="path-line" id="paths"></p>
|
||||
</article>
|
||||
</div>
|
||||
</details>
|
||||
|
||||
<main class="analysis-shell">
|
||||
<div class="report-column">
|
||||
<section class="panel workspace" aria-label="Layer performance report">
|
||||
<section class="chart-panel" aria-labelledby="chart-title">
|
||||
<div class="panel-header">
|
||||
<div>
|
||||
<div class="panel-title-line">
|
||||
<h2 class="panel-title" id="chart-title">Layers</h2>
|
||||
<button class="panel-help-button" type="button" aria-label="Show help for Layers" aria-expanded="false" aria-controls="panel-help-popover" data-help-title="Layers" data-help="Layer latency/type breakdown and searchable layer table.">?</button>
|
||||
</div>
|
||||
<p class="panel-kicker" id="chart-kicker"></p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="type-summary" id="type-summary"></div>
|
||||
<div class="chart-body">
|
||||
<div class="chart-wrap">
|
||||
<svg class="pie" id="type-pie" viewBox="0 0 320 320" role="img" aria-labelledby="chart-title"></svg>
|
||||
</div>
|
||||
<div class="legend" id="legend"></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="layers-panel" aria-labelledby="layers-title">
|
||||
<div class="table-toolbar">
|
||||
<div class="table-title">
|
||||
<strong id="layers-title">Layers</strong>
|
||||
<span id="layers-count"></span>
|
||||
</div>
|
||||
<label class="sr-only" for="layer-filter">Filter layers</label>
|
||||
<input class="filter-input" id="layer-filter" type="search" placeholder="Filter layers" autocomplete="off">
|
||||
</div>
|
||||
<div class="table-scroll">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">
|
||||
<button class="table-sort-button" type="button" data-layer-sort-key="topo">
|
||||
<span>Topo</span>
|
||||
<span class="sort-indicator" aria-hidden="true"></span>
|
||||
</button>
|
||||
</th>
|
||||
<th scope="col">Layer</th>
|
||||
<th scope="col">
|
||||
<button class="table-sort-button" type="button" data-layer-sort-key="avg">
|
||||
<span>Avg ms</span>
|
||||
<span class="sort-indicator" aria-hidden="true"></span>
|
||||
</button>
|
||||
</th>
|
||||
<th scope="col">
|
||||
<button class="table-sort-button" type="button" data-layer-sort-key="total">
|
||||
<span>Total ms</span>
|
||||
<span class="sort-indicator" aria-hidden="true"></span>
|
||||
</button>
|
||||
</th>
|
||||
<th scope="col">
|
||||
<button class="table-sort-button" type="button" data-layer-sort-key="share">
|
||||
<span>Share</span>
|
||||
<span class="sort-indicator" aria-hidden="true"></span>
|
||||
</button>
|
||||
</th>
|
||||
<th scope="col">Tags</th>
|
||||
<th scope="col">
|
||||
<button class="table-sort-button" type="button" data-layer-sort-key="stream">
|
||||
<span>Stream</span>
|
||||
<span class="sort-indicator" aria-hidden="true"></span>
|
||||
</button>
|
||||
</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="layer-rows"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div class="empty" id="empty-state" hidden>No matching layers</div>
|
||||
</section>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<aside class="panel graph-panel" aria-labelledby="graph-title">
|
||||
<div class="panel-header graph-header">
|
||||
<div>
|
||||
<div class="panel-title-line">
|
||||
<h2 class="panel-title" id="graph-title">Model Graph</h2>
|
||||
<button class="panel-help-button" type="button" aria-label="Show help for Model Graph" aria-expanded="false" aria-controls="panel-help-popover" data-help-title="Model Graph" data-help="Visual model graph linked to selected layers.">?</button>
|
||||
</div>
|
||||
<p class="panel-kicker" id="graph-kicker">Rows with matching graph nodes can be selected.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="netron-views" id="netron-views"></div>
|
||||
</aside>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<div class="panel-help-popover" id="panel-help-popover" role="tooltip" hidden>
|
||||
<strong id="panel-help-popover-title"></strong>
|
||||
<p id="panel-help-popover-text"></p>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
window.trtReportOptions = {
|
||||
dataName: 'analyze-data.json',
|
||||
analyzeName: 'analyze.md',
|
||||
graphMode: 'onnx-json',
|
||||
modelName: 'model.trt_inspect.onnx.json',
|
||||
netronBase: 'netron/',
|
||||
netronFrameMode: 'srcdoc'
|
||||
};
|
||||
</script>
|
||||
<script src="app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,596 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="description" content="Visualizer for neural network, deep learning and machine learning models." />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, viewport-fit=cover">
|
||||
<meta http-equiv="Content-Security-Policy" content="script-src 'self'">
|
||||
<meta name="version" content="9.1.1">
|
||||
<meta name="date" content="2026-06-05 14:43:12">
|
||||
<title>Netron</title>
|
||||
<link rel="stylesheet" type="text/css" href="grapher.css">
|
||||
<link rel="icon" href="data:,">
|
||||
<script type="text/javascript" src="netron.js"></script>
|
||||
<style>
|
||||
html { touch-action: none; overflow: hidden; width: 100%; height: 100%; -ms-text-size-adjust: 100%; -webkit-text-size-adjust: 100%; text-rendering: optimizeLegibility; -webkit-text-rendering: optimizeLegibility; -moz-text-rendering: optimizeLegibility; -ms-text-rendering: optimizeLegibility; -o-text-rendering: optimizeLegibility; -webkit-font-smoothing: antialiased; -moz-font-smoothing: antialiased; -ms-font-smoothing: antialiased; -o-font-smoothing: antialiased; }
|
||||
body { touch-action: none; overflow: hidden; width: 100%; height: 100%; margin: 0; font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", "Ubuntu", "Droid Sans", sans-serif, "PingFang SC"; font-size: 12px; text-rendering: geometricPrecision; }
|
||||
button { font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", "Ubuntu", "Droid Sans", sans-serif, "PingFang SC"; }
|
||||
.center { position: absolute; margin: auto; top: 0; right: 0; bottom: 0; left: 0; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.select { user-select: text; -webkit-user-select: text; -moz-user-select: text; }
|
||||
.target { display: flex; height: 100%; width: 100%; overflow: auto; outline: none; touch-action: pan-x pan-y; }
|
||||
.canvas { margin: auto; flex-shrink: 0; text-rendering: geometricPrecision; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.open-file-dialog { display: none; }
|
||||
.default { background-color: #ffffff; }
|
||||
.default .logo { display: none; }
|
||||
.default .target { display: flex; opacity: 1; }
|
||||
.default .toolbar { display: flex; }
|
||||
.toolbar { display: flex; align-items: center; position: absolute; bottom: 10px; left: 10px; padding: 0; margin: 0; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.toolbar button:focus { outline: 0; }
|
||||
.toolbar-button { background: None; border-radius: 6px; border: 0; margin: 0; margin-right: 1px; padding: 0; fill: None; stroke: #777; cursor: pointer; width: 32px; height: 32px; user-select: none; }
|
||||
.toolbar-path { display: flex; align-items: center; }
|
||||
.toolbar-select { background: transparent; position: relative; width: 170px; margin: 4px 4px 4px 4px; }
|
||||
.toolbar-select select { width: 100%; appearance: none; -webkit-appearance: none; -moz-appearance: none; font-family: inherit; font-size: 12px; line-height: 16px; border: 1px solid; padding: 3px 18px 3px 10px; border-radius: 6px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; box-sizing: border-box; }
|
||||
.toolbar-select select { background: #777; border-color: #777; color: #fff; }
|
||||
.toolbar-select-arrow { position: absolute; top: 50%; right: 8px; transform: translateY(-50%); pointer-events: none; }
|
||||
.toolbar-select-arrow { fill: #ffffff; }
|
||||
.toolbar-select select:hover { background: #000000; border-color: #000000; }
|
||||
.toolbar-select select:focus { outline: 0; }
|
||||
.toolbar-path-back-button { background: #777; border-top-left-radius: 6px; border-bottom-left-radius: 6px; border: 1px solid; border-color: #777; margin: 4px 0px 4px 4px; padding: 5px 8px 5px 8px; cursor: pointer; color: #ffffff; font-size: 12px; line-height: 12px; transition: 0.1s; }
|
||||
.toolbar-path-back-button:hover { background: #000000; border-color: #000000; }
|
||||
.toolbar-path-name-button { background: #777; border: 0px; border-color: #777; color: #ffffff; border-left: 1px; border-left-color: #ffffff; margin: 4px 0 4px 1px; padding: 6px 8px 6px 8px; cursor: pointer; width: auto; font-size: 12px; line-height: 12px; transition: 0.1s; }
|
||||
.toolbar-path-name-button:hover { background: #000000; border-color: #000000; }
|
||||
.toolbar-path-name-button:last-child { border-top-right-radius: 6px; border-bottom-right-radius: 6px; }
|
||||
.toolbar-icon .border { stroke: #fff; }
|
||||
.toolbar-icon .stroke { stroke: #808080; }
|
||||
.toolbar-icon .fill { fill: #808080; }
|
||||
.toolbar-icon:hover .stroke { stroke: #000000; }
|
||||
.toolbar-icon:hover .fill { fill: #000000; }
|
||||
.message { display: none; opacity: 0; position: absolute; top: 0; left: 0; width: 100%; height: 100%; flex-direction: column; justify-content: flex-start; }
|
||||
.message-text { display: inline; text-align: center; width: 562px; font-size: 13px; line-height: 20px; margin-top: 50vh; padding-top: 56px; padding-bottom: 20px; margin-left: auto; margin-right: auto; user-select: text; -webkit-user-select: text; -moz-user-select: text; }
|
||||
.welcome .message-text a { text-decoration: none; color: #666666; }
|
||||
.welcome .message-text a:visited { color: inherit; }
|
||||
.welcome .message-text a:hover { color: #242424; text-decoration: underline; }
|
||||
.message-button { display: inline; text-align: center; width: 125px; margin-left: auto; margin-right: auto; }
|
||||
.logo-text { top: -57px; width: 582px; transition: 0.1s; }
|
||||
.logo-name { top: -170px; width: 582px; transition: 0.1s; }
|
||||
.logo-icon { left: 248px; top: -18px; width: 106px; height: 106px; transition: 0.1s; }
|
||||
.logo-spinner { left: 248px; top: -18px; width: 106px; height: 106px; display: none; }
|
||||
.logo-stroke { stroke: #444444; }
|
||||
.logo-fill { fill: #444444; }
|
||||
.logo-border { stroke: #555555; }
|
||||
.logo-glyph { fill: #444444; }
|
||||
.logo-button { font-size: 12px; font-weight: bold; line-height: 1.25; text-align: center; vertical-align: middle; min-width: 5em; height: 2.7em; border-radius: 1.3em; transition: 0.1s; user-select: none; -webkit-user-select: none; -moz-user-select: none; color: #444444; background-color: #ececec; border: 1px solid #444444; }
|
||||
.logo-button:hover { color: #ececec; background-color: #444444; cursor: pointer; transition: 0.2s; }
|
||||
.logo-button:focus { outline: 0; }
|
||||
.logo-message { display: none; height: 0px; }
|
||||
.logo-github { display: none; }
|
||||
.open-file-button { top: 170px; left: 0px; width: 10.5em; }
|
||||
.progress { top: 120px; height: 2px; width: 400px; }
|
||||
.progress-bar { height: 100%; width: 0%; background-color: #444444; }
|
||||
.notification .logo-name { display: none; }
|
||||
.notification .open-file-button { display: none; }
|
||||
.notification .progress { display: none; }
|
||||
.welcome body { background-color: #ececec; }
|
||||
.welcome { background-color: #ececec; color: #242424; }
|
||||
.welcome .message-text { display: none; opacity: 0; }
|
||||
.welcome .message-button { display: none; opacity: 0; }
|
||||
.welcome .target { display: none; opacity: 0; }
|
||||
.welcome .menu { background-color: #ffffff; }
|
||||
.welcome.spinner .logo-spinner { display: block; -webkit-animation: orbit 0.5s infinite linear; animation: orbit 0.5s infinite linear; cursor: wait; }
|
||||
.welcome.spinner .menu-button { display: none; }
|
||||
.welcome.notification .menu-button { display: none; }
|
||||
.notification body { background-color: #ececec; }
|
||||
.notification .message { display: flex; opacity: 1; }
|
||||
.notification .message-text { display: inline; opacity: 1; }
|
||||
.notification .message-button { display: inline; opacity: 1; }
|
||||
.alert { background-color: #ececec; color: #242424; }
|
||||
.alert .target { display: none; opacity: 0; }
|
||||
.alert .toolbar { display: none; opacity: 0; }
|
||||
.alert .menu { display: none; opacity: 0; }
|
||||
.alert .logo { display: none; opacity: 0; }
|
||||
.alert .message { display: flex; opacity: 1; }
|
||||
.alert .message-text { display: inline; opacity: 1; width: 50%; padding-top: 0px; }
|
||||
.alert .message-button { display: inline; opacity: 1; }
|
||||
.about { overflow: hidden; }
|
||||
.about .toolbar { display: none; }
|
||||
.about .logo { display: block; background-color: #ececec; color: #666666; }
|
||||
.about .logo-message { display: block; top: 132px; font-size: 14px; }
|
||||
.about .logo-github { display: block; top: 340px; width: 48px; height: 48px; }
|
||||
.about a { text-decoration: none; color: #666666; }
|
||||
.about a:visited { color: inherit; }
|
||||
.about a:hover { color: #242424; }
|
||||
.about .open-file-button { display: none; }
|
||||
.about .logo-name { display: none; }
|
||||
.about .notification { display: none; }
|
||||
.about .progress { display: none; }
|
||||
.about .menu-button { display: none; }
|
||||
.titlebar { color: #aaaaaa; display: none; height: 32px; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 2; -webkit-app-region: drag; }
|
||||
.titlebar-visible { display: block; }
|
||||
.titlebar-content { display: block; padding: 0 142px; height: 100%; text-align: center; font-size: 14px; line-height: 32px; transition: all .1s ease-in-out; user-select: none; }
|
||||
.titlebar-content-text { display: block; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.spinner .titlebar-content { opacity: 0; }
|
||||
.active .titlebar { color: #464646; transition: all 0.05s ease-in-out; }
|
||||
.titlebar-control-box { display: none; align-items: center; flex-direction: row-reverse; height: 100%; position: absolute; top: 0; right: 0; width: 138px; }
|
||||
.titlebar-control-box-visible { display: flex; }
|
||||
.titlebar-icon { width: 1em; height: 1em; vertical-align: -0.15em; fill: currentColor; overflow: hidden; }
|
||||
.titlebar-button { display: flex; justify-content: center; align-items: center; width: 46px; height: 32px; user-select: none; -webkit-app-region: no-drag; }
|
||||
.titlebar-button:hover { color: #000000; background-color: rgba(0, 0, 0, 0.15); }
|
||||
.titlebar-button-close:hover { color: #ffffff; background-color: #b43029; }
|
||||
.menu-button { display: flex; justify-content: center; align-items: center; color: #aaaaaa; font-size: 20px; height: 32px; width: 32px; position: fixed; top: 0; left: 0; right: 0; bottom: 0; z-index: 2; -webkit-app-region: no-drag; -webkit-app-region: no-drag; user-select: none; }
|
||||
.menu-button:hover { color: #000000; }
|
||||
.menu { display: block; position: absolute; left: -17em; width: 17em; top: 0; height: 100%; z-index: 2; background-color: #ececec; border-right: 1px solid rgba(255, 255, 255, 0.5); padding-top: 40px; padding-bottom: 2px; margin-left: 0; margin-top: 0; overflow: hidden; transition: 0.1s; }
|
||||
.menu .menu-group { margin-bottom: 12px; }
|
||||
.menu .menu-group .menu-group-header { display: block; border: none; border-radius: 0; color: black; width: 100%; text-align: left; margin: 4px 12px 5px 12px; white-space: no-wrap; font-size: 11px; font-weight: bold; color: #bbbbbb; white-space: nowrap; }
|
||||
.menu .menu-group .menu-command { display: block; border: none; border-radius: 0; background-color: transparent; color: black; width: 100%; text-align: left; padding: 4px 12px 5px 12px; font-size: 12px; }
|
||||
.menu .menu-group .menu-command:focus { color: #ffffff; background-color: #2e6bd2; outline: none; }
|
||||
.menu .menu-group .menu-command:disabled { color: #888888; }
|
||||
.menu .menu-group .menu-command .menu-label { display: block; overflow: hidden; white-space: nowrap; text-overflow: ellipsis; }
|
||||
.menu .menu-group .menu-command .menu-shortcut { display: block; float: right; margin-left: 25px; color: #888888; }
|
||||
.menu .menu-group .menu-separator { border-top: 1px; border-bottom: 0; border-style: solid; border-color: #e5e5e5; margin-left: 12px; margin-right: 12px; }
|
||||
.about .titlebar-visible { opacity: 0; }
|
||||
@-webkit-keyframes orbit { 0% { -webkit-transform: rotate(0deg); transform: rotate(0deg); } 100% { -webkit-transform: rotate(360deg); transform: rotate(360deg); } }
|
||||
@keyframes orbit { 0% { -webkit-transform: rotate(0deg); transform: rotate(0deg); } 100% { -webkit-transform: rotate(360deg); transform: rotate(360deg); } }
|
||||
.welcome.spinner .logo-spinner-stroke { stroke: #ececec; }
|
||||
.welcome.spinner .logo-name { display: none; }
|
||||
.welcome.spinner .open-file-button { display: none; }
|
||||
.welcome.spinner .target { display: flex; opacity: 0; }
|
||||
.welcome .notification .logo-name { display: none; }
|
||||
.welcome .toolbar { display: none; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root { color-scheme: dark; }
|
||||
.default { background-color: #404040; }
|
||||
.target { background-color: #404040; }
|
||||
.welcome { background-color: #1e1e1e; color: #888888; }
|
||||
.alert { background-color: #1e1e1e; color: #888888; }
|
||||
.logo-stroke { stroke: #888888; }
|
||||
.logo-fill { fill: #888888; }
|
||||
.logo-border { stroke: #000000; }
|
||||
.logo-glyph { fill: #888888; }
|
||||
.logo-spinner-stroke { stroke: #ffffff; }
|
||||
.logo-button { color: #888888; background-color: #1e1e1e; border-color: #888888; }
|
||||
.logo-button:hover { color: #1e1e1e; background-color: #888888; }
|
||||
.welcome .progress-bar { background-color: #888888; }
|
||||
.welcome .menu { background-color: #2d2d2d }
|
||||
.about .logo { background-color: #1e1e1e; color: #888888; }
|
||||
.about a { color: #c6c6c6; }
|
||||
.about a:hover { color: #565656; }
|
||||
.welcome .message-text a { color: #c6c6c6; }
|
||||
.welcome .message-text a:visited { color: #c6c6c6; }
|
||||
.welcome .message-text a:hover { color: #565656; }
|
||||
.toolbar-icon .border { stroke: #333333; }
|
||||
.toolbar-icon .stroke { stroke: #aaaaaa; }
|
||||
.toolbar-icon .fill { fill: #aaaaaa; }
|
||||
.toolbar-icon:hover .stroke { stroke: #dfdfdf; }
|
||||
.toolbar-icon:hover .fill { fill: #dfdfdf; }
|
||||
.toolbar-path-back-button { background: #aaaaaa; border-color: #aaaaaa; color: #333333; }
|
||||
.toolbar-path-back-button:hover { background: #dfdfdf; border-color: #dfdfdf; }
|
||||
.toolbar-path-name-button { background: #aaaaaa ; border-color: #aaaaaa; color: #404040; }
|
||||
.toolbar-path-name-button:hover { background: #dfdfdf; border-color: #dfdfdf; }
|
||||
.toolbar-select select { background: #aaaaaa; border-color: #aaaaaa; color: #404040; }
|
||||
.toolbar-select select:hover { background: #dfdfdf; border-color: #dfdfdf; color: #404040; }
|
||||
.toolbar-select-arrow { fill: #404040; }
|
||||
.titlebar { color: #949494; }
|
||||
.welcome body { background-color: #1e1e1e; }
|
||||
.default body { background-color: #404040; }
|
||||
.active .titlebar { color: #c4c4c4; }
|
||||
.titlebar-button:hover { color: #ffffff; background-color: rgba(0, 0, 0, 0.15); }
|
||||
.titlebar-button-close:hover { color: #ffffff; background-color: #b43029; }
|
||||
.menu-button { color: #aaaaaa; }
|
||||
.menu-button:hover { color: #ffffff; }
|
||||
.menu { background-color: #2d2d2d; border-color: rgba(0, 0, 0, 0); }
|
||||
.menu .menu-group .menu-group-header { color: #666666; }
|
||||
.menu .menu-group .menu-command { color: #ffffff; }
|
||||
.menu .menu-group .menu-command:focus { color: #ffffff; background-color: #2e6bd2; }
|
||||
.menu .menu-group .menu-command:disabled { color: #888888; }
|
||||
.menu .menu-group .menu-command .shortcut { color: #888888; }
|
||||
.menu .menu-group .menu-separator { border-color: #363636; }
|
||||
}
|
||||
@media all and (max-width: 640px) {
|
||||
.logo { width: 240px; }
|
||||
.logo-text { opacity: 0; }
|
||||
.logo-name { opacity: 0; }
|
||||
.logo-icon { left: 0; width: 128px; height: 128px; }
|
||||
.logo-spinner { left: 0; width: 128px; height: 128px; }
|
||||
.logo .open-file-button { top: 204px; left: 0; }
|
||||
.message-text { padding-top: 68px; width: 320px; }
|
||||
.progress { top: 160px; height: 2px; width: 100px; }
|
||||
.about .logo { width: 100%; padding-left: 0; padding-right: 0; }
|
||||
.about .logo-message { top: 175px; font-size: 12px; }
|
||||
.about .logo-github { top: 370px; }
|
||||
}
|
||||
.sidebar { display: flex; flex-direction: column; font-family: -apple-system, BlinkMacSystemFont, "Segoe WPC", "Segoe UI", "Ubuntu", "Droid Sans", sans-serif; font-size: 12px; height: 100%; right: -100%; position: fixed; transition: 0.1s; top: 0; background-color: #ececec; color: #242424; overflow: hidden; border-left: 1px solid rgba(255, 255, 255, 0.5); opacity: 0; }
|
||||
.sidebar-title { font-weight: bold; font-size: 12px; letter-spacing: 0.5px; text-transform: uppercase; height: 20px; margin: 0; padding: 20px; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.sidebar-closebutton { padding: 8px 8px 8px 32px; text-decoration: none; font-size: 25px; color: #777777; opacity: 1.0; display: block; transition: 0.2s; position: absolute; top: 0; right: 15px; margin-left: 50px; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.sidebar-closebutton:hover { color: #242424; }
|
||||
.sidebar-content { display: flex; flex-direction: column; flex-grow: 1; height: 0; }
|
||||
.sidebar-header { font-weight: bold; font-size: 12px; letter-spacing: 0.5px; text-transform: uppercase; height: 20px; margin: 0; margin-top: 30px; padding-top: 10px; padding-bottom: 10px; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.sidebar-section { font-weight: bold; font-size: 11px; text-transform: uppercase; line-height: 1.25; margin-top: 16px; margin-bottom: 16px; display: block; user-select: none; -webkit-user-select: none; -moz-user-select: none; cursor: default; }
|
||||
.sidebar-object { flex-grow: 1; padding: 0px 20px 20px 20px; overflow-y: auto; }
|
||||
.sidebar-item { margin-bottom: 0px; display: block; }
|
||||
.sidebar-item-name { float: left; font-size: 11px; min-width: 95px; max-width: 95px; padding-right: 5px; padding-top: 7px; display: block; }
|
||||
.sidebar-item-name input { color: #777; font-family: inherit; font-size: inherit; color: inherit; background-color: inherit; width: 100%; text-align: right; margin: 0; padding: 0; border: 0; outline: none; text-overflow: ellipsis; }
|
||||
.sidebar-item-value-list { margin: 0; margin-left: 105px; overflow: hidden; display: block; padding: 0; }
|
||||
.sidebar-item-value { font-size: 11px; background-color: #fcfcfc; border-radius: 2px; border: 1px solid #fcfcfc; margin-top: 3px; margin-bottom: 3px; overflow: auto; }
|
||||
.sidebar-item-value-content { background-color: #f8f8f8; border: 1px solid #f8f8f8; }
|
||||
.sidebar-item-value b { font-weight: bold; }
|
||||
.sidebar-item-value code { font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
|
||||
.sidebar-item-value pre { font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace; margin: 0; overflow: auto; white-space: pre; word-wrap: normal; display: block; }
|
||||
.sidebar-item-value-line { padding: 4px 6px 4px 6px; overflow-x: auto; white-space: pre; }
|
||||
.sidebar-item-value-line-wrap { white-space: pre-wrap; overflow-x: hidden; overflow-wrap: break-word; }
|
||||
.sidebar-item-value-line-break { padding: 4px 6px 4px 6px; overflow-x: auto; white-space: pre; }
|
||||
.sidebar-item-value-line-link { padding: 4px 6px 4px 6px; cursor: default; overflow-x: auto; white-space: nowrap; }
|
||||
.sidebar-item-value-line-link:hover { text-decoration: underline; }
|
||||
.sidebar-item-value-line-border { padding: 4px 6px 4px 6px; border-top: 1px solid rgba(27, 31, 35, 0.05); }
|
||||
.sidebar-item-value-line-content { white-space: pre; word-wrap: normal; overflow: auto; display: block; }
|
||||
.sidebar-item-value-expander { font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace; float: right; color: #aaa; cursor: pointer; user-select: none; -webkit-user-select: none; -moz-user-select: none; padding: 4px 6px 4px 4px; }
|
||||
.sidebar-item-value-expander:hover { color: #333; }
|
||||
.sidebar-item-value-button { display: flex; justify-content: center; align-items: center; font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace; width: 22px; height: 22px; cursor: pointer; user-select: none; -webkit-user-select: none; -moz-user-select: none; color: #aaa; }
|
||||
.sidebar-item-value-button svg use { fill: #aaa; stroke: #aaa; }
|
||||
.sidebar-item-value-button:hover svg use { fill: #333; stroke: #333; }
|
||||
.sidebar-item-value-button-tool { float: right; padding-left: 3px; }
|
||||
.sidebar-item-value-button-context { float: right; }
|
||||
.sidebar-item-selector {
|
||||
font-family: inherit; font-size: 12px;
|
||||
background-color: #fcfcfc; border: #fcfcfc; color: #333;
|
||||
border-radius: 2px; width: 100%; height: 23px; padding: 3px 12px 3px 7px;
|
||||
margin-top: 3px; margin-bottom: 3px; outline: none;
|
||||
box-sizing: border-box; -moz-box-sizing: border-box;
|
||||
appearance: none; -webkit-appearance: none; -moz-appearance: none;
|
||||
background-image: linear-gradient(45deg, transparent 50%, #333 50%), linear-gradient(135deg, #333 50%, transparent 50%);
|
||||
background-position: calc(100% - 12px) calc(10px), calc(100% - 7px) calc(10px);
|
||||
background-size: 5px 5px, 5px 5px;
|
||||
background-repeat: no-repeat;
|
||||
}
|
||||
.sidebar-separator { margin-bottom: 20px; }
|
||||
.sidebar-find-search { display: flex; align-items: center; background: #fff; border-radius: 16px; margin: 0px 20px 8px 20px; padding: 0px 8px 0px 8px; }
|
||||
.sidebar-find-query { width: 100vw; background: none; font-family: inherit; font-size: 13px; padding: 8px 8px 8px 8px; border: 0; outline: 0; }
|
||||
.sidebar-find-toggle { margin-left: auto; margin-right: 2px; width: 16px; height: 16px; cursor: pointer; }
|
||||
.sidebar-find-toggle input[type="checkbox"] { display: none; }
|
||||
.sidebar-find-toggle input[type="checkbox"]:not(:checked) + svg { fill: #ccc; stroke: #ccc; }
|
||||
.sidebar-find-toggle input[type="checkbox"]:checked + svg { fill: #555; stroke: #555; }
|
||||
.sidebar-find-toggle-icon { stroke: #555; fill: #555; width: 16px; height: 16px; }
|
||||
.sidebar-find-content { flex-grow: 1; padding: 0px 20px 20px 20px; overflow-y: auto; list-style-type: none; margin: 0; outline: 0; }
|
||||
.sidebar-find-content li *:first-child { margin-right: 2px; }
|
||||
.sidebar-find-content li { color: #666; font-size: 13px; height: 22px; line-height: 22px; padding: 0 12px 0 12px; outline: 0; white-space: nowrap; border-radius: 3px; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.sidebar-find-content li.focus { background: #e5e5e5; color: #000; }
|
||||
.sidebar-find-content-icon { stroke: #555; fill: #555; float: left; width: 16px; height: 16px; padding: 3px; pointer-events: none; }
|
||||
.sidebar-documentation { flex-grow: 1; padding: 0px 20px 20px 20px; overflow-y: auto; font-size: 13px; line-height: 1.5; margin: 0; }
|
||||
.sidebar-documentation h1 { font-weight: bold; font-size: 13px; line-height: 1.25; border-bottom: 1px solid #e8e8e8; padding-bottom: 0.3em; margin-top: 0; margin-bottom: 16px; }
|
||||
.sidebar-documentation h2 { font-weight: bold; font-size: 13px; line-height: 1.25; margin-top: 20px; margin-bottom: 16px; text-transform: uppercase; border: 0; }
|
||||
.sidebar-documentation h3 { font-weight: bold; font-size: 11px; line-height: 1.25; }
|
||||
.sidebar-documentation p { margin-top: 4px; margin-bottom: 4px; margin-left: 0px; }
|
||||
.sidebar-documentation a { color: #237; }
|
||||
.sidebar-documentation code { font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 12px; background-color: rgba(27, 31, 35, 0.05); padding: 0.2em 0.4em; margin: 0; border-radius: 3px; }
|
||||
.sidebar-documentation pre { font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 12px; padding: 16px; overflow: auto; line-height: 1.45; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; }
|
||||
.sidebar-documentation pre code { font-size: 13px; padding: 16px; line-height: 1.45; background-color: transparent; padding: 0; border-radius: 0; }
|
||||
.sidebar-documentation tt { font-family: 'SFMono-Regular', Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-weight: bold; font-size: 90%; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px; padding: 0.2em 0.4em; margin: 0; }
|
||||
.sidebar-documentation dl dt { font-size: 13px; font-weight: bold; padding: 0; margin-top: 16px; margin-left: 0px; }
|
||||
.sidebar-documentation dd { padding: 0 16px; margin-left: 0; margin-bottom: 16px; }
|
||||
.sidebar-documentation ul { margin-top: 6px; margin-bottom: 6px; padding-left: 20px; }
|
||||
.sidebar-documentation blockquote { margin-left: 15px; margin-right: 15px; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.sidebar html { color: #dfdfdf; }
|
||||
.sidebar { background-color: #2d2d2d; color: #dfdfdf; border-left: 1px solid rgba(0, 0, 0, 0); }
|
||||
.sidebar-closebutton { padding: 8px 8px 8px 32px; text-decoration: none; font-size: 25px; color: #777777; opacity: 1.0; display: block; transition: 0.2s; position: absolute; top: 0; right: 15px; margin-left: 50px; user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.sidebar-closebutton:hover { color: #ffffff; }
|
||||
.sidebar-item-value { background-color: #383838; border-color: #383838; }
|
||||
.sidebar-item-value-content { background-color: #3e3e3e; border-color: #3e3e3e; }
|
||||
.sidebar-item-value-line-border { border-color: rgba(0, 0, 0, 0.09); }
|
||||
.sidebar-item-selector { background-color: #383838; border: #383838; color: #dfdfdf; background-image: linear-gradient(45deg, transparent 50%, #aaa 50%), linear-gradient(135deg, #aaa 50%, transparent 50%); }
|
||||
.sidebar-item-disable-select { user-select: none; -webkit-user-select: none; -moz-user-select: none; }
|
||||
.sidebar-header { border-bottom-color: #2d2d2d; color: #dfdfdf; }
|
||||
.sidebar-documentation h1 { border-bottom: 1px solid #424242; color: #dfdfdf; }
|
||||
.sidebar-documentation h2 { color: #dfdfdf; }
|
||||
.sidebar-documentation p { color: #aaaaaa; }
|
||||
.sidebar-documentation a { color: #6688aa; }
|
||||
.sidebar-documentation tt { background-color:#1e1e1e; }
|
||||
.sidebar-documentation code { background-color: #1e1e1e; }
|
||||
.sidebar-documentation pre { background-color: #1e1e1e; }
|
||||
.sidebar-find-search { background: #383838; color: #dfdfdf; border-color: #424242; }
|
||||
.sidebar-find-toggle input[type="checkbox"]:not(:checked) + svg { fill: #555; stroke: #555; }
|
||||
.sidebar-find-toggle input[type="checkbox"]:checked + svg { fill: #aaa; stroke: #aaa; }
|
||||
.sidebar-find-content li { color: #aaaaaa; }
|
||||
.sidebar-find-content li.focus { background: #383838; color: #dfdfdf; }
|
||||
.sidebar-find-content-icon { stroke: #888888; fill: #888888; }
|
||||
.sidebar-item-value-expander { color: #888; }
|
||||
.sidebar-item-value-expander:hover { color: #e5e5e5; }
|
||||
.sidebar-item-value-button { color: #888; }
|
||||
.sidebar-item-value-button svg use { fill: #888; stroke: #888; }
|
||||
.sidebar-item-value-button:hover svg use { fill: #e5e5e5; stroke: #e5e5e5; }
|
||||
}
|
||||
@media screen and (prefers-reduced-motion: reduce) {
|
||||
.menu { transition: none; }
|
||||
.sidebar { transition: none; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body class="welcome spinner">
|
||||
<div id="target" class="target" tabindex="0">
|
||||
</div>
|
||||
<div id="sidebar" class="sidebar">
|
||||
<h1 id="sidebar-title" class="sidebar-title"></h1>
|
||||
<a id="sidebar-closebutton" class="sidebar-closebutton" href="javascript:void(0)" draggable="false">×</a>
|
||||
<div id="sidebar-content" class="sidebar-content"></div>
|
||||
<svg width="0" height="0" display="none">
|
||||
<defs>
|
||||
<symbol id="sidebar-icon-node" viewBox="0 0 20 20">
|
||||
<circle cx="10" cy="10" r="6" stroke-width="2" fill="none"/>
|
||||
</symbol>
|
||||
<symbol id="sidebar-icon-connection" viewBox="0 0 20 20">
|
||||
<line x1="4" y1="10" x2="15" y2="10" stroke-width="2" />
|
||||
<polyline points="11,6 15,10 11,14" stroke-width="2" />
|
||||
</symbol>
|
||||
<symbol id="sidebar-icon-weight" viewBox="0 0 20 20">
|
||||
<circle cx="5" cy="5" r="1" />
|
||||
<circle cx="10" cy="5" r="1" />
|
||||
<circle cx="15" cy="5" r="1" />
|
||||
<circle cx="5" cy="10" r="1" />
|
||||
<circle cx="10" cy="10" r="1" />
|
||||
<circle cx="15" cy="10" r="1" />
|
||||
<circle cx="5" cy="15" r="1" />
|
||||
<circle cx="10" cy="15" r="1" />
|
||||
<circle cx="15" cy="15" r="1" />
|
||||
</symbol>
|
||||
</defs>
|
||||
</svg>
|
||||
</div>
|
||||
<div id="toolbar" class="toolbar">
|
||||
<button id="zoom-in-button" class="toolbar-button" title="Zoom In">
|
||||
<svg class="toolbar-icon" viewbox="0 0 100 100">
|
||||
<circle class="border" cx="50" cy="50" r="35" stroke-width="8" stroke="#fff"></circle>
|
||||
<line class="border" x1="50" y1="38" x2="50" y2="62" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="38" y1="50" x2="62" y2="50" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="78" y1="78" x2="82" y2="82" stroke-width="12" stroke-linecap="square" stroke="#fff"></line>
|
||||
<circle class="stroke" cx="50" cy="50" r="35" stroke-width="4"></circle>
|
||||
<line class="stroke" x1="50" y1="38" x2="50" y2="62" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="38" y1="50" x2="62" y2="50" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="78" y1="78" x2="82" y2="82" stroke-width="8" stroke-linecap="square"></line>
|
||||
</svg>
|
||||
</button>
|
||||
<button id="zoom-out-button" class="toolbar-button" title="Zoom Out">
|
||||
<svg class="toolbar-icon" viewbox="0 0 100 100">
|
||||
<circle class="border" cx="50" cy="50" r="35" stroke-width="8" stroke="#fff"></circle>
|
||||
<line class="border" x1="38" y1="50" x2="62" y2="50" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="78" y1="78" x2="82" y2="82" stroke-width="12" stroke-linecap="square" stroke="#fff"></line>
|
||||
<circle class="stroke" cx="50" cy="50" r="35" stroke-width="4"></circle>
|
||||
<line class="stroke" x1="38" y1="50" x2="62" y2="50" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="78" y1="78" x2="82" y2="82" stroke-width="8" stroke-linecap="square"></line>
|
||||
</svg>
|
||||
</button>
|
||||
<button id="sidebar-model-button" class="toolbar-button" title="Model Properties">
|
||||
<svg class="toolbar-icon" viewbox="0 0 100 100">
|
||||
<rect class="border" x="12" y="12" width="76" height="76" rx="16" ry="16" stroke-width="8"></rect>
|
||||
<line class="border" x1="28" y1="37" x2="32" y2="37" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="28" y1="50" x2="32" y2="50" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="28" y1="63" x2="32" y2="63" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="40" y1="37" x2="70" y2="37" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="40" y1="50" x2="70" y2="50" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<line class="border" x1="40" y1="63" x2="70" y2="63" stroke-width="8" stroke-linecap="round" stroke="#fff"></line>
|
||||
<rect class="stroke" x="12" y="12" width="76" height="76" rx="16" ry="16" stroke-width="4"></rect>
|
||||
<line class="stroke" x1="28" y1="37" x2="32" y2="37" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="28" y1="50" x2="32" y2="50" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="28" y1="63" x2="32" y2="63" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="40" y1="37" x2="70" y2="37" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="40" y1="50" x2="70" y2="50" stroke-width="4" stroke-linecap="round"></line>
|
||||
<line class="stroke" x1="40" y1="63" x2="70" y2="63" stroke-width="4" stroke-linecap="round"></line>
|
||||
</svg>
|
||||
</button>
|
||||
<button id="sidebar-target-button" class="toolbar-button" title="Target Properties">
|
||||
<svg class="toolbar-icon" viewBox="0 0 100 100">
|
||||
<rect class="border" x="12" y="12" width="76" height="76" rx="16" ry="16" stroke-width="8"></rect>
|
||||
<circle class="border" cx="50" cy="50" r="18" stroke-width="8"></circle>
|
||||
<circle class="border" cx="50" cy="50" r="3" stroke-width="8"></circle>
|
||||
<rect class="stroke" x="12" y="12" width="76" height="76" rx="16" ry="16" stroke-width="4"></rect>
|
||||
<circle class="stroke" cx="50" cy="50" r="18" stroke-width="4"></circle>
|
||||
<circle class="stroke fill" cx="50" cy="50" r="3" stroke-width="4"></circle>
|
||||
</svg>
|
||||
</button>
|
||||
<div id="toolbar-navigator" class="toolbar-select">
|
||||
<select id="toolbar-target-selector">
|
||||
<option value="Graphs" disabled="true">— Graphs ——</option>
|
||||
<option value="add">add</option>
|
||||
<option value="subtract">subtract</option>
|
||||
<option value="multiply">multiply</option>
|
||||
<option value="Functions" disabled="true">— Functions ——</option>
|
||||
<option value="line">foo_bar.foo</option>
|
||||
</select>
|
||||
<svg class="toolbar-select-arrow" viewBox="0 0 10 6" width="10" height="6">
|
||||
<path d="M0 0L5 6L10 0Z" />
|
||||
</svg>
|
||||
</div>
|
||||
<div id="toolbar-path" class="toolbar-path">
|
||||
<button id="toolbar-path-back-button" class="toolbar-path-back-button" title="Back">
|
||||
❮
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div id="logo" class="center logo">
|
||||
<a href="https://github.com/lutzroeder/netron" target="blank_">
|
||||
<svg class="center logo-text" viewbox="0 0 5120 1024">
|
||||
<g transform="scale(9) translate(-44,-15)">
|
||||
<g transform="matrix(100,0,0,100,60.9965,126)">
|
||||
<path class="logo-glyph" d="M0.089,0L0.089,-0.745L0.595,-0.147L0.595,-0.715L0.656,-0.715L0.656,0.021L0.15,-0.578L0.15,0L0.089,0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(100,0,0,100,164.341,126)">
|
||||
<path class="logo-glyph" d="M0.089,0L0.089,-0.715L0.443,-0.715L0.443,-0.654L0.154,-0.654L0.154,-0.43L0.443,-0.43L0.443,-0.369L0.154,-0.369L0.154,-0.061L0.443,-0.061L0.443,0L0.089,0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(100,0,0,100,244.491,126)">
|
||||
<path class="logo-glyph" d="M0.216,0L0.216,-0.654L0.019,-0.654L0.019,-0.715L0.478,-0.715L0.478,-0.654L0.281,-0.654L0.281,0L0.216,0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(100,0,0,100,323.031,126)">
|
||||
<path class="logo-glyph" d="M0.154,-0.658L0.154,-0.394L0.219,-0.394C0.28,-0.394 0.322,-0.404 0.346,-0.423C0.37,-0.442 0.382,-0.475 0.382,-0.522C0.382,-0.571 0.369,-0.606 0.345,-0.627C0.32,-0.648 0.278,-0.658 0.219,-0.658L0.154,-0.658ZM0.523,0L0.444,0L0.193,-0.341L0.154,-0.341L0.154,0L0.089,0L0.089,-0.715L0.22,-0.715C0.298,-0.715 0.356,-0.699 0.394,-0.667C0.433,-0.634 0.452,-0.585 0.452,-0.52C0.452,-0.464 0.436,-0.421 0.403,-0.389C0.37,-0.357 0.324,-0.341 0.266,-0.341L0.523,0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(100,0,0,100,520.979,126)">
|
||||
<path class="logo-glyph" d="M0.089,0L0.089,-0.745L0.595,-0.147L0.595,-0.715L0.656,-0.715L0.656,0.021L0.15,-0.578L0.15,0L0.089,0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
<svg class="center logo-icon" viewbox="0 0 1024 1024">
|
||||
<circle class="logo-stroke" cx="512" cy="512" r="431" fill="none" stroke-width="32"></circle>
|
||||
<circle class="logo-border" cx="512" cy="512" r="450" fill="none" stroke-width="6"></circle>
|
||||
<circle class="logo-border" cx="512" cy="512" r="412" fill="none" stroke-width="6"></circle>
|
||||
<line class="logo-stroke" x1="296" y1="392" x2="540" y2="280" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="296" y1="632" x2="540" y2="280" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="296" y1="392" x2="540" y2="435" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="296" y1="632" x2="540" y2="435" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="296" y1="392" x2="540" y2="590" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="296" y1="632" x2="540" y2="590" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="296" y1="392" x2="540" y2="744" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="296" y1="632" x2="540" y2="744" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="540" y1="280" x2="785" y2="512" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="540" y1="590" x2="785" y2="512" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="540" y1="435" x2="785" y2="512" stroke-width="12"></line>
|
||||
<line class="logo-stroke" x1="540" y1="744" x2="785" y2="512" stroke-width="12"></line>
|
||||
<g transform="translate(296, 392)">
|
||||
<circle class="logo-fill" cx="0" cy="0" r="51"></circle>
|
||||
<circle class="logo-border" cx="0" cy="0" r="51" fill="none" stroke-width="6"></circle>
|
||||
</g>
|
||||
<g transform="translate(296, 632)">
|
||||
<circle class="logo-fill" cx="0" cy="0" r="51"></circle>
|
||||
<circle class="logo-border" cx="0" cy="0" r="51" fill="none" stroke-width="6"></circle>
|
||||
</g>
|
||||
<g transform="translate(540, 280)">
|
||||
<circle class="logo-fill" cx="0" cy="0" r="51"></circle>
|
||||
<circle class="logo-border" cx="0" cy="0" r="51" fill="none" stroke-width="6"></circle>
|
||||
</g>
|
||||
<g transform="translate(540, 435)">
|
||||
<circle class="logo-fill" cx="0" cy="0" r="51"></circle>
|
||||
<circle class="logo-border" cx="0" cy="0" r="51" fill="none" stroke-width="6"></circle>
|
||||
</g>
|
||||
<g transform="translate(540, 590)">
|
||||
<circle class="logo-fill" cx="0" cy="0" r="51"></circle>
|
||||
<circle class="logo-border" cx="0" cy="0" r="51" fill="none" stroke-width="6"></circle>
|
||||
</g>
|
||||
<g transform="translate(540, 744)">
|
||||
<circle class="logo-fill" cx="0" cy="0" r="51"></circle>
|
||||
<circle class="logo-border" cx="0" cy="0" r="51" fill="none" stroke-width="6"></circle>
|
||||
</g>
|
||||
<g transform="translate(785, 512)">
|
||||
<circle class="logo-fill" cx="0" cy="0" r="51"></circle>
|
||||
<circle class="logo-border" cx="0" cy="0" r="51" fill="none" stroke-width="6"></circle>
|
||||
</g>
|
||||
</svg>
|
||||
<svg id="logo-spinner" class="center logo-spinner" viewbox="0 0 1024 1024">
|
||||
<g transform="translate(512, 512)" style="opacity: 1">
|
||||
<path class="logo-spinner-stroke" d="M-431,0 A-431,-431 0 0,1 0,-431" stroke-width="24" fill="None"></path>
|
||||
</g>
|
||||
</svg>
|
||||
</a>
|
||||
<a href="https://www.lutzroeder.com" target="blank_">
|
||||
<svg class="center logo-name" viewbox="0 0 5120 300">
|
||||
<g transform="scale(5.8) translate(20, 0)">
|
||||
<g transform="matrix(30,0,0,30,18.9123,38)">
|
||||
<path class="logo-glyph" d="M0.089,-0L0.089,-0.715L0.154,-0.715L0.154,-0.061L0.399,-0.061L0.399,-0L0.089,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,46.7613,38)">
|
||||
<path class="logo-glyph" d="M0.086,-0.715L0.15,-0.715L0.15,-0.248C0.15,-0.177 0.166,-0.125 0.198,-0.091C0.23,-0.056 0.28,-0.039 0.346,-0.039C0.412,-0.039 0.46,-0.056 0.493,-0.091C0.525,-0.125 0.541,-0.177 0.541,-0.248L0.541,-0.715L0.606,-0.715L0.606,-0.269C0.606,-0.172 0.584,-0.1 0.542,-0.052C0.499,-0.005 0.433,0.019 0.346,0.019C0.259,0.019 0.193,-0.005 0.15,-0.052C0.107,-0.1 0.086,-0.172 0.086,-0.269L0.086,-0.715Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,83.5133,38)">
|
||||
<path class="logo-glyph" d="M0.216,-0L0.216,-0.654L0.019,-0.654L0.019,-0.715L0.478,-0.715L0.478,-0.654L0.281,-0.654L0.281,-0L0.216,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,114.421,38)">
|
||||
<path class="logo-glyph" d="M0.012,-0L0.437,-0.656L0.074,-0.656L0.074,-0.715L0.548,-0.715L0.125,-0.06L0.505,-0.06L0.505,-0L0.012,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,171.777,38)">
|
||||
<path class="logo-glyph" d="M0.154,-0.658L0.154,-0.394L0.219,-0.394C0.28,-0.394 0.322,-0.404 0.346,-0.423C0.37,-0.442 0.382,-0.475 0.382,-0.522C0.382,-0.571 0.369,-0.606 0.345,-0.627C0.32,-0.648 0.278,-0.658 0.219,-0.658L0.154,-0.658ZM0.523,-0L0.444,-0L0.193,-0.341L0.154,-0.341L0.154,-0L0.089,-0L0.089,-0.715L0.22,-0.715C0.298,-0.715 0.356,-0.699 0.394,-0.667C0.433,-0.634 0.452,-0.585 0.452,-0.52C0.452,-0.464 0.436,-0.421 0.403,-0.389C0.37,-0.357 0.324,-0.341 0.266,-0.341L0.523,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,203.607,38)">
|
||||
<path class="logo-glyph" d="M0.437,-0.039C0.479,-0.039 0.519,-0.047 0.557,-0.063C0.595,-0.078 0.629,-0.101 0.659,-0.131C0.689,-0.161 0.712,-0.196 0.727,-0.234C0.743,-0.273 0.751,-0.313 0.751,-0.356C0.751,-0.399 0.743,-0.44 0.728,-0.478C0.712,-0.516 0.689,-0.55 0.659,-0.581C0.63,-0.611 0.596,-0.634 0.558,-0.649C0.52,-0.665 0.48,-0.673 0.437,-0.673C0.395,-0.673 0.355,-0.665 0.317,-0.649C0.28,-0.634 0.246,-0.611 0.216,-0.581C0.186,-0.55 0.163,-0.516 0.147,-0.478C0.132,-0.44 0.124,-0.399 0.124,-0.356C0.124,-0.313 0.132,-0.272 0.147,-0.234C0.163,-0.196 0.186,-0.161 0.216,-0.131C0.246,-0.101 0.279,-0.078 0.316,-0.062C0.354,-0.047 0.394,-0.039 0.437,-0.039ZM0.82,-0.356C0.82,-0.306 0.81,-0.258 0.791,-0.212C0.772,-0.167 0.744,-0.126 0.708,-0.091C0.671,-0.055 0.63,-0.028 0.583,-0.009C0.537,0.01 0.488,0.019 0.437,0.019C0.386,0.019 0.337,0.01 0.291,-0.009C0.245,-0.028 0.203,-0.055 0.167,-0.091C0.131,-0.127 0.103,-0.168 0.084,-0.213C0.065,-0.258 0.055,-0.306 0.055,-0.356C0.055,-0.407 0.065,-0.455 0.084,-0.501C0.103,-0.546 0.131,-0.587 0.167,-0.623C0.203,-0.659 0.244,-0.685 0.29,-0.704C0.335,-0.722 0.385,-0.731 0.437,-0.731C0.49,-0.731 0.539,-0.722 0.585,-0.703C0.631,-0.685 0.672,-0.658 0.708,-0.623C0.744,-0.587 0.772,-0.546 0.791,-0.501C0.81,-0.455 0.82,-0.407 0.82,-0.356Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,245.853,38)">
|
||||
<path class="logo-glyph" d="M0.089,-0L0.089,-0.715L0.443,-0.715L0.443,-0.654L0.154,-0.654L0.154,-0.43L0.443,-0.43L0.443,-0.369L0.154,-0.369L0.154,-0.061L0.443,-0.061L0.443,-0L0.089,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,277.243,38)">
|
||||
<path class="logo-glyph" d="M0.154,-0.056L0.245,-0.056C0.319,-0.056 0.371,-0.06 0.402,-0.069C0.433,-0.077 0.459,-0.091 0.481,-0.111C0.511,-0.139 0.534,-0.174 0.549,-0.215C0.564,-0.257 0.572,-0.305 0.572,-0.358C0.572,-0.413 0.564,-0.461 0.549,-0.502C0.533,-0.544 0.51,-0.578 0.48,-0.605C0.457,-0.625 0.429,-0.64 0.396,-0.648C0.364,-0.657 0.306,-0.661 0.224,-0.661L0.154,-0.661L0.154,-0.056ZM0.089,-0L0.089,-0.715L0.2,-0.715C0.299,-0.715 0.37,-0.71 0.412,-0.7C0.453,-0.69 0.489,-0.674 0.519,-0.65C0.559,-0.618 0.589,-0.578 0.61,-0.528C0.631,-0.478 0.641,-0.421 0.641,-0.357C0.641,-0.293 0.631,-0.236 0.61,-0.186C0.589,-0.136 0.559,-0.096 0.52,-0.066C0.489,-0.042 0.454,-0.025 0.414,-0.015C0.374,-0.005 0.31,-0 0.222,-0L0.089,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,314.142,38)">
|
||||
<path class="logo-glyph" d="M0.089,-0L0.089,-0.715L0.443,-0.715L0.443,-0.654L0.154,-0.654L0.154,-0.43L0.443,-0.43L0.443,-0.369L0.154,-0.369L0.154,-0.061L0.443,-0.061L0.443,-0L0.089,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,345.532,38)">
|
||||
<path class="logo-glyph" d="M0.154,-0.658L0.154,-0.394L0.219,-0.394C0.28,-0.394 0.322,-0.404 0.346,-0.423C0.37,-0.442 0.382,-0.475 0.382,-0.522C0.382,-0.571 0.369,-0.606 0.345,-0.627C0.32,-0.648 0.278,-0.658 0.219,-0.658L0.154,-0.658ZM0.523,-0L0.444,-0L0.193,-0.341L0.154,-0.341L0.154,-0L0.089,-0L0.089,-0.715L0.22,-0.715C0.298,-0.715 0.356,-0.699 0.394,-0.667C0.433,-0.634 0.452,-0.585 0.452,-0.52C0.452,-0.464 0.436,-0.421 0.403,-0.389C0.37,-0.357 0.324,-0.341 0.266,-0.341L0.523,-0Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,376.911,38)">
|
||||
<path class="logo-glyph" d="M0.06,-0.468L0.155,-0.731L0.228,-0.699L0.1,-0.452L0.06,-0.468Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
<g transform="matrix(30,0,0,30,401.549,38)">
|
||||
<path class="logo-glyph" d="M0.034,-0.12L0.09,-0.15C0.1,-0.115 0.118,-0.087 0.144,-0.068C0.169,-0.049 0.2,-0.039 0.236,-0.039C0.281,-0.039 0.316,-0.052 0.342,-0.079C0.367,-0.106 0.38,-0.143 0.38,-0.19C0.38,-0.224 0.371,-0.253 0.354,-0.276C0.337,-0.299 0.3,-0.325 0.244,-0.355C0.172,-0.393 0.124,-0.427 0.101,-0.456C0.077,-0.485 0.065,-0.519 0.065,-0.56C0.065,-0.611 0.082,-0.652 0.116,-0.684C0.151,-0.716 0.195,-0.732 0.25,-0.732C0.286,-0.732 0.317,-0.724 0.344,-0.709C0.37,-0.694 0.392,-0.671 0.408,-0.641L0.358,-0.611C0.347,-0.631 0.333,-0.647 0.314,-0.658C0.295,-0.668 0.272,-0.674 0.246,-0.674C0.211,-0.674 0.183,-0.663 0.162,-0.643C0.141,-0.622 0.131,-0.594 0.131,-0.559C0.131,-0.509 0.172,-0.462 0.255,-0.419C0.27,-0.411 0.281,-0.405 0.289,-0.401C0.35,-0.367 0.391,-0.336 0.411,-0.306C0.432,-0.276 0.442,-0.237 0.442,-0.19C0.442,-0.126 0.423,-0.075 0.386,-0.037C0.348,0 0.297,0.019 0.233,0.019C0.186,0.019 0.146,0.007 0.113,-0.016C0.079,-0.039 0.053,-0.074 0.034,-0.12Z" style="fill-rule:nonzero;"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
</a>
|
||||
<div class="center logo-message">
|
||||
<div style="height: 30px; text-align: center;">Version <span id="version" class="select">{version}</span></div>
|
||||
<div style="height: 30px; text-align: center;">Copyright © <a href="https://www.lutzroeder.com" target="blank_">Lutz Roeder</a></div>
|
||||
</div>
|
||||
<a id="logo-github" class="center logo-github" href="https://github.com/lutzroeder/netron" target="blank_">
|
||||
<svg viewbox="0 0 438.549 438.549">
|
||||
<path class="logo-fill" d="M409.132,114.573c-19.608-33.596-46.205-60.194-79.798-79.8C295.736,15.166,259.057,5.365,219.271,5.365
|
||||
c-39.781,0-76.472,9.804-110.063,29.408c-33.596,19.605-60.192,46.204-79.8,79.8C9.803,148.168,0,184.854,0,224.63
|
||||
c0,47.78,13.94,90.745,41.827,128.906c27.884,38.164,63.906,64.572,108.063,79.227c5.14,0.954,8.945,0.283,11.419-1.996
|
||||
c2.475-2.282,3.711-5.14,3.711-8.562c0-0.571-0.049-5.708-0.144-15.417c-0.098-9.709-0.144-18.179-0.144-25.406l-6.567,1.136
|
||||
c-4.187,0.767-9.469,1.092-15.846,1c-6.374-0.089-12.991-0.757-19.842-1.999c-6.854-1.231-13.229-4.086-19.13-8.559
|
||||
c-5.898-4.473-10.085-10.328-12.56-17.556l-2.855-6.57c-1.903-4.374-4.899-9.233-8.992-14.559
|
||||
c-4.093-5.331-8.232-8.945-12.419-10.848l-1.999-1.431c-1.332-0.951-2.568-2.098-3.711-3.429c-1.142-1.331-1.997-2.663-2.568-3.997
|
||||
c-0.572-1.335-0.098-2.43,1.427-3.289c1.525-0.859,4.281-1.276,8.28-1.276l5.708,0.853c3.807,0.763,8.516,3.042,14.133,6.851
|
||||
c5.614,3.806,10.229,8.754,13.846,14.842c4.38,7.806,9.657,13.754,15.846,17.847c6.184,4.093,12.419,6.136,18.699,6.136
|
||||
c6.28,0,11.704-0.476,16.274-1.423c4.565-0.952,8.848-2.383,12.847-4.285c1.713-12.758,6.377-22.559,13.988-29.41
|
||||
c-10.848-1.14-20.601-2.857-29.264-5.14c-8.658-2.286-17.605-5.996-26.835-11.14c-9.235-5.137-16.896-11.516-22.985-19.126
|
||||
c-6.09-7.614-11.088-17.61-14.987-29.979c-3.901-12.374-5.852-26.648-5.852-42.826c0-23.035,7.52-42.637,22.557-58.817
|
||||
c-7.044-17.318-6.379-36.732,1.997-58.24c5.52-1.715,13.706-0.428,24.554,3.853c10.85,4.283,18.794,7.952,23.84,10.994
|
||||
c5.046,3.041,9.089,5.618,12.135,7.708c17.705-4.947,35.976-7.421,54.818-7.421s37.117,2.474,54.823,7.421l10.849-6.849
|
||||
c7.419-4.57,16.18-8.758,26.262-12.565c10.088-3.805,17.802-4.853,23.134-3.138c8.562,21.509,9.325,40.922,2.279,58.24
|
||||
c15.036,16.18,22.559,35.787,22.559,58.817c0,16.178-1.958,30.497-5.853,42.966c-3.9,12.471-8.941,22.457-15.125,29.979
|
||||
c-6.191,7.521-13.901,13.85-23.131,18.986c-9.232,5.14-18.182,8.85-26.84,11.136c-8.662,2.286-18.415,4.004-29.263,5.146
|
||||
c9.894,8.562,14.842,22.077,14.842,40.539v60.237c0,3.422,1.19,6.279,3.572,8.562c2.379,2.279,6.136,2.95,11.276,1.995
|
||||
c44.163-14.653,80.185-41.062,108.068-79.226c27.88-38.161,41.825-81.126,41.825-128.906
|
||||
C438.536,184.851,428.728,148.168,409.132,114.573z"/>
|
||||
</svg>
|
||||
</a>
|
||||
<button id="open-file-button" class="center logo-button open-file-button" tabindex="0">Open Model…</button>
|
||||
<div class="center progress">
|
||||
<div id="progress-bar" class="progress-bar"></div>
|
||||
</div>
|
||||
<input type="file" id="open-file-dialog" class="open-file-dialog" multiple="false" accept="">
|
||||
<!-- Preload fonts to workaround Chrome SVG layout issue -->
|
||||
<div style="font-weight: normal; color: rgba(0, 0, 0, 0.01); user-select: none;">.</div>
|
||||
<div style="font-weight: bold; color: rgba(0, 0, 0, 0.01); user-select: none;">.</div>
|
||||
<div style="font-weight: bold; color: rgba(0, 0, 0, 0.01); user-select: none;">.</div>
|
||||
</div>
|
||||
<div id="message" class="message">
|
||||
<div id="message-text" class="message-text"></div>
|
||||
<button id="message-button" class="logo-button message-button" tabindex="0">OK</button>
|
||||
</div>
|
||||
<div id="titlebar" class="titlebar">
|
||||
<svg style="position: absolute; width: 0px; height: 0px; overflow: hidden;" aria-hidden="true">
|
||||
<symbol id="icon-arrow-right" viewBox="0 0 1024 1024">
|
||||
<path d="M698.75712 565.02272l-191.488 225.4848a81.73568 81.73568 0 0 1-62.48448 28.89728 81.89952 81.89952 0 0 1-62.40256-134.94272l146.432-172.4416-146.432-172.4416a81.92 81.92 0 0 1 124.88704-106.06592l191.488 225.4848a81.87904 81.87904 0 0 1 0 106.02496z"></path>
|
||||
</symbol>
|
||||
</svg>
|
||||
<div id="titlebar-content" class="titlebar-content">
|
||||
<span id="titlebar-content-text" class="titlebar-content-text"></span>
|
||||
</div>
|
||||
<div id="titlebar-control-box" class="titlebar-control-box">
|
||||
<div id="titlebar-close" class="titlebar-button titlebar-button-close" title="Close">
|
||||
<svg class="titlebar-icon" aria-hidden="true">
|
||||
<path d="M 0,0 0,0.7 4.3,5 0,9.3 0,10 0.7,10 5,5.7 9.3,10 10,10 10,9.3 5.7,5 10,0.7 10,0 9.3,0 5,4.3 0.7,0 Z"></path>
|
||||
</svg>
|
||||
</div>
|
||||
<div id="titlebar-toggle" class="titlebar-button" title="Maximize">
|
||||
<svg id="titlebar-maximize" class="titlebar-icon" aria-hidden="true" style="position: absolute;">
|
||||
<path d="M 0,0 0,10 10,10 10,0 Z M 1,1 9,1 9,9 1,9 Z"></path>
|
||||
</svg>
|
||||
<svg id="titlebar-restore" class="titlebar-icon" aria-hidden="true" style="position: absolute;">
|
||||
<path d="m 2,1e-5 0,2 -2,0 0,8 8,0 0,-2 2,0 0,-8 z m 1,1 6,0 0,6 -1,0 0,-5 -5,0 z m -2,2 6,0 0,6 -6,0 z"></path>
|
||||
</svg>
|
||||
</div>
|
||||
<div id="titlebar-minimize" class="titlebar-button" title="Minimize">
|
||||
<svg class="titlebar-icon" aria-hidden="true">
|
||||
<path d="M 0,5 10,5 10,6 0,6 Z"></path>
|
||||
</svg>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div id="menu" class="menu"></div>
|
||||
<div id="menu-button" class="menu-button">≡</div>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,832 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://codex.local/trt-perf-analysis/analyze-data.schema.json",
|
||||
"title": "TensorRT Performance Analysis Data",
|
||||
"description": "Structured JSON emitted by scripts/analyze_trt_perf.py and consumed by the report app.",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"schema_version",
|
||||
"folder",
|
||||
"validation",
|
||||
"results"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"schema_version": {
|
||||
"const": "1.0"
|
||||
},
|
||||
"folder": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"model_name": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"model_name_confidence": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high"
|
||||
]
|
||||
},
|
||||
"model_name_evidence": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"model_components": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"validation": {
|
||||
"$ref": "#/$defs/top_level_validation"
|
||||
},
|
||||
"results": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"$ref": "#/$defs/result"
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"top_level_validation": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"status",
|
||||
"backend_count",
|
||||
"passed_count",
|
||||
"failed_count",
|
||||
"warnings"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"status": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"passed",
|
||||
"partial",
|
||||
"failed"
|
||||
]
|
||||
},
|
||||
"backend_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"passed_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"failed_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"warnings": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
}
|
||||
}
|
||||
},
|
||||
"layer_profile_validation": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"status",
|
||||
"analysis_mode",
|
||||
"messages",
|
||||
"source_paths",
|
||||
"report_available"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"status": {
|
||||
"const": "passed"
|
||||
},
|
||||
"analysis_mode": {
|
||||
"const": "layer_profile"
|
||||
},
|
||||
"messages": {
|
||||
"$ref": "#/$defs/validation_messages"
|
||||
},
|
||||
"source_paths": {
|
||||
"$ref": "#/$defs/source_paths"
|
||||
},
|
||||
"report_available": {
|
||||
"const": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"layer_only_validation": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"status",
|
||||
"analysis_mode",
|
||||
"messages",
|
||||
"source_paths",
|
||||
"report_available"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"status": {
|
||||
"const": "passed"
|
||||
},
|
||||
"analysis_mode": {
|
||||
"const": "layer_only"
|
||||
},
|
||||
"messages": {
|
||||
"$ref": "#/$defs/validation_messages"
|
||||
},
|
||||
"source_paths": {
|
||||
"$ref": "#/$defs/source_paths"
|
||||
},
|
||||
"report_available": {
|
||||
"const": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"failed_validation": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"status",
|
||||
"analysis_mode",
|
||||
"messages",
|
||||
"source_paths",
|
||||
"report_available"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"status": {
|
||||
"const": "failed"
|
||||
},
|
||||
"analysis_mode": {
|
||||
"const": "unusable"
|
||||
},
|
||||
"messages": {
|
||||
"$ref": "#/$defs/validation_messages"
|
||||
},
|
||||
"source_paths": {
|
||||
"$ref": "#/$defs/source_paths"
|
||||
},
|
||||
"report_available": {
|
||||
"const": false
|
||||
}
|
||||
}
|
||||
},
|
||||
"source_paths": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"layer",
|
||||
"profile"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"layer": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"profile": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"validation_messages": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/validation_message"
|
||||
}
|
||||
},
|
||||
"validation_message": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"level",
|
||||
"message"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"level": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"error",
|
||||
"warning",
|
||||
"info"
|
||||
]
|
||||
},
|
||||
"message": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"result": {
|
||||
"oneOf": [
|
||||
{
|
||||
"$ref": "#/$defs/layer_profile_result"
|
||||
},
|
||||
{
|
||||
"$ref": "#/$defs/layer_only_result"
|
||||
},
|
||||
{
|
||||
"$ref": "#/$defs/failure_result"
|
||||
}
|
||||
]
|
||||
},
|
||||
"layer_profile_result": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"label",
|
||||
"validation",
|
||||
"layers",
|
||||
"profile",
|
||||
"layer_count",
|
||||
"profile_count",
|
||||
"graph",
|
||||
"count",
|
||||
"count_source",
|
||||
"timing",
|
||||
"model",
|
||||
"engines",
|
||||
"records",
|
||||
"type_breakdown",
|
||||
"tag_breakdown",
|
||||
"issues",
|
||||
"positives"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"validation": {
|
||||
"$ref": "#/$defs/layer_profile_validation"
|
||||
},
|
||||
"layers": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/layer"
|
||||
}
|
||||
},
|
||||
"profile": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/profile_entry"
|
||||
}
|
||||
},
|
||||
"layer_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"profile_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"graph": {
|
||||
"$ref": "#/$defs/graph"
|
||||
},
|
||||
"count": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 0
|
||||
},
|
||||
"count_source": {
|
||||
"type": "string"
|
||||
},
|
||||
"timing": {
|
||||
"$ref": "#/$defs/timing"
|
||||
},
|
||||
"model": {
|
||||
"$ref": "#/$defs/model"
|
||||
},
|
||||
"engines": {
|
||||
"$ref": "#/$defs/engines"
|
||||
},
|
||||
"records": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/record"
|
||||
}
|
||||
},
|
||||
"type_breakdown": {
|
||||
"$ref": "#/$defs/breakdown"
|
||||
},
|
||||
"tag_breakdown": {
|
||||
"$ref": "#/$defs/breakdown"
|
||||
},
|
||||
"issues": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/issue"
|
||||
}
|
||||
},
|
||||
"positives": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
}
|
||||
}
|
||||
},
|
||||
"layer_only_result": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"label",
|
||||
"validation",
|
||||
"layers",
|
||||
"layer_count",
|
||||
"graph",
|
||||
"model",
|
||||
"engines"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"validation": {
|
||||
"$ref": "#/$defs/layer_only_validation"
|
||||
},
|
||||
"layers": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/layer"
|
||||
}
|
||||
},
|
||||
"layer_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"graph": {
|
||||
"$ref": "#/$defs/graph"
|
||||
},
|
||||
"model": {
|
||||
"$ref": "#/$defs/model"
|
||||
},
|
||||
"engines": {
|
||||
"$ref": "#/$defs/engines"
|
||||
}
|
||||
}
|
||||
},
|
||||
"failure_result": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"label",
|
||||
"validation"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"label": {
|
||||
"type": "string"
|
||||
},
|
||||
"validation": {
|
||||
"$ref": "#/$defs/failed_validation"
|
||||
},
|
||||
"layer_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"profile_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"layer": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"Name",
|
||||
"LayerType",
|
||||
"Inputs",
|
||||
"Outputs"
|
||||
],
|
||||
"additionalProperties": true,
|
||||
"properties": {
|
||||
"Name": {
|
||||
"type": "string"
|
||||
},
|
||||
"LayerType": {
|
||||
"type": "string"
|
||||
},
|
||||
"Inputs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/tensor"
|
||||
}
|
||||
},
|
||||
"Outputs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/tensor"
|
||||
}
|
||||
},
|
||||
"Constants": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/tensor"
|
||||
}
|
||||
},
|
||||
"Metadata": {
|
||||
"type": [
|
||||
"string",
|
||||
"object",
|
||||
"array",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"StreamId": {
|
||||
"type": [
|
||||
"integer",
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"TacticName": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"tensor": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"Name"
|
||||
],
|
||||
"additionalProperties": true,
|
||||
"properties": {
|
||||
"Name": {
|
||||
"type": "string"
|
||||
},
|
||||
"Datatype": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"Dimensions": {
|
||||
"$ref": "#/$defs/number_array"
|
||||
},
|
||||
"Format": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"StrideOrder": {
|
||||
"$ref": "#/$defs/number_array"
|
||||
},
|
||||
"Strides": {
|
||||
"$ref": "#/$defs/number_array"
|
||||
}
|
||||
}
|
||||
},
|
||||
"profile_entry": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"name",
|
||||
"timeMs",
|
||||
"averageMs",
|
||||
"medianMs",
|
||||
"percentage"
|
||||
],
|
||||
"additionalProperties": true,
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"timeMs": {
|
||||
"type": "number"
|
||||
},
|
||||
"averageMs": {
|
||||
"type": "number"
|
||||
},
|
||||
"medianMs": {
|
||||
"type": "number"
|
||||
},
|
||||
"percentage": {
|
||||
"type": "number"
|
||||
}
|
||||
}
|
||||
},
|
||||
"record": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"name",
|
||||
"layer_type",
|
||||
"raw_layer_type",
|
||||
"tactic",
|
||||
"time_ms",
|
||||
"avg_ms",
|
||||
"median_ms",
|
||||
"percentage",
|
||||
"tags",
|
||||
"layer"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"layer_type": {
|
||||
"type": "string"
|
||||
},
|
||||
"raw_layer_type": {
|
||||
"type": "string"
|
||||
},
|
||||
"tactic": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"time_ms": {
|
||||
"type": "number"
|
||||
},
|
||||
"avg_ms": {
|
||||
"type": "number"
|
||||
},
|
||||
"median_ms": {
|
||||
"type": "number"
|
||||
},
|
||||
"percentage": {
|
||||
"type": "number"
|
||||
},
|
||||
"tags": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"layer": {
|
||||
"$ref": "#/$defs/layer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"graph": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"edge_count",
|
||||
"external_inputs",
|
||||
"graph_outputs"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"edge_count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"external_inputs": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"graph_outputs": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
}
|
||||
}
|
||||
},
|
||||
"timing": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"average_inference_ms",
|
||||
"total_time_ms",
|
||||
"total_percentage"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"average_inference_ms": {
|
||||
"type": "number"
|
||||
},
|
||||
"total_time_ms": {
|
||||
"type": "number"
|
||||
},
|
||||
"total_percentage": {
|
||||
"type": "number"
|
||||
}
|
||||
}
|
||||
},
|
||||
"model": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"category",
|
||||
"confidence",
|
||||
"evidence",
|
||||
"config_path",
|
||||
"hidden_size",
|
||||
"intermediate_size",
|
||||
"num_layers",
|
||||
"num_heads",
|
||||
"max_position_embeddings",
|
||||
"sequence_length",
|
||||
"dynamic_dims",
|
||||
"input_shapes"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"name_confidence": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high"
|
||||
]
|
||||
},
|
||||
"name_evidence": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"components": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"category": {
|
||||
"type": "string"
|
||||
},
|
||||
"confidence": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high"
|
||||
]
|
||||
},
|
||||
"evidence": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"config_path": {
|
||||
"type": [
|
||||
"string",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"hidden_size": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"intermediate_size": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"num_layers": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"num_heads": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"max_position_embeddings": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"sequence_length": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
]
|
||||
},
|
||||
"dynamic_dims": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"input_shapes": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"$ref": "#/$defs/number_array"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"engines": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"engine_files",
|
||||
"subgraphs",
|
||||
"streams",
|
||||
"inferred_count",
|
||||
"source"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"engine_files": {
|
||||
"$ref": "#/$defs/string_array"
|
||||
},
|
||||
"subgraphs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": [
|
||||
"integer",
|
||||
"string"
|
||||
]
|
||||
}
|
||||
},
|
||||
"streams": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": [
|
||||
"integer",
|
||||
"string"
|
||||
]
|
||||
}
|
||||
},
|
||||
"inferred_count": {
|
||||
"type": [
|
||||
"integer",
|
||||
"null"
|
||||
],
|
||||
"minimum": 0
|
||||
},
|
||||
"source": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"breakdown": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/breakdown_item"
|
||||
}
|
||||
},
|
||||
"breakdown_item": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"name",
|
||||
"count",
|
||||
"time_ms",
|
||||
"percentage"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string"
|
||||
},
|
||||
"count": {
|
||||
"type": "integer",
|
||||
"minimum": 0
|
||||
},
|
||||
"time_ms": {
|
||||
"type": "number"
|
||||
},
|
||||
"percentage": {
|
||||
"type": "number"
|
||||
}
|
||||
}
|
||||
},
|
||||
"issue": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"score",
|
||||
"title",
|
||||
"evidence",
|
||||
"next_check",
|
||||
"confidence"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"score": {
|
||||
"type": "number"
|
||||
},
|
||||
"title": {
|
||||
"type": "string"
|
||||
},
|
||||
"evidence": {
|
||||
"type": "string"
|
||||
},
|
||||
"next_check": {
|
||||
"type": "string"
|
||||
},
|
||||
"confidence": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"low",
|
||||
"medium",
|
||||
"high"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"string_array": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"number_array": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "number"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Validate and extract structured TensorRT layer/profile analysis data.
|
||||
|
||||
The script intentionally uses only Python built-in modules so it can run in
|
||||
minimal benchmark environments.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from typing import Optional, Sequence
|
||||
|
||||
from trt_perf.data import DataError, extract_perf_data
|
||||
from trt_perf.render_json import render_results_json
|
||||
|
||||
|
||||
def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Validate and analyze TensorRT layer/profile JSON performance data."
|
||||
)
|
||||
parser.add_argument(
|
||||
"path",
|
||||
nargs="?",
|
||||
help="Folder directly containing one or more layers_*.json/profile_*.json backend pairs. Omit when using --data.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--data",
|
||||
dest="data_specs",
|
||||
action="append",
|
||||
nargs="+",
|
||||
metavar="PATH",
|
||||
help="Explicit backend input. Use --data layers.json for layer-only analysis or --data layers.json profile.json for a layer/profile pair. Repeat for multiple backends.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model-name",
|
||||
help="Optional model name from the user prompt or caller context. It is cleaned before serialization.",
|
||||
)
|
||||
parser.add_argument("--output", help="Optional path to write the generated analysis JSON.")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
args = parse_args(argv)
|
||||
|
||||
try:
|
||||
data = extract_perf_data(args.path, args.data_specs, args.model_name)
|
||||
except DataError as exc:
|
||||
sys.stderr.write(f"error: {exc}\n")
|
||||
return 2
|
||||
|
||||
report = render_results_json(data)
|
||||
|
||||
if args.output:
|
||||
try:
|
||||
with open(args.output, "w", encoding="utf-8") as handle:
|
||||
handle.write(report)
|
||||
except OSError as exc:
|
||||
sys.stderr.write(f"error: unable to write {args.output}: {exc}\n")
|
||||
return 2
|
||||
else:
|
||||
sys.stdout.write(report)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,319 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Create a standalone TensorRT performance report folder.
|
||||
|
||||
This script keeps report packaging deterministic and cross-platform: it runs
|
||||
the analyzer, validates the generated JSON, then copies the browser report
|
||||
template without relying on shell tools such as rsync.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Sequence
|
||||
|
||||
from trt_perf.data import infer_model_name_metadata
|
||||
|
||||
|
||||
def script_dir() -> Path:
|
||||
return Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def skill_root() -> Path:
|
||||
return script_dir().parent
|
||||
|
||||
|
||||
def default_template_dir() -> Path:
|
||||
return skill_root() / "report_template"
|
||||
|
||||
|
||||
def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate and package a standalone TensorRT performance report."
|
||||
)
|
||||
parser.add_argument(
|
||||
"path",
|
||||
nargs="?",
|
||||
help="Folder directly containing layers_*.json and/or profile_*.json inputs. Pass one component/report folder per invocation. Omit when using --analyze-data.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--analyze-data",
|
||||
help="Existing analyze-data.json to validate and package in place. The file must be named analyze-data.json.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-parent",
|
||||
help="Directory where an auto-named report folder should be created. Defaults to the current working directory unless that is inside the skill.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--report-dir",
|
||||
help="Exact report directory to create. For new analysis this directory must be empty or absent; useful for component-specific reports.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model-name",
|
||||
help="Model name from the user prompt or caller context. It is cleaned and stored in analyze-data.json.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--timestamp",
|
||||
help="Timestamp for auto-generated report folder names, formatted as YYYYMMDD_HHMMSS. Defaults to local time.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--template-dir",
|
||||
default=str(default_template_dir()),
|
||||
help="Report template directory. Defaults to report_template in this skill.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--analyze-md",
|
||||
help="AI-generated Markdown analysis to copy into the report folder as analyze.md after the final analysis is written.",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def input_source_paths(input_path: Path) -> list[str]:
|
||||
paths: list[Path] = []
|
||||
for pattern in ("layer*.json", "profile*.json"):
|
||||
paths.extend(input_path.glob(pattern))
|
||||
return [str(path) for path in sorted(paths)]
|
||||
|
||||
|
||||
def inferred_analyzer_model_name(input_path: Path, explicit_name: Optional[str]) -> Optional[str]:
|
||||
metadata = infer_model_name_metadata(str(input_path), explicit_name, input_source_paths(input_path))
|
||||
return metadata["name"]
|
||||
|
||||
|
||||
def infer_report_model_name(input_path: Path, explicit_name: Optional[str]) -> str:
|
||||
metadata = infer_model_name_metadata(str(input_path), explicit_name, input_source_paths(input_path))
|
||||
return metadata["name"] or "unknown_model"
|
||||
|
||||
|
||||
def is_relative_to(path: Path, parent: Path) -> bool:
|
||||
try:
|
||||
path.relative_to(parent)
|
||||
return True
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def default_output_parent() -> Path:
|
||||
cwd = Path.cwd().resolve()
|
||||
root = skill_root().resolve()
|
||||
if is_relative_to(cwd, root):
|
||||
return Path(tempfile.gettempdir()).resolve()
|
||||
return cwd
|
||||
|
||||
|
||||
def parse_timestamp(value: Optional[str]) -> str:
|
||||
if value is None:
|
||||
return datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
if not re.fullmatch(r"\d{8}_\d{6}", value):
|
||||
raise ValueError("--timestamp must use YYYYMMDD_HHMMSS format")
|
||||
return value
|
||||
|
||||
|
||||
def has_files(path: Path) -> bool:
|
||||
return path.exists() and any(path.iterdir())
|
||||
|
||||
|
||||
def has_input_files(path: Path) -> bool:
|
||||
return any(path.glob("layers_*.json")) or any(path.glob("profile_*.json"))
|
||||
|
||||
|
||||
def next_available_report_dir(output_parent: Path, model_name: str, timestamp: str) -> Path:
|
||||
base_name = f"report_{model_name}_{timestamp}"
|
||||
candidate = output_parent / base_name
|
||||
if not has_files(candidate):
|
||||
return candidate
|
||||
|
||||
index = 2
|
||||
while True:
|
||||
candidate = output_parent / f"{base_name}_{index}"
|
||||
if not has_files(candidate):
|
||||
return candidate
|
||||
index += 1
|
||||
|
||||
|
||||
def resolve_new_report_dir(args: argparse.Namespace, input_path: Path) -> Path:
|
||||
if args.report_dir:
|
||||
report_dir = Path(args.report_dir).expanduser()
|
||||
if has_files(report_dir):
|
||||
raise ValueError(f"report directory is not empty: {report_dir}")
|
||||
return report_dir
|
||||
|
||||
output_parent = Path(args.output_parent).expanduser() if args.output_parent else default_output_parent()
|
||||
timestamp = parse_timestamp(args.timestamp)
|
||||
model_name = infer_report_model_name(input_path, args.model_name)
|
||||
return next_available_report_dir(output_parent, model_name, timestamp)
|
||||
|
||||
|
||||
def validate_template_dir(template_dir: Path) -> None:
|
||||
required = [
|
||||
"index.html",
|
||||
"app.js",
|
||||
"styles.css",
|
||||
"netron/grapher.css",
|
||||
"netron/index.html",
|
||||
"netron/netron.js",
|
||||
"netron/worker.js",
|
||||
]
|
||||
missing = [name for name in required if not (template_dir / name).is_file()]
|
||||
if missing:
|
||||
raise ValueError(f"report template is missing required file(s): {', '.join(missing)}")
|
||||
|
||||
|
||||
def run_command(command: Sequence[str]) -> int:
|
||||
process = subprocess.run(command, text=True, check=False)
|
||||
return process.returncode
|
||||
|
||||
|
||||
def run_analyzer(input_path: Path, output_path: Path, model_name: Optional[str]) -> int:
|
||||
analyzer = script_dir() / "analyze_trt_perf.py"
|
||||
command = [
|
||||
sys.executable,
|
||||
str(analyzer),
|
||||
str(input_path),
|
||||
"--output",
|
||||
str(output_path),
|
||||
]
|
||||
if model_name:
|
||||
command.extend(["--model-name", model_name])
|
||||
return run_command(command)
|
||||
|
||||
|
||||
def run_validator(analyze_data_path: Path) -> int:
|
||||
validator = script_dir() / "validate_analyze_data.py"
|
||||
return run_command([sys.executable, str(validator), str(analyze_data_path)])
|
||||
|
||||
|
||||
def copy_template(template_dir: Path, report_dir: Path) -> None:
|
||||
validate_template_dir(template_dir)
|
||||
report_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
for item in template_dir.iterdir():
|
||||
if item.name in {"analyze-data.json", "analyze.md"}:
|
||||
continue
|
||||
|
||||
destination = report_dir / item.name
|
||||
if item.is_dir():
|
||||
shutil.copytree(item, destination, dirs_exist_ok=True)
|
||||
else:
|
||||
shutil.copy2(item, destination)
|
||||
|
||||
|
||||
def copy_analyze_markdown(analyze_md: Optional[str], report_dir: Path) -> Optional[Path]:
|
||||
if not analyze_md:
|
||||
return None
|
||||
|
||||
source = Path(analyze_md).expanduser()
|
||||
if not source.is_file():
|
||||
raise ValueError(f"analyze markdown file not found: {source}")
|
||||
|
||||
destination = report_dir / "analyze.md"
|
||||
if source.resolve() == destination.resolve():
|
||||
return destination
|
||||
|
||||
shutil.copy2(source, destination)
|
||||
return destination
|
||||
|
||||
|
||||
def package_existing_json(args: argparse.Namespace) -> int:
|
||||
analyze_data_path = Path(args.analyze_data).expanduser()
|
||||
if not analyze_data_path.is_file():
|
||||
sys.stderr.write(f"error: analyze-data file not found: {analyze_data_path}\n")
|
||||
return 2
|
||||
if analyze_data_path.name != "analyze-data.json":
|
||||
sys.stderr.write("error: --analyze-data must point to a file named analyze-data.json\n")
|
||||
return 2
|
||||
|
||||
validate_result = run_validator(analyze_data_path)
|
||||
if validate_result != 0:
|
||||
return validate_result
|
||||
|
||||
try:
|
||||
copy_template(Path(args.template_dir).expanduser(), analyze_data_path.parent)
|
||||
analyze_markdown_path = copy_analyze_markdown(args.analyze_md, analyze_data_path.parent)
|
||||
except (OSError, ValueError) as exc:
|
||||
sys.stderr.write(f"error: unable to package report template: {exc}\n")
|
||||
return 2
|
||||
|
||||
print(f"report_dir: {analyze_data_path.parent}")
|
||||
print(f"analyze_data: {analyze_data_path}")
|
||||
if analyze_markdown_path:
|
||||
print(f"analyze_md: {analyze_markdown_path}")
|
||||
return 0
|
||||
|
||||
|
||||
def package_input_folder(args: argparse.Namespace, input_path: Path) -> int:
|
||||
try:
|
||||
report_dir = resolve_new_report_dir(args, input_path)
|
||||
except ValueError as exc:
|
||||
sys.stderr.write(f"error: {exc}\n")
|
||||
return 2
|
||||
|
||||
report_dir.mkdir(parents=True, exist_ok=True)
|
||||
analyze_data_path = report_dir / "analyze-data.json"
|
||||
|
||||
model_name = inferred_analyzer_model_name(input_path, args.model_name)
|
||||
analyze_result = run_analyzer(input_path, analyze_data_path, model_name)
|
||||
if analyze_result != 0:
|
||||
return analyze_result
|
||||
|
||||
validate_result = run_validator(analyze_data_path)
|
||||
if validate_result != 0:
|
||||
return validate_result
|
||||
|
||||
try:
|
||||
copy_template(Path(args.template_dir).expanduser(), report_dir)
|
||||
analyze_markdown_path = copy_analyze_markdown(args.analyze_md, report_dir)
|
||||
except (OSError, ValueError) as exc:
|
||||
sys.stderr.write(f"error: unable to package report template: {exc}\n")
|
||||
return 2
|
||||
|
||||
print(f"report_dir: {report_dir}")
|
||||
print(f"analyze_data: {analyze_data_path}")
|
||||
if analyze_markdown_path:
|
||||
print(f"analyze_md: {analyze_markdown_path}")
|
||||
return 0
|
||||
|
||||
|
||||
def package_new_analysis(args: argparse.Namespace) -> int:
|
||||
if not args.path:
|
||||
sys.stderr.write("error: provide an input folder, or use --analyze-data\n")
|
||||
return 2
|
||||
|
||||
input_path = Path(args.path).expanduser()
|
||||
if not input_path.is_dir():
|
||||
sys.stderr.write(f"error: input folder not found: {input_path}\n")
|
||||
return 2
|
||||
|
||||
if has_input_files(input_path):
|
||||
return package_input_folder(args, input_path)
|
||||
|
||||
sys.stderr.write(
|
||||
f"error: no layers_*.json or profile_*.json files found directly in {input_path}. "
|
||||
"package_report.py creates exactly one report per invocation; pass a single "
|
||||
"component/report folder instead of a parent folder.\n"
|
||||
)
|
||||
return 2
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
args = parse_args(argv)
|
||||
|
||||
if args.path and args.analyze_data:
|
||||
sys.stderr.write("error: use either an input folder or --analyze-data, not both\n")
|
||||
return 2
|
||||
|
||||
if args.analyze_data:
|
||||
return package_existing_json(args)
|
||||
return package_new_analysis(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,76 @@
|
||||
@echo off
|
||||
:: SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
:: SPDX-License-Identifier: Apache-2.0
|
||||
setlocal
|
||||
|
||||
if "%~1"=="" (
|
||||
echo usage: run.cmd ^<python-script^> [args...] 1>&2
|
||||
exit /b 2
|
||||
)
|
||||
|
||||
set "PY_CMD="
|
||||
set "PY_ARG="
|
||||
|
||||
if not defined SKILL_PYTHON goto after_skill_python
|
||||
call :select_plain "%SKILL_PYTHON%"
|
||||
if defined PY_CMD goto run_python
|
||||
:after_skill_python
|
||||
|
||||
if not defined PYTHON goto after_python_env
|
||||
call :select_plain "%PYTHON%"
|
||||
if defined PY_CMD goto run_python
|
||||
:after_python_env
|
||||
|
||||
if not defined PYTHON3 goto after_python3_env
|
||||
call :select_plain "%PYTHON3%"
|
||||
if defined PY_CMD goto run_python
|
||||
:after_python3_env
|
||||
|
||||
call :select_with_arg py -3
|
||||
if defined PY_CMD goto run_python
|
||||
call :select_plain python
|
||||
if defined PY_CMD goto run_python
|
||||
call :select_plain python3
|
||||
if defined PY_CMD goto run_python
|
||||
call :select_plain python3.12
|
||||
if defined PY_CMD goto run_python
|
||||
call :select_plain python3.11
|
||||
if defined PY_CMD goto run_python
|
||||
call :select_plain python3.10
|
||||
if defined PY_CMD goto run_python
|
||||
call :select_plain python3.9
|
||||
if defined PY_CMD goto run_python
|
||||
call :select_plain python3.8
|
||||
if defined PY_CMD goto run_python
|
||||
|
||||
echo error: unable to find Python 3.8+ on PATH. Set SKILL_PYTHON to a Python executable. 1>&2
|
||||
exit /b 127
|
||||
|
||||
:run_python
|
||||
if "%PY_ARG%"=="" (
|
||||
"%PY_CMD%" %*
|
||||
) else (
|
||||
"%PY_CMD%" "%PY_ARG%" %*
|
||||
)
|
||||
exit /b %ERRORLEVEL%
|
||||
|
||||
:select_plain
|
||||
set "PY_CMD=%~1"
|
||||
set "PY_ARG="
|
||||
if "%PY_CMD%"=="" goto clear_selection
|
||||
"%PY_CMD%" -c "import sys; raise SystemExit(0 if sys.version_info >= (3, 8) else 1)" >nul 2>nul
|
||||
if errorlevel 1 goto clear_selection
|
||||
exit /b 0
|
||||
|
||||
:select_with_arg
|
||||
set "PY_CMD=%~1"
|
||||
set "PY_ARG=%~2"
|
||||
if "%PY_CMD%"=="" goto clear_selection
|
||||
"%PY_CMD%" "%PY_ARG%" -c "import sys; raise SystemExit(0 if sys.version_info >= (3, 8) else 1)" >nul 2>nul
|
||||
if errorlevel 1 goto clear_selection
|
||||
exit /b 0
|
||||
|
||||
:clear_selection
|
||||
set "PY_CMD="
|
||||
set "PY_ARG="
|
||||
exit /b 1
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env sh
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
set -u
|
||||
|
||||
if [ $# -lt 1 ]; then
|
||||
printf '%s\n' "usage: run.sh <python-script> [args...]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
SCRIPT_PATH=$1
|
||||
shift
|
||||
|
||||
try_python() {
|
||||
candidate=$1
|
||||
launcher_arg=$2
|
||||
shift 2
|
||||
|
||||
if [ -z "$candidate" ]; then
|
||||
return 1
|
||||
fi
|
||||
if ! command -v "$candidate" >/dev/null 2>&1; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ -n "$launcher_arg" ]; then
|
||||
"$candidate" "$launcher_arg" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 8) else 1)' >/dev/null 2>&1 || return 1
|
||||
exec "$candidate" "$launcher_arg" "$SCRIPT_PATH" "$@"
|
||||
fi
|
||||
|
||||
"$candidate" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 8) else 1)' >/dev/null 2>&1 || return 1
|
||||
exec "$candidate" "$SCRIPT_PATH" "$@"
|
||||
}
|
||||
|
||||
if [ -n "${SKILL_PYTHON:-}" ]; then
|
||||
try_python "$SKILL_PYTHON" "" "$@"
|
||||
fi
|
||||
if [ -n "${PYTHON:-}" ]; then
|
||||
try_python "$PYTHON" "" "$@"
|
||||
fi
|
||||
if [ -n "${PYTHON3:-}" ]; then
|
||||
try_python "$PYTHON3" "" "$@"
|
||||
fi
|
||||
|
||||
try_python python3 "" "$@"
|
||||
try_python python "" "$@"
|
||||
try_python py -3 "$@"
|
||||
try_python python3.12 "" "$@"
|
||||
try_python python3.11 "" "$@"
|
||||
try_python python3.10 "" "$@"
|
||||
try_python python3.9 "" "$@"
|
||||
try_python python3.8 "" "$@"
|
||||
|
||||
printf '%s\n' "error: unable to find Python 3.8+ on PATH. Set SKILL_PYTHON to a Python executable." >&2
|
||||
exit 127
|
||||
@@ -0,0 +1,4 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Implementation modules for the TensorRT performance analysis scripts."""
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,536 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Process TensorRT layer-info JSON and write visualization targets."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
STANDARD_LAYER_KEYS = {
|
||||
"Name",
|
||||
"LayerType",
|
||||
"Inputs",
|
||||
"Constants",
|
||||
"Outputs",
|
||||
"TacticName",
|
||||
"StreamId",
|
||||
"Metadata",
|
||||
"_subgraph",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProcessedInitializer:
|
||||
name: str
|
||||
desc: dict[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProcessedNode:
|
||||
name: str
|
||||
op_type: str
|
||||
inputs: list[str]
|
||||
outputs: list[str]
|
||||
attrs: dict[str, Any]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProcessedLayerInfo:
|
||||
source_path: Path
|
||||
layers: list[dict[str, Any]]
|
||||
nodes: list[ProcessedNode]
|
||||
graph_inputs: list[str]
|
||||
graph_outputs: list[str]
|
||||
graph_value_info: list[str]
|
||||
value_descriptors: dict[str, dict[str, Any]]
|
||||
initializers: list[ProcessedInitializer]
|
||||
layer_type_counts: Counter[str]
|
||||
node_op_type_counts: Counter[str]
|
||||
extra_layer_fields: list[str]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TargetSpec:
|
||||
name: str
|
||||
module_name: str
|
||||
default_suffix: str
|
||||
description: str
|
||||
aliases: tuple[str, ...] = ()
|
||||
|
||||
|
||||
TARGET_SPECS = (
|
||||
TargetSpec(
|
||||
name="html",
|
||||
module_name="trt_perf.layer_info_html",
|
||||
default_suffix=".html",
|
||||
description="Standalone HTML layer graph summary",
|
||||
),
|
||||
)
|
||||
|
||||
TARGETS_BY_NAME: dict[str, TargetSpec] = {}
|
||||
for target_spec in TARGET_SPECS:
|
||||
TARGETS_BY_NAME[target_spec.name] = target_spec
|
||||
for alias in target_spec.aliases:
|
||||
TARGETS_BY_NAME[alias] = target_spec
|
||||
|
||||
|
||||
class NameRegistry:
|
||||
"""Keep target value names valid under SSA while preserving TRT names."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.used: set[str] = set()
|
||||
|
||||
def reserve(self, preferred: Any, fallback: str) -> str:
|
||||
base = str(preferred).strip() if preferred is not None else ""
|
||||
if not base:
|
||||
base = fallback
|
||||
|
||||
candidate = base
|
||||
suffix = 1
|
||||
while candidate in self.used:
|
||||
candidate = f"{base}__trt_dup{suffix}"
|
||||
suffix += 1
|
||||
self.used.add(candidate)
|
||||
return candidate
|
||||
|
||||
|
||||
def compact_json(value: Any) -> str:
|
||||
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
|
||||
|
||||
|
||||
def load_layers(path: Path) -> list[dict[str, Any]]:
|
||||
with path.open("r", encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
|
||||
if isinstance(payload, list):
|
||||
layers = payload
|
||||
elif isinstance(payload, dict):
|
||||
for key in ("Layers", "layers", "LayerInfo", "layerInfo"):
|
||||
if isinstance(payload.get(key), list):
|
||||
layers = payload[key]
|
||||
break
|
||||
else:
|
||||
raise ValueError("JSON object does not contain a layer list")
|
||||
else:
|
||||
raise ValueError("top-level JSON value must be a list or object")
|
||||
|
||||
bad_indexes = [idx for idx, layer in enumerate(layers) if not isinstance(layer, dict)]
|
||||
if bad_indexes:
|
||||
preview = ", ".join(str(idx) for idx in bad_indexes[:5])
|
||||
raise ValueError(f"layer entries must be objects; invalid indexes: {preview}")
|
||||
|
||||
return layers
|
||||
|
||||
|
||||
def as_descriptor_list(value: Any) -> list[dict[str, Any]]:
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
return [item for item in value if isinstance(item, dict)]
|
||||
|
||||
|
||||
def tensor_name(desc: dict[str, Any], fallback: str) -> str:
|
||||
name = desc.get("Name")
|
||||
if name is None or str(name).strip() == "":
|
||||
return fallback
|
||||
return str(name)
|
||||
|
||||
|
||||
def safe_dim_param(text: Any, fallback: str) -> str:
|
||||
value = re.sub(r"[^0-9A-Za-z_]+", "_", str(text)).strip("_")
|
||||
if not value:
|
||||
value = fallback
|
||||
if value[0].isdigit():
|
||||
value = f"d_{value}"
|
||||
return value
|
||||
|
||||
|
||||
def tensor_shape(desc: dict[str, Any], tensor: str) -> list[int | str] | None:
|
||||
dims = desc.get("Dimensions")
|
||||
if dims is None:
|
||||
return None
|
||||
if not isinstance(dims, list):
|
||||
return None
|
||||
|
||||
shape: list[int | str] = []
|
||||
prefix = safe_dim_param(tensor, "tensor")
|
||||
for idx, dim in enumerate(dims):
|
||||
if isinstance(dim, bool):
|
||||
shape.append(int(dim))
|
||||
elif isinstance(dim, int) and dim >= 0:
|
||||
shape.append(dim)
|
||||
else:
|
||||
shape.append(f"{prefix}_dim{idx}")
|
||||
return shape
|
||||
|
||||
|
||||
def sanitize_op_type(layer_type: Any) -> str:
|
||||
text = str(layer_type).strip() if layer_type is not None else ""
|
||||
op_type = re.sub(r"[^0-9A-Za-z_]+", "_", text).strip("_")
|
||||
if not op_type:
|
||||
op_type = "Layer"
|
||||
if not re.match(r"[A-Za-z_]", op_type[0]):
|
||||
op_type = f"_{op_type}"
|
||||
return op_type[:200]
|
||||
|
||||
|
||||
KGEN_TOKEN_MAP = {
|
||||
"and": "And",
|
||||
"add": "Add",
|
||||
"cast": "Cast",
|
||||
"div": "Div",
|
||||
"erf": "Erf",
|
||||
"gath": "Gather",
|
||||
"gemm": "Gemm",
|
||||
"mean": "Mean",
|
||||
"mha": "MHA",
|
||||
"move": "Move",
|
||||
"mul": "Mul",
|
||||
"resh": "Reshape",
|
||||
"sele": "Select",
|
||||
"slic": "Slice",
|
||||
"sqrt": "Sqrt",
|
||||
"sub": "Sub",
|
||||
"tanh": "Tanh",
|
||||
"tran": "Transpose",
|
||||
}
|
||||
|
||||
|
||||
def strip_kgen_signature_noise(text: str) -> str:
|
||||
text = re.sub(r"_0x[0-9A-Fa-f]+$", "", text)
|
||||
text = re.sub(r"_myl\d+(?:_\d+)?$", "", text)
|
||||
text = re.sub(r"^__?myl_", "", text)
|
||||
return text.strip("_")
|
||||
|
||||
|
||||
def split_kgen_signature(signature: str) -> list[str]:
|
||||
if not signature:
|
||||
return []
|
||||
if "_" in signature:
|
||||
return [part for part in signature.split("_") if part]
|
||||
return re.findall(r"[A-Z][a-z0-9]*|[a-z0-9]+", signature)
|
||||
|
||||
|
||||
def normalize_kgen_token(token: str) -> str:
|
||||
lowered = token.lower()
|
||||
if lowered in KGEN_TOKEN_MAP:
|
||||
return KGEN_TOKEN_MAP[lowered]
|
||||
if lowered.startswith("v") and lowered[1:].isdigit():
|
||||
return lowered.upper()
|
||||
return token[:1].upper() + token[1:]
|
||||
|
||||
|
||||
def kgen_details(layer: dict[str, Any]) -> tuple[str, list[str], str]:
|
||||
candidates = [str(layer.get("TacticName", "")), layer_name(layer, 0)]
|
||||
for candidate in candidates:
|
||||
signature = strip_kgen_signature_noise(candidate)
|
||||
tokens = [normalize_kgen_token(token) for token in split_kgen_signature(signature)]
|
||||
if tokens:
|
||||
return signature, tokens, f"KGEN_{'_'.join(tokens)}"
|
||||
return "", [], "kgen"
|
||||
|
||||
|
||||
def node_op_type(layer: dict[str, Any]) -> tuple[str, dict[str, Any]]:
|
||||
raw_layer_type = layer.get("LayerType", "Layer")
|
||||
if str(raw_layer_type).lower() != "kgen":
|
||||
return sanitize_op_type(raw_layer_type), {}
|
||||
|
||||
signature, tokens, op_type = kgen_details(layer)
|
||||
attrs: dict[str, Any] = {}
|
||||
if signature:
|
||||
attrs["trt_kgen_signature"] = signature
|
||||
if tokens:
|
||||
attrs["trt_kgen_ops_json"] = compact_json(tokens)
|
||||
attrs["trt_kgen_op_count"] = len(tokens)
|
||||
return sanitize_op_type(op_type), attrs
|
||||
|
||||
|
||||
def layer_name(layer: dict[str, Any], index: int) -> str:
|
||||
raw = layer.get("Name")
|
||||
if raw is None or str(raw).strip() == "":
|
||||
return f"trt_layer_{index}"
|
||||
return str(raw)
|
||||
|
||||
|
||||
def int_attr(value: Any, default: int = 0) -> int:
|
||||
if isinstance(value, bool):
|
||||
return int(value)
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
try:
|
||||
return int(str(value))
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def process_layers(layers: list[dict[str, Any]], source_path: Path) -> ProcessedLayerInfo:
|
||||
registry = NameRegistry()
|
||||
external_by_original: dict[str, str] = {}
|
||||
latest_by_original: dict[str, str] = {}
|
||||
descriptors_by_value: dict[str, dict[str, Any]] = {}
|
||||
initializers_by_original: dict[str, str] = {}
|
||||
initializers: list[ProcessedInitializer] = []
|
||||
produced_values: list[str] = []
|
||||
consumed_values: set[str] = set()
|
||||
nodes: list[ProcessedNode] = []
|
||||
|
||||
def map_external(desc: dict[str, Any], fallback: str) -> str:
|
||||
original = tensor_name(desc, fallback)
|
||||
if original not in external_by_original:
|
||||
value_name = registry.reserve(original, fallback)
|
||||
external_by_original[original] = value_name
|
||||
descriptors_by_value[value_name] = desc
|
||||
return external_by_original[original]
|
||||
|
||||
def map_input(desc: dict[str, Any], fallback: str) -> str:
|
||||
original = tensor_name(desc, fallback)
|
||||
value_name = latest_by_original.get(original)
|
||||
if value_name is None:
|
||||
value_name = map_external(desc, fallback)
|
||||
descriptors_by_value.setdefault(value_name, desc)
|
||||
consumed_values.add(value_name)
|
||||
return value_name
|
||||
|
||||
def map_constant(desc: dict[str, Any], fallback: str) -> str:
|
||||
original = tensor_name(desc, fallback)
|
||||
if original not in initializers_by_original:
|
||||
value_name = registry.reserve(original, fallback)
|
||||
initializers_by_original[original] = value_name
|
||||
descriptors_by_value[value_name] = desc
|
||||
initializers.append(ProcessedInitializer(value_name, desc))
|
||||
value_name = initializers_by_original[original]
|
||||
descriptors_by_value.setdefault(value_name, desc)
|
||||
consumed_values.add(value_name)
|
||||
return value_name
|
||||
|
||||
def map_output(desc: dict[str, Any], fallback: str) -> str:
|
||||
original = tensor_name(desc, fallback)
|
||||
value_name = registry.reserve(original, fallback)
|
||||
latest_by_original[original] = value_name
|
||||
descriptors_by_value[value_name] = desc
|
||||
produced_values.append(value_name)
|
||||
return value_name
|
||||
|
||||
layer_type_counts: Counter[str] = Counter()
|
||||
node_op_type_counts: Counter[str] = Counter()
|
||||
unmapped_layer_fields: set[str] = set()
|
||||
|
||||
for index, layer in enumerate(layers):
|
||||
inputs = as_descriptor_list(layer.get("Inputs"))
|
||||
constants = as_descriptor_list(layer.get("Constants"))
|
||||
outputs = as_descriptor_list(layer.get("Outputs"))
|
||||
|
||||
node_inputs = [
|
||||
map_input(desc, f"trt_layer_{index}_input_{pos}")
|
||||
for pos, desc in enumerate(inputs)
|
||||
]
|
||||
node_inputs.extend(
|
||||
map_constant(desc, f"trt_layer_{index}_constant_{pos}")
|
||||
for pos, desc in enumerate(constants)
|
||||
)
|
||||
node_outputs = [
|
||||
map_output(desc, f"trt_layer_{index}_output_{pos}")
|
||||
for pos, desc in enumerate(outputs)
|
||||
]
|
||||
|
||||
raw_layer_type = layer.get("LayerType", "Layer")
|
||||
layer_type_counts[str(raw_layer_type)] += 1
|
||||
op_type, op_type_attrs = node_op_type(layer)
|
||||
node_op_type_counts[op_type] += 1
|
||||
extra = {key: value for key, value in layer.items() if key not in STANDARD_LAYER_KEYS}
|
||||
unmapped_layer_fields.update(extra)
|
||||
|
||||
attrs: dict[str, Any] = {
|
||||
"trt_layer_index": index,
|
||||
"trt_layer_type": str(raw_layer_type),
|
||||
"trt_input_count": len(inputs),
|
||||
"trt_constant_count": len(constants),
|
||||
"trt_output_count": len(outputs),
|
||||
"trt_inputs_json": compact_json(inputs),
|
||||
"trt_constants_json": compact_json(constants),
|
||||
"trt_outputs_json": compact_json(outputs),
|
||||
}
|
||||
attrs.update(op_type_attrs)
|
||||
|
||||
if "TacticName" in layer:
|
||||
attrs["trt_tactic_name"] = str(layer.get("TacticName", ""))
|
||||
if "StreamId" in layer:
|
||||
attrs["trt_stream_id"] = int_attr(layer.get("StreamId"))
|
||||
if "_subgraph" in layer:
|
||||
attrs["trt_subgraph"] = int_attr(layer.get("_subgraph"))
|
||||
if "Metadata" in layer:
|
||||
attrs["trt_metadata"] = str(layer.get("Metadata", ""))
|
||||
if extra:
|
||||
attrs["trt_extra_json"] = compact_json(extra)
|
||||
|
||||
nodes.append(
|
||||
ProcessedNode(
|
||||
name=layer_name(layer, index),
|
||||
op_type=op_type,
|
||||
inputs=node_inputs,
|
||||
outputs=node_outputs,
|
||||
attrs=attrs,
|
||||
)
|
||||
)
|
||||
|
||||
terminal_values = [value_name for value_name in produced_values if value_name not in consumed_values]
|
||||
if not terminal_values and produced_values:
|
||||
terminal_values = [produced_values[-1]]
|
||||
|
||||
graph_output_names = set(terminal_values)
|
||||
graph_input_names = set(external_by_original.values())
|
||||
graph_value_info = [
|
||||
value_name
|
||||
for value_name in descriptors_by_value
|
||||
if value_name not in graph_input_names and value_name not in graph_output_names
|
||||
]
|
||||
|
||||
return ProcessedLayerInfo(
|
||||
source_path=source_path,
|
||||
layers=layers,
|
||||
nodes=nodes,
|
||||
graph_inputs=list(external_by_original.values()),
|
||||
graph_outputs=terminal_values,
|
||||
graph_value_info=graph_value_info,
|
||||
value_descriptors=descriptors_by_value,
|
||||
initializers=initializers,
|
||||
layer_type_counts=layer_type_counts,
|
||||
node_op_type_counts=node_op_type_counts,
|
||||
extra_layer_fields=sorted(unmapped_layer_fields),
|
||||
)
|
||||
|
||||
|
||||
def load_and_process_layer_info(path: Path) -> ProcessedLayerInfo:
|
||||
return process_layers(load_layers(path), path)
|
||||
|
||||
|
||||
def resolve_target_spec(target: str) -> TargetSpec:
|
||||
try:
|
||||
return TARGETS_BY_NAME[target]
|
||||
except KeyError as exc:
|
||||
valid = ", ".join(sorted(TARGETS_BY_NAME))
|
||||
raise ValueError(f"unknown target {target!r}; valid targets: {valid}") from exc
|
||||
|
||||
|
||||
def default_output_path(input_path: Path, target: str = "html") -> Path:
|
||||
spec = resolve_target_spec(target)
|
||||
return input_path.with_name(f"{input_path.stem}{spec.default_suffix}")
|
||||
|
||||
|
||||
def target_names_help() -> str:
|
||||
return ", ".join(spec.name for spec in TARGET_SPECS)
|
||||
|
||||
|
||||
def load_target_module(spec: TargetSpec) -> Any:
|
||||
return importlib.import_module(spec.module_name)
|
||||
|
||||
|
||||
def write_target(processed: ProcessedLayerInfo, spec: TargetSpec, output_path: Path) -> None:
|
||||
module = load_target_module(spec)
|
||||
writer = getattr(module, "write", None)
|
||||
if writer is None:
|
||||
raise NotImplementedError(
|
||||
f"target {spec.name!r} is not implemented yet; "
|
||||
f"expected write(processed, output_path) in {spec.module_name}"
|
||||
)
|
||||
writer(processed, output_path)
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Process TensorRT layer-info JSON and write one or more visualization targets."
|
||||
)
|
||||
parser.add_argument("input", type=Path, nargs="?", help="Path to TensorRT layer-info JSON")
|
||||
parser.add_argument(
|
||||
"-t",
|
||||
"--target",
|
||||
action="append",
|
||||
dest="targets",
|
||||
metavar="TARGET",
|
||||
help=f"Output target; repeat for multiple outputs. Default: html. Available: {target_names_help()}",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-o",
|
||||
"--output",
|
||||
type=Path,
|
||||
help="Output path. Only valid when writing one target.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=Path,
|
||||
help="Directory for target-specific default filenames.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--list-targets",
|
||||
action="store_true",
|
||||
help="List supported target names and exit.",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def list_targets() -> None:
|
||||
seen: set[str] = set()
|
||||
for spec in TARGET_SPECS:
|
||||
aliases = f" (aliases: {', '.join(spec.aliases)})" if spec.aliases else ""
|
||||
print(f"{spec.name}{aliases}: {spec.description}")
|
||||
seen.add(spec.name)
|
||||
|
||||
alias_only = sorted(name for name, spec in TARGETS_BY_NAME.items() if spec.name not in seen)
|
||||
if alias_only:
|
||||
print(f"aliases: {', '.join(alias_only)}")
|
||||
|
||||
|
||||
def resolve_requested_targets(targets: list[str] | None) -> list[TargetSpec]:
|
||||
requested = targets or ["html"]
|
||||
specs: list[TargetSpec] = []
|
||||
seen: set[str] = set()
|
||||
for target in requested:
|
||||
spec = resolve_target_spec(target)
|
||||
if spec.name in seen:
|
||||
continue
|
||||
specs.append(spec)
|
||||
seen.add(spec.name)
|
||||
return specs
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(sys.argv[1:] if argv is None else argv)
|
||||
if args.list_targets:
|
||||
list_targets()
|
||||
return 0
|
||||
if args.input is None:
|
||||
print("error: input is required unless --list-targets is used", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
try:
|
||||
target_specs = resolve_requested_targets(args.targets)
|
||||
if args.output is not None and len(target_specs) != 1:
|
||||
raise ValueError("--output can only be used when writing one target")
|
||||
|
||||
processed = load_and_process_layer_info(args.input)
|
||||
written: list[Path] = []
|
||||
for spec in target_specs:
|
||||
if args.output is not None:
|
||||
output_path = args.output
|
||||
elif args.output_dir is not None:
|
||||
output_path = args.output_dir / f"{args.input.stem}{spec.default_suffix}"
|
||||
else:
|
||||
output_path = default_output_path(args.input, spec.name)
|
||||
|
||||
write_target(processed, spec, output_path)
|
||||
written.append(output_path)
|
||||
except Exception as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
for output_path in written:
|
||||
print(f"wrote {output_path}")
|
||||
return 0
|
||||
@@ -0,0 +1,220 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Write a standalone HTML visualization target for TensorRT layer-info data."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def escape(value: Any) -> str:
|
||||
return html.escape(str(value), quote=True)
|
||||
|
||||
|
||||
def compact_json(value: Any) -> str:
|
||||
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
|
||||
|
||||
|
||||
def short(value: Any, limit: int = 180) -> str:
|
||||
text = str(value)
|
||||
if len(text) <= limit:
|
||||
return text
|
||||
return f"{text[: limit - 1]}..."
|
||||
|
||||
|
||||
def table_rows(items: list[tuple[str, Any]]) -> str:
|
||||
rows = []
|
||||
for key, value in items:
|
||||
rows.append(
|
||||
"<tr>"
|
||||
f"<th>{escape(key)}</th>"
|
||||
f"<td><code>{escape(value)}</code></td>"
|
||||
"</tr>"
|
||||
)
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
def write(processed: Any, output_path: Path) -> None:
|
||||
layer_rows = []
|
||||
for node in processed.nodes:
|
||||
attrs = node.attrs
|
||||
layer_rows.append(
|
||||
"<tr>"
|
||||
f"<td>{escape(attrs.get('trt_layer_index', ''))}</td>"
|
||||
f"<td><code>{escape(short(node.name, 140))}</code></td>"
|
||||
f"<td>{escape(attrs.get('trt_layer_type', node.op_type))}</td>"
|
||||
f"<td>{escape(node.op_type)}</td>"
|
||||
f"<td>{escape(len(node.inputs))}</td>"
|
||||
f"<td>{escape(len(node.outputs))}</td>"
|
||||
f"<td><code>{escape(short(attrs.get('trt_tactic_name', ''), 160))}</code></td>"
|
||||
"</tr>"
|
||||
)
|
||||
|
||||
summary_rows = table_rows(
|
||||
[
|
||||
("source", processed.source_path),
|
||||
("layers", len(processed.layers)),
|
||||
("nodes", len(processed.nodes)),
|
||||
("graph inputs", len(processed.graph_inputs)),
|
||||
("graph outputs", len(processed.graph_outputs)),
|
||||
("initializers", len(processed.initializers)),
|
||||
("layer types", compact_json(dict(processed.layer_type_counts))),
|
||||
("node op types", compact_json(dict(processed.node_op_type_counts))),
|
||||
]
|
||||
)
|
||||
|
||||
graph_inputs = "\n".join(f"<li><code>{escape(name)}</code></li>" for name in processed.graph_inputs)
|
||||
graph_outputs = "\n".join(f"<li><code>{escape(name)}</code></li>" for name in processed.graph_outputs)
|
||||
|
||||
document = f"""<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>TensorRT Layer Info: {escape(processed.source_path.name)}</title>
|
||||
<style>
|
||||
:root {{
|
||||
color-scheme: light dark;
|
||||
--bg: #f8f9fb;
|
||||
--fg: #17202a;
|
||||
--muted: #586476;
|
||||
--line: #d7dce5;
|
||||
--panel: #ffffff;
|
||||
--accent: #0f766e;
|
||||
--code: #0b4a64;
|
||||
}}
|
||||
@media (prefers-color-scheme: dark) {{
|
||||
:root {{
|
||||
--bg: #111417;
|
||||
--fg: #edf1f5;
|
||||
--muted: #aab4c0;
|
||||
--line: #303942;
|
||||
--panel: #181d22;
|
||||
--accent: #5eead4;
|
||||
--code: #93c5fd;
|
||||
}}
|
||||
}}
|
||||
* {{ box-sizing: border-box; }}
|
||||
body {{
|
||||
margin: 0;
|
||||
background: var(--bg);
|
||||
color: var(--fg);
|
||||
font: 14px/1.5 system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
}}
|
||||
main {{
|
||||
width: min(1400px, calc(100vw - 32px));
|
||||
margin: 0 auto;
|
||||
padding: 28px 0 48px;
|
||||
}}
|
||||
h1 {{
|
||||
margin: 0 0 6px;
|
||||
font-size: 24px;
|
||||
letter-spacing: 0;
|
||||
}}
|
||||
h2 {{
|
||||
margin: 28px 0 10px;
|
||||
font-size: 17px;
|
||||
letter-spacing: 0;
|
||||
}}
|
||||
p {{ margin: 0 0 16px; color: var(--muted); }}
|
||||
code {{
|
||||
color: var(--code);
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||||
font-size: 12px;
|
||||
word-break: break-word;
|
||||
}}
|
||||
table {{
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--line);
|
||||
}}
|
||||
th, td {{
|
||||
border-bottom: 1px solid var(--line);
|
||||
padding: 8px 10px;
|
||||
text-align: left;
|
||||
vertical-align: top;
|
||||
}}
|
||||
th {{
|
||||
color: var(--muted);
|
||||
font-weight: 600;
|
||||
white-space: nowrap;
|
||||
}}
|
||||
thead th {{
|
||||
position: sticky;
|
||||
top: 0;
|
||||
background: var(--panel);
|
||||
z-index: 1;
|
||||
}}
|
||||
tbody tr:last-child td, tbody tr:last-child th {{ border-bottom: 0; }}
|
||||
.summary th {{ width: 160px; }}
|
||||
.split {{
|
||||
display: grid;
|
||||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||||
gap: 16px;
|
||||
}}
|
||||
.panel {{
|
||||
background: var(--panel);
|
||||
border: 1px solid var(--line);
|
||||
padding: 12px 14px;
|
||||
}}
|
||||
ul {{
|
||||
margin: 0;
|
||||
padding-left: 18px;
|
||||
}}
|
||||
@media (max-width: 780px) {{
|
||||
main {{ width: min(100vw - 20px, 1400px); padding-top: 18px; }}
|
||||
.split {{ grid-template-columns: 1fr; }}
|
||||
th, td {{ padding: 7px 8px; }}
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>TensorRT Layer Info</h1>
|
||||
<p>{escape(processed.source_path)}</p>
|
||||
|
||||
<h2>Summary</h2>
|
||||
<table class="summary"><tbody>
|
||||
{summary_rows}
|
||||
</tbody></table>
|
||||
|
||||
<h2>Graph Endpoints</h2>
|
||||
<div class="split">
|
||||
<section class="panel">
|
||||
<h2>Inputs</h2>
|
||||
<ul>{graph_inputs}</ul>
|
||||
</section>
|
||||
<section class="panel">
|
||||
<h2>Outputs</h2>
|
||||
<ul>{graph_outputs}</ul>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<h2>Layers</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>#</th>
|
||||
<th>Name</th>
|
||||
<th>Layer Type</th>
|
||||
<th>Op Type</th>
|
||||
<th>Inputs</th>
|
||||
<th>Outputs</th>
|
||||
<th>Tactic</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{chr(10).join(layer_rows)}
|
||||
</tbody>
|
||||
</table>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(document, encoding="utf-8")
|
||||
@@ -0,0 +1,22 @@
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Serialize extracted TensorRT performance data as JSON."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
def json_ready(value: Any) -> Any:
|
||||
if isinstance(value, dict):
|
||||
return {str(key): json_ready(item) for key, item in value.items()}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [json_ready(item) for item in value]
|
||||
if isinstance(value, (str, int, float, bool)) or value is None:
|
||||
return value
|
||||
return str(value)
|
||||
|
||||
|
||||
def render_results_json(data: Dict[str, Any], indent: int = 2) -> str:
|
||||
return json.dumps(json_ready(data), indent=indent, sort_keys=True) + "\n"
|
||||
@@ -0,0 +1,246 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Validate analyzer JSON output against schemas/analyze-data.schema.json.
|
||||
|
||||
This intentionally implements only the JSON Schema keywords used by the local
|
||||
schema so it can run without third-party dependencies in benchmark environments.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from typing import Any, List, Optional, Sequence
|
||||
|
||||
|
||||
class ValidationError(Exception):
|
||||
"""Raised for schema loading or schema reference errors."""
|
||||
|
||||
|
||||
def default_schema_path() -> str:
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
return os.path.abspath(os.path.join(script_dir, "..", "schemas", "analyze-data.schema.json"))
|
||||
|
||||
|
||||
def load_json(path: str) -> Any:
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
except FileNotFoundError as exc:
|
||||
raise ValidationError(f"file not found: {path}") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValidationError(f"invalid JSON in {path}: line {exc.lineno}, column {exc.colno}: {exc.msg}") from exc
|
||||
except OSError as exc:
|
||||
raise ValidationError(f"unable to read {path}: {exc}") from exc
|
||||
|
||||
|
||||
def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Validate TensorRT analysis JSON generated by analyze_trt_perf.py."
|
||||
)
|
||||
parser.add_argument("input", help="Path to generated analysis JSON.")
|
||||
parser.add_argument(
|
||||
"--schema",
|
||||
default=default_schema_path(),
|
||||
help="Schema path. Defaults to schemas/analyze-data.schema.json.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-errors",
|
||||
type=int,
|
||||
default=40,
|
||||
help="Maximum validation errors to print. Defaults to 40.",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def decode_ref_token(token: str) -> str:
|
||||
return token.replace("~1", "/").replace("~0", "~")
|
||||
|
||||
|
||||
def resolve_ref(ref: str, root_schema: Any) -> Any:
|
||||
if not ref.startswith("#/"):
|
||||
raise ValidationError(f"only local JSON Schema refs are supported: {ref}")
|
||||
|
||||
current = root_schema
|
||||
for token in ref[2:].split("/"):
|
||||
key = decode_ref_token(token)
|
||||
if not isinstance(current, dict) or key not in current:
|
||||
raise ValidationError(f"unresolved schema ref: {ref}")
|
||||
current = current[key]
|
||||
return current
|
||||
|
||||
|
||||
def type_label(instance: Any) -> str:
|
||||
if instance is None:
|
||||
return "null"
|
||||
if isinstance(instance, bool):
|
||||
return "boolean"
|
||||
if isinstance(instance, int):
|
||||
return "integer"
|
||||
if isinstance(instance, float):
|
||||
return "number"
|
||||
if isinstance(instance, str):
|
||||
return "string"
|
||||
if isinstance(instance, list):
|
||||
return "array"
|
||||
if isinstance(instance, dict):
|
||||
return "object"
|
||||
return type(instance).__name__
|
||||
|
||||
|
||||
def matches_type(instance: Any, expected: str) -> bool:
|
||||
if expected == "null":
|
||||
return instance is None
|
||||
if expected == "boolean":
|
||||
return isinstance(instance, bool)
|
||||
if expected == "integer":
|
||||
return isinstance(instance, int) and not isinstance(instance, bool)
|
||||
if expected == "number":
|
||||
return isinstance(instance, (int, float)) and not isinstance(instance, bool)
|
||||
if expected == "string":
|
||||
return isinstance(instance, str)
|
||||
if expected == "array":
|
||||
return isinstance(instance, list)
|
||||
if expected == "object":
|
||||
return isinstance(instance, dict)
|
||||
raise ValidationError(f"unsupported schema type: {expected}")
|
||||
|
||||
|
||||
def matches_any_type(instance: Any, expected: Any) -> bool:
|
||||
if isinstance(expected, str):
|
||||
return matches_type(instance, expected)
|
||||
if isinstance(expected, list):
|
||||
return any(matches_type(instance, item) for item in expected)
|
||||
raise ValidationError(f"invalid schema type declaration: {expected!r}")
|
||||
|
||||
|
||||
def append_error(errors: List[str], path: str, message: str) -> None:
|
||||
errors.append(f"{path}: {message}")
|
||||
|
||||
|
||||
def append_nearest_errors(errors: List[str], matches: List[List[str]], limit: int = 5) -> None:
|
||||
if not matches:
|
||||
return
|
||||
nearest = min(matches, key=lambda item: (sum("expected constant" in error for error in item), len(item)))
|
||||
for error in nearest[:limit]:
|
||||
errors.append(f" nearest schema: {error}")
|
||||
|
||||
|
||||
def validate(instance: Any, schema: Any, path: str, root_schema: Any) -> List[str]:
|
||||
if not isinstance(schema, dict):
|
||||
raise ValidationError(f"{path}: schema node is not an object")
|
||||
|
||||
if "$ref" in schema:
|
||||
return validate(instance, resolve_ref(str(schema["$ref"]), root_schema), path, root_schema)
|
||||
|
||||
errors: List[str] = []
|
||||
|
||||
if "allOf" in schema:
|
||||
for subschema in schema["allOf"]:
|
||||
errors.extend(validate(instance, subschema, path, root_schema))
|
||||
|
||||
if "anyOf" in schema:
|
||||
matches = [validate(instance, subschema, path, root_schema) for subschema in schema["anyOf"]]
|
||||
if not any(not match_errors for match_errors in matches):
|
||||
append_error(errors, path, "does not match any allowed schema")
|
||||
append_nearest_errors(errors, matches)
|
||||
|
||||
if "oneOf" in schema:
|
||||
matches = [validate(instance, subschema, path, root_schema) for subschema in schema["oneOf"]]
|
||||
match_count = sum(1 for match_errors in matches if not match_errors)
|
||||
if match_count != 1:
|
||||
append_error(errors, path, f"matches {match_count} schemas, expected exactly one")
|
||||
if match_count == 0:
|
||||
append_nearest_errors(errors, matches)
|
||||
|
||||
if "const" in schema and instance != schema["const"]:
|
||||
append_error(errors, path, f"expected constant {schema['const']!r}, got {instance!r}")
|
||||
|
||||
if "enum" in schema and instance not in schema["enum"]:
|
||||
append_error(errors, path, f"expected one of {schema['enum']!r}, got {instance!r}")
|
||||
|
||||
expected_type = schema.get("type")
|
||||
if expected_type is not None and not matches_any_type(instance, expected_type):
|
||||
append_error(errors, path, f"expected type {expected_type!r}, got {type_label(instance)}")
|
||||
return errors
|
||||
|
||||
if isinstance(instance, dict):
|
||||
validate_object(instance, schema, path, root_schema, errors)
|
||||
elif isinstance(instance, list):
|
||||
validate_array(instance, schema, path, root_schema, errors)
|
||||
elif isinstance(instance, (int, float)) and not isinstance(instance, bool):
|
||||
validate_number(instance, schema, path, errors)
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
def validate_object(instance: dict, schema: dict, path: str, root_schema: Any, errors: List[str]) -> None:
|
||||
properties = schema.get("properties", {})
|
||||
required = schema.get("required", [])
|
||||
additional = schema.get("additionalProperties", True)
|
||||
|
||||
for key in required:
|
||||
if key not in instance:
|
||||
append_error(errors, path, f"missing required property {key!r}")
|
||||
|
||||
for key, value in instance.items():
|
||||
child_path = f"{path}.{key}"
|
||||
if key in properties:
|
||||
errors.extend(validate(value, properties[key], child_path, root_schema))
|
||||
elif additional is False:
|
||||
append_error(errors, child_path, "additional property is not allowed")
|
||||
elif isinstance(additional, dict):
|
||||
errors.extend(validate(value, additional, child_path, root_schema))
|
||||
|
||||
|
||||
def validate_array(instance: list, schema: dict, path: str, root_schema: Any, errors: List[str]) -> None:
|
||||
min_items = schema.get("minItems")
|
||||
if isinstance(min_items, int) and len(instance) < min_items:
|
||||
append_error(errors, path, f"expected at least {min_items} items, got {len(instance)}")
|
||||
|
||||
max_items = schema.get("maxItems")
|
||||
if isinstance(max_items, int) and len(instance) > max_items:
|
||||
append_error(errors, path, f"expected at most {max_items} items, got {len(instance)}")
|
||||
|
||||
items_schema = schema.get("items")
|
||||
if isinstance(items_schema, dict):
|
||||
for index, value in enumerate(instance):
|
||||
errors.extend(validate(value, items_schema, f"{path}[{index}]", root_schema))
|
||||
|
||||
|
||||
def validate_number(instance: float, schema: dict, path: str, errors: List[str]) -> None:
|
||||
if "minimum" in schema and instance < schema["minimum"]:
|
||||
append_error(errors, path, f"expected >= {schema['minimum']}, got {instance}")
|
||||
if "maximum" in schema and instance > schema["maximum"]:
|
||||
append_error(errors, path, f"expected <= {schema['maximum']}, got {instance}")
|
||||
|
||||
|
||||
def main(argv: Optional[Sequence[str]] = None) -> int:
|
||||
args = parse_args(argv)
|
||||
|
||||
try:
|
||||
data = load_json(args.input)
|
||||
schema = load_json(args.schema)
|
||||
errors = validate(data, schema, "$", schema)
|
||||
except ValidationError as exc:
|
||||
sys.stderr.write(f"error: {exc}\n")
|
||||
return 2
|
||||
|
||||
if errors:
|
||||
sys.stderr.write(f"{args.input}: FAIL ({len(errors)} schema error(s))\n")
|
||||
for error in errors[: args.max_errors]:
|
||||
sys.stderr.write(f"- {error}\n")
|
||||
remaining = len(errors) - args.max_errors
|
||||
if remaining > 0:
|
||||
sys.stderr.write(f"... {remaining} more error(s) omitted\n")
|
||||
return 1
|
||||
|
||||
sys.stdout.write(f"{args.input}: PASS\n")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Standalone TensorRT layer-info HTML visualizer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from trt_perf.layer_info import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,418 @@
|
||||
---
|
||||
name: trt-strong-typing-migration
|
||||
description: >-
|
||||
Migrate a TensorRT build from weak typing (deprecated 10.12, removed 11.0) to
|
||||
strong typing — across Python INetworkDefinition builders, the trtexec CLI, and
|
||||
C++ builder code. Use when a TRT 11 upgrade breaks a weakly-typed build. Triggers: weakly typed to strongly typed,
|
||||
kSTRONGLY_TYPED, weak typing deprecated, kFP16/kINT8 removed, setPrecision rejected,
|
||||
setComputePrecision deprecated, do I still need --stronglyTyped, how to add the
|
||||
kSTRONGLY_TYPED flag, ModelOpt autocast, INT8 on TRT 11. NOT for ONNX import
|
||||
(`trt-onnx-quickstart`), Torch-TRT (`trt-torch-quickstart`), or C++ deploy
|
||||
(`trt-cpp-runtime-quickstart`).
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: NVIDIA Corporation
|
||||
version: "1.0"
|
||||
tags:
|
||||
- migration
|
||||
- strong-typing
|
||||
- autocast
|
||||
- modelopt
|
||||
- builder
|
||||
---
|
||||
|
||||
# Migrate Weakly-Typed to Strongly-Typed TensorRT Build
|
||||
|
||||
Strong typing is the recommended build mode for TensorRT 11.x; weak typing was
|
||||
deprecated in 10.12 and is scheduled for removal. This skill migrates a weakly-typed
|
||||
build flow to a strongly-typed equivalent, one path per entry point (Python network,
|
||||
`trtexec`, C++ builder), plus the ModelOpt AutoCast pre-step needed when the source
|
||||
is an ONNX file that must run in mixed precision.
|
||||
|
||||
**Version behavior.** On **TensorRT 11.x strong typing is unconditional** — every
|
||||
network is strongly typed, the `kSTRONGLY_TYPED` creation flag is accepted but
|
||||
ignored, and `trtexec --stronglyTyped` is a no-op (default-on). On **10.12–10.x**
|
||||
you opt in explicitly via that flag / `--stronglyTyped`. Setting it is therefore
|
||||
always safe: required on 10.x, harmless on 11.x. Either way the migration is the
|
||||
same — remove the precision *hints* (gone in 11.x) and move precision into the graph.
|
||||
|
||||
## When to Use
|
||||
|
||||
| Scenario | Use this skill? |
|
||||
|----------|-----------------|
|
||||
| Existing Python build calls `config.set_flag(trt.BuilderFlag.FP16 / INT8 / TF32)` | Yes |
|
||||
| Existing C++ build calls `config->setFlag(BuilderFlag::kFP16 / kINT8 / kTF32)` | Yes |
|
||||
| Existing `trtexec` line uses `--fp16`, `--int8`, `--best`, `--bf16` and the developer wants strong typing | Yes |
|
||||
| Build emits "weak typing is deprecated, use STRONGLY_TYPED" warning at runtime | Yes |
|
||||
| Build fails with "flag X is incompatible with strongly typed networks" | Yes |
|
||||
| Developer has a FP32 ONNX file and wants a strongly-typed mixed FP16 engine | Yes — start at Step 1 (AutoCast) |
|
||||
| Developer is importing a brand-new ONNX file with no existing build flow | No — use `trt-onnx-quickstart` first; come back here to switch the flag |
|
||||
| Developer's engine produces wrong outputs *after* migration | No — use `trt-quantization-accuracy` |
|
||||
| Developer is on TensorRT 10.11 or earlier | No — strong typing is fully available from 10.12 onward; upgrade before migrating |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **TensorRT 10.12 or later (11.x recommended).** Check:
|
||||
```bash
|
||||
python3 -c "import tensorrt; print(tensorrt.__version__)"
|
||||
trtexec --help 2>&1 | grep -i stronglyTyped
|
||||
```
|
||||
The version check is the determinant — strong typing is available from 10.12
|
||||
on. The `trtexec` probe just confirms the CLI flag for the trtexec path; a
|
||||
missing `--stronglyTyped` means a pre-10.12 install.
|
||||
|
||||
2. **ModelOpt installed when the source is an ONNX file requiring mixed precision.**
|
||||
```bash
|
||||
pip install nvidia-modelopt onnx onnxruntime
|
||||
```
|
||||
If the ONNX file is already strongly typed (contains explicit `Cast` nodes
|
||||
and FP16 initializers), AutoCast is not needed — skip directly to Step 2.
|
||||
|
||||
3. **A working baseline.** Capture the weakly-typed engine's outputs on a known
|
||||
input set before migrating. Strong typing is stricter, and any silent precision
|
||||
substitutions weak typing was making will become observable.
|
||||
|
||||
4. **The verifier script.** `scripts/verify.sh` runs `migrate.py` against a
|
||||
committed sample and asserts the rewrite is well-formed. Run it before editing
|
||||
anything to confirm the helper works.
|
||||
|
||||
## Migration paths
|
||||
|
||||
There are three migration paths — Python builder, `trtexec`, C++ builder.
|
||||
**Infer the path from the query when signals are clear; ask only if genuinely
|
||||
uncertain.** Match the strongest signal:
|
||||
|
||||
| Signal in the request | Path | Entry point |
|
||||
|------------------------|------|-------------|
|
||||
| Python tokens: `trt.BuilderFlag`, `set_flag(`, `create_network(`, `.py` | **A** | Python `INetworkDefinition` builder |
|
||||
| Standalone `trtexec --fp16` / `--int8` / `--best` / `--bf16` | **B** | `trtexec` CLI |
|
||||
| C++ tokens: `BuilderFlag::`, `createNetworkV2`, `->setFlag`, `.cpp` / `.h` | **C** | C++ builder |
|
||||
| Multiple signals | — | do each, most-specific first |
|
||||
| No usable signal (generic "how do I migrate") | — | ask which entry point, or proceed with Python and say you assumed it |
|
||||
|
||||
The migration preserves existing precisions (kFP16 → FP16 cast, kINT8 → Q/DQ) and is
|
||||
identical whether the network came from a parser or was hand-built — only Step 1
|
||||
(AutoCast) is ONNX-specific, so source and precision goal rarely need a question.
|
||||
|
||||
Inferring the path is a low-stakes default — it does **not** extend to destructive
|
||||
operations, which always need confirmation. `migrate.py` defaults to dry-run
|
||||
(prints a diff); the `--write` flag is the human-in-the-loop confirmation. Treat
|
||||
any other destructive step (overwriting a `.plan`, replacing an ONNX) the same way:
|
||||
show, confirm, then act.
|
||||
|
||||
For a strongly-typed INT8 model straight from an ONNX file (not a migration of
|
||||
existing builder code), jump to Step 1's ModelOpt quantization recipe.
|
||||
|
||||
## Step 1 — (ONNX source only) AutoCast the model to mixed precision
|
||||
|
||||
Skip if the source is not ONNX, or if the ONNX file is already strongly typed.
|
||||
|
||||
A FP32 ONNX file built for a weakly-typed `--fp16` flow carries no precision
|
||||
information — TensorRT was free to pick FP16 or FP32 per layer. Strong typing
|
||||
requires the graph itself to declare precision via `Cast` operators and typed
|
||||
initializers. ModelOpt's AutoCast inserts these.
|
||||
|
||||
Use the ModelOpt CLI — it's the supported entry point and takes `.npz` calibration
|
||||
directly. Use `--calibration_data` for both tools (autocast's canonical flag;
|
||||
quantization's abbreviation of `--calibration_data_path`):
|
||||
|
||||
```bash
|
||||
# FP16 / BF16 mixed-precision AutoCast
|
||||
python -m modelopt.onnx.autocast \
|
||||
--onnx_path model.fp32.onnx \
|
||||
--output_path model.mixed.onnx \
|
||||
--low_precision_type fp16 \
|
||||
--calibration_data calib.npz
|
||||
|
||||
# INT8 / FP8 strongly-typed quantization (produces ONNX with explicit Q/DQ)
|
||||
python -m modelopt.onnx.quantization \
|
||||
--onnx_path model.fp32.onnx \
|
||||
--output_path model.int8.onnx \
|
||||
--quantize_mode int8 \
|
||||
--high_precision_dtype fp16 \
|
||||
--calibration_data calib.npz
|
||||
```
|
||||
|
||||
Keep accuracy-sensitive ops in FP32 via AutoCast's node/op-type exclusion lists
|
||||
(e.g. exclude `MatMul`/`LayerNormalization` on transformer-heavy networks), and keep
|
||||
FP32 graph I/O when downstream stages expect it. See the
|
||||
[Model Optimizer docs](https://nvidia.github.io/Model-Optimizer) for the exact
|
||||
module and flags for your ModelOpt version.
|
||||
|
||||
Verify the converted model in ONNX Runtime before passing it to TRT:
|
||||
|
||||
```python
|
||||
import onnxruntime, numpy as np
|
||||
sess = onnxruntime.InferenceSession("model.mixed.onnx", providers=["CPUExecutionProvider"])
|
||||
# Compare a few outputs against the FP32 model with np.allclose(..., rtol=5e-3, atol=5e-3).
|
||||
```
|
||||
|
||||
If parity fails here, **stop** — the conversion is wrong and no amount of TRT
|
||||
engine work will recover it. Re-run with tighter exclusion lists.
|
||||
|
||||
**Next step — build the strongly-typed plan from the converted ONNX:**
|
||||
|
||||
```bash
|
||||
# For the INT8 case (modelopt.onnx.quantization output)
|
||||
trtexec --onnx=model.int8.onnx --stronglyTyped --saveEngine=model.int8.plan
|
||||
|
||||
# For the mixed-precision case (modelopt.onnx.autocast output)
|
||||
trtexec --onnx=model.mixed.onnx --stronglyTyped --saveEngine=model.fp16.plan
|
||||
```
|
||||
|
||||
**Critical**: do NOT pass `--int8` or `--fp16` alongside `--stronglyTyped`. Those flags are removed in TRT 11, and on TRT 10.12+ they conflict with strong typing. The precision is already encoded in the ONNX (via explicit Cast / Q / DQ nodes); the builder honors it.
|
||||
|
||||
The reference recipe for the end-to-end INT8 flow is `samples/python/strongly_type_autocast/sample.py` in `tensorrt-oss`.
|
||||
|
||||
## Step 2A — Migrate a Python `INetworkDefinition` builder
|
||||
|
||||
Mechanical replacements:
|
||||
|
||||
| Weakly typed | Strongly typed |
|
||||
|--------------|----------------|
|
||||
| `builder.create_network(0)` or `builder.create_network(1 << int(NetworkDefinitionCreationFlag.EXPLICIT_BATCH))` | `builder.create_network(1 << int(NetworkDefinitionCreationFlag.STRONGLY_TYPED))` |
|
||||
| `config.set_flag(trt.BuilderFlag.FP16)` | **Remove.** Types come from the network. |
|
||||
| `config.set_flag(trt.BuilderFlag.BF16)` | **Remove.** |
|
||||
| `config.set_flag(trt.BuilderFlag.INT8)` | **Remove.** Quantization expressed via Q/DQ nodes in the graph. |
|
||||
| `config.set_flag(trt.BuilderFlag.TF32)` | **Keep.** `kTF32` is retained in 11.x — it allows (not requires) TF32 for FP32 tensors and is orthogonal to typing. |
|
||||
| `layer.precision = trt.float16` | **Remove.** Cast the input tensor instead (`network.add_cast`). |
|
||||
| `layer.set_output_type(0, trt.float16)` | **Remove.** Output type follows the network; for cast/quantize/dequantize/fill layers use `set_to_type` instead. |
|
||||
| `builder.platform_has_fast_int8 / has_fast_fp16` checks gating the above | **Remove the gating logic** along with the flag. |
|
||||
|
||||
Worked rewrite (the MNIST builder from `samples/python/network_api_pytorch_mnist/sample.py`
|
||||
already uses the strongly-typed form — note `NetworkDefinitionCreationFlag.STRONGLY_TYPED`
|
||||
and the absence of any `BuilderFlag.FP16` call):
|
||||
|
||||
```python
|
||||
builder = trt.Builder(TRT_LOGGER)
|
||||
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))
|
||||
config = builder.create_builder_config()
|
||||
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, common.GiB(1))
|
||||
# ... populate network ...
|
||||
plan = builder.build_serialized_network(network, config)
|
||||
```
|
||||
|
||||
If the network is built via the ONNX parser, the parser respects strong typing
|
||||
automatically — no code change is needed beyond the flag.
|
||||
|
||||
Where a removed `precision` / `set_output_type` hint actually mattered, encode it
|
||||
in the graph: cast a tensor's dtype with `add_cast`, or insert an explicit
|
||||
Quantize/Dequantize pair for INT8 (the scale is a build-time constant; ModelOpt
|
||||
derives it from calibration, or supply it by hand):
|
||||
|
||||
```python
|
||||
# kFP16 hint -> explicit cast on the tensor:
|
||||
cast = network.add_cast(some_tensor, trt.float16)
|
||||
|
||||
# kINT8 hint -> explicit Q/DQ pair (per-tensor scalar scale). NOTE: trt.Weights does
|
||||
# not copy the numpy buffer, so q_scale must stay alive until the engine is built.
|
||||
q_scale = np.array(1.0 / 127.0, dtype=np.float32).reshape(()) # rank-0 scalar
|
||||
scale = network.add_constant(shape=(), weights=trt.Weights(q_scale)).get_output(0)
|
||||
q = network.add_quantize(some_tensor, scale, trt.int8).get_output(0)
|
||||
dq = network.add_dequantize(q, scale, trt.float32).get_output(0)
|
||||
# Feed dq into the consumer; TensorRT fuses the Q/DQ into it so it runs in INT8.
|
||||
```
|
||||
|
||||
Drive the rewrite from `scripts/migrate.py`:
|
||||
|
||||
```bash
|
||||
python3 scripts/migrate.py path/to/build.py # dry-run diff
|
||||
python3 scripts/migrate.py path/to/build.py --write # rewrite in place
|
||||
```
|
||||
|
||||
The script edits AST nodes (not regexes) — it understands `create_network` and
|
||||
`set_flag` call shapes, so it is safe on files that mix conventions.
|
||||
|
||||
## Step 2B — Migrate a `trtexec` command line
|
||||
|
||||
Replacements:
|
||||
|
||||
| Weakly typed flag | Strongly typed equivalent |
|
||||
|-------------------|---------------------------|
|
||||
| `--fp16` | **Remove.** Precision comes from the ONNX. See the `--stronglyTyped` note below. |
|
||||
| `--bf16` | **Remove.** Same as `--fp16`. |
|
||||
| `--int8` | **Remove.** Q/DQ must be present in the ONNX. |
|
||||
| `--best` | **Remove.** Not allowed with `--stronglyTyped` on 10.12–10.x (see Common Errors). |
|
||||
| `--stronglyTyped` | **Version-gated.** Required on 10.12–10.x to opt into strong typing; a no-op (default-on) on 11.x — drop it or leave it, your choice. |
|
||||
| `--noTF32` | **Keep.** `kTF32` is retained in 11.x; `--noTF32` still disables TF32 and is orthogonal to typing. |
|
||||
| `--precisionConstraints=obey` / `prefer` | **Remove.** Constraints are expressed in the graph. |
|
||||
| `--layerPrecisions=...` | **Remove.** Use ONNX `Cast` nodes / AutoCast. |
|
||||
| `--layerOutputTypes=...` | **Remove.** |
|
||||
|
||||
Worked rewrite:
|
||||
|
||||
```bash
|
||||
# Before (weakly typed)
|
||||
trtexec --onnx=model.onnx --fp16 --saveEngine=model.fp16.plan \
|
||||
--minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 \
|
||||
--maxShapes=input:16x3x224x224
|
||||
|
||||
# After (strongly typed; ONNX has been AutoCast'd to mixed FP16)
|
||||
# On 11.x --stronglyTyped is the default and a no-op — the line below is equivalent
|
||||
# with or without it. On 10.12–10.x it is REQUIRED, or the build stays weakly typed.
|
||||
trtexec --onnx=model.mixed.onnx --stronglyTyped --saveEngine=model.fp16.plan \
|
||||
--minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 \
|
||||
--maxShapes=input:16x3x224x224
|
||||
```
|
||||
|
||||
The `trtexec` README's [Example 6](https://github.com/NVIDIA/TensorRT/blob/main/samples/trtexec/README.md#example-6-create-a-strongly-typed-plan-file)
|
||||
is the canonical reference.
|
||||
|
||||
## Step 2C — Migrate a C++ builder
|
||||
|
||||
Replacements mirror the Python path:
|
||||
|
||||
| Weakly typed (C++) | Strongly typed (C++) |
|
||||
|---|---|
|
||||
| `builder->createNetworkV2(0)` | `builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED))` |
|
||||
| `config->setFlag(BuilderFlag::kFP16)` | **Remove.** |
|
||||
| `config->setFlag(BuilderFlag::kBF16)` | **Remove.** |
|
||||
| `config->setFlag(BuilderFlag::kINT8)` | **Remove.** |
|
||||
| `config->setFlag(BuilderFlag::kTF32)` | **Keep.** `kTF32` is retained in 11.x — orthogonal to typing. |
|
||||
| `config->setFlag(BuilderFlag::kOBEY_PRECISION_CONSTRAINTS / kPREFER_PRECISION_CONSTRAINTS)` | **Remove.** No per-layer precision exists to constrain. |
|
||||
| `layer->setPrecision(DataType::kHALF)` | **Remove.** Cast the input tensor instead (`addCast`). |
|
||||
| `layer->setComputePrecision(...)` | **Remove.** Deprecated 10.16, removed; compute precision follows input dtypes. |
|
||||
| `layer->setOutputType(0, DataType::kHALF)` | **Remove.** For cast/quantize/dequantize/fill layers use `setToType()` instead. |
|
||||
|
||||
Worked rewrite:
|
||||
|
||||
```cpp
|
||||
// Before (10.x weakly typed) — the builder picks precision from these hints:
|
||||
auto network = std::unique_ptr<INetworkDefinition>(
|
||||
builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH)));
|
||||
auto config = std::unique_ptr<IBuilderConfig>(builder->createBuilderConfig());
|
||||
config->setFlag(BuilderFlag::kFP16); // hint: builder may use FP16
|
||||
config->setFlag(BuilderFlag::kINT8); // hint: builder may use INT8 (+ a calibrator)
|
||||
config->setFlag(BuilderFlag::kOBEY_PRECISION_CONSTRAINTS); // force the per-layer overrides below
|
||||
someLayer->setPrecision(DataType::kHALF); // per-layer override
|
||||
someLayer->setOutputType(0, DataType::kHALF); // per-layer override
|
||||
```
|
||||
|
||||
Each hint above maps to a concrete graph edit — there is no longer a "tell the
|
||||
builder to prefer X" knob, so state the dtype directly:
|
||||
|
||||
```cpp
|
||||
// After (11.x strongly typed) — precision lives in the graph; every line above is gone.
|
||||
auto network = std::unique_ptr<INetworkDefinition>(
|
||||
builder->createNetworkV2(1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)));
|
||||
// kSTRONGLY_TYPED: required on 10.12–10.x, accepted-but-ignored on 11.x (0 works too).
|
||||
auto config = std::unique_ptr<IBuilderConfig>(builder->createBuilderConfig());
|
||||
// No precision setFlag, no setPrecision / setOutputType.
|
||||
|
||||
// kFP16 hint -> cast the tensor's dtype explicitly:
|
||||
auto* cast = network->addCast(*someTensor, DataType::kHALF);
|
||||
// feed cast->getOutput(0) into the consumer layer
|
||||
|
||||
// kINT8 hint -> insert an explicit Quantize/Dequantize pair. The scale is a
|
||||
// build-time constant (per-tensor scalar here); ModelOpt derives it from calibration,
|
||||
// or supply it by hand. NOTE: TensorRT does not copy Weights, so qScale must stay
|
||||
// alive until the engine is built.
|
||||
float qScale = 1.0f / 127.0f;
|
||||
auto* scale = network->addConstant(Dims{0, {}},
|
||||
Weights{DataType::kFLOAT, &qScale, 1})->getOutput(0);
|
||||
auto* q = network->addQuantize(*someTensor, *scale, DataType::kINT8)->getOutput(0);
|
||||
auto* dq = network->addDequantize(*q, *scale, DataType::kFLOAT)->getOutput(0);
|
||||
// feed dq into the consumer; TensorRT fuses the Q/DQ into it so it runs in INT8.
|
||||
|
||||
// kOBEY_PRECISION_CONSTRAINTS + setPrecision/setOutputType -> nothing to port:
|
||||
// the dtype is now fixed by the cast / Q-DQ above, so there is no constraint to obey.
|
||||
```
|
||||
|
||||
`setToType()` is the only surviving per-layer type control, and is valid **only**
|
||||
on `ICastLayer`, `IDequantizeLayer`, `IDynamicQuantizeLayer`, `IFillLayer`, and
|
||||
`IQuantizeLayer` — not on arbitrary layers.
|
||||
|
||||
If the build pipeline parses ONNX via `nvonnxparser::IParser`, no parser change is
|
||||
required — the parser honors the strong typing flag set on the network.
|
||||
|
||||
## Step 3 — Rebuild and validate
|
||||
|
||||
1. Rebuild the engine with the migrated flow.
|
||||
2. Run the captured baseline inputs through the new engine.
|
||||
3. Compare against the weakly-typed baseline:
|
||||
- **Outputs match within `atol=5e-3, rtol=5e-3`**: migration is complete.
|
||||
- **Outputs diverge but stay within model accuracy tolerance**: weak typing
|
||||
was opportunistically using FP16/INT8 where the new graph keeps FP32 (or vice
|
||||
versa). Acceptable if the downstream metric (top-1, BLEU, WER) is unchanged;
|
||||
confirm with the original accuracy harness.
|
||||
- **Outputs diverge beyond tolerance**: see Common Errors below.
|
||||
|
||||
If the source was ONNX, also run the AutoCast'd ONNX through ONNX Runtime and
|
||||
compare against the FP32 ONNX before blaming TRT — graph-level conversion errors
|
||||
should be caught there.
|
||||
|
||||
## Migration Patterns Reference
|
||||
|
||||
| Pattern | Action |
|
||||
|---------|--------|
|
||||
| Network created with no flags (legacy implicit batch) | First migrate to explicit batch; only then add `STRONGLY_TYPED` |
|
||||
| Multiple `set_flag` calls (FP16 + INT8 + TF32 + REFIT) | Keep `REFIT`, `SPARSE_WEIGHTS`, `kTF32`; drop the precision-hint flags (FP16/BF16/INT8/FP8) and the removed `kOBEY`/`kPREFER_PRECISION_CONSTRAINTS` |
|
||||
| `set_calibration_profile` / `IInt8Calibrator` | **Remove.** PTQ calibration is replaced by Q/DQ in the ONNX. Use ModelOpt's PTQ for the model first. |
|
||||
| Mixed FP16/FP32 per-layer overrides via `layer.precision` | Express the same intent with `Cast` ops in the ONNX, or in the Python network construction code |
|
||||
| Plugin layers with explicit precision | Plugins must implement `IPluginV3OneBuild::getOutputDataTypes`; the network reads types from the plugin |
|
||||
| Engine consumed by downstream framework expecting FP32 I/O | Set `keep_io_types=True` in AutoCast, or add explicit `Cast` to FP32 at the network outputs |
|
||||
|
||||
## Common Errors
|
||||
|
||||
- **`"BuilderFlag::kFP16 is not compatible with a strongly typed network"`** —
|
||||
a `setFlag`/`set_flag` call survived the migration. Re-run `migrate.py` or grep
|
||||
for `BuilderFlag\.` / `BuilderFlag::` in the migrated source.
|
||||
- **`"--best is incompatible with --stronglyTyped"`** — drop `--best` from the
|
||||
`trtexec` line; precision choices live in the graph now.
|
||||
- **`"Network has no marked outputs"` immediately after switching the flag** —
|
||||
some builders only call `mark_output` on the path conditioned by
|
||||
`platform_has_fast_fp16`. Remove the conditional.
|
||||
- **AutoCast'd ONNX fails the parser with "If subgraph type mismatch"** — AutoCast
|
||||
can corrupt certain `If` node subgraphs. Add the affected `If` node names to
|
||||
`nodes_to_exclude` so they stay FP32.
|
||||
- **Engine output is FP32 zeros after migration** — `keep_io_types=False` was used
|
||||
but the host code still feeds FP32 buffers. Either set `keep_io_types=True` or
|
||||
cast the host input before binding.
|
||||
- **Accuracy regression of ~1-3% after migration** — known case for
|
||||
transformer-heavy networks: leftover `Cast(to=FLOAT32)` from AutoCast combined
|
||||
with FP16 weights causes thrash. Work around by excluding the residual cast op
|
||||
types in `op_types_to_exclude`.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Do not partially migrate.** Half-migrated builds (some `set_flag` calls
|
||||
removed, `STRONGLY_TYPED` not set) silently fall back to weak typing and the
|
||||
warning is easy to miss. Check the engine build log for `"Strongly typed network
|
||||
detected"` to confirm the flag took effect.
|
||||
- **Do not migrate without a baseline.** Weak typing's per-layer precision choices
|
||||
are non-deterministic across TRT versions; without the old outputs you have no
|
||||
ground truth and cannot tell whether the new engine is correct.
|
||||
- **Do not run `--stronglyTyped` against a FP32 ONNX expecting FP16 speedup.**
|
||||
Strong typing builds exactly what the graph specifies — a FP32 ONNX produces a
|
||||
FP32 engine. Run AutoCast first.
|
||||
- **Plugins must declare types.** Old plugins that left output types as "follow
|
||||
input" will fail to build under strong typing. Update them to the
|
||||
`IPluginV3OneBuild` interface, the only plugin API that supports strong typing
|
||||
in 11.x.
|
||||
- **Refit + strong typing**: refittable strongly-typed engines work, but the refit
|
||||
weights must match the precision declared in the network. A FP16 layer cannot be
|
||||
refit with FP32 weights — convert host weights before calling `setNamedWeights`.
|
||||
- **Don't trust grep alone for completeness.** `migrate.py` uses AST matching, so
|
||||
it catches the precision flags regardless of how `tensorrt` is imported or
|
||||
aliased via `as` — something string-level grep misses. It matches direct
|
||||
attribute access (`trt.BuilderFlag.FP16`); dynamic forms such as
|
||||
`getattr(trt.BuilderFlag, 'FP16')` are *not* rewritten and need manual review.
|
||||
Run the AST helper as the source of truth.
|
||||
|
||||
## References
|
||||
|
||||
- **Migration guide:** [TensorRT 10.x → 11.x](https://docs.nvidia.com/deeplearning/tensorrt/latest/api/migration/tensorrt-10x-to-11x.html)
|
||||
— the authoritative description of the weak → strong typing change.
|
||||
- **Operator type rules:** the [TensorRT Operators Reference](https://docs.nvidia.com/deeplearning/tensorrt/latest/_static/operators/)
|
||||
— each operator's page lists its supported input/output types, i.e. how its output
|
||||
dtype follows its inputs in a strongly-typed graph.
|
||||
- **API reference:** the TensorRT C++/Python API docs for `NetworkDefinitionCreationFlag`,
|
||||
`BuilderFlag`, `ILayer::setToType`, and `INetworkDefinition::addCast` / `addQuantize` /
|
||||
`addDequantize`.
|
||||
- **TensorRT Model Optimizer:** <https://nvidia.github.io/Model-Optimizer> — AutoCast and
|
||||
quantization tooling for getting precision into the model.
|
||||
- **Sample code** (TensorRT OSS, <https://github.com/NVIDIA/TensorRT>):
|
||||
[`samples/trtexec/README.md` Example 6](https://github.com/NVIDIA/TensorRT/blob/main/samples/trtexec/README.md#example-6-create-a-strongly-typed-plan-file)
|
||||
(strongly-typed plan), `samples/python/strongly_type_autocast/sample.py`
|
||||
(AutoCast → strongly-typed INT8), and `samples/python/network_api_pytorch_mnist/sample.py`
|
||||
(strongly-typed Python builder).
|
||||
@@ -0,0 +1,65 @@
|
||||
# trt-strong-typing-migration helper scripts
|
||||
|
||||
Scripts that automate the mechanical parts of the SKILL.md walkthrough.
|
||||
|
||||
## `migrate.py`
|
||||
|
||||
AST-based rewriter for Python TensorRT builder code. Detects the
|
||||
weakly-typed builder pattern (`create_network` + `set_flag(BuilderFlag.FP16/...)`)
|
||||
and rewrites it to the strongly-typed form.
|
||||
|
||||
```bash
|
||||
# Show what would change (dry-run): prints a unified diff; exits 1 if changes are pending, 0 if nothing to do.
|
||||
python3 migrate.py path/to/build.py
|
||||
|
||||
# Rewrite the file in place:
|
||||
python3 migrate.py path/to/build.py --write
|
||||
|
||||
# Process every .py file in a directory tree:
|
||||
python3 migrate.py path/to/project/ --write
|
||||
```
|
||||
|
||||
The script uses `ast.NodeTransformer` so it understands call shapes, not just
|
||||
text — it correctly handles `set_flag` accessed via aliased imports, multi-flag
|
||||
construction in `create_network`, and per-layer `precision`/`set_output_type`
|
||||
assignments. Unrelated logic and docstrings are left intact, but note: because
|
||||
the rewrite round-trips through `ast.unparse`, **regular `#` comments and the
|
||||
original formatting are not preserved**. Review the dry-run diff before
|
||||
`--write`, and re-add any comments you need afterward.
|
||||
|
||||
### What it changes
|
||||
|
||||
| Before | After |
|
||||
|--------|-------|
|
||||
| `builder.create_network(0)` | `builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))` |
|
||||
| `builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))` | `builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED))` |
|
||||
| `config.set_flag(trt.BuilderFlag.FP16 / BF16 / INT8 / FP8)` | (line removed) |
|
||||
| `layer.precision = trt.float16` | (statement removed) |
|
||||
| `layer.set_output_type(0, trt.float16)` | (statement removed) |
|
||||
|
||||
### What it does NOT change
|
||||
|
||||
- `set_flag` calls for non-precision-hint flags (`REFIT`, `SPARSE_WEIGHTS`,
|
||||
`DISABLE_TIMING_CACHE`, and `TF32`). These are orthogonal to typing — `kTF32`
|
||||
is kept in TRT 11.
|
||||
- Calibration setup (`set_calibration_profile`, `IInt8Calibrator`). These must
|
||||
be removed manually because the correct replacement (Q/DQ in the ONNX) lives
|
||||
outside the build script.
|
||||
- Conditional logic gating on `platform_has_fast_fp16` / `platform_has_fast_int8`
|
||||
— the body of those branches becomes a no-op after the flag is removed, but
|
||||
the conditional remains. Clean up manually if desired.
|
||||
|
||||
## `verify.sh`
|
||||
|
||||
End-to-end check that `migrate.py` is doing the right thing. Writes a
|
||||
representative weakly-typed sample to a temp directory, runs `migrate.py
|
||||
--write` on it, then asserts the rewritten file matches the strongly-typed
|
||||
contract (flag present, precision flags absent, valid Python).
|
||||
|
||||
```bash
|
||||
bash verify.sh # run and clean up
|
||||
bash verify.sh --keep # keep the temp workspace for inspection
|
||||
```
|
||||
|
||||
Exits 0 on success, 1 on any assertion failure. Run this before applying
|
||||
`migrate.py` to a real codebase.
|
||||
@@ -0,0 +1,305 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""AST-based migrator for weakly-typed TensorRT Python builders.
|
||||
|
||||
Rewrites:
|
||||
|
||||
- `builder.create_network(...)` -> uses NetworkDefinitionCreationFlag.STRONGLY_TYPED
|
||||
- `config.set_flag(trt.BuilderFlag.{FP16,BF16,INT8,FP8})` -> removed
|
||||
- `layer.precision = ...` and `layer.set_output_type(...)` -> removed
|
||||
|
||||
The rewrites are intentionally conservative: ambiguous patterns are skipped and
|
||||
reported so the user can resolve them manually. Non-precision flags (REFIT,
|
||||
SPARSE_WEIGHTS, TF32, etc.) are preserved — kTF32 is kept in TRT 11 and is
|
||||
orthogonal to typing.
|
||||
|
||||
Usage:
|
||||
python3 migrate.py path/to/file.py # dry-run, print unified diff
|
||||
python3 migrate.py path/to/file.py --write # rewrite in place
|
||||
python3 migrate.py path/to/dir/ --write # recurse over .py files
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import difflib
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
# Precision-hint BuilderFlag attribute names that must be removed. NOTE: TF32 is
|
||||
# deliberately NOT here — kTF32 is kept in TRT 11 (orthogonal to typing), like REFIT.
|
||||
PRECISION_FLAGS = frozenset({"FP16", "BF16", "INT8", "FP8"})
|
||||
|
||||
# NetworkDefinitionCreationFlag attribute names that map to STRONGLY_TYPED.
|
||||
WEAK_NETWORK_FLAGS = frozenset({"EXPLICIT_BATCH"})
|
||||
|
||||
|
||||
def _is_attr(node: ast.AST, *, attr_chain: tuple[str, ...]) -> bool:
|
||||
"""Return True if `node` is an attribute access matching the trailing chain.
|
||||
|
||||
Walks `node.attr -> node.value.attr -> ...` and compares to attr_chain
|
||||
in reverse. The leading element (e.g. module / object name) is not checked,
|
||||
so this matches `trt.BuilderFlag.FP16`, `tensorrt.BuilderFlag.FP16`, or any
|
||||
aliased import.
|
||||
"""
|
||||
cur = node
|
||||
for expected in reversed(attr_chain):
|
||||
if not isinstance(cur, ast.Attribute):
|
||||
return False
|
||||
if cur.attr != expected:
|
||||
return False
|
||||
cur = cur.value
|
||||
return True
|
||||
|
||||
|
||||
def _builder_flag_name(node: ast.AST) -> str | None:
|
||||
"""If `node` is `<anything>.BuilderFlag.<NAME>`, return NAME."""
|
||||
if isinstance(node, ast.Attribute) and isinstance(node.value, ast.Attribute):
|
||||
if node.value.attr == "BuilderFlag":
|
||||
return node.attr
|
||||
return None
|
||||
|
||||
|
||||
def _network_flag_name(node: ast.AST) -> str | None:
|
||||
"""If `node` is `<anything>.NetworkDefinitionCreationFlag.<NAME>`, return NAME."""
|
||||
if isinstance(node, ast.Attribute) and isinstance(node.value, ast.Attribute):
|
||||
if node.value.attr == "NetworkDefinitionCreationFlag":
|
||||
return node.attr
|
||||
return None
|
||||
|
||||
|
||||
def _strongly_typed_arg() -> ast.expr:
|
||||
"""Build the AST for `1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)`.
|
||||
|
||||
Uses bare `trt.` qualifier — matches the canonical import alias. Callers
|
||||
whose code uses `tensorrt.` directly will get a working but visually
|
||||
inconsistent line; that is acceptable as a one-line manual cleanup.
|
||||
"""
|
||||
return ast.BinOp(
|
||||
left=ast.Constant(value=1),
|
||||
op=ast.LShift(),
|
||||
right=ast.Call(
|
||||
func=ast.Name(id="int", ctx=ast.Load()),
|
||||
args=[
|
||||
ast.Attribute(
|
||||
value=ast.Attribute(
|
||||
value=ast.Name(id="trt", ctx=ast.Load()),
|
||||
attr="NetworkDefinitionCreationFlag",
|
||||
ctx=ast.Load(),
|
||||
),
|
||||
attr="STRONGLY_TYPED",
|
||||
ctx=ast.Load(),
|
||||
)
|
||||
],
|
||||
keywords=[],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class Migrator(ast.NodeTransformer):
|
||||
def __init__(self) -> None:
|
||||
self.removed_flag_calls = 0
|
||||
self.removed_precision_assigns = 0
|
||||
self.removed_set_output_type = 0
|
||||
self.rewrote_create_network = 0
|
||||
self.skipped: list[str] = []
|
||||
|
||||
# ---- create_network rewrites ----------------------------------------
|
||||
|
||||
def visit_Call(self, node: ast.Call) -> ast.AST:
|
||||
self.generic_visit(node)
|
||||
func = node.func
|
||||
if isinstance(func, ast.Attribute) and func.attr == "create_network":
|
||||
# Replace the single positional argument with the STRONGLY_TYPED flag.
|
||||
if len(node.args) == 0:
|
||||
# No positional arg: the flag may be passed as `flags=` keyword.
|
||||
# Blindly appending a positional arg would collide with it
|
||||
# (TypeError: got multiple values for argument 'flags').
|
||||
flags_kw = next((kw for kw in node.keywords if kw.arg == "flags"), None)
|
||||
if flags_kw is not None:
|
||||
if self._is_weak_network_arg(flags_kw.value):
|
||||
flags_kw.value = _strongly_typed_arg()
|
||||
self.rewrote_create_network += 1
|
||||
elif self._already_strongly_typed(flags_kw.value):
|
||||
pass # idempotent
|
||||
else:
|
||||
self.skipped.append(
|
||||
f"create_network at line {node.lineno}: unrecognized flags= "
|
||||
"argument shape, review manually"
|
||||
)
|
||||
elif not node.keywords:
|
||||
# Truly empty call create_network(): defaults to weak typing.
|
||||
node.args = [_strongly_typed_arg()]
|
||||
self.rewrote_create_network += 1
|
||||
else:
|
||||
self.skipped.append(
|
||||
f"create_network at line {node.lineno}: keyword args present "
|
||||
"without flags=, review manually"
|
||||
)
|
||||
elif len(node.args) == 1:
|
||||
arg = node.args[0]
|
||||
if self._is_weak_network_arg(arg):
|
||||
node.args[0] = _strongly_typed_arg()
|
||||
self.rewrote_create_network += 1
|
||||
elif self._already_strongly_typed(arg):
|
||||
pass # idempotent
|
||||
else:
|
||||
self.skipped.append(
|
||||
f"create_network at line {node.lineno}: unrecognized argument shape, "
|
||||
"review manually"
|
||||
)
|
||||
return node
|
||||
|
||||
@staticmethod
|
||||
def _is_weak_network_arg(arg: ast.expr) -> bool:
|
||||
# Match `0`
|
||||
if isinstance(arg, ast.Constant) and arg.value == 0:
|
||||
return True
|
||||
# Match `1 << int(...EXPLICIT_BATCH)` (any weak-network flag name)
|
||||
if isinstance(arg, ast.BinOp) and isinstance(arg.op, ast.LShift):
|
||||
inner = arg.right
|
||||
if isinstance(inner, ast.Call) and isinstance(inner.func, ast.Name) and inner.func.id == "int":
|
||||
if inner.args:
|
||||
name = _network_flag_name(inner.args[0])
|
||||
if name in WEAK_NETWORK_FLAGS:
|
||||
return True
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _already_strongly_typed(arg: ast.expr) -> bool:
|
||||
if isinstance(arg, ast.BinOp) and isinstance(arg.op, ast.LShift):
|
||||
inner = arg.right
|
||||
if isinstance(inner, ast.Call) and isinstance(inner.func, ast.Name) and inner.func.id == "int":
|
||||
if inner.args and _network_flag_name(inner.args[0]) == "STRONGLY_TYPED":
|
||||
return True
|
||||
return False
|
||||
|
||||
# ---- set_flag / precision assignment removals -----------------------
|
||||
|
||||
def visit_Expr(self, node: ast.Expr) -> ast.AST | None:
|
||||
# `config.set_flag(trt.BuilderFlag.FP16)` as an Expr statement.
|
||||
self.generic_visit(node)
|
||||
if isinstance(node.value, ast.Call):
|
||||
call = node.value
|
||||
if isinstance(call.func, ast.Attribute) and call.func.attr == "set_flag":
|
||||
if call.args:
|
||||
flag = _builder_flag_name(call.args[0])
|
||||
if flag in PRECISION_FLAGS:
|
||||
self.removed_flag_calls += 1
|
||||
return None
|
||||
# `layer.set_output_type(0, trt.float16)` — drop unconditionally.
|
||||
if isinstance(call.func, ast.Attribute) and call.func.attr == "set_output_type":
|
||||
self.removed_set_output_type += 1
|
||||
return None
|
||||
return node
|
||||
|
||||
def visit_Assign(self, node: ast.Assign) -> ast.AST | None:
|
||||
# `something.precision = trt.float16` (or any value) — drop.
|
||||
self.generic_visit(node)
|
||||
if len(node.targets) == 1:
|
||||
tgt = node.targets[0]
|
||||
if isinstance(tgt, ast.Attribute) and tgt.attr == "precision":
|
||||
self.removed_precision_assigns += 1
|
||||
return None
|
||||
return node
|
||||
|
||||
# ---- if-statement cleanup ------------------------------------------
|
||||
|
||||
def visit_If(self, node: ast.If) -> ast.AST | None:
|
||||
"""Drop `if builder.platform_has_fast_{fp16,int8,bf16}:` blocks that become empty."""
|
||||
self.generic_visit(node)
|
||||
test = node.test
|
||||
gating_attrs = {"platform_has_fast_fp16", "platform_has_fast_int8",
|
||||
"platform_has_fast_bf16", "platform_has_fast_fp8"}
|
||||
if isinstance(test, ast.Attribute) and test.attr in gating_attrs:
|
||||
# If body becomes empty (because set_flag was the only statement) and
|
||||
# there is no else clause, the whole if is dead.
|
||||
if not node.body and not node.orelse:
|
||||
return None
|
||||
if not node.body:
|
||||
# Body emptied but orelse remains — replace with the orelse block.
|
||||
return node.orelse if len(node.orelse) > 1 else (
|
||||
node.orelse[0] if node.orelse else None
|
||||
)
|
||||
return node
|
||||
|
||||
|
||||
def _process_source(src: str) -> tuple[str, Migrator]:
|
||||
tree = ast.parse(src)
|
||||
m = Migrator()
|
||||
new_tree = m.visit(tree)
|
||||
ast.fix_missing_locations(new_tree)
|
||||
return ast.unparse(new_tree), m
|
||||
|
||||
|
||||
def _iter_files(paths: Iterable[Path]) -> Iterable[Path]:
|
||||
for p in paths:
|
||||
if p.is_dir():
|
||||
yield from sorted(p.rglob("*.py"))
|
||||
elif p.suffix == ".py":
|
||||
yield p
|
||||
|
||||
|
||||
def _process_file(path: Path, *, write: bool) -> bool:
|
||||
"""Return True iff the file needs (or got) a migration.
|
||||
|
||||
Keyed on the migrator's transform counts, not on text equality: `ast.unparse`
|
||||
reformats and drops comments on every round-trip, so a file that needs no
|
||||
migration would otherwise look "changed" and have its comments stripped.
|
||||
"""
|
||||
original = path.read_text()
|
||||
try:
|
||||
new_src, m = _process_source(original)
|
||||
except SyntaxError as e:
|
||||
print(f"[skip] {path}: syntax error: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
migrated = (m.rewrote_create_network + m.removed_flag_calls
|
||||
+ m.removed_precision_assigns + m.removed_set_output_type) > 0
|
||||
if not migrated:
|
||||
for note in m.skipped:
|
||||
print(f"[note] {path}: {note}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
if write:
|
||||
path.write_text(new_src)
|
||||
print(
|
||||
f"[write] {path}: create_network={m.rewrote_create_network} "
|
||||
f"set_flag={m.removed_flag_calls} "
|
||||
f"precision_assign={m.removed_precision_assigns} "
|
||||
f"set_output_type={m.removed_set_output_type}"
|
||||
)
|
||||
else:
|
||||
diff = difflib.unified_diff(
|
||||
original.splitlines(keepends=True),
|
||||
new_src.splitlines(keepends=True),
|
||||
fromfile=f"{path} (weakly typed)",
|
||||
tofile=f"{path} (strongly typed)",
|
||||
)
|
||||
sys.stdout.writelines(diff)
|
||||
for note in m.skipped:
|
||||
print(f"[note] {path}: {note}", file=sys.stderr)
|
||||
return True
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("paths", nargs="+", type=Path, help="Files or directories to process.")
|
||||
parser.add_argument("--write", action="store_true",
|
||||
help="Rewrite files in place. Default is dry-run with unified diff.")
|
||||
args = parser.parse_args()
|
||||
|
||||
any_changed = False
|
||||
for path in _iter_files(args.paths):
|
||||
if _process_file(path, write=args.write):
|
||||
any_changed = True
|
||||
|
||||
# In dry-run, exit non-zero when changes are pending so callers/CI can gate
|
||||
# on it (like `black --check`). `--write` applies the changes and exits 0.
|
||||
return 1 if any_changed and not args.write else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
#
|
||||
# Verify the `migrate.py` helper on an inline weakly-typed sample.
|
||||
# Run from anywhere; resolves paths relative to this script.
|
||||
#
|
||||
# Usage:
|
||||
# bash scripts/verify.sh # run the verification end-to-end
|
||||
# bash scripts/verify.sh --keep # keep the temp workspace for inspection
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
MIGRATE="${SCRIPT_DIR}/migrate.py"
|
||||
|
||||
if [[ ! -f "${MIGRATE}" ]]; then
|
||||
echo "ERROR: migrate.py not found at ${MIGRATE}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
KEEP=0
|
||||
if [[ "${1:-}" == "--keep" ]]; then
|
||||
KEEP=1
|
||||
fi
|
||||
|
||||
WORK="$(mktemp -d -t trt-w2s-verify-XXXXXX)"
|
||||
|
||||
cleanup_workspace() {
|
||||
if [[ ${KEEP} -eq 1 ]]; then
|
||||
echo "Kept workspace at ${WORK}"
|
||||
else
|
||||
rm -rf "${WORK}"
|
||||
fi
|
||||
}
|
||||
trap cleanup_workspace EXIT
|
||||
|
||||
SAMPLE="${WORK}/sample_build.py"
|
||||
|
||||
# A representative weakly-typed Python builder. Contains every transform path
|
||||
# migrate.py is expected to handle: EXPLICIT_BATCH network flag, set_flag for
|
||||
# FP16/INT8 (removed) plus TF32/REFIT (preserved), per-layer precision overrides,
|
||||
# and the platform_has_fast_* gating that becomes dead code after migration.
|
||||
cat > "${SAMPLE}" <<'PY'
|
||||
import tensorrt as trt
|
||||
|
||||
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
|
||||
|
||||
def build():
|
||||
builder = trt.Builder(TRT_LOGGER)
|
||||
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
|
||||
config = builder.create_builder_config()
|
||||
|
||||
if builder.platform_has_fast_fp16:
|
||||
config.set_flag(trt.BuilderFlag.FP16)
|
||||
config.set_flag(trt.BuilderFlag.TF32) # kept in TRT 11 (orthogonal to typing); must survive
|
||||
config.set_flag(trt.BuilderFlag.INT8)
|
||||
config.set_flag(trt.BuilderFlag.REFIT) # not a precision flag; must survive
|
||||
|
||||
parser = trt.OnnxParser(network, TRT_LOGGER)
|
||||
with open("model.onnx", "rb") as f:
|
||||
parser.parse(f.read())
|
||||
|
||||
last = network.get_layer(network.num_layers - 1)
|
||||
last.precision = trt.float16
|
||||
last.set_output_type(0, trt.float16)
|
||||
|
||||
plan = builder.build_serialized_network(network, config)
|
||||
return plan
|
||||
PY
|
||||
|
||||
echo "[verify] sample written to ${SAMPLE}"
|
||||
|
||||
echo "[verify] dry-run diff (exit 1 = changes pending, expected here):"
|
||||
python3 "${MIGRATE}" "${SAMPLE}" || true
|
||||
|
||||
echo "[verify] writing rewrite in place:"
|
||||
python3 "${MIGRATE}" "${SAMPLE}" --write
|
||||
|
||||
# Assertions on the rewritten file.
|
||||
fail=0
|
||||
assert_present() {
|
||||
local needle="$1"
|
||||
if ! grep -qF "${needle}" "${SAMPLE}"; then
|
||||
echo "ASSERT FAIL: expected to find: ${needle}" >&2
|
||||
fail=1
|
||||
fi
|
||||
}
|
||||
assert_absent() {
|
||||
local needle="$1"
|
||||
if grep -qF "${needle}" "${SAMPLE}"; then
|
||||
echo "ASSERT FAIL: expected to be removed: ${needle}" >&2
|
||||
fail=1
|
||||
fi
|
||||
}
|
||||
|
||||
assert_present "NetworkDefinitionCreationFlag.STRONGLY_TYPED"
|
||||
assert_absent "NetworkDefinitionCreationFlag.EXPLICIT_BATCH"
|
||||
assert_absent "BuilderFlag.FP16"
|
||||
assert_absent "BuilderFlag.INT8"
|
||||
assert_present "BuilderFlag.TF32" # kTF32 is kept in TRT 11; must NOT be stripped
|
||||
assert_present "BuilderFlag.REFIT" # non-precision flag must remain
|
||||
assert_absent ".precision = trt.float16"
|
||||
assert_absent "set_output_type(0, trt.float16)"
|
||||
|
||||
# Syntactic validity.
|
||||
python3 -c "import ast, sys; ast.parse(open('${SAMPLE}').read())" || {
|
||||
echo "ASSERT FAIL: rewritten file is not valid Python" >&2
|
||||
fail=1
|
||||
}
|
||||
|
||||
if [[ ${fail} -ne 0 ]]; then
|
||||
echo "[verify] FAIL — see assertions above. Workspace: ${WORK}"
|
||||
KEEP=1
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[verify] OK — migrate.py produced a well-formed strongly-typed rewrite"
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
name: trt-torch-quickstart
|
||||
description: >
|
||||
Compile a PyTorch model to a TensorRT engine via Torch-TensorRT — AOT or
|
||||
JIT — under the new strong-typing default. Use when the user
|
||||
compiles PyTorch to TensorRT without ONNX, hits "enabled_precisions
|
||||
should not be used when use_explicit_typing=True", sees Dynamo graph
|
||||
breaks or PyTorch fallback, debugs ABI errors at import torch_tensorrt, or
|
||||
needs the compatible torch / torch_tensorrt / tensorrt-cu13 version pins for
|
||||
TensorRT 11. Triggers: torch_tensorrt, torch_tensorrt.dynamo.compile,
|
||||
torch.compile backend torch_tensorrt, pytorch to tensorrt, ExportedProgram,
|
||||
Dynamo graph break, use_explicit_typing, enabled_precisions,
|
||||
torch_tensorrt.Input, min_block_size, truncate_double, tensorrt-cu13,
|
||||
version pinning, version compatibility. Adjacent skills: `trt-onnx-quickstart`,
|
||||
`trt-cpp-runtime-quickstart`. LLM token generation belongs in TensorRT-LLM.
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: NVIDIA Corporation
|
||||
version: "1.0"
|
||||
tags:
|
||||
- torch-tensorrt
|
||||
- dynamo
|
||||
- pytorch
|
||||
- quickstart
|
||||
- fp16
|
||||
---
|
||||
|
||||
# Torch-TensorRT Quickstart
|
||||
|
||||
Convert a PyTorch `nn.Module` to a TensorRT engine using `torch_tensorrt` (the [Torch-TensorRT](https://github.com/pytorch/TensorRT) frontend). Covers the AOT path (for production and C++ deploy) and the JIT path (for Python-only inference).
|
||||
|
||||
## When to Use
|
||||
|
||||
| Scenario | Use this skill? |
|
||||
|----------|-----------------|
|
||||
| PyTorch `nn.Module`; want a TensorRT engine without writing an ONNX intermediate | Yes |
|
||||
| Model uses ops that don't export cleanly to ONNX (custom autograd, dynamic control flow) | Yes |
|
||||
| Want Torch-TensorRT's automatic fallback of unsupported subgraphs to PyTorch | Yes |
|
||||
| Need a serialized engine for C++ deploy from a PyTorch source | Yes — use AOT path here, then `trt-cpp-runtime-quickstart` for the C++ load |
|
||||
| Have a working ONNX file already | No — use `trt-onnx-quickstart` |
|
||||
| LLM token generation (Llama, Mistral, Qwen text gen) | **No** — route to [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) |
|
||||
| Have a `.plan` file already and want to run it from C++ | No — use `trt-cpp-runtime-quickstart` |
|
||||
| Migrating an existing weakly-typed TRT network to strongly-typed | No — use `trt-strong-typing-migration` |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Torch-TensorRT versions are tightly coupled to a (torch, tensorrt-cu13, CUDA) triple; mismatches produce ABI errors at `import torch_tensorrt`. **Pin matrix source of truth:** the [pytorch/TensorRT releases page](https://github.com/pytorch/TensorRT/releases) — each release lists the exact `torch` / `tensorrt` / CUDA versions it was built against. Do not invent pins from memory.
|
||||
|
||||
For TensorRT 11.0:
|
||||
|
||||
| Component | Required | Notes |
|
||||
|-----------|----------|-------|
|
||||
| TensorRT | 11.x (`tensorrt-cu13` wheel) | Mixing `-cu12` and `-cu13` wheels breaks imports. |
|
||||
| CUDA toolkit | 13.x | Driver R590+. |
|
||||
| Python | ≥ 3.10 | TRT 11 dropped 3.9 and earlier. |
|
||||
| `torch` | Match the `torch_tensorrt` release notes for the chosen `torch_tensorrt` version. | |
|
||||
| `torch_tensorrt` | The release matching the TRT 11.0 RC — see [pytorch/TensorRT releases](https://github.com/pytorch/TensorRT/releases). | |
|
||||
|
||||
**Recommended environment**: NGC TensorRT container with PyTorch (`nvcr.io/nvidia/pytorch:<tag>` or `nvcr.io/nvidia/tensorrt:<tag>` with `torch_tensorrt` pip-installed on top). Run with `--gpus all`.
|
||||
|
||||
Verification:
|
||||
```bash
|
||||
python3 -c "import torch, torch_tensorrt, tensorrt; print(torch.__version__, torch_tensorrt.__version__, tensorrt.__version__)"
|
||||
```
|
||||
|
||||
## Step 1 — Load and prepare the model
|
||||
|
||||
Use a `torch.nn.Module` in eval mode. Torch-TensorRT will trace through it.
|
||||
|
||||
```python
|
||||
import torch
|
||||
import torchvision.models as models
|
||||
|
||||
model = models.resnet50(weights=None).eval().cuda()
|
||||
example = torch.randn(1, 3, 224, 224, device="cuda")
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `weights=None` skips the download — fine for shape/perf testing. For accuracy work, load real weights.
|
||||
- Model must be on CUDA before compile. Torch-TensorRT does not move it for you.
|
||||
- `.eval()` matters: BatchNorm and Dropout behave differently in train mode and can produce different engines.
|
||||
|
||||
## Step 2 — Compile to a TensorRT engine (AOT)
|
||||
|
||||
Two compile paths — pick by deployment target:
|
||||
|
||||
| Goal | Path | API |
|
||||
|------|------|-----|
|
||||
| Serialized engine (production, C++ deploy, reuse) | **AOT** — this section | `torch_tensorrt.dynamo.compile` → `torch_tensorrt.save` |
|
||||
| In-process Python callable only (no serialized engine) | **JIT** — end of this section | `torch.compile(backend="torch_tensorrt")` |
|
||||
|
||||
The AOT path uses `torch_tensorrt.dynamo.compile` and returns a serializable `ExportedProgram`.
|
||||
|
||||
**Important — strong typing is the default in torch_tensorrt ≥ 2.12.** With `use_explicit_typing=True` (the default), engine precision is **inferred from the exported model's dtype** — passing `enabled_precisions={torch.float16}` raises `AssertionError`. To compile in FP16, cast the model and example tensor to FP16 before exporting:
|
||||
|
||||
```python
|
||||
import torch
|
||||
import torch_tensorrt
|
||||
|
||||
model = MyModel().eval().cuda().half() # cast to FP16
|
||||
example = torch.randn(1, 3, 224, 224, device="cuda", dtype=torch.float16)
|
||||
|
||||
trt_gm = torch_tensorrt.dynamo.compile(
|
||||
torch.export.export(model, (example,)),
|
||||
inputs=[example],
|
||||
# No enabled_precisions — precision comes from the model dtype.
|
||||
truncate_double=True,
|
||||
min_block_size=1,
|
||||
)
|
||||
|
||||
# Serialize for later / C++ loading — use torch_tensorrt.save, NOT raw bytes.
|
||||
torch_tensorrt.save(trt_gm, "resnet50_trt.ep", inputs=[example])
|
||||
```
|
||||
|
||||
**Do not extract raw engine bytes via `submod.engine` and write them with `open(...).write(...)`.** That bypasses Torch-TensorRT's metadata wrapper: the blob will not round-trip through `torch_tensorrt.load(...)` and silently drops the dispatch graph that handles partial-fallback subgraphs. `torch_tensorrt.save(trt_gm, path, inputs=...)` is the only supported serialization path.
|
||||
|
||||
For mixed precision or to override the model dtype, set `use_explicit_typing=False` and `enabled_precisions` applies as before (the weakly-typed path, deprecated in TRT 11):
|
||||
|
||||
```python
|
||||
trt_gm = torch_tensorrt.dynamo.compile(
|
||||
torch.export.export(model, (example,)),
|
||||
inputs=[example],
|
||||
use_explicit_typing=False, # weakly-typed (deprecated)
|
||||
enabled_precisions={torch.float16, torch.float32},
|
||||
truncate_double=True,
|
||||
)
|
||||
```
|
||||
|
||||
Key arguments:
|
||||
- `truncate_double=True`: silently downcasts FP64 constants. Without it, any FP64 op forces a partition boundary.
|
||||
- `min_block_size`: smallest subgraph (in node count) worth handing to TRT. `1` is aggressive; default `5` avoids tiny TRT subgraphs that don't pay back their launch overhead.
|
||||
- `workspace_size`: bytes of scratch memory the TRT builder can use. Leave unset to let TRT decide.
|
||||
|
||||
**JIT alternative (Python-only inference):** if you just want a callable and not a serialized engine, use `torch.compile` with the Torch-TensorRT backend. Under the strong-typing default, precision comes from the model dtype — `.half()` the model for FP16 rather than passing `enabled_precisions`:
|
||||
```python
|
||||
model = model.half() # FP16 inferred from model dtype (strong typing default)
|
||||
example = example.half()
|
||||
trt_model = torch.compile(model, backend="torch_tensorrt")
|
||||
out = trt_model(example) # compiles lazily on the first call, then runs through TRT in-process
|
||||
```
|
||||
This skips `torch.export` and gives a `torch.compile`-wrapped callable. It cannot be loaded from C++ — use the AOT path above if you need that.
|
||||
|
||||
## Step 3 — Dynamic shapes
|
||||
|
||||
Wrap inputs in `torch_tensorrt.Input` with `min_shape`/`opt_shape`/`max_shape`:
|
||||
|
||||
```python
|
||||
dynamic_input = torch_tensorrt.Input(
|
||||
min_shape=(1, 3, 224, 224),
|
||||
opt_shape=(8, 3, 224, 224),
|
||||
max_shape=(32, 3, 224, 224),
|
||||
dtype=torch.float16,
|
||||
)
|
||||
trt_gm = torch_tensorrt.dynamo.compile(
|
||||
torch.export.export(model, (example,), dynamic_shapes={"x": {0: torch.export.Dim("batch", min=1, max=32)}}),
|
||||
inputs=[dynamic_input],
|
||||
# Strong typing: FP16 comes from the input/model dtype, not enabled_precisions.
|
||||
)
|
||||
```
|
||||
|
||||
Pitfalls:
|
||||
- The `dynamic_shapes=` argument to `torch.export.export` and the `Input` ranges must agree. A mismatch produces a builder error 2–3 minutes into compilation, not at export time.
|
||||
- `opt_shape` is what TRT tunes kernels for. Set it to the most common runtime shape, not the midpoint.
|
||||
- Only mark dimensions dynamic that actually vary at runtime; unnecessary dynamism degrades performance.
|
||||
|
||||
## Step 4 — Run inference and verify
|
||||
|
||||
Raw-tensor comparison (`assert_close`) is fine for FP32 but **misleading for FP16** — a few outlier positions exceed the tolerance even when downstream behavior is identical. Use a **semantic check** appropriate to the model type:
|
||||
|
||||
```python
|
||||
with torch.inference_mode():
|
||||
torch_out = model(example)
|
||||
trt_out = trt_gm(example)
|
||||
```
|
||||
|
||||
**Image classification** — top-1 (or top-5) class match, plus mean-softmax-distance:
|
||||
|
||||
```python
|
||||
assert torch.equal(torch_out.argmax(-1), trt_out.argmax(-1)), "top-1 mismatch"
|
||||
mean_prob_diff = (torch.softmax(torch_out.float(), -1) - torch.softmax(trt_out.float(), -1)).abs().mean()
|
||||
assert mean_prob_diff < 1e-3
|
||||
```
|
||||
|
||||
**Sentence / token embeddings** — cosine similarity per token, mean across the batch:
|
||||
|
||||
```python
|
||||
cos = torch.nn.functional.cosine_similarity(torch_out.flatten(0, -2), trt_out.flatten(0, -2), dim=-1)
|
||||
assert cos.mean() > 0.999
|
||||
```
|
||||
|
||||
**Detection / regression heads** — bounded raw-tensor closeness on the meaningful sub-tensor (e.g., bbox coordinates), softmax check on the class logits.
|
||||
|
||||
**FP32 sanity** — for any model, an FP32 (no `.half()`) run should pass `torch.testing.assert_close(trt_out, torch_out, rtol=1e-4, atol=1e-4)`. If that fails, the issue is in the compilation, not numeric precision.
|
||||
|
||||
## Loading the serialized engine
|
||||
|
||||
**From Python**:
|
||||
```python
|
||||
import torch_tensorrt
|
||||
loaded = torch_tensorrt.load("resnet50_trt.ep").module()
|
||||
out = loaded(example)
|
||||
```
|
||||
|
||||
**From C++**: this skill does NOT cover C++ loading. The `.ep` file `torch_tensorrt.save` produces by default is a `torch.export` archive for the **Python** `torch_tensorrt.load(...)` path — it is *not* directly consumable by the plain TensorRT C++ runtime (`IRuntime` / `deserializeCudaEngine`), which expects a serialized TensorRT engine (`.plan`). Two supported C++ routes:
|
||||
|
||||
- Save with `output_format="torchscript"` and deploy the TorchScript module with libtorch + the Torch-TensorRT C++ runtime.
|
||||
- If you only need the raw TensorRT engine in C++, build it through the ONNX path (`trt-onnx-quickstart`) and load the resulting `.plan` with `trt-cpp-runtime-quickstart` (modern `IRuntime` + `enqueueV3` + `setTensorAddress`).
|
||||
|
||||
Do not inline a C++ snippet here — it will diverge from the canonical pattern.
|
||||
|
||||
## Common issues
|
||||
|
||||
**`RuntimeError: Trying to create tensor with negative dimension`** during export — the model has a shape-dependent control flow path that `torch.export` can't trace. Either rewrite with `torch.cond` / `torch.where`, or fall back to `ir="dynamo"` + `min_block_size=1` so the unsupported region runs in PyTorch.
|
||||
|
||||
**`Unsupported operator` warnings** — Torch-TensorRT partitions around them and runs those nodes in PyTorch. Set `require_full_compilation=True` to turn this into an error; use during development to find what's actually falling back.
|
||||
|
||||
## Diagnosing excessive PyTorch fallback
|
||||
|
||||
If the compile completes but most of the model fell back to PyTorch (only a few subgraphs run on TRT), three knobs in order:
|
||||
|
||||
1. **Identify what fell back.** Run with `TORCH_LOGS="graph_breaks"` and `require_full_compilation=True` to escalate fallbacks to errors that name the offending op. Without this you don't know what the autotuner rejected.
|
||||
```bash
|
||||
TORCH_LOGS="graph_breaks" python3 your_compile.py
|
||||
```
|
||||
2. **Reconsider `min_block_size`.** This is a real trade-off — not just a knob.
|
||||
- Default `min_block_size=5`: Torch-TensorRT only hands a subgraph to TRT if it has ≥5 nodes. Tiny TRT subgraphs cost more in kernel-launch overhead than they save.
|
||||
- `min_block_size=1`: aggressive — every supported op goes to TRT, even single-node subgraphs. Useful to see what's *theoretically* supported, but commonly slower at inference due to the per-subgraph launch tax.
|
||||
- Recommended: leave at default `5` for production; drop to `1` only during diagnosis to see maximum TRT coverage.
|
||||
3. **Look for shape-dependent control flow.** `torch.export` traces only one branch of `if x.shape[0] > 0:` style code; rewrite with `torch.where`, `torch.cond`, or lift the condition out of the model. The trace error names the offending op.
|
||||
|
||||
If fallback persists after all three, you have a genuine unsupported op — write a converter (see Torch-TensorRT upstream docs) or switch to the ONNX path (`trt-onnx-quickstart`).
|
||||
|
||||
**Engine builds but output is garbage** — almost always a dtype issue. Check:
|
||||
1. Model/input dtype matches your pipeline — under strong typing precision follows the model dtype (`.half()` for FP16); `enabled_precisions` applies only under weak typing (`use_explicit_typing=False`).
|
||||
2. `truncate_double=True` if the model has any FP64 constants (common in positional encodings).
|
||||
3. The model was `.eval()` before export.
|
||||
|
||||
**Compile takes >10 minutes** — turn on the builder log to see what's going on:
|
||||
```python
|
||||
import torch_tensorrt.logging as ttlog
|
||||
ttlog.set_reportable_log_level(ttlog.Level.Info)
|
||||
```
|
||||
Usually it's the autotuner exploring kernel variants for a heavy GEMM/conv. To cap exploration, set `workspace_size` smaller (less scratch → fewer candidates).
|
||||
|
||||
**`ImportError: cannot import name 'XYZ' from 'torch_tensorrt'`** — version mismatch. Confirm the pin matrix; the `torch_tensorrt` Python API moves between minor versions.
|
||||
|
||||
## Numerical debugging
|
||||
|
||||
If `assert_close` fails by more than ~5× tolerance:
|
||||
1. Re-run in FP32 — drop the model's `.half()` (or under weak typing, `enabled_precisions={torch.float32}`). If that matches PyTorch, the issue is FP16 accumulator drift in a specific op — usually a softmax or LayerNorm. The pragmatic fix is to wrap the offending submodule in a module excluded from the compile (`min_block_size` won't help; explicit exclusion is needed). See the [Torch-TensorRT lowering guide](https://pytorch.org/TensorRT/) for the version-specific exclusion API.
|
||||
2. Sanity-check at FP32 first, then re-introduce FP16 and bisect — almost always reveals one offending op.
|
||||
3. For FP8: check that the calibration distribution actually covers the runtime distribution. FP8 has no headroom for outliers.
|
||||
|
||||
See `trt-strong-typing-migration` for the broader weak-vs-strong typing discussion. By default Torch-TensorRT produces strongly-typed engines in TRT 11.0 — types inferred from the exported program — but you can override.
|
||||
|
||||
## What this skill is not
|
||||
|
||||
- Not a guide to writing custom Torch-TensorRT converters. Use the upstream Torch-TensorRT docs for that.
|
||||
- Not for TensorRT-LLM / LLM inference — use the TRT-LLM `examples/` flow instead.
|
||||
- Not for QAT (quantization-aware training) — Torch-TensorRT consumes the QAT model but the training loop is upstream.
|
||||
|
||||
## References
|
||||
|
||||
- Upstream Torch-TensorRT: https://pytorch.org/TensorRT/
|
||||
- Version pin matrix: [pytorch/TensorRT releases](https://github.com/pytorch/TensorRT/releases)
|
||||
- Related skills: `trt-onnx-quickstart`, `trt-strong-typing-migration`, `trt-cpp-runtime-quickstart`
|
||||
Reference in New Issue
Block a user