chore: import upstream snapshot with attribution

.claude/rules/general-code-style.md

@@ -0,0 +1,19 @@
+---
+paths:
+  - "**/*.py"
+---
+
+# General Code Style
+
+Default conventions for new and modified Python code. Prefer these unless there is a concrete reason not to; call out deviations in review.
+
+- **Prefer stateless.** Favor pure functions over methods that mutate instance state; pass inputs in, return outputs out.
+- **Prefer immutable.** Default to immutable data (frozen structs, tuples, read-only values); mutate only when there is a clear need.
+- **Extract init-static values at construction.** When a derived value's inputs are frozen for the object's lifetime (typically configuration: constructor args, env vars, server args), compute it once in `__init__` and store it as a well-named attribute (`self.mtp_enabled`, `self.needs_cpu_seq_lens`); later code reads the attribute instead of re-deriving it. Input immutability is the hard precondition — if inputs can change, recompute in place or funnel mutation through a single override point (the frozen `ServerArgs.override()` pattern). If you can't give the value a meaningful name, the boundary is wrong — don't cache unnameable subexpressions.
+- **Functions stay small.** Keep each function under ~100 LOC; split larger ones into named helpers.
+- **Files stay small.** Keep each file under ~2k LOC; split larger modules along cohesive boundaries.
+- **Core functions read like pseudocode.** The main / orchestration function of a unit should be short and read like algorithm pseudocode — push detail into well-named helpers so the top-level flow is obvious.
+- **Avoid mixins.** Don't add behavior via mixin classes; prefer explicit composition (hold a collaborator and call it) or plain functions.
+- **Prefer protected over public.** Default methods to protected (`_name`); expose only what callers actually use.
+- **Prefer keyword arguments.** Call functions of 2+ args by keyword, and design APIs to be called that way.
+- **Pass what you need, not the god object.** Give a callee the specific values it uses (by keyword), not a whole large object (`ModelRunner`, `Scheduler`); reserve passing the whole object for a leaf whose contract genuinely requires it. Even then, keep it read-only — read fields off it and return results for the caller to assign, rather than writing fields back through it.

.claude/rules/modify-component-must-read.md

@@ -0,0 +1,9 @@
+# Must-Read Skills Before Modifying Components
+
+Before modifying the following components, read the listed skill first.
+
+- **Speculative decoding code** (anything under `python/sglang/srt/speculative/`, related attention backends, scheduler accumulators, IPC fields, observability metrics, or CLI flags) → [`speculative-naming`](../skills/speculative-naming/SKILL.md)
+- **`Scheduler` / `TokenizerManager` / `ModelRunner` `__init__`** (`python/sglang/srt/managers/scheduler.py`, `python/sglang/srt/managers/tokenizer_manager.py`, `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
+- **Any edit to a frozen core file** (currently `python/sglang/srt/model_executor/model_runner.py`) → [`large-class-style`](../skills/large-class-style/SKILL.md)
+- **Environment variables** (adding, renaming, or reviewing any `SGLANG_*` env var, migrating a legacy `SGL_*` alias, or touching `python/sglang/srt/environ.py`) → [`env-var-conventions`](../skills/env-var-conventions/SKILL.md)
+- **Scripted runtime** (anything related to the scripted runtime) → [`scripted-runtime-notes`](../skills/scripted-runtime-notes/SKILL.md)

.claude/rules/no-dataclasses.md

@@ -0,0 +1,23 @@
+---
+paths:
+  - "**/*.py"
+---
+
+# Use `msgspec.Struct`, not `@dataclass`
+
+Define data containers as `msgspec.Struct`. Do not add new
+`dataclasses.dataclass` (or `attrs`) — they weaken strict type checking and
+don't translate cleanly for multi-language support (e.g. the planned Rust migration).
+
+```python
+import msgspec
+
+class LoadSnapshot(msgspec.Struct):   # prefer frozen= and omit_defaults=; kw_only= as needed
+    dp_rank: int = 0
+    tokens: list[int] = []            # mutable defaults are safe
+```
+
+- Methods / `@classmethod` constructors go on the `Struct`; see
+  `python/sglang/srt/managers/load_snapshot.py`.
+- New code only. Existing `@dataclass` is grandfathered — migrate opportunistically
+  while editing the file, not in drive-by sweeps.

.claude/rules/no-getattr-defensive.md

@@ -0,0 +1,39 @@
+---
+paths:
+  - "**/*.py"
+---
+
+# Don't use `getattr` / `hasattr` for defensive access
+
+Over-defensive `getattr(obj, "field", default)` / `hasattr(obj, "field")` hide
+errors and defeat strict type checking. If a field is always present, accessing
+it defensively is confusing and masks real bugs. Prefer:
+
+1. **`isinstance` for type narrowing** — check the type, then access fields directly:
+
+   ```python
+   if (
+       isinstance(obj, (TokenizedGenerateReqInput, TokenizedEmbeddingReqInput))
+       and obj.mm_inputs
+   ):
+   ```
+   (see `python/sglang/srt/managers/mm_utils.py`)
+
+2. **Always set the field (to `None` if needed), then do a `None` check** — the
+   field should always exist, so a `None` / non-`None` check is enough:
+
+   ```python
+   obj.field = None   # in __init__ / construction
+   ...
+   if obj.field is not None:
+       ...
+   ```
+
+Bad — `server_args` always has `revision`, so `getattr` is misleading and swallows
+a real `AttributeError` if the field is ever renamed:
+
+```python
+revision=getattr(server_args, "revision", None),   # BAD
+revision=server_args.revision,                     # GOOD
+```
+(see `python/sglang/srt/managers/template_detection.py`)

.claude/rules/unit-test-admission.md

@@ -0,0 +1,72 @@
+---
+paths:
+  - "test/**/*.py"
+---
+
+# Unit Test Admission Criteria
+
+Every test case must have a concrete answer to: "what future diff would turn
+this case red?" If the only answer is "editing the test itself", delete it.
+
+A new unit test case must fall into one of these categories:
+
+1. **Bug regression.** Guards a bug that actually happened (CI failure, issue,
+   incident). Before committing, verify the case fails on the pre-fix code and
+   passes on the fix. Describe the bug mechanism in the docstring in black-box
+   terms. For concurrency bugs, reproduce the exact interleaving
+   deterministically (`create_task` + `sleep(0)` scheduling); do not rely on
+   probabilistic stress -- a stress loop that cannot hit the bug even on the
+   buggy code has zero guard value.
+
+2. **Derived property.** Pins down a conclusion that required reasoning to
+   establish -- boundary/alignment math, invariants, protocol semantics (FIFO
+   fairness, idempotency, round-trip). Protects against "looks equivalent"
+   rewrites that silently break the derivation.
+
+3. **Critical-path bookkeeping.** Defends conventions that are easy to break by
+   forgetting to sync -- registry completeness, field lifecycle, serialization
+   compatibility. Enumerating assertions are fine here; the guarded failure
+   mode is "someone extended X without updating Y". Example: the ratchet tests
+   (`test/registered/unit/test_module_state_ratchet.py`).
+
+Not admissible:
+
+- Happy-path tautologies that re-assert what the implementation trivially does.
+- Mirror tests that restate the implementation logic as assertions.
+- Probabilistic stress that cannot reproduce the failure it claims to guard.
+
+**Distinguishing test — does deletion leave a silent-failure path?** A case
+that *looks* like a tautology/mirror is still admissible when it guards a
+failure mode no other case covers. The criterion is not "is the code under
+test simple?" but "would some regression pass every remaining test if this
+case were deleted?"
+
+Keep (bookkeeping, not mirror) when the assertion guards one of:
+
+- An **external-source literal** — a value copied from an outside spec
+  (OTel semantic conventions, a protocol field name, a vendor API shape).
+  Deleting it removes the only guard against silently copying the spec wrong.
+  Example: `assertEqual(SpanAttributes.GEN_AI_LATENCY_E2E, "gen_ai.latency.e2e")`
+  stays — the string is dictated by the OTel spec, not by this repo's code.
+- A **completeness / negative-branch contract** — "all builtins are
+  registered", "a non-matching id does *not* trigger", "the default is
+  applied when the input is absent". Even if the code is a one-liner, the
+  failure mode is "someone added X without updating Y" or "a predicate
+  degraded to always-true". Example: `test_abort_non_matching_rid` (asserts
+  an unmatched rid is *not* aborted) stays because no positive-match test
+  covers the no-op branch.
+
+Delete (true mirror/tautology) when the assertion merely echoes an
+**isolated** implementation output — changing it breaks nothing outside the
+line itself, so the test has no independent guard value. Example:
+`assertEqual(MixedPrecisionConfig.get_min_capability(),
+Fp4Config.get_min_capability())` goes — the source body is literally
+`return Fp4Config.get_min_capability()`, and flipping it is an isolated
+change that every dependent test catches anyway.
+
+One strong case beats several weak ones: each additional case must guard a
+distinct failure mode. Ask "which bug escapes if I delete this case?" -- no
+answer means delete it.
+
+Test mechanics (placement, CI registration, fixtures) live in
+[`write-sglang-test`](../skills/write-sglang-test/SKILL.md).

.claude/skills/add-jit-kernel/SKILL.md

@@ -0,0 +1,650 @@
+---
+name: add-jit-kernel
+description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module
+---
+
+# Tutorial: Adding a New JIT Kernel to SGLang
+
+This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
+
+## Goal
+
+Add a new operation that scales each element of a tensor by a scalar factor:
+
+- Input: tensor `x` (CUDA) and scalar `factor` (float, passed at runtime)
+- Output: `x * factor` (element-wise), allocated internally
+- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
+
+## When to use JIT vs AOT (`sgl-kernel`)
+
+- **JIT (`jit_kernel`)**: prefer this first for kernels that do **not** depend on CUTLASS or another large C++ project. It is the default choice for lightweight kernels that benefit from rapid iteration and first-use compilation.
+- **AOT (`sgl-kernel`)**: prefer this when the kernel **does** depend on CUTLASS or another large C++ project, or when it should live in `sgl-kernel/` and participate in the wheel build / torch op registration flow.
+- **Exception**: kernels that depend on `flashinfer`, or on CUTLASS that is already provided through `flashinfer`, can still be implemented as `jit_kernel`.
+
+---
+
+## Common Abstractions in `python/sglang/jit_kernel/include/sgl_kernel/`
+
+**Always prefer these abstractions over raw CUDA primitives.** They provide safety, readability, and consistency with the rest of the codebase.
+
+**Important include rule:** for every `#include <sgl_kernel/...>` line, add a short trailing comment explaining why that header is included (for example `// For TensorMatcher, SymbolicSize, SymbolicDevice`). This matches the current JIT kernel style and keeps include usage self-documenting.
+
+### `utils.h` — Host-side utilities
+
+```cpp
+#include <sgl_kernel/utils.h>
+```
+
+- **`host::RuntimeCheck(cond, args...)`** — Assert a condition at runtime; throws `PanicError` with file/line info on failure. Prefer this over bare `assert`.
+- **`host::Panic(args...)`** — Unconditionally throw a `PanicError` with a descriptive message.
+- **`host::div_ceil(a, b)`** — Integer ceiling division `(a + b - 1) / b`.
+- **`host::irange(n)`** / **`host::irange(start, end)`** — Range views for cleaner loops.
+- **`host::pointer::offset(ptr, offsets...)`** — Byte-safe pointer arithmetic on `void*`. Use this instead of raw casts.
+
+### `utils.cuh` — Device-side utilities + `LaunchKernel`
+
+```cpp
+#include <sgl_kernel/utils.cuh>
+```
+
+- **Type aliases**: `fp16_t`, `bf16_t`, `fp32_t`, `fp8_e4m3_t`, `fp8_e5m2_t` and their packed variants `fp16x2_t`, `bf16x2_t`, `fp32x2_t`, etc.
+- **`SGL_DEVICE`** — Expands to `__forceinline__ __device__`. Use on all device functions.
+- **`device::kWarpThreads`** — Constant `32`.
+- **`device::load_as<T>(ptr, offset)`** / **`device::store_as<T>(ptr, val, offset)`** — Type-safe loads/stores from `void*`.
+- **`device::pointer::offset(ptr, offsets...)`** — Pointer arithmetic on device.
+- **`host::LaunchKernel(grid, block, device_or_stream [, smem])`** — RAII kernel launcher that:
+  - Resolves the CUDA stream from a `DLDevice` via TVM-FFI automatically.
+  - Checks the CUDA error with file/line info after launch via `operator()(kernel, args...)`.
+  - Supports `.enable_pdl(bool)` for PDL (Programmatic Dependent Launch, SM90+).
+- **`host::RuntimeDeviceCheck(cudaError_t)`** — Check a CUDA error; throw on failure.
+
+### `tensor.h` — Tensor validation (`TensorMatcher`, Symbolic types)
+
+```cpp
+#include <sgl_kernel/tensor.h>
+```
+
+This is the **primary validation API** for all kernel launchers. Use it to validate every `tvm::ffi::TensorView` argument.
+
+- **`host::SymbolicSize{"name"}`** — A named symbolic dimension. Call `.set_value(n)` to pin it, `.unwrap()` to extract after verification.
+- **`host::SymbolicDType`** — Symbolic dtype. Use `.set_options<Ts...>()` to restrict allowed types.
+- **`host::SymbolicDevice`** — Symbolic device. Use `.set_options<kDLCUDA>()` to restrict to CUDA.
+- **`host::TensorMatcher({dims...})`** — Fluent builder for tensor validation:
+  - `.with_dtype<T>()` — require a specific C++ type (e.g. `fp16_t`)
+  - `.with_dtype<T1, T2, ...>()` — allow a set of types
+  - `.with_device<kDLCUDA>(device_sym)` — require CUDA and bind the checked device to a `SymbolicDevice`
+  - `.with_strides({strides...})` — validate strides (omit to require contiguous)
+  - `.verify(tensor_view)` — execute the check; throws `PanicError` with full context on failure; **chainable** (`verify(a).verify(b)` to check multiple tensors with the same shape)
+
+**Typical pattern:**
+```cpp
+auto N = SymbolicSize{"num_elements"};
+auto device = SymbolicDevice{};
+device.set_options<kDLCUDA>();
+TensorMatcher({N})  //
+    .with_dtype<fp16_t>()
+    .with_device<kDLCUDA>(device)
+    .verify(dst)
+    .verify(src);  // same shape, dtype, device as dst
+const size_t n = N.unwrap();
+const DLDevice dev = device.unwrap();
+```
+
+### `type.cuh` — `dtype_trait<T>` and `packed_t<T>`
+
+```cpp
+#include <sgl_kernel/type.cuh>
+```
+
+- **`dtype_trait<T>`** — Static trait struct for each scalar type. Provides:
+  - `dtype_trait<T>::from(value)` — convert from another type (e.g. `fp32_t` → `fp16_t`)
+  - `dtype_trait<T>::abs/sqrt/rsqrt/exp/sin/cos(x)` — type-dispatched unary math (primarily for `fp32_t`)
+  - `dtype_trait<T>::max/min(x, y)` — type-dispatched binary math (primarily for `fp32_t`)
+- **`packed_t<T>`** — Two-element packed alias: `packed_t<fp16_t>` = `fp16x2_t`, `packed_t<bf16_t>` = `bf16x2_t`, `packed_t<fp32_t>` = `fp32x2_t`. Use for vectorized loads/stores.
+- **`device::cast<To, From>(value)`** — Type-safe cast using `dtype_trait`, e.g. `cast<fp32x2_t, fp16x2_t>(v)`.
+
+### `vec.cuh` — Vectorized memory access (`AlignedVector`)
+
+```cpp
+#include <sgl_kernel/vec.cuh>
+```
+
+- **`device::AlignedVector<T, N>`** — Aligned storage for N elements of type T. N must be a power of two, `sizeof(T)*N <= 32`. Enables vectorized loads/stores for bandwidth efficiency. In terms of API/codegen constraints, the upper bound is 256-bit; in practice, 128-bit is the portable default, while 256-bit vectorization is typically only viable on `SM100+` and should be gated by an architecture check when needed.
+  - `.load(ptr, offset)` — vectorized load from `ptr[offset]`
+  - `.store(ptr, offset)` — vectorized store to `ptr[offset]`
+  - `.fill(value)` — fill all lanes
+  - `operator[](i)` — element access
+
+### `tile.cuh` — `tile::Memory` (strided memory access pattern)
+
+```cpp
+#include <sgl_kernel/tile.cuh>
+```
+
+- `tile::Memory<T>` is fundamentally a **1D cooperative accessor** over a contiguous region.
+- **`device::tile::Memory<T>::cta(blockDim.x)`** — Creates a tile accessor where each thread handles `tid = threadIdx.x` with stride `tsize` (for `cta(blockDim.x)`, this is `blockDim.x`). Common for loops over a 1D array.
+- **`.load(ptr, offset)`** — loads `ptr[tid + offset * tsize]`
+- **`.store(ptr, val, offset)`** — stores to `ptr[tid + offset * tsize]`
+- **`.in_bound(n, offset)`** — boundary check
+
+For a **2D tile**, either flatten `(row, col)` into a linear tile index first, or compute the address manually with `ptr[row * stride + col]` using your thread/block coordinates.
+
+### `math.cuh` — Device math (`device::math::`)
+
+```cpp
+#include <sgl_kernel/math.cuh>
+```
+
+- `device::math::max/min<T>(a, b)` — type-dispatched binary math via `dtype_trait`
+- `device::math::abs/sqrt/rsqrt/exp/sin/cos<T>(x)` — type-dispatched unary math via `dtype_trait`
+
+### `warp.cuh` — Warp-level primitives
+
+```cpp
+#include <sgl_kernel/warp.cuh>
+```
+
+- `device::warp::reduce_sum<T>(value)` — warp-level sum reduction via `__shfl_xor_sync`
+- `device::warp::reduce_max<T>(value)` — warp-level max reduction
+
+### `cta.cuh` — CTA-level primitives
+
+```cpp
+#include <sgl_kernel/cta.cuh>
+```
+
+- `device::cta::reduce_max<T>(value, smem, min_value)` — CTA-wide max using shared memory + warp reduction. Caller is responsible for a `__syncthreads()` after if the result in `smem[0]` is needed.
+
+### `atomic.cuh` — Atomic operations
+
+```cpp
+#include <sgl_kernel/atomic.cuh>
+```
+
+- `device::atomic::max(float* addr, float value)` — float atomic max (handles negative values correctly via bit tricks).
+
+### `runtime.cuh` — Occupancy and device info
+
+```cpp
+#include <sgl_kernel/runtime.cuh>
+```
+
+- `host::runtime::get_blocks_per_sm(kernel, block_dim)` — max active blocks per SM (occupancy)
+- `host::runtime::get_sm_count(device_id)` — number of SMs on the device
+- `host::runtime::get_cc_major(device_id)` — compute capability major version
+
+**Persistent kernel pattern** (cap blocks to SM count × occupancy):
+```cpp
+static const uint32_t max_occ = runtime::get_blocks_per_sm(kernel, kBlockSize);
+static const uint32_t num_sm  = runtime::get_sm_count(device.unwrap().device_id);
+const auto num_blocks = std::min(num_sm * max_occ, div_ceil(n, kBlockSize));
+LaunchKernel(num_blocks, kBlockSize, device.unwrap())(kernel, params);
+```
+
+---
+
+## Step 0 (optional): Generate a `.clangd` config for better IDE support
+
+```bash
+python -m sglang.jit_kernel -h  # for verbose help info about clangd configuration
+python -m sglang.jit_kernel
+python -m sglang.jit_kernel --dep cutlass flashinfer  # with cutlass/flashinfer dependency
+```
+
+---
+
+## Step 1: Implement the CUDA kernel in `jit_kernel/csrc/`
+
+Create `python/sglang/jit_kernel/csrc/elementwise/scale.cuh`.
+
+The implementation fully uses the project abstractions described above:
+
+```cpp
+// NOTE: Comments for headers are not common in practice.
+// It is only shown here for tutorial purposes to highlight the key abstractions.
+#include <sgl_kernel/tensor.h>   // For TensorMatcher, SymbolicSize, SymbolicDevice
+#include <sgl_kernel/type.cuh>   // For dtype_trait, fp16_t, bf16_t, fp32_t
+#include <sgl_kernel/utils.h>    // For RuntimeCheck, div_ceil
+#include <sgl_kernel/utils.cuh>  // For LaunchKernel, SGL_DEVICE
+#include <sgl_kernel/vec.cuh>    // For AlignedVector
+
+#include <dlpack/dlpack.h>
+#include <tvm/ffi/container/tensor.h>
+
+namespace {
+
+// ----------------------------------------------------------------
+// Kernel: element-wise scale using vectorized 128-bit loads/stores
+// T       = fp16_t | bf16_t | fp32_t
+// kVecN   = number of elements per vector load (e.g. 8 for fp16)
+// factor  = runtime scale factor
+// ----------------------------------------------------------------
+template <typename T, int kVecN, bool kUsePDL>
+__global__ void scale_kernel(T* __restrict__ dst,
+                              const T* __restrict__ src,
+                              float factor,
+                              uint32_t n_total) {
+  using vec_t = device::AlignedVector<T, kVecN>;
+  const uint32_t n_vecs = n_total / kVecN;
+
+  // If using PDL, wait for primary kernel before any global memory load.
+  // This is NOT a synchronization point, which means some threads can early exit before this.
+  device::PDLWaitPrimary<kUsePDL>();
+
+  // --- vectorised body ---
+  const uint32_t vec_stride = blockDim.x * gridDim.x;
+  for (uint32_t vi = blockIdx.x * blockDim.x + threadIdx.x;
+       vi < n_vecs;
+       vi += vec_stride) {
+    vec_t v;
+    v.load(src, vi);
+#pragma unroll
+    for (int i = 0; i < kVecN; ++i) {
+      v[i] = static_cast<T>(static_cast<float>(v[i]) * factor);
+    }
+    v.store(dst, vi);
+  }
+
+  // --- scalar tail ---
+  const uint32_t base = n_vecs * kVecN;
+  const uint32_t scalar_stride = blockDim.x * gridDim.x;
+  for (uint32_t i = blockIdx.x * blockDim.x + threadIdx.x;
+       base + i < n_total;
+       i += scalar_stride) {
+    dst[base + i] = static_cast<T>(static_cast<float>(src[base + i]) * factor);
+  }
+
+  // If using PDL, signal for the secondary kernel to start after all threads have finished
+  // This is NOT a synchronization point, which means some threads can early exit before this.
+  device::PDLTriggerSecondary<kUsePDL>();
+}
+
+// ----------------------------------------------------------------
+// Launcher: validates tensors, selects vector width, launches kernel
+// ----------------------------------------------------------------
+template <typename T, bool kUsePDL>
+void scale(tvm::ffi::TensorView dst, tvm::ffi::TensorView src, float factor) {
+  using namespace host;
+
+  // 1. Validate input tensors with TensorMatcher
+  SymbolicSize N = {"num_elements"};
+  SymbolicDevice device_;
+  device_.set_options<kDLCUDA>();
+
+  TensorMatcher({N})  //
+      .with_dtype<T>()
+      .with_device<kDLCUDA>(device_)
+      .verify(dst)
+      .verify(src);  // same shape / dtype / device as dst
+
+  const uint32_t n = static_cast<uint32_t>(N.unwrap());
+  const DLDevice device = device_.unwrap();
+
+  RuntimeCheck(n > 0, "scale: num_elements must be > 0, got ", n);
+
+  // 2. Choose vector width for 128-bit loads (16 bytes)
+  //    fp16/bf16: 8 elements x 2 bytes = 16 bytes
+  //    fp32:      4 elements x 4 bytes = 16 bytes
+  // We encourage using `device::kMaxVecBytes`, which will change according to
+  // the target architecture and can enable 256-bit vectorization on SM100+ if desired.
+  // But 128-bit is more commonly adapted for better compatibility,
+  // so it's still ok to hardcode 16 here just for simplicity.
+  constexpr int kVecN = 16 / sizeof(T);
+  const uint32_t n_work_items = div_ceil(n, static_cast<uint32_t>(kVecN));
+
+  // 3. Launch
+  constexpr uint32_t kBlockSize = 256;
+  const uint32_t grid = div_ceil(n_work_items, kBlockSize);
+
+  // PDL feature is 100% optional. Without `enable_pdl`, the code should still be correct.
+  // Try to enable it if profiling shows that it can benefit the performance of this kernel.
+  LaunchKernel(grid, kBlockSize, device).enable_pdl(kUsePDL)(
+      scale_kernel<T, kVecN, kUsePDL>,
+      static_cast<T*>(dst.data_ptr()),
+      static_cast<const T*>(src.data_ptr()),
+      factor,
+      n);
+}
+
+}  // namespace
+```
+
+**Key points:**
+
+- Include headers from `sgl_kernel/` — **not** raw CUDA headers for anything already covered
+- Add a short trailing `// For ...` explanation to every `#include <sgl_kernel/...>` line
+- Use `TensorMatcher` for all tensor validation; never manually check shape/dtype/device
+- Use `AlignedVector` for vectorised 128-bit loads/stores — significant bandwidth win
+- Use `LaunchKernel` — it resolves the stream and checks errors automatically
+- Use `RuntimeCheck` for runtime assertions with useful error messages
+- Prefer passing runtime scalars like `factor` directly unless compile-time specialisation is genuinely required
+- `fp16_t` / `bf16_t` / `fp32_t` are the project's type aliases (from `utils.cuh`)
+- `device::cast<To, From>` or `dtype_trait<T>::from(val)` for cross-type conversions
+- `device::math::` functions for device math instead of bare `__` intrinsics if possible.
+- Try to use `PDL` feature. In some cases, this will benefit the performance.
+
+---
+
+## Step 2: Add the Python wrapper in `jit_kernel/`
+
+Create `python/sglang/jit_kernel/scale.py`:
+
+```python
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import torch
+
+from sglang.jit_kernel.utils import (
+    cache_once,
+    is_arch_support_pdl,
+    load_jit,
+    make_cpp_args,
+)
+
+if TYPE_CHECKING:
+    from tvm_ffi.module import Module
+
+
+@cache_once
+def _jit_scale_module(dtype: torch.dtype) -> Module:
+    """Compile and cache the JIT scale module for a given dtype."""
+    args = make_cpp_args(dtype, is_arch_support_pdl())
+    return load_jit(
+        "scale",
+        *args,
+        cuda_files=["elementwise/scale.cuh"],
+        cuda_wrappers=[("scale", f"scale<{args}>")],
+    )
+
+
+def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) -> torch.Tensor:
+    """
+    Element-wise scale: dst = src * factor.
+
+    Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
+
+    Parameters
+    ----------
+    src    : CUDA tensor (FP16 / BF16 / FP32)
+    factor : scale factor
+    out    : optional pre-allocated output tensor (same shape/dtype as src)
+
+    Returns
+    -------
+    Scaled tensor (dst = src * factor).
+    """
+    # DO NOT add too much proactive validation here.
+    # Keep the Python wrapper thin, only enforce the preconditions
+    # that the current JIT/FFI path (C++ side) does not reject on its own.
+    if src.dtype not in (torch.float16, torch.bfloat16, torch.float32):
+        raise RuntimeError(
+            f"Unsupported dtype {src.dtype}. Supported: float16, bfloat16, float32"
+        )
+    if out is None:
+        out = torch.empty_like(src)
+
+    module = _jit_scale_module(src.dtype)
+    module.scale(out, src, factor)
+    return out
+```
+
+**Key points:**
+
+- Use `cache_once` — **not** `functools.lru_cache` (incompatible with `torch.compile`)
+- `load_jit` first arg(s) form the unique build marker; same marker = same cached binary
+- Only include compile-time specialisation knobs in the build marker; runtime values like `factor` should stay runtime unless the kernel truly needs templating
+- `cuda_wrappers`: `(export_name, kernel_symbol)` — `export_name` is called from Python
+- `make_cpp_args(dtype, ...)` converts `torch.dtype` to C++ type alias:
+- `is_arch_support_pdl()` checks if the current architecture supports PDL, which is typically passed as a template argument to the kernel.
+- Keep Python launchers thin, but still validate the basic invariants (`is_cuda`, supported dtype, `out` metadata). In the current JIT/FFI path, invalid tensors are not always rejected safely before launch
+
+| `torch.dtype`      | C++ type   |
+|--------------------|------------|
+| `torch.float16`    | `fp16_t`   |
+| `torch.bfloat16`   | `bf16_t`   |
+| `torch.float32`    | `fp32_t`   |
+
+---
+
+## Step 3 (optional): Tune JIT build flags
+
+If your kernel uses some math functions like `expf` or `sinf`, consider enabling `--use_fast_math` for better performance (with a potential precision tradeoff):
+
+```python
+return load_jit(
+    "scale",
+    *args,
+    cuda_files=["elementwise/scale.cuh"],
+    cuda_wrappers=[("scale", f"scale<{args}>")],
+    extra_cuda_cflags=["-O3", "--use_fast_math"],
+)
+```
+
+If your kernel requires SM90+, raise a clear Python error before calling `load_jit`:
+
+```python
+if torch.cuda.get_device_capability()[0] < 9:
+    raise RuntimeError("This kernel requires SM90 (Hopper) or later")
+```
+
+---
+
+## Step 4: Write tests (required)
+
+JIT kernel correctness tests and benchmarks live under `test/registered/jit/` and `test/registered/jit/benchmark/` (NOT inside the `sglang` package -- a `register_*_ci(...)` call anywhere under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook). Only their test-only helpers (e.g. `benchmark/marker.py`) stay alongside the kernel source under `python/sglang/jit_kernel/` and are imported by absolute path. **CI does not run `pytest` in those directories directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` and `bench_*.py` under `test/registered/`, collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check.
+
+- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `base-b-kernel-unit-test-1-gpu-large` on H100 and `base-b-kernel-unit-test-4-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `base-b-kernel-unit-test-8-gpu-h200`.
+- **Nightly kernel suite**: `nightly-kernel-1-gpu` with `--nightly` — typically used with `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` in CI for expanded parameter grids (see `python/sglang/jit_kernel/utils.py` → `should_run_full_tests` / `get_ci_test_range`). Wired in `.github/workflows/nightly-test-nvidia.yml` (e.g. `python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error`).
+
+Registration pattern (module level, **literal** `est_time`, `stage`, and `runner_config` values — required for AST parsing):
+
+```python
+from sglang.test.ci.ci_register import register_cuda_ci
+
+register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
+# Optional B200/SM100 registration for tests that cover Blackwell-specific code paths
+# register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
+# Optional second registration: same file also listed under the nightly kernel suite
+# (nightly suites use the legacy single-string suite=, not stage/runner_config)
+# register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True)
+```
+
+CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. The single-string `suite=` form is only for nightly/stress/weekly suites.
+
+Keep `est_time`, `stage`, `runner_config`, and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery.
+
+Use `register_cuda_ci(..., disabled="reason")` if the file must stay in-tree but should be skipped in CI (e.g. multi-GPU only).
+
+**Run like CI** (from repo root):
+
+```bash
+(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-1-gpu-large)
+# For B200/SM100-specific coverage:
+(cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-4-gpu-b200)
+```
+
+For fast iteration you can still run `pytest` on a single file locally; CI coverage is via `run_suite.py`.
+
+Create `test/registered/jit/test_scale.py`:
+
+```python
+import pytest
+import torch
+from sglang.jit_kernel.scale import scale
+from sglang.test.ci.ci_register import register_cuda_ci
+
+register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
+
+
+@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
+@pytest.mark.parametrize("size", [1, 127, 128, 1024, 4097])  # cover tail remainder
+@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0, 3.0])
+def test_scale_correctness(dtype, size, factor):
+    src = torch.randn(size, dtype=dtype, device="cuda")
+    out = scale(src, factor)
+    expected = src * factor
+
+    rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
+    torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
+
+
+@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
+def test_scale_out_param(dtype):
+    src = torch.randn(1024, dtype=dtype, device="cuda")
+    out = torch.empty_like(src)
+    result = scale(src, 2.0, out=out)
+    assert result is out
+    torch.testing.assert_close(out, src * 2.0, rtol=1e-2, atol=1e-2)
+
+
+def test_scale_cpu_error():
+    src = torch.randn(128, dtype=torch.float16)  # CPU tensor
+    with pytest.raises(RuntimeError, match="CUDA"):
+        scale(src, 2.0)
+
+
+def test_scale_unsupported_dtype():
+    src = torch.randint(0, 10, (128,), dtype=torch.int32, device="cuda")
+    with pytest.raises(RuntimeError, match="dtype"):
+        scale(src, 2.0)
+
+
+if __name__ == "__main__":
+    import sys
+    sys.exit(pytest.main([__file__, "-v", "-s"]))
+```
+
+---
+
+## Step 5: Add a benchmark (required)
+
+Benchmarks are `bench_*.py` files under `test/registered/jit/benchmark/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`base-b-kernel-benchmark-test-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large`).
+
+Benchmarks use the project's own `marker` framework (in `python/sglang/jit_kernel/benchmark/marker.py`) — **do not** use `triton.testing.perf_report` / `triton.testing.do_bench` directly. The marker framework provides (public names: `benchmark`, `parametrize`, `do_bench`, `skip`, `BenchResult`, `BenchSkip`):
+
+- **`@marker.benchmark(line_arg, line_vals, *, unit="us")`** — the **innermost** decorator (bottom of the stack, directly above `def benchmark`). Declares the column axis: each value in `line_vals` becomes a result column, and `line_arg` is the parameter name passed into the benchmark function. `unit` is one of `"us" | "ms" | "s"`.
+- **`@marker.parametrize(names, vals, ci_vals=None)`** — stackable decorator that adds a row axis (pytest-style). Each `@parametrize` adds one (or more, correlated) parameter the benchmark is swept over (Cartesian product across all `parametrize` decorators). `names` may be a single name (`"size"`) or a comma-separated correlated tuple axis (`"h,d"`, with `vals` then a list of tuples like `[(1, 64), (2, 128)]`). Pass the optional third `ci_vals` for a smaller sweep that is auto-selected under `is_in_ci()` — this is the built-in CI-shrinking mechanism, so you usually don't need `get_benchmark_range` for swept axes.
+- **`marker.do_bench(fn, *, input_args=(), input_kwargs={}, ...)`** — runs `fn` under CUDA graph (default) or a naive loop, returns a `BenchResult`. Key knobs:
+  - `memory_args`: defaults to `"all"` (footprint derived from all input args/kwargs). Pass an explicit tuple of tensors (e.g. `(k, v, indices)`) to count only the inputs the kernel actually touches.
+  - `memory_output`: defaults to `"out"` — re-runs `fn` once to capture its **returned** tensor and counts it. For in-place kernels (which return `None`), pass the written tensors explicitly (e.g. `memory_output=(k, v)`); the re-run is then skipped. Set to `None` to count no output.
+  - Together `memory_args` + `memory_output` give the GB/s column; with both defaults a function `out = f(src)` already reports `bytes(src) + bytes(out)`.
+  - `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"` — pass an iterable of indices/keys to limit to the *read* args (writes don't need cloning).
+  - `use_cuda_graph=False` for kernels that can't be captured.
+  - `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column).
+  - `disable_log_bandwidth` (defaults from `SGLANG_KERNEL_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely.
+- **`utils.create_random(*shape)` / `utils.create_empty(*shape)`** — shorthand for `torch.randn` / `torch.empty` with `DEFAULT_DTYPE` (`bfloat16`) and `DEFAULT_DEVICE` (`"cuda"`). Override via the `dtype=` / `device=` kwargs.
+- **`utils.get_benchmark_range(full_range, ci_range)`** — returns the smaller `ci_range` under CI (`is_in_ci()`), the `full_range` locally. Still available for the `benchmark(...)` column axis (which has no `ci_vals`); for `parametrize` row axes prefer the built-in `ci_vals` argument.
+
+Create `test/registered/jit/benchmark/bench_scale.py`:
+
+```python
+import torch
+
+from sglang.jit_kernel.benchmark import marker
+from sglang.jit_kernel.benchmark.utils import create_random
+from sglang.jit_kernel.scale import scale as jit_scale
+from sglang.test.ci.ci_register import register_cuda_ci
+
+register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")
+
+
+@torch.compile()
+def torch_impl_scale(src: torch.Tensor, factor: float) -> torch.Tensor:
+    return src * factor
+
+
+FN_MAP = {
+    "jit": jit_scale,
+    "torch": torch_impl_scale,
+}
+
+
+# `parametrize(name, full_vals, ci_vals)`: the 3rd arg is the smaller sweep
+# auto-selected under CI; the full range runs locally.
+@marker.parametrize("size", [2**n for n in range(10, 20)], [4096, 65536])  # 1K … 512K
+@marker.benchmark("impl", ["jit", "torch"])
+def benchmark(size: int, impl: str):
+    src = create_random(size)
+    factor = 2.0
+    return marker.do_bench(
+        FN_MAP[impl],
+        input_args=(src, factor),
+        # `src` is read -> clone it per iter to avoid L2 reuse; factor is a scalar.
+        graph_clone_args=(0,),
+        # Defaults already report bandwidth: memory_args="all" counts src,
+        # memory_output="out" counts the returned tensor -> bytes(src)+bytes(out).
+    )
+
+
+if __name__ == "__main__":
+    benchmark.run()
+```
+
+**Key points:**
+
+- The `line_arg` name passed to `benchmark` (`"impl"` here) must match a parameter on `benchmark(...)`; same for every `parametrize` name (`"size"`).
+- Stack `@parametrize` once per swept axis. The required `@marker.benchmark` is the **innermost** decorator (bottom of the stack, directly above the function) — `@parametrize` rows go above it.
+- Prefer `create_random` / `create_empty` from `utils.py` over open-coding `torch.randn(..., dtype=..., device=...)`.
+- The GB/s column appears by default (`memory_args="all"` + `memory_output="out"`). For memory-bound kernels it's the most informative number; scope `memory_args` / `memory_output` to the tensors actually touched if the defaults over- or under-count. For compute-bound kernels where bandwidth is misleading, set `SGLANG_KERNEL_DISABLE_LOG_BANDWIDTH=1` (or `disable_log_bandwidth=True`).
+- For in-place kernels (which return `None`), pass the written tensors via `memory_output=(...)` since the `"out"` default would capture nothing.
+- Tune `graph_clone_args` / `graph_clone_kwargs` to all the arguments that might be read by the kernel. We can only skip cloning for write-only args. For in-place modified args, we still need to clone them to get accurate timing (reusing the same buffer keeps it L2-hot and skews results).
+- Call `benchmark.run()` (no `print_data=` kwarg — the marker framework prints directly).
+
+Run locally:
+
+```bash
+python test/registered/jit/benchmark/bench_scale.py
+```
+
+Run the benchmark suite the way CI does:
+
+```bash
+cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large
+```
+
+---
+
+## Troubleshooting
+
+- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config` (and optional `nightly=True`); starred args and non-literal values break AST collection
+- **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/jit_kernel/csrc/`; reduce template argument combinations
+- **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...`
+- **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. Make sure `graph_clone_args` covers every *read* tensor — reusing a single buffer keeps it L2-hot and skews results
+- **Missing GB/s column**: the column is on by default; check that `SGLANG_KERNEL_DISABLE_LOG_BANDWIDTH` is not `1` and `disable_log_bandwidth` is not `True`. For in-place kernels (return `None`) the `memory_output="out"` default counts nothing — pass the written tensors via `memory_output=(...)`
+
+---
+
+## References
+
+- `docs/developer_guide/development_jit_kernel_guide.md`
+- `test/run_suite.py` — suite names, discovery of `test/registered/`, execution entrypoint for CI
+- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
+- `python/sglang/jit_kernel/utils.py` — `cache_once`, `load_jit`, `make_cpp_args`, `should_run_full_tests`, `get_ci_test_range`
+- `python/sglang/jit_kernel/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`
+- `python/sglang/jit_kernel/include/sgl_kernel/utils.cuh` — type aliases, `LaunchKernel`, `SGL_DEVICE`
+- `python/sglang/jit_kernel/include/sgl_kernel/vec.cuh` — `AlignedVector`
+- `python/sglang/jit_kernel/include/sgl_kernel/tile.cuh` — `tile::Memory`
+- `python/sglang/jit_kernel/include/sgl_kernel/type.cuh` — `dtype_trait`, `packed_t`, `device::cast`
+- `python/sglang/jit_kernel/include/sgl_kernel/math.cuh` — `device::math::`
+- `python/sglang/jit_kernel/include/sgl_kernel/warp.cuh` — `warp::reduce_sum/max`
+- `python/sglang/jit_kernel/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
+- `python/sglang/jit_kernel/include/sgl_kernel/atomic.cuh` — `atomic::max`
+- `python/sglang/jit_kernel/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
+- `python/sglang/jit_kernel/csrc/add_constant.cuh` — minimal runnable reference
+- `python/sglang/jit_kernel/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
+- `python/sglang/jit_kernel/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
+- `python/sglang/jit_kernel/benchmark/marker.py` — `benchmark`, `parametrize`, `do_bench`, `BenchResult`
+- `python/sglang/jit_kernel/benchmark/utils.py` — `create_random` / `create_empty` / `get_benchmark_range` helpers and `DEFAULT_DTYPE` / `DEFAULT_DEVICE`
+- `test/registered/jit/benchmark/bench_qknorm.py` — real example: multi-axis `parametrize` (with `ci_vals`) + in-place `memory_output`
+- `test/registered/jit/benchmark/bench_store_cache.py` — real example: scoped `memory_args` / `memory_output` + selective `graph_clone_args`
+
+## Summary of Files Created
+
+```
+python/sglang/jit_kernel/csrc/elementwise/scale.cuh   # NEW: CUDA kernel
+python/sglang/jit_kernel/scale.py                     # NEW: Python wrapper
+test/registered/jit/test_scale.py                     # NEW: Tests
+test/registered/jit/benchmark/bench_scale.py          # NEW: Benchmark
+```

.claude/skills/add-sgl-kernel/SKILL.md

@@ -0,0 +1,367 @@
+---
+name: add-sgl-kernel
+description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel to sgl-kernel (including tests & benchmarks)
+---
+
+# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
+
+This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
+
+## Goal
+
+Add a new operation that scales each element of a tensor by a scalar factor:
+
+- Input: tensor `x` (CUDA) and scalar `factor` (float)
+- Output: `x * factor` (element-wise, in-place or into pre-allocated `out`)
+- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
+  - Dispatched via `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro (defined in `sgl-kernel/include/utils.h`)
+
+## Two rules of thumb (must follow)
+
+1. **Prefer `python/sglang/jit_kernel` first** when the kernel does **not** depend on CUTLASS or another large C++ project. This is the default path for lightweight kernels that benefit from rapid iteration.
+2. **Prefer `sgl-kernel`** when the kernel **does** depend on CUTLASS or another large C++ project, or when it should be part of the AOT wheel / torch op registration flow.
+3. **Exception**: if the dependency is `flashinfer`, or CUTLASS that is already provided through `flashinfer`, the kernel can still be implemented as `jit_kernel`.
+
+In addition, every new kernel must ship with:
+
+- **Tests** (pytest)
+- **A benchmark script** (triton.testing)
+
+---
+
+## Repository integration map
+
+You will typically touch these files/areas:
+
+- Implementation: `sgl-kernel/csrc/elementwise/scale.cu` (pick the right subdirectory)
+- Public declarations: `sgl-kernel/include/sgl_kernel_ops.h`
+- Torch extension registration: `sgl-kernel/csrc/common_extension.cc`
+- Build: `sgl-kernel/CMakeLists.txt` (`set(SOURCES ...)`)
+- Python API: `sgl-kernel/python/sgl_kernel/` and `sgl-kernel/python/sgl_kernel/__init__.py`
+- Tests: `sgl-kernel/tests/test_scale.py`
+- Benchmarks: `sgl-kernel/benchmark/bench_scale.py`
+
+---
+
+## Step 1: Implement the kernel in `csrc/`
+
+Pick the right subdirectory:
+
+- `csrc/elementwise/` — for element-wise ops (our example)
+- `csrc/gemm/`, `csrc/attention/`, `csrc/moe/` — for other categories
+
+Create `sgl-kernel/csrc/elementwise/scale.cu`:
+
+```cpp
+#include <ATen/cuda/CUDAContext.h>
+#include <c10/cuda/CUDAGuard.h>
+#include <torch/all.h>
+
+#include "utils.h"  // DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16
+
+// scale_kernel: out[i] = input[i] * factor
+// Supports float, half (__half), __nv_bfloat16 via template T
+template <typename T>
+__global__ void scale_kernel(T* __restrict__ out,
+                              const T* __restrict__ input,
+                              float factor,
+                              int64_t n) {
+  int64_t idx = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
+  if (idx < n) {
+    out[idx] = static_cast<T>(static_cast<float>(input[idx]) * factor);
+  }
+}
+
+void scale(at::Tensor& out, const at::Tensor& input, double factor) {
+  TORCH_CHECK(input.is_cuda(),       "input must be a CUDA tensor");
+  TORCH_CHECK(input.is_contiguous(), "input must be contiguous");
+  TORCH_CHECK(out.is_cuda(),         "out must be a CUDA tensor");
+  TORCH_CHECK(out.is_contiguous(),   "out must be contiguous");
+  TORCH_CHECK(out.sizes() == input.sizes(),  "out and input must have the same shape");
+  TORCH_CHECK(out.scalar_type() == input.scalar_type(),
+              "out and input must have the same dtype");
+
+  const int64_t n = input.numel();
+  const int threads = 256;
+  const int blocks  = (n + threads - 1) / threads;
+
+  const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
+  const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
+
+  // Dispatches over float, float16, bfloat16
+  DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16(input.scalar_type(), c_type, [&] {
+    scale_kernel<c_type><<<blocks, threads, 0, stream>>>(
+        static_cast<c_type*>(out.data_ptr()),
+        static_cast<const c_type*>(input.data_ptr()),
+        static_cast<float>(factor),
+        n);
+    cudaError_t status = cudaGetLastError();
+    TORCH_CHECK(status == cudaSuccess,
+                "scale_kernel launch failed: ", cudaGetErrorString(status));
+    return true;
+  });
+}
+```
+
+**Key points:**
+
+- Use `at::Tensor` (PyTorch tensors), `TORCH_CHECK` for validation, `at::cuda::getCurrentCUDAStream()` for stream
+- Keep Python wrappers thin; do shape/dtype/device validation in C++ right around the launch path
+- `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` covers `float`, `half` (FP16), `__nv_bfloat16` (BF16)
+- Add device error checking after every kernel launch
+- If a kernel only works on certain architectures, enforce that with `TORCH_CHECK` and skip logic in tests
+
+---
+
+## Step 2: Add a C++ declaration in `include/sgl_kernel_ops.h`
+
+Edit `sgl-kernel/include/sgl_kernel_ops.h`, add to the elementwise section:
+
+```cpp
+void scale(at::Tensor& out, const at::Tensor& input, double factor);
+```
+
+---
+
+## Step 3: Register the op in `csrc/common_extension.cc`
+
+Edit `sgl-kernel/csrc/common_extension.cc`, inside `TORCH_LIBRARY_FRAGMENT(sgl_kernel, m)`:
+
+```cpp
+// From csrc/elementwise
+m.def("scale(Tensor! out, Tensor input, float factor) -> ()");
+m.impl("scale", torch::kCUDA, &scale);
+```
+
+**Key points:**
+
+- `Tensor!` means in-place / mutable output argument
+- The schema is important for `torch.compile` and for consistent call signatures
+- Keep the torch schema in PyTorch scalar types (`float` here), but note that the C++ launcher signature still needs `double` for scalar arguments accepted by `torch::Library`
+
+---
+
+## Step 4: Add the new source file to `CMakeLists.txt`
+
+Edit `sgl-kernel/CMakeLists.txt`, add to `set(SOURCES ...)`:
+
+```cmake
+csrc/elementwise/scale.cu
+```
+
+**Key points:**
+
+- Keep the list **alphabetically sorted** (the file explicitly requires this)
+- If the kernel has arch constraints, reflect that in tests/benchmarks via skip logic
+
+---
+
+## Step 5: Expose a Python API under `sgl-kernel/python/sgl_kernel/`
+
+Prefer following the existing module organization first. For elementwise kernels, the usual pattern is:
+
+- implement the Python wrapper in `sgl-kernel/python/sgl_kernel/elementwise.py`
+- then re-export it from `sgl-kernel/python/sgl_kernel/__init__.py`
+
+For example, in `sgl-kernel/python/sgl_kernel/elementwise.py`, add:
+
+```python
+import torch
+
+def scale(
+    input: torch.Tensor,
+    factor: float,
+    out: torch.Tensor | None = None,
+) -> torch.Tensor:
+    """
+    Element-wise scale: out = input * factor.
+
+    Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
+
+    Parameters
+    ----------
+    input  : CUDA input tensor
+    factor : scale factor (float)
+    out    : optional pre-allocated CUDA output tensor (same shape/dtype as input)
+    """
+    if out is None:
+        out = torch.empty_like(input)
+    torch.ops.sgl_kernel.scale.default(out, input, factor)
+    return out
+```
+
+Then re-export it from `sgl-kernel/python/sgl_kernel/__init__.py` following the existing import style used by other kernels.
+
+---
+
+## Step 6: Write tests (required)
+
+Create `sgl-kernel/tests/test_scale.py`:
+```python
+import pytest
+
+import torch
+import sgl_kernel
+
+@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
+@pytest.mark.parametrize("size", [128, 1024, 4096, 65536])
+@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0])
+def test_scale_correctness(dtype, size, factor):
+    input = torch.randn(size, dtype=dtype, device="cuda")
+    out   = torch.empty_like(input)
+
+    result = sgl_kernel.scale(input, factor, out=out)
+    assert result is out
+
+    expected = input * factor
+    rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
+    torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
+
+
+def test_scale_shape_mismatch():
+    input = torch.randn(128, dtype=torch.float16, device="cuda")
+    out   = torch.empty(256, dtype=torch.float16, device="cuda")
+    with pytest.raises(RuntimeError, match="same shape"):
+        sgl_kernel.scale(input, 2.0, out=out)
+
+
+def test_scale_cpu_input():
+    input = torch.randn(128, dtype=torch.float16)  # CPU
+    out   = torch.empty_like(input)
+    with pytest.raises(RuntimeError, match="CUDA"):
+        sgl_kernel.scale(input, 2.0, out=out)
+
+
+if __name__ == "__main__":
+    import sys
+    sys.exit(pytest.main([__file__, "-q"]))
+```
+
+---
+
+## Step 7: Add a benchmark (required)
+
+Create `sgl-kernel/benchmark/bench_scale.py`:
+
+```python
+import itertools
+
+import torch
+import triton
+import triton.testing
+
+import sgl_kernel
+from sglang.utils import is_in_ci
+
+IS_CI = is_in_ci()
+
+dtypes  = [torch.float16] if IS_CI else [torch.float16, torch.bfloat16, torch.float32]
+sizes   = [4096] if IS_CI else [2**n for n in range(10, 20)]  # 1K … 512K
+factors = [2.0]
+
+configs = list(itertools.product(dtypes, sizes))
+
+
+def torch_scale(input: torch.Tensor, factor: float) -> torch.Tensor:
+    return input * factor
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["dtype", "size"],
+        x_vals=configs,
+        line_arg="provider",
+        line_vals=["sglang", "torch"],
+        line_names=["SGL Kernel", "PyTorch"],
+        styles=[("green", "-"), ("red", "--")],
+        ylabel="µs (median)",
+        plot_name="scale-performance",
+        args={},
+    )
+)
+def benchmark(dtype, size, provider):
+    input  = torch.randn(size, dtype=dtype, device="cuda")
+    out    = torch.empty_like(input)
+    factor = 2.0
+
+    if provider == "sglang":
+        fn = lambda: sgl_kernel.scale(input, factor, out=out)
+    else:
+        fn = lambda: torch_scale(input, factor)
+
+    ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
+        fn, quantiles=[0.5, 0.2, 0.8]
+    )
+    return 1000 * ms, 1000 * max_ms, 1000 * min_ms
+
+
+if __name__ == "__main__":
+    benchmark.run(print_data=True)
+```
+
+---
+
+## Step 8: Build
+
+Build:
+
+```bash
+cd sgl-kernel
+make build -j16
+```
+
+If you need to limit host resource usage:
+
+```bash
+cd sgl-kernel
+make build -j1 MAX_JOBS=2 CMAKE_ARGS="-DSGL_KERNEL_COMPILE_THREADS=1"
+```
+
+---
+
+## Step 9: Validate
+
+After building successfully, run the test and benchmark:
+
+```bash
+pytest sgl-kernel/tests/test_scale.py -q
+python sgl-kernel/benchmark/bench_scale.py
+```
+
+PR CI also runs `pr-test-sgl-kernel.yml`, including the B200 job
+`sgl-kernel-b200-test` when kernel changes are detected. Use that job as the
+Blackwell coverage signal for AOT `sgl-kernel` changes.
+
+---
+
+## Troubleshooting
+
+- **Async CUDA errors**: `CUDA_LAUNCH_BLOCKING=1`
+- **Memory errors**: `compute-sanitizer --tool memcheck python ...`
+- **Build is too slow / OOM**: reduce `MAX_JOBS` and `SGL_KERNEL_COMPILE_THREADS`
+- **Binary bloat**: use `sgl-kernel/analyze_whl_kernel_sizes.py`
+- **CMake sources list**: if your `.cu` file is missing from `SOURCES`, the symbol will be undefined at link time
+
+---
+
+## References
+
+- `sgl-kernel/README.md`
+- `sgl-kernel/include/sgl_kernel_ops.h`
+- `sgl-kernel/csrc/common_extension.cc`
+- `sgl-kernel/CMakeLists.txt`
+- `sgl-kernel/include/utils.h` — `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro and friends
+- `sgl-kernel/csrc/elementwise/activation.cu` — reference for the FP16/BF16/FP32 dispatch pattern
+
+## Summary of Files Created/Modified
+
+```
+sgl-kernel/csrc/elementwise/scale.cu          # NEW: CUDA kernel + launcher
+sgl-kernel/include/sgl_kernel_ops.h           # MODIFIED: C++ declaration
+sgl-kernel/csrc/common_extension.cc           # MODIFIED: schema + dispatch registration
+sgl-kernel/CMakeLists.txt                     # MODIFIED: add source file (alphabetical)
+sgl-kernel/python/sgl_kernel/elementwise.py   # MODIFIED: Python wrapper
+sgl-kernel/python/sgl_kernel/__init__.py      # MODIFIED: re-export Python API
+sgl-kernel/tests/test_scale.py                # NEW: tests
+sgl-kernel/benchmark/bench_scale.py           # NEW: benchmark
+```

.claude/skills/ci-workflow-guide/SKILL.md

@@ -0,0 +1,406 @@
+---
+name: ci-workflow-guide
+description: Guide to SGLang CI workflow orchestration — stage ordering, fast-fail, gating, partitioning, execution modes, and debugging CI failures. Use when modifying CI workflows, adding stages, debugging CI pipeline issues, or understanding how tests are dispatched and gated across stages.
+---
+
+# SGLang CI Workflow Orchestration Guide
+
+This skill covers the CI **infrastructure** layer — how tests are dispatched, gated, and fast-failed across stages. For test authoring (templates, fixtures, registration, model selection), see the [write-sglang-test skill](../write-sglang-test/SKILL.md).
+
+---
+
+## Naming Conventions
+
+- **Suite**: `base-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `base-b-test-1-gpu-small`)
+- **Test group**: Directory-level registered test group under `test/registered/` (e.g., `hicache` maps to `test/registered/hicache/test_*.py`)
+- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
+
+---
+
+## Key Files
+
+| File | Role |
+|------|------|
+| `.github/workflows/pr-test.yml` | Main workflow — all stages, jobs, conditions, matrix definitions |
+| `.github/workflows/pr-test-extra.yml` | Extra workflow — gated by BOTH `run-ci` and `run-ci-extra` labels |
+| `.github/workflows/pr-gate.yml` | PR gating: draft check, `run-ci` label, per-user rate limiting |
+| `.github/actions/check-pr-test-health/action.yml` | Cross-job fast-fail: queries API for any failed job |
+| `.github/actions/wait-for-jobs/action.yml` | Stage gating: polls API until stage jobs complete |
+| `.github/actions/check-maintenance/action.yml` | Maintenance mode check |
+| `test/run_suite.py` | Suite runner: collects, filters, partitions, executes tests |
+| `python/sglang/test/ci/ci_register.py` | Test registration (AST-parsed markers), LPT auto-partition |
+| `python/sglang/test/ci/ci_utils.py` | `run_unittest_files()`: execution, retry, continue-on-error |
+| `scripts/ci/utils/slash_command_handler.py` | Handles slash commands from PR comments |
+
+---
+
+## Architecture Overview
+
+```
+ ┌──────────────┐
+ │ build kernel │
+ └──────┬───────┘
+        │
+        ├─ check-changes ──── detects which packages changed
+        │                      (main_package, sgl_kernel, jit_kernel, multimodal_gen)
+        │
+        ├─ call-gate ──────── pr-gate.yml (draft? label? rate limit?)
+        │
+        ├─────────────────────────────────────────────────────┐
+        │                                                     │
+        ▼                                                     │
+ ┌─────────────────────────────────────┐                      │
+ │          Base A (~3 min)            │                      │
+ │         pre-flight check            │                      │
+ │                                     │                      │
+ │  ┌─────────────────────────────┐    │                      │
+ │  │ base-a-test-1-gpu-small    │    │                      │
+ │  │ (small GPUs)                │    │                      │
+ │  └─────────────────────────────┘    │                      │
+ │  ┌─────────────────────────────┐    │                      │
+ │  │ base-a-test-cpu            │    │                      │
+ │  │ (CPU)                       │    │                      │
+ │  └─────────────────────────────┘    │                      │
+ └──────┬──────────────────────────────┘                      │
+        │                                                     │
+        ▼                                                     ▼
+ ┌─────────────────────────────────────┐          ┌──────────────────────────┐
+ │          Base B (~30 min)           │          │      kernel test         │
+ │            base tests               │          └──────────────────────────┘
+ │                                     │          ┌──────────────────────────┐
+ │  ┌─────────────────────────────┐    │          │   multimodal gen test    │
+ │  │ base-b-test-1-gpu-small    │    │          └──────────────────────────┘
+ │  │ (small GPUs, e.g. 5090)     │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ base-b-test-1-gpu-large    │    │
+ │  │ (large GPUs, e.g. H100)     │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ base-b-test-2-gpu-large    │    │
+ │  │ (large GPUs, e.g. H100)     │    │
+ │  └─────────────────────────────┘    │
+ └──────┬──────────────────────────────┘
+        │
+        ▼
+ ┌─────────────────────────────────────┐
+ │          Base C (~30 min)           │
+ │          advanced tests             │
+ │                                     │
+ │  ┌─────────────────────────────┐    │
+ │  │ base-c-test-4-gpu-h100     │    │
+ │  │ (H100 GPUs)                 │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ base-c-test-8-gpu-h200     │    │
+ │  │ (8 x H200 GPUs)             │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ base-c-test-4-gpu-b200     │    │
+ │  │ (4 x B200 GPUs)             │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ Other advanced tests        │    │
+ │  │ (DeepEP, PD Disagg, GB300)  │    │
+ │  └─────────────────────────────┘    │
+ └──────┬──────────────────────────────┘
+        │
+        ▼
+ ┌─────────────────────────────────────┐
+ │         pr-test-finish              │
+ │  aggregates all results, fails if   │
+ │  any job failed/cancelled           │
+ └─────────────────────────────────────┘
+```
+
+**Every stage test job** includes a `check-pr-test-health` step after checkout — if any job in the run has already failed, the job fast-fails (red X) with a root cause annotation.
+
+**Scheduled runs** skip `wait-for-base-*` jobs, running all stages in parallel. Fast-fail is also disabled.
+
+---
+
+## Fast-Fail Layers
+
+4 layers of fast-fail, from fine to coarse:
+
+| Layer | Mechanism | Granularity | Disabled on schedule? |
+|-------|-----------|-------------|----------------------|
+| **1. Test method → file** | `unittest -f` (failfast) | One test method fails → entire test file stops immediately | Yes |
+| **2. File → suite** | `run_unittest_files()` default | One test file fails → entire suite stops (`--continue-on-error` off) | Yes |
+| **3. Job → job (same stage)** | `check-pr-test-health` action | One job fails → other waiting jobs in same stage fast-fail (red X) | Yes |
+| **4. Stage → stage (cross-stage)** | `wait-for-base-*` + `needs` | Base A fails → base B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) |
+
+- **Layer 1**: `-f` flag appended to all `python3 -m pytest` / `unittest` invocations in `ci_utils.py`
+- **Layer 2**: `--continue-on-error` flag in `run_suite.py` — off for PRs, on for scheduled runs
+- **Layer 3**: `check-pr-test-health` auto-detects `schedule` event and skips; filters out cascade failures to show only root cause jobs
+- **Layer 4**: `wait-for-base-*` jobs are conditioned on `github.event_name == 'pull_request'` — skipped for scheduled runs
+
+---
+
+## Execution Modes
+
+| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | `/rerun-stage` (`workflow_dispatch`) |
+|--------|---------------------|------------------------------|--------------------------------------|
+| **Stage ordering** | Sequential: A → B → C via `wait-for-base-*` | Parallel (all at once) | Single target stage only |
+| **Cross-job fast-fail** | Yes (`check-pr-test-health`) | Yes | Yes |
+| **continue-on-error** | No (stop at first failure within suite) | Yes (run all tests) | No |
+| **Retry** | Enabled | Enabled | Enabled |
+| **max_parallel** | 3 (default), 14 if `high priority` label | 14 | 3 (default), 14 if `high priority` |
+| **PR gate** | Yes (draft, label, rate limit) | Skipped | Skipped |
+| **Concurrency** | `cancel-in-progress: true` per branch | Queue (no cancel) | Isolated per stage+SHA |
+
+---
+
+## Stage Gating (`wait-for-jobs` action)
+
+`wait-for-base-a` and `wait-for-base-b` are lightweight `ubuntu-latest` jobs that poll the GitHub Actions API.
+
+**How it works:**
+1. Calls `listJobsForWorkflowRun` to list all jobs in the current run
+2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `base-b-test-1-gpu-small (3)`)
+3. If any matched job has `conclusion === 'failure'` → fail immediately (fast-fail)
+4. If all matched jobs are completed and count matches `expected_count` → success
+5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
+6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
+
+**Job specs example** (base-b):
+```json
+[
+  {"prefix": "base-b-test-1-gpu-small", "expected_count": 8},
+  {"prefix": "base-b-test-1-gpu-large", "expected_count": 14},
+  {"prefix": "base-b-test-2-gpu-large", "expected_count": 4},
+  {"prefix": "base-b-test-4-gpu-b200", "expected_count": 1}
+]
+```
+
+> **Critical**: `expected_count` must match the matrix size. If you add/remove matrix entries, update the wait job's spec accordingly.
+
+**PR only**: Condition `github.event_name == 'pull_request' && !inputs.target_stage` — scheduled runs and `/rerun-stage` skip these entirely, allowing parallel execution.
+
+---
+
+## Cross-Job Fast-Fail (`check-pr-test-health` action)
+
+Composite action called after checkout in every stage test job (21 jobs total across `pr-test.yml`, `pr-test-multimodal-gen.yml`, `pr-test-sgl-kernel.yml`, `pr-test-jit-kernel.yml`).
+
+**How it works:**
+1. Queries `listJobsForWorkflowRun` for the current workflow run
+2. Filters for **root cause failures only** — jobs with `conclusion === 'failure'` whose failing step is NOT `check-pr-test-health` (excludes cascade failures)
+3. If root cause failures found → calls `core.setFailed()` with the list of root cause job names
+4. If none → does nothing (step succeeds)
+
+**Cascade filtering**: When job A fast-fails due to health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fast-fail. The filter checks each failed job's `steps` array — if the failing step name contains `check-pr-test-health` or `Check PR test health`, it's excluded from the root cause list.
+
+**Usage pattern:**
+```yaml
+steps:
+  - name: Checkout code
+    uses: actions/checkout@v4
+    ...
+
+  - uses: ./.github/actions/check-pr-test-health
+    id: pr-test-health
+
+  - name: Install dependencies        # skipped automatically if health check failed
+    ...                                # (default if: success() is false)
+
+  - name: Run test                     # also skipped
+    ...
+```
+
+**Visual effect**: Job shows **red X** (failure) with error annotation showing root cause job names. Subsequent steps are naturally skipped (default `if: success()` is false after a failed step). No per-step `if` guards needed.
+
+**No stage filtering**: Checks ALL jobs in the run, not just the current stage. Any failure anywhere triggers fast-fail.
+
+**Error message example:**
+```
+Fast-fail: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1)
+```
+
+---
+
+## Within-Suite Failure Handling
+
+Controlled by `run_unittest_files()` in `python/sglang/test/ci/ci_utils.py`.
+
+### Flags
+
+| Flag | PR default | Scheduled default | Effect |
+|------|------------|-------------------|--------|
+| `--continue-on-error` | Off | On | Off: stop at first failure. On: run all files, report all failures at end |
+| `--enable-retry` | On | On | Retry retriable failures (accuracy/perf assertions) |
+| `--max-attempts` | 2 | 2 | Max attempts per file including initial run |
+
+### Retry Classification
+
+When a test fails and retry is enabled, the output is classified:
+
+**Non-retriable** (checked first — real code errors):
+`SyntaxError`, `ImportError`, `ModuleNotFoundError`, `NameError`, `TypeError`, `AttributeError`, `RuntimeError`, `CUDA out of memory`, `OOM`, `Segmentation fault`, `core dumped`, `ConnectionRefusedError`, `FileNotFoundError`
+
+**Retriable** (accuracy/performance):
+`AssertionError` with comparison patterns (`not greater than`, `not less than`, `not equal to`), `accuracy`, `score`, `latency`, `throughput`, `timeout`
+
+**Default**: Unknown `AssertionError` → retriable. Other unknown failures → not retriable.
+
+### How `continue_on_error` is set
+
+In `pr-test.yml`'s `check-changes` job:
+- `schedule` runs or `run_all_tests` flag → `continue_on_error = 'true'`
+- PR runs → `continue_on_error = 'false'`
+
+Each test job propagates via:
+```yaml
+env:
+  CONTINUE_ON_ERROR_FLAG: ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
+run: |
+  python3 run_suite.py --hw cuda --suite <name> $CONTINUE_ON_ERROR_FLAG
+```
+
+---
+
+## Test Partitioning
+
+Large suites are split across matrix jobs using the **LPT (Longest Processing Time) heuristic** in `ci_register.py:auto_partition()`:
+
+1. Sort tests by `est_time` descending, filename as tie-breaker (deterministic)
+2. Greedily assign each test to the partition with smallest cumulative time
+3. Result: roughly equal total time per partition
+
+**Partition table** (CUDA per-commit suites):
+
+| Suite | Partitions | Runner | max_parallel |
+|-------|-----------|--------|-------------|
+| `base-a-test-1-gpu-small` | 1 (no matrix) | `1-gpu-5090` | — |
+| `base-a-test-cpu` | 4 | `ubuntu-latest` | — |
+| `base-b-test-1-gpu-small` | 8 | `1-gpu-5090` | 8 |
+| `base-b-test-1-gpu-large` | 14 | `1-gpu-h100` | dynamic (3 or 14) |
+| `base-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
+| `base-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
+| `base-b-kernel-unit-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
+| `base-b-kernel-unit-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
+| `base-b-kernel-unit-test-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
+| `base-b-kernel-benchmark-test-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
+| `base-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
+| `base-c-test-8-gpu-h200` | 4 | `8-gpu-h200` | — |
+| `base-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
+| `base-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
+| `base-c-test-4-gpu-b200` | 3 | `4-gpu-b200` | — |
+| `base-c-test-4-gpu-b200-small` | 3 | `4-gpu-b200-low-disk` | — |
+| `base-c-test-8-gpu-b200` | registered only | `8-gpu-b200` | — |
+| `base-c-test-4-gpu-gb200` | registered only | `4-gpu-gb200` | — |
+
+> **Suite names are generated**, not hand-written: each comes from a test's `register_*_ci(stage=..., runner_config=...)` as `{stage}-test-{runner_config}`, and `runner_config` maps to the `Runner` column via `scripts/ci/runner_configs.yml`.
+>
+> **Note**: Kernel suites (`base-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. `base-c-test-8-gpu-b200` is registered in `test/run_suite.py` but not wired to PR CI. The GB200 job is currently commented out in `pr-test.yml` until a company-owned runner is provisioned. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
+
+**Workflow usage:**
+```yaml
+strategy:
+  matrix:
+    partition: [0, 1, 2, 3, 4, 5, 6, 7]
+steps:
+  - run: python3 run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
+           --auto-partition-id ${{ matrix.partition }} --auto-partition-size 8
+```
+
+---
+
+## check-changes Job
+
+Determines which test suites to run based on file changes.
+
+### Detection Methods
+
+| Trigger | Method | Details |
+|---------|--------|---------|
+| `pull_request` | `dorny/paths-filter` | Detects changes via GitHub diff |
+| `workflow_dispatch` (with `pr_head_sha`) | GitHub API | `repos/{repo}/compare/main...{sha}` |
+| `schedule` / `run_all_tests` | Force all true | Runs everything |
+
+### Output Flags
+
+| Output | Triggers |
+|--------|----------|
+| `main_package` | Base A/B/C test suites |
+| `sgl_kernel` | Kernel wheel builds + kernel test suites; also switches B200 jobs to kernel-build runner labels outside `target_stage` mode |
+| `jit_kernel` | JIT kernel test workflow |
+| `multimodal_gen` | Multimodal-gen test workflow |
+
+> **Note**: In `target_stage` mode, `sgl_kernel` is only active when `include_wheel_build=true`. Without that opt-in, kernel-change reruns fail validation instead of running a target stage without freshly built wheels. Outside `target_stage`, `sgl_kernel=true` switches B200 jobs from `4-gpu-b200` / `4-gpu-b200-low-disk` to `4-gpu-b200-kernel` / `4-gpu-b200-kernel-low-disk`.
+
+---
+
+## Concurrency Control
+
+```
+group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
+```
+
+| Segment | Source | Purpose |
+|---------|--------|---------|
+| `event_name` | `github.event_name` | Prevents scheduled runs colliding with fork PRs named `main` |
+| `branch` | `github.head_ref \|\| github.ref_name` | Per-branch isolation |
+| `pr_sha` | `inputs.pr_head_sha \|\| 'current'` | Isolates `/rerun-stage` from main runs |
+| `stage` | `inputs.target_stage \|\| 'all'` | Allows parallel stage dispatches |
+
+`cancel-in-progress: true` for `pull_request` events (new push cancels old run), `false` for `workflow_call`.
+
+---
+
+## How To: Add a New Stage Job
+
+1. Define the job in `pr-test.yml` with `needs: [check-changes, call-gate, wait-for-base-X, ...]`
+2. Copy the `if:` condition pattern from an existing same-stage job (handles `target_stage`, `schedule`, `main_package`)
+3. Add `checkout` step
+4. Add `check-pr-test-health` step (after checkout) — if any prior job failed, `core.setFailed()` fires and all subsequent steps auto-skip via default `if: success()`
+5. Add `check-maintenance` step
+6. Add `download-artifact` step if `sgl_kernel` changed
+7. Add `install dependencies` step
+8. Add `run test` step with `$CONTINUE_ON_ERROR_FLAG`
+9. Add `upload-cuda-coredumps` step with `if: always()`
+10. Register the suite name in `PER_COMMIT_SUITES` in `test/run_suite.py`
+11. If using matrix, add `--auto-partition-id` and `--auto-partition-size` to the run command
+12. **Update `wait-for-base-X`** job spec with the new job name and `expected_count` (if matrix)
+13. **Add the job to `pr-test-finish.needs`** list
+
+---
+
+## How To: Debug CI Failures
+
+| Symptom | Likely cause | What to check |
+|---------|-------------|---------------|
+| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
+| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
+| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
+| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
+| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
+| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
+
+---
+
+## Slash Commands
+
+| Command | Effect |
+|---------|--------|
+| `/tag-run-ci-label` | Adds `run-ci` label to PR |
+| `/tag-run-ci-label extra` | Adds both `run-ci` and `run-ci-extra` labels |
+| `/rerun-failed-ci` | Reruns failed jobs in the latest workflow run |
+| `/tag-and-rerun-ci` | Adds `run-ci` label + reruns failed |
+| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
+| `/rerun-stage <stage>` | Deprecated; posts deprecation notice |
+| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
+| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
+
+Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
+
+### Label-gated workflow dispatch (pr-test, pr-test-extra)
+
+`pr-test.yml` and `pr-test-extra.yml` both listen for `pull_request.labeled` (in addition to `opened`/`synchronize`/`reopened`). The `check-changes.if` gate has two clauses:
+
+1. **For `labeled` events**: the just-added label must be one of the gating labels (`run-ci` for pr-test, `run-ci` or `run-ci-extra` for pr-test-extra) — otherwise every unrelated label addition would dispatch a full CI run.
+2. **All events**: the PR must currently carry the required labels.
+
+This is what lets `/tag-run-ci-label` (and the `extra` variant) trigger a fresh CI run without an extra push.
+
+**Caveat — skipped runs cannot be un-skipped by `run.rerun()`:** GitHub's rerun API reuses the original event payload, so rerunning a `pull_request`-event run that was skipped because of missing labels will skip again (label set in the frozen payload doesn't update). The only way to recover a label-skipped run is to add the missing label, which fires a fresh `labeled` event with the current label set. `handle_rerun_failed_ci` in the slash handler is for rerunning failed/non-label-skipped runs; it cannot revive label-skipped ones.

.claude/skills/clean-startup-log/SKILL.md

@@ -0,0 +1,282 @@
+---
+name: clean-startup-log
+description: Clean up noisy startup warnings and spurious prints in SGLang server logs. Use when users ask to clean up unwanted warnings, deprecation messages, or third-party noise in the server startup output.
+disable-model-invocation: true
+---
+
+# Clean Up SGLang Server Startup Logs
+
+Goal: ensure the server startup log is clean and minimal, with no spurious warnings, deprecation messages, or unformatted prints from third-party libraries.
+
+## Workflow
+
+### 1. Launch a server and capture the log
+
+```bash
+uv run sglang serve --model-path Qwen/Qwen3-8B 2>&1 | tee /tmp/startup_log.txt
+```
+
+Wait until the server prints `The server is fired up and ready to roll!`, then Ctrl-C.
+
+For TP>1 testing:
+```bash
+uv run sglang serve --model-path Qwen/Qwen3-8B --tp 2 2>&1 | tee /tmp/startup_log.txt
+```
+
+For MoE / hybrid-SWA models (e.g. gpt-oss), test separately — they exercise different code paths:
+```bash
+uv run sglang serve --model-path openai/gpt-oss-20b 2>&1 | tee /tmp/startup_log.txt
+```
+
+### 2. Compare against the clean reference log
+
+Read `/tmp/startup_log.txt` and compare it against the reference log at the bottom of this file. Identify lines that:
+
+- Do NOT have the `[timestamp]` or `[timestamp TPx]` logger prefix
+- Contain `WARNING`, `deprecated`, `is deprecated`, or similar noise
+- Are printed by third-party libraries (transformers, torchao, NCCL, Gloo, tqdm, etc.)
+- Are duplicate/redundant with information already logged by SGLang
+- Appear multiple times due to `ModelConfig` being constructed in multiple processes
+
+### 3. Classify each noisy line
+
+For each noisy line, determine:
+
+| Category | Action |
+|----------|--------|
+| **SGLang code using wrong API** | Fix the SGLang code (e.g., replace deprecated API with new one) |
+| **SGLang code logging at wrong level** | Change log level (e.g., warning -> debug for non-actionable messages) |
+| **Duplicated across processes** | Downgrade to debug — info logged in one process becomes noise in 3-4 |
+| **Third-party lib prints at import time** | Suppress the logger or redirect stdout during that import |
+| **C-level print from .so library** | Redirect fd 1 during the specific C call, or accept it if too invasive |
+| **Real warning the user should see** | Keep it |
+
+### 4. Present findings before fixing
+
+List all noisy lines with their source and proposed fix. Ask the user to review before making changes.
+
+### 5. Apply fixes and verify
+
+After approval, apply fixes one at a time, re-launch the server, and verify each fix works.
+
+## Key Architecture: Why Logs Repeat
+
+`ModelConfig` is constructed **3-4 times** during startup across different processes:
+1. Main process: `ServerArgs.__post_init__()` → `get_model_config()` → `ModelConfig()`
+2. Scheduler subprocess: `Scheduler.init_model_config()` → `ModelConfig.from_server_args()`
+3. Scheduler subprocess: `TpModelWorker._init_model_config()` → `ModelConfig.from_server_args()`
+4. Main process: `TokenizerManager.init_model_config()` → `ModelConfig.from_server_args()`
+
+Similarly, `get_tokenizer()` is called **5 times** across processes:
+1. `resolve_auto_parsers` (main) — `template_detection.py`
+2. `Scheduler.init_tokenizer()` (scheduler subprocess) — `scheduler.py`
+3. `DetokenizerManager` (detokenizer subprocess) — `detokenizer_manager.py`
+4. `TpModelWorker.__init__()` (scheduler subprocess) — `tp_worker.py`
+5. `TokenizerManager` (main) — `tokenizer_manager.py`
+
+Any `logger.info()` or `logger.warning()` in `ModelConfig.__init__()` or `get_tokenizer()` will appear 3-5 times. **Keep these at `logger.debug()`.**
+
+## Known Noise Sources and Fixes (from past sessions)
+
+### 1. torchao "Skipping import of cpp extensions due to incompatible torch version"
+
+- **Source:** `torchao/__init__.py` — printed via `logger.warning()` when torch version < 2.11.0
+- **Trigger:** `sglang/__init__.py` -> `_apply_hf_patches()` -> `_patch_removed_symbols()` -> `from transformers.models.llama import modeling_llama` -> deep import chain -> `transformers/quantizers/auto.py` -> `from .quantizer_torchao import TorchAoHfQuantizer` -> imports torchao
+- **Fix:** In `hf_transformers_patches.py::_patch_removed_symbols()`, temporarily set the `torchao` logger level to `ERROR` around the `modeling_llama` import:
+  ```python
+  _torchao_logger = logging.getLogger("torchao")
+  _prev_level = _torchao_logger.level
+  _torchao_logger.setLevel(logging.ERROR)
+  try:
+      from transformers.models.llama import modeling_llama
+  finally:
+      _torchao_logger.setLevel(_prev_level)
+  ```
+
+### 2. "`torch_dtype` is deprecated! Use `dtype` instead!" (PARTIALLY FIXED)
+
+- **Source:** `transformers/configuration_utils.py` — the `torch_dtype` property warns via `logger.warning_once()`
+- **Trigger:** Model files accessing `config.torch_dtype` instead of `config.dtype`
+- **Fix applied so far:** Only `models/gpt_oss.py` (lines 222, 471) — tested with `openai/gpt-oss-20b`.
+- **Remaining files that still use `config.torch_dtype`** (fix each only after testing with the corresponding model):
+  - `models/bailing_moe.py` (line 302)
+  - `models/llada2.py` (line 313)
+  - `models/qwen3_next.py` (lines 192, 209)
+  - `models/qwen3_5.py` (line 245)
+  - `models/nano_nemotron_vl.py` (lines 79, 102, 284)
+  - `models/llava.py` (lines 732, 734-737)
+  - `model_loader/loader.py` (line 649)
+- **Note:** `common.py` was already fixed in a prior session. If new model files are added with `config.torch_dtype`, the warning will reappear — grep for `\.torch_dtype` to find them.
+- **Important:** Only change `config.torch_dtype` → `config.dtype` for models you have actually tested. The `dtype` property should return the same value, but verify per-model to avoid regressions.
+
+### 3. "`BaseImageProcessorFast` is deprecated"
+
+- **Source:** `transformers/utils/import_utils.py` — the lazy module `__getattr__` warns when `BaseImageProcessorFast` is accessed
+- **Trigger:** `base_processor.py` and `ernie45_vl.py` have `from transformers import BaseImageProcessorFast` at top level. These are imported eagerly via `tokenizer_manager.py` -> `multimodal_processor.py` -> `base_processor.py`, even for non-multimodal models.
+- **Fix:** Replace `from transformers import BaseImageProcessorFast` with `from transformers import BaseImageProcessor` and update all `isinstance(..., BaseImageProcessorFast)` checks to `isinstance(..., BaseImageProcessor)`
+
+### 4. "No platform detected. Using base SRTPlatform with defaults."
+
+- **Source:** `sglang/srt/platforms/__init__.py` — `logger.warning()`
+- **Fix:** Change to `logger.debug()` — this is expected on machines without a platform plugin and not actionable.
+
+### 5. `NCCL version 2.27.7+cuda13.0`
+
+- **Source:** C-level print from `libnccl.so` during `ncclCommInitRank()` call
+- **Status:** Accepted as-is. SGLang already logs the version via `sglang is using nccl==X.Y.Z`. The C-level print cannot be suppressed without redirecting stdout fd, which is too invasive. `NCCL_DEBUG=WARN` does not suppress it in NCCL 2.27+.
+
+### 6. `[Gloo] Rank X is connected to Y peer ranks`
+
+- **Source:** C++ Gloo library print during process group init
+- **Status:** Accepted as-is. From C++ code inside PyTorch's Gloo backend.
+
+### 7. `torchao SyntaxWarning: invalid escape sequence`
+
+- **Source:** `torchao/quantization/quant_api.py` — a raw string with unescaped `\.`
+- **Status:** Upstream torchao bug. Cannot fix from SGLang side.
+
+### 8. tqdm progress bars (e.g., `Multi-thread loading shards`, `Capturing batches`)
+
+- **Status:** These are expected and useful. They show progress during weight loading and CUDA graph capture. Keep them.
+
+### 9. CUTE_DSL "Unexpected error during package walk" — double-logged (FIXED)
+
+- **Source:** `nvidia-cutlass-dsl` package at `.venv/.../cutlass/cutlass_dsl/cutlass.py`, line 391. Logger named `CUTE_DSL` with its own `StreamHandler`.
+- **Trigger:** During CUDA graph capture, cutlass DSL walks packages and hits an unexpected error for `cutlass.cute.experimental`.
+- **Root cause of double-logging:** The CUTE_DSL logger has `propagate=True` (default), so the warning is emitted by both the CUTE_DSL handler (with its format) and the root logger (SGLang's format).
+- **Fix applied:** In `entrypoints/engine.py`, changed `CUTE_DSL_LOG_LEVEL` from `"30"` (WARNING) to `"40"` (ERROR). This suppresses the WARNING at both the CUTE_DSL logger and root propagation levels. The env var controls both `logger.setLevel()` and `console_handler.setLevel()` in cutlass's `setup_log()`.
+
+### 10. ModelConfig init logs repeated 3x (FIXED)
+
+- **Lines:** `"Downcasting torch.float32 to ..."`, `"Hybrid swa model: ..."`, `"DeepGemm is enabled but ..."`
+- **Source:** `configs/model_config.py` — `_get_and_verify_dtype()` (line 1457), `_derive_hybrid_model()` (line 497), `_verify_quantization()` (line 1236)
+- **Root cause:** `ModelConfig.__init__()` is called 3-4 times in different processes (see "Key Architecture" above). Each construction fires the same log lines.
+- **Fix applied:** Downgraded all three from `logger.info()`/`logger.warning()` to `logger.debug()`. The dtype is already visible in `server_args` and `Load weight end`. Hybrid SWA info appears in `Tree cache initialized`. DeepGemm is not actionable.
+
+### 11. Tokenizer retry/fallback messages repeated 3-4x (FIXED)
+
+- **Lines:** `"Tokenizer loaded as generic TokenizersBackend ... retrying"`, `"Loading tokenizer ... directly as PreTrainedTokenizerFast"`, `"Tokenizer for ... loaded as generic TokenizersBackend. Set --trust-remote-code"`
+- **Source:** `utils/hf_transformers/tokenizer.py` — `_resolve_tokenizers_backend()` (line 215), `_load_tokenizer_by_declared_class()` (line 110), final warning (line 244)
+- **Root cause:** 5 separate `get_tokenizer()` calls across processes (see "Key Architecture" above). Each produces 3 log lines. Concurrent subprocess launches cause interleaved/doubled output.
+- **Fix applied:** Downgraded all three from `logger.warning()`/`logger.info()` to `logger.debug()`.
+
+### 12. Template detection logs — 5 lines consolidated to 1 (FIXED)
+
+- **Lines:** `"Detected reasoning config '...' from template rule '...'"`, `"Detected reasoning parser '...' from template rule '...'"`, `"Detected tool-call parser '...' from template rule '...'"`, `"Auto-detected reasoning parser: ..."`, `"Auto-detected tool-call parser: ..."`
+- **Source:** `managers/template_detection.py` (lines 337, 370) logged each detection rule match. `managers/template_manager.py` (lines 177-182) logged summary lines that duplicated the detection logs.
+- **Fix applied:** Removed per-rule logs from `template_detection.py`. Consolidated the 5 lines in `template_manager.py` into a single summary: `"Auto-detected template features: reasoning_config=..., reasoning_parser=..., tool_call_parser=..."`
+
+### 13. KV cache dtype logged separately from allocation (FIXED)
+
+- **Lines:** `"Using KV cache dtype: torch.bfloat16"` then `"KV Cache is allocated. #tokens: ..., K size: ..., V size: ..."`
+- **Source:** `model_executor/model_runner.py` (line 2217) and `mem_cache/memory_pool.py` (line 740)
+- **Fix applied:** Removed the standalone dtype log from `model_runner.py`. Added `dtype` field to the allocation log in `memory_pool.py`: `"KV Cache is allocated. dtype: torch.bfloat16, #tokens: ..., K size: ..., V size: ..."`
+
+### 14. CUTLASS backend warning — B200 → SM100, warning → info (FIXED)
+
+- **Line:** `"CUTLASS backend is disabled when piecewise cuda graph is enabled due to TMA descriptor initialization issues on B200."`
+- **Source:** `layers/attention/flashinfer_backend.py` (line 249)
+- **Fix applied:** Changed "B200" to "SM100 GPUs" (the condition checks `is_sm100_supported()` which matches SM10x, not just B200). Downgraded from `logger.warning()` to `logger.info()` since it's an expected automatic fallback.
+
+### 15. `max_total_num_tokens` and `Tree cache initialized` log ordering
+
+- **Issue:** `max_total_num_tokens=...` appears before `Tree cache initialized:...` even though tree cache is conceptually part of memory setup.
+- **Root cause:** `max_total_num_tokens` is logged inside `init_model_worker()` (scheduler.py:972), which runs before `build_kv_cache()` (scheduler.py:425) where tree cache is created.
+- **Status:** Not fixed — reordering was reverted. Acceptable as-is.
+
+### 16. `Ignore import error when loading sglang.srt.models.midashenglm`
+
+- **Source:** `models/registry.py` (line 109) — `logger.warning()` during `import_model_classes()` which iterates all model modules via `pkgutil.iter_modules`
+- **Trigger:** The `midashenglm` model depends on `torchaudio`, which fails to load
+- **Status:** Should be downgraded to `logger.debug()` — not actionable when loading an unrelated model. Same pattern exists in `managers/multimodal_processor.py`, `dllm/algorithm/__init__.py`, `multimodal_gen/runtime/models/registry.py`.
+
+### 17. `Multiple NUMA nodes found for GPU X`
+
+- **Source:** `utils/numa_utils.py` (line 112) — `logger.warning()`
+- **Status:** Could be downgraded to `logger.info()`. The situation is handled gracefully ("Using the first one") and not actionable.
+
+### 18. Warmup `/model_info` access log
+
+- **Source:** Uvicorn access log, triggered by SGLang's own warmup at `entrypoints/http_server.py` (line 1877)
+- **Status:** SGLang talking to itself. Could suppress uvicorn access logger during warmup, or exclude `/model_info` from warmup access logging.
+
+## Investigation Techniques
+
+### Trace what triggers an import
+```python
+import sys
+_real_import = __builtins__.__import__
+def _tracing_import(name, *args, **kwargs):
+    if 'TARGET_MODULE' in name:
+        import traceback
+        print(f'=== Importing {name} ===')
+        traceback.print_stack()
+    return _real_import(name, *args, **kwargs)
+__builtins__.__import__ = _tracing_import
+```
+
+### Trace what triggers a logger warning
+```python
+import logging, traceback
+class TraceHandler(logging.Handler):
+    def emit(self, record):
+        if 'SEARCH_STRING' in record.getMessage():
+            traceback.print_stack()
+h = TraceHandler()
+h.setLevel(logging.WARNING)
+logging.getLogger('TARGET_LOGGER_NAME').addHandler(h)
+```
+
+### Find C-level prints in .so files
+```bash
+strings /path/to/library.so | grep "SEARCH_STRING"
+```
+
+### Find all config.torch_dtype accesses (for deprecation warning)
+```bash
+grep -rn '\.torch_dtype' python/sglang/srt/models/ python/sglang/srt/model_loader/ python/sglang/srt/utils/hf_transformers/
+```
+
+## Reference: Clean Startup Log (TP=1, Qwen3-8B)
+
+```
+[2026-05-24 00:52:39] Attention backend not specified. Use trtllm_mha backend by default.
+[2026-05-24 00:52:39] TensorRT-LLM MHA only supports page_size of 16, 32 or 64, changing page_size from None to 64.
+[2026-05-24 00:52:40] server_args=ServerArgs(model_path='Qwen/Qwen3-8B', ...)
+[2026-05-24 00:52:40] Multiple NUMA nodes found for GPU 0: [...]. Using the first one.
+[2026-05-24 00:52:42] Using default HuggingFace chat template with detected content format: string
+[2026-05-24 00:52:42] Auto-detected template features: reasoning_config=..., reasoning_parser=qwen3, tool_call_parser=qwen
+[2026-05-24 00:52:50] Init torch distributed begin.
+[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
+[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
+[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
+[2026-05-24 00:52:50] Init torch distributed ends. elapsed=0.21 s, mem usage=0.10 GB
+[2026-05-24 00:52:51] Load weight begin. avail mem=275.75 GB
+[2026-05-24 00:52:51] Found local HF snapshot for Qwen/Qwen3-8B at ...; skipping download.
+Multi-thread loading shards: 100% Completed | 5/5 [00:01<00:00,  2.62it/s]
+[2026-05-24 00:52:54] Load weight end. elapsed=2.62 s, type=Qwen3ForCausalLM, avail mem=260.48 GB, mem usage=15.28 GB.
+[2026-05-24 00:52:54] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 1707904, K size: 117.28 GB, V size: 117.28 GB
+[2026-05-24 00:52:54] Memory pool end. avail mem=25.28 GB
+[2026-05-24 00:52:54] CUTLASS backend is disabled when piecewise cuda graph is enabled due to TMA descriptor initialization issues on SM100 GPUs. Using auto backend instead for stability.
+[2026-05-24 00:52:54] Capture cuda graph begin. This can take up to several minutes. avail mem=24.16 GB
+[2026-05-24 00:52:54] Capture cuda graph bs [1, 2, 4, ...]
+Capturing batches (bs=1 avail_mem=23.56 GB): 100% | 52/52 [00:05<00:00, 10.36it/s]
+[2026-05-24 00:53:00] Capture cuda graph end. Time elapsed: 5.38 s. mem usage=0.60 GB. avail mem=23.56 GB.
+[2026-05-24 00:53:00] Capture piecewise CUDA graph begin. avail mem=23.56 GB
+[2026-05-24 00:53:00] Capture cuda graph num tokens [4, 8, 12, ...]
+Compiling num tokens (num_tokens=4): 100% | 74/74 [00:09<00:00, 7.44it/s]
+Capturing num tokens (num_tokens=4 avail_mem=21.24 GB): 100% | 74/74 [00:07<00:00, 10.44it/s]
+[2026-05-24 00:53:18] Capture piecewise CUDA graph end. Time elapsed: 18.18 s. mem usage=2.32 GB. avail mem=21.24 GB.
+[2026-05-24 00:53:20] Tree cache initialized: source=default impl=RadixCache hybrid_swa=False hybrid_ssm=False hierarchical=False streaming_wrapped=False
+[2026-05-24 00:53:20] max_total_num_tokens=1707904, chunked_prefill_size=16384, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=21.24 GB
+[2026-05-24 00:53:20] INFO:     Started server process [1964249]
+[2026-05-24 00:53:20] INFO:     Waiting for application startup.
+[2026-05-24 00:53:20] Using default chat sampling params from model generation config: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
+[2026-05-24 00:53:20] INFO:     Application startup complete.
+[2026-05-24 00:53:20] INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
+[2026-05-24 00:53:21] Prefill batch, #new-seq: 1, #new-token: 64, ...
+[2026-05-24 00:53:21] INFO:     127.0.0.1:... - "POST /generate HTTP/1.1" 200 OK
+[2026-05-24 00:53:21] The server is fired up and ready to roll!
+```
+
+Note: `[Gloo]` messages and tqdm progress bars are acceptable. The key is no warnings or deprecation messages from transformers, torchao, or other third-party libraries. The `CUTLASS backend is disabled` message is now `info` level, not a warning.

.claude/skills/cookbook-add-model/SKILL.md

@@ -0,0 +1,226 @@
+---
+name: cookbook-add-model
+description: Add a new model to the SGLang Cookbook (docs_new/, Mintlify), config-driven format — instantiate the model-agnostic template into a per-model config (+ benchmarks) JSX under src/snippets/configs/, an MDX page, the docs.json nav entry, NEW-tag hygiene, and the homepage vendor card. Interactive, multi-phase. Run with /cookbook-add-model.
+disable-model-invocation: true
+---
+
+# Add a model to the SGLang Cookbook
+
+> Migrating an **existing legacy-template page** (one that imports a monolithic
+> `…/autoregressive/<slug>-deployment.jsx` generator)? Use the
+> `cookbook-migrate-model` skill instead — same target format, but the legacy
+> page (not the user) is the source of truth.
+
+The cookbook is **config-driven**: two shared engines contain NO model-specific code —
+`docs_new/src/snippets/_deployment.jsx` (the 5-dim deploy matrix) and
+`_playground.jsx` (the diff-based override Playground). Adding a model = adding **data**:
+a per-model `config` (+ optional `benchmarks`) consumed by both engines, plus an MDX page
+that imports them. No engine edits.
+
+**Instantiate the model-agnostic template** (NOT a clone of any live cookbook — the
+template is decoupled and covers all hardware + all axes):
+- `templates/config.jsx.tmpl` → `docs_new/src/snippets/configs/<hf-org>/<model-slug>.jsx`
+- `templates/benchmarks.jsx.tmpl` → `…/<model-slug>-benchmarks.jsx` (skip if no numbers)
+- `templates/page.mdx.tmpl` → `docs_new/cookbook/<category>/<Vendor>/<ModelName>.mdx`
+
+The template uses explicit `__TOKEN__` placeholders; you fill them, prune what the model
+lacks, and replace the EXAMPLE cells with verified recipes. DeepSeek-V4 is a populated
+*instance* you can consult, but is not the template.
+
+**Deep references (read on demand, don't inline):**
+- [references/authoring-reference.md](references/authoring-reference.md) — field-by-field config / cells / playground / MDX contract.
+- [references/mintlify-authoring.md](references/mintlify-authoring.md) — MDX rules (forbidden syntax, JSX tables, labeled fences) + invocation-example patterns. Read before writing §1–§3 prose.
+- [references/engine-axis.md](references/engine-axis.md) — adding a new Playground feature axis (rare engine work).
+- [references/vendor-logo.md](references/vendor-logo.md) — new-vendor card logo: ask the user for the brand logo, then generate the icon-only 940×525 RGBA PNG (spec + Pillow recipe + `git add -f`).
+
+## Architecture at a glance
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│  cookbook/<category>/<Vendor>/<Model>.mdx                       │
+│  import { Deployment } from "/src/snippets/_deployment.jsx";    │
+│  import { Playground } from "/src/snippets/_playground.jsx";    │
+│  import { config }     from "/src/snippets/configs/.../X.jsx";  │
+│  <Deployment config={config} />   <Playground config={config} />│
+└─────────────────────────────────────────────────────────────────┘
+              │ (config passed as React prop)
+              ▼
+┌─────────────────────────────────────────────────────────────────┐
+│  src/snippets/configs/<vendor>/<model>.jsx                      │
+│  export const config = {                                        │
+│    supportedHardware, variants, quantizations, strategies, ...  │
+│    cells: [ { match:{hw,variant,quant,strategy,nodes},          │
+│              env:[...], flags:[...] }, ... ],   // 5-dim matrix  │
+│    playgroundFeatures: { attention, moe, parsers, ... },        │
+│  };                                                             │
+└─────────────────────────────────────────────────────────────────┘
+              │ (consumed by BOTH engines — no model code in engines)
+              ▼
+┌──────────────────────────────────┬──────────────────────────────┐
+│  _deployment.jsx                 │  _playground.jsx             │
+│  Renders the verified matrix;    │  Renders override chips +    │
+│  one cell → its env/flags.       │  diff against the cell.      │
+└──────────────────────────────────┴──────────────────────────────┘
+```
+
+The two widgets stay in sync via: the **URL hash** (deploy mirrors its selection;
+playground reads it), the **`sglang-deploy-sel` custom event** (deploy dispatches on
+every change; playground listens — `replaceState` doesn't fire `hashchange`), and the
+shared **`sglang-deploy-env` localStorage key** (HOST/PORT placeholders).
+
+> The template is **autoregressive**. Diffusion / omni pages follow their own category
+> structure — don't force the config-driven template on them; still obey the Mintlify /
+> NEW-tag / docs.json / category-card / validation rules below.
+
+---
+
+**Interactive, multi-step workflow. Collect inputs incrementally — don't ask for
+everything upfront.** The real work is the verified `cells[]` recipes + measured
+benchmarks (Phases 2 + 4); everything else is filling the template.
+
+## Phase 1 — Collect inputs
+
+1. **Model card** — HuggingFace repo/URL. **Fetch the page** and extract description,
+   param count, architecture, context length, **license**. (Fetching guards factual bugs
+   like an off-by-a-few-B param count.) If the model isn't public, ask the user.
+2. **Variants / quantizations** — keep separate: variants are size/mode (e.g. Flash/Pro,
+   Instruct/Thinking); quantizations come from the HF card / linked repos (BF16/FP8/FP4/…).
+   Default to BF16 when a full-precision repo exists.
+3. **Tested hardware + parallelism** — which platforms are actually tested, and TP/EP/DP
+   for each. List only tested hw (unlisted greys out).
+4. **Verified launch recipes** — the full `sglang serve` flags per
+   (hw × variant × quant × strategy × nodes) combo → these become `cells[]`. Rewrite any
+   `python -m sglang.launch_server` to `sglang serve` form.
+5. **sglang version / image tag** — ask which sglang build the recipes + benchmarks ran on
+   (a release like `0.5.x`, or `main`/nightly). **Never guess or hallucinate it.** This one
+   tag fills `dockerImages` and the benchmarks' `sglang_version`; when the user is unsure,
+   default the image to `lmsysorg/sglang:dev` (nightly) rather than inventing a release.
+6. **Pre-flight**: `gh pr list --repo sgl-project/sglang --search "<model>"` (dup check).
+
+**Hardware reference** (the shared `HARDWARE_CATALOG` in `_deployment.jsx`). A GPU **not** in
+this table (RTX PRO 6000, GH200, future chips) goes in the model's own `config.hardware`
+(`{id,label,vram,vendor}`) — the engine merges it in; don't edit the engine catalog:
+
+| Platform | Vendor | VRAM | Docker image |
+|---|---|---|---|
+| H100 | NVIDIA | 80GB | `lmsysorg/sglang:<ver>` |
+| H200 | NVIDIA | 141GB | `lmsysorg/sglang:<ver>` |
+| B200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` |
+| B300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130` when required) |
+| GB200 | NVIDIA | 192GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
+| GB300 | NVIDIA | 288GB | `lmsysorg/sglang:<ver>` (or `-cu130`) |
+| MI300X | AMD | 192GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
+| MI325X | AMD | 256GB | `lmsysorg/sglang:<ver>-rocm720-mi30x` |
+| MI350X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
+| MI355X | AMD | 288GB | `lmsysorg/sglang:<ver>-rocm720-mi35x` |
+
+- **Image tag (`<ver>`)**: don't guess — ask the user for the tag the recipes ran on, or
+  default to `dev` (nightly). The same tag goes in `dockerImages` and benchmarks'
+  `sglang_version`; the engine falls back to `lmsysorg/sglang:dev` for any unmapped hw.
+- **TP sizing** (sanity-check recipes): `weight_GB / gpu_mem`, round up to a power of 2,
+  ~20–30% headroom. BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5. MoE → **total** weight, not
+  active params. FP4 is Blackwell-only (B200/B300/GB200/GB300). GB200/GB300 single-node
+  hosts are typically **4 GPUs** (TP=4 ceiling).
+- **Platform flags**: Blackwell may need `--attention-backend trtllm_mha`; AMD typically
+  needs `--attention-backend triton` + env `SGLANG_USE_AITER=1` /
+  `SGLANG_ROCM_FUSED_DECODE_MLA=0` (check AITER TP constraints, e.g. `heads_per_gpu % 16 == 0`).
+- **EP** (MoE): 8-GPU NVIDIA `--tp 8 --ep 8`; AMD `EP = TP`; small NVIDIA (TP≤4) omit
+  `--ep` unless benchmarked. (The template's AMD example cell shows these.)
+
+## Phase 2 — Instantiate the template
+
+1. **Copy** the three template files to their target paths (above). Note the two
+   vendor-folder conventions: under `configs/` the folder is the **HuggingFace org**
+   (`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`).
+2. **Replace every `__TOKEN__`**: `__MODEL_DISPLAY__`, `__MODEL_SLUG__`, `__HF_ORG__`,
+   `__HF_REPO__`, `__REASONING_PARSER__`, `__TOOLCALL_PARSER__`, `__ONE_LINER__`. Verify
+   none remain: `grep -rn '__[A-Z_]*__' <new files>`.
+3. **Prune** to what the model supports (delete, don't stub) — using
+   [references/authoring-reference.md](references/authoring-reference.md):
+   - `supportedHardware` + the EXAMPLE cells: keep your tested families; **delete the
+     `mi*` ids + AMD example cell if no AMD recipe**, etc. A GPU not in the shared catalog
+     (e.g. RTX PRO 6000) → declare it in `config.hardware` and add its id here.
+   - `playgroundFeatures` axes: remove the `megamoe` backend option + the
+     `megamoeQuant` block from the `moe` axis (non-Blackwell-MoE), delete `hisparse`
+     (non-DSA), `pdDisagg`/`router` (no PD), the `parsers` axis (no parsers), etc.
+   - `quantizations` / `variants`: drop what the model doesn't ship; collapse `variants`
+     to single `default` if there's no variant axis (then drop the `variant` half of
+     `modelNames`/`defaultAccuracy` keys).
+4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell;
+   set `verified: true` only on tested combos), and `modelNames` with real HF slugs,
+   `dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never
+   a guessed release; key by `hw`, or `hw|quant` when one quant on a shared GPU needs its own
+   image), `multiNodeHints` only for fabric-specific hw (e.g. gb200).
+
+### Site-wiring (do all three)
+
+- **`docs_new/docs.json`** — add the page under Cookbook → `<category>` → `<Vendor>`, at
+  the **top** of that vendor's `pages` (root-relative, no `.mdx`:
+  `cookbook/<category>/<Vendor>/<Model>`). New vendor group → insert in the section's
+  local ordering.
+- **NEW-tag hygiene** — the new page keeps `tag: NEW` (from the template). Scan the
+  vendor dir for existing NEW and strip it from siblings; verify ≤1:
+  `grep -rn 'tag: NEW' docs_new/cookbook/<category>/<Vendor>/` → at most one result. (Scan
+  files; don't assume the first `docs.json` entry holds NEW.)
+- **Homepage card** — `docs_new/cookbook/<category>/intro.mdx`: if the org already has a
+  `<Card>`, update only its `href` (keep `img`). If the org is **new**, add a `<Card>`
+  (title = nav-group name; keep card order aligned with `docs.json`) **and create its logo**:
+  ask the user for the brand logo, then generate the conforming **icon-only 940×525 RGBA
+  transparent** PNG → `docs_new/cards/logos/<org-slug>.png` per
+  [references/vendor-logo.md](references/vendor-logo.md) (track with `git add -f` — `*.png`
+  is gitignored repo-wide). Never invent or copy a logo.
+
+## Phase 3 — Validate
+
+```bash
+cd docs_new
+mint validate        # frontmatter, missing nav entries, MDX/JSX errors
+mint broken-links
+mint dev             # visual smoke test at http://localhost:3000/cookbook/<category>/<Vendor>/<Model>
+```
+
+Spot-check: cells render sensible commands; URL-hash nav persists across reload; the
+Playground inherits the Deploy selection live; each axis toggle produces the expected
+diff; Docker mode wraps in `docker run` with the right image; multi-node cells emit the
+hints + `--nnodes N`; cURL resolves the model name; the NEW badge shows on the new page
+and is gone from same-vendor siblings; the homepage card points to the new model.
+
+## Phase 4 — Interactive testing
+
+The user deploys each cell, runs the benches, and pastes results; you fill the data:
+- mark each tested `cells[]` entry `verified: true` (absent = yellow/unverified badge);
+- fill the `<model>-benchmarks.jsx` entries (one per cell `match`) with measured
+  speed/accuracy + the `sglang_version` the user reports (don't invent one — the template's
+  `0.0.0` is a deliberate TODO); set model-level `defaultAccuracy` per variant. Leave a
+  cell's entry as a bare `match` stub if it has no numbers yet (the card shows "pending").
+
+## Phase 5 — Prose & config tips
+
+**Read [references/mintlify-authoring.md](references/mintlify-authoring.md) first** (it
+carries the parser-output-shape / thinking-mode / Output-Example / no-hardcoded-sampling
+rules + the Mintlify forbidden-syntax list). Then rewrite the MDX prose from the HF card +
+user notes: §1 Model Introduction (description, links, params, license, variants table),
+§2 Configuration Tips (hw-specific tuning, caveats), §3 Advanced Usage (Reasoning /
+Tool-Calling / HiCache — keep only what applies; match the reasoning example to the
+parser's output shape; each runnable block gets an `**Output Example:**`).
+
+## Phase 6 — Review
+
+```
+/cookbook-review-pr <PR number>
+```
+
+## Git workflow
+
+Always branch — never commit to main directly.
+
+```bash
+git checkout -b add-<model>-cookbook
+git add docs_new/src/snippets/configs/<hf-org>/<slug>.jsx \
+        docs_new/src/snippets/configs/<hf-org>/<slug>-benchmarks.jsx \
+        docs_new/cookbook/<category>/<Vendor>/<Model>.mdx \
+        docs_new/docs.json docs_new/cookbook/<category>/intro.mdx
+git commit -m "Add <Display-Name> cookbook"
+git push -u origin add-<model>-cookbook
+gh pr create --title "Add <Display-Name> cookbook" --body "..."
+```

.claude/skills/cookbook-add-model/references/authoring-reference.md

@@ -0,0 +1,246 @@
+# Cookbook config reference (fields · cells · playground · MDX)
+
+Loaded on demand by the `cookbook-add-model` skill. This is the field-by-field
+contract for when the clone needs more than a rename. The two engine files are
+the canonical specs — read their headers first:
+
+- [`_deployment.jsx`](../../../../docs_new/src/snippets/_deployment.jsx) — the 5-dim matrix widget; lists every config field.
+- [`_playground.jsx`](../../../../docs_new/src/snippets/_playground.jsx) — the diff-based override widget; lists the `playgroundFeatures` axes + the `AXIS_HANDLERS` interface.
+
+Engine extension (adding a new playground axis) lives in [engine-axis.md](engine-axis.md).
+
+---
+
+## 2.1 Create the config file
+
+**Path**: `docs_new/src/snippets/configs/<vendor>/<model>.jsx`. The vendor folder is
+the HuggingFace org (`deepseek-ai`, `Qwen`, `moonshotai`, ...); the file
+name is a short hyphenated model id (`deepseek-v4`, `qwen3.5`, ...).
+
+**Shape**: must be a single `export const config = { ... }` literal. Do not
+use function calls, spreads, fragment refs, or IIFE — Mintlify re-evaluates
+this export at hydration time with module-level identifiers out of scope,
+and any non-literal value crashes with `ReferenceError`.
+
+**Required fields** (engine reads these — see the `_deployment.jsx` header for
+the full contract):
+
+| Field | Type | Purpose |
+|---|---|---|
+| `modelName` | string | Display label only. Not used for HF slug — see `modelNames`. |
+| `supportedHardware` | `string[]` | Which hw ids appear in the catalog. Subset of `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. Listing an id makes its button appear; if no cell uses it, the engine greys it out automatically. |
+| `hardware` | `{id,label,vram,vendor}[]` | Optional. GPUs the shared `HARDWARE_CATALOG` doesn't carry (workstation / desktop / future chips, e.g. RTX PRO 6000). The engine merges these into the catalog, so a model-specific GPU is config data — **no engine-catalog edit**. Also add the id to `supportedHardware`. |
+| `variants` | `{id, label, subtitle?}[]` | 2nd-dim option list. Use `default` / single-element if the model has no variant axis. |
+| `quantizations` | `{id, label}[]` | 3rd-dim option list. |
+| `strategies` | `{id, label}[]` | 4th-dim option list. Canonical ids: `low-latency` / `balanced` / `high-throughput` (never model-specific ids like `mtp`). **The count follows the page's operating points**: one recipe → a single `balanced`; two → `low-latency` + `high-throughput`; three → the full trio (the ideal). Tiers apply per (hw × variant × quant) combination — a single-recipe combination parks under its semantically honest tier (clear slant → that tier, e.g. DSv4's RTX 6000 → `low-latency`; no slant → `balanced`, e.g. Qwen3.5's Xeon); the page's list is the union and the engine greys unused chips per selection. Never invent a recipe just to fill chips. When two recipes differ by MTP / speculative decoding, the assignment is deterministic: spec ON → `low-latency`, spec OFF → `high-throughput` (at saturation the draft+verify overhead outweighs the speedup — same reason DSv4's high-throughput recipes disable MTP). The recurring markers in the other direction: dp-attention ON (MLA-attention models) and EP / DP+EP ON (MoE models) → `high-throughput`. |
+| `nodesOptions` | `{id, label}[]` | 5th-dim option list. The `id` MUST be `single` or `multi-N` — the engine parses N from the id for `--nnodes`. |
+| `cells` | `{match, verified?, env, flags}[]` | One per supported (hw × variant × quant × strategy × nodes) combination. See §2.2. |
+| `modelNames` | `{[key]: string}` | HF slug lookup. Keys are either `hw\|variant\|quant` (most specific) or `variant\|quant` (fallback). |
+| `placeholders` | `{[key]: {target, label, default?}}` | `{{KEY}}` interpolation map for command + curl. `target` is `'command'` or `'curl'`. Editable through the Env modal. |
+| `curl` | string | cURL template. Uses `{{MODEL_NAME}}` + placeholder keys. |
+
+**Optional fields**:
+
+| Field | Type | Purpose |
+|---|---|---|
+| `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. |
+| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. |
+| `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. |
+| `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. Optional; the button only appears when this AND `benchmarks` are present. |
+| `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `accuracyLabels` (below) + `benchmarkCommands.accuracy`. |
+| `accuracyLabels` | `[key, label, unit][]` | The eval set rendered in the benchmark card and the "⚡ Reproduce" modal — **the engine ships no default**, every config declares its own (e.g. DSv4: GPQA/AIME25/GSM8K; Qwen3.5: GSM8K/MMMU). Required whenever the benchmarks carry accuracy data; without it the accuracy rows silently don't render. Every key used in `benchmarks[].accuracy`, `defaultAccuracy`, and `benchmarkCommands.accuracy` must appear here. |
+| `latencyPercentile` | `"Mean" \| "P50"` | Optional, **temporary**; the percentile the benchmark TTFT/TPOT values are. **Default `"P50"`** — the card renders `TTFT (<pct>)` / `TPOT (<pct>)`. Set `"Mean"` only for legacy data recorded as Mean (being re-measured to P50). `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU** = `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. |
+| `github` | `{owner?, repo?, issueTemplate?, cookbookModel?}` | Overrides for the "Submit verified cell" CTA in the playground. Defaults: `sgl-project/sglang` + `3-playground-verified-cell.yml` + `"deepseek-ai/deepseek-v4"`. Set `cookbookModel` to the model's HF id (`<hf-org>/<model-slug>`); it prefills the issue template's free-form `model` input when the issue opens. **Don't prune this block** — without it the engine falls back to `deepseek-ai/deepseek-v4` and submissions from your page get mislabeled. |
+
+## 2.2 Author the 5-dim matrix (`cells[]`)
+
+Each cell describes one verified (or auto-estimated) launch recipe.
+
+```js
+{
+  match: { hw: "b200", variant: "flash", quant: "fp4",
+           strategy: "low-latency", nodes: "single" },
+  verified: true,                  // green "Verified" badge; absence = yellow
+  env: [
+    "SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024",
+  ],
+  flags: [
+    "--trust-remote-code",
+    "--model-path {{MODEL_NAME}}", // {{MODEL_NAME}} resolves from modelNames
+    "--tp 4",
+    "--moe-runner-backend flashinfer_mxfp4",
+    "--host {{HOST_IP}}",
+    "--port {{PORT}}",
+  ],
+},
+```
+
+**Rules**:
+
+- `match` MUST contain exactly the 5 keys: `hw`, `variant`, `quant`,
+  `strategy`, `nodes`. The engine looks up cells by tuple equality.
+- `env` and `flags` are FLAT literals. The engine does NOT expand
+  fragments, aliases, or templates — it consumes them verbatim
+  (only `{{PLACEHOLDER}}` substitutions happen at render time).
+- DO NOT include `--nnodes` / `--node-rank` / `--dist-init-addr` in
+  `cell.flags` for multi-node cells. The renderer injects them
+  automatically from `match.nodes` (`multi-N` → N nodes).
+- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
+  `{{PORT}}` placeholders so users can override through the Env modal.
+- Order flags as: `--model-path` first (after any `--trust-remote-code`),
+  then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
+  flags, then tuning knobs, with `--host` / `--port` last. The playground
+  engine assumes this ordering when inserting overrides (its anchors target
+  `--model-path` / `--tp` / etc., and inserts before the `--host` tail).
+- Accuracy-degrading flags don't belong in cells by default: a cell's
+  output quality should be exactly what its quantization chip declares.
+  Runtime quant below the checkpoint's precision (e.g. MegaMoE **W4A4** —
+  DSv4 gates it behind the Playground's `megamoeQuant` opt-in) and lossy
+  KV-cache dtypes (`--kv-cache-dtype fp8_e4m3` over a higher-precision-KV
+  checkpoint) default to Playground opt-ins or §2-tips material. If the
+  model's recipe genuinely needs one in a cell, **flag it to the user and
+  get explicit confirmation** — never ship it silently. (Migrations are the
+  sanctioned exception: a flag baked into the legacy recipe's default
+  command keeps verbatim — see the migrate skill.)
+
+**Cells are denormalized on purpose** — common flags repeat across cells.
+This makes each cell self-contained and easy to verify. When sweeping a
+common change, edit every cell.
+
+**Avoid premature cells**: only add a cell for a (hw × variant × quant ×
+strategy × nodes) combination if you have a recipe that has been tested or
+at least sanity-checked. The engine greys out un-listed combinations
+automatically.
+
+## 2.3 Configure `playgroundFeatures` (optional)
+
+The Playground is **opt-out, not opt-in**: every cookbook ships the general
+axes by default — `attention` (TP/CP/DP-Attn), `moe` (backend + EP, for MoE
+models), `parsers`, `speculative`, `pdDisagg`, `hicache` — then adds
+model-specific axes (e.g. MegaMoE for DeepSeek-V4) and deletes ONLY the axes
+this model genuinely cannot use (e.g. `hisparse` on non-DSA models, `moe` on a
+pure-dense model). Knobs that don't apply to a subset of variants/hw get
+`disable` + `disableReason`, not removal. Recognised axis keys and their
+schemas (full reference in the `_playground.jsx` header):
+
+| Axis key | Widget | Use when |
+|---|---|---|
+| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
+| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
+| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
+| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
+| `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a separate companion block and retargets the cURL modal to `router.port` (clients hit the router, not the role servers). |
+| `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. |
+| `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. |
+| `flagSelects` | A config-declared **list** of single-selects, each `{ id, title, stripPrefixes, options }` (option = `{ id, label, flags?, hide?, disable?, disableReason? }`); a flagless option is the "none"/accuracy-safe choice | A titled single-select that picks one value of a flag family the other axes don't model — e.g. KV-cache dtype (`--kv-cache-dtype`), mamba scheduler strategy (`--mamba-scheduler-strategy`). Generic: no engine change to add another. |
+
+**Per-chip constraints**: any chip entry in any axis can be wrapped with
+`hide` / `disable` constraint objects:
+
+```js
+{ value: 16, disable: { nodes: ["single"] },
+  disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." }
+```
+
+- `hide` — chip omitted entirely (use for hard impossibilities).
+- `disable` — chip greyed out with tooltip (soft warning).
+- Constraints are AND across keys, OR within each key's array.
+- Bare `disabled: true` / `disable: true` is a static always-disabled form
+  (used for "Coming soon" chips).
+
+## 2.4 Create the MDX page
+
+Path: `docs_new/cookbook/<category>/<Vendor>/<Model>.mdx`. Import both widgets and
+the per-model config, render them inside the relevant sections:
+
+```mdx
+## Deployment
+
+import { Deployment } from "/src/snippets/_deployment.jsx";
+import { config }     from "/src/snippets/configs/<vendor>/<model>.jsx";
+import { benchmarks } from "/src/snippets/configs/<vendor>/<model>-benchmarks.jsx";
+
+{/* Install is a PREREQUISITE — keep it compact + collapsed at the top of the
+    Deploy section (NOT a numbered section). Tabs mirror the widget's
+    Python/Docker toggle. */}
+<a id="install" />
+<Accordion title="Install SGLang">
+  <Tabs>
+    <Tab title="Python (pip / uv)">…pip / uv install…</Tab>
+    <Tab title="Docker">…docker pull + a `docker run … sglang serve` example…</Tab>
+  </Tabs>
+</Accordion>
+
+<Deployment config={config} benchmarks={benchmarks} />
+
+[model-specific tuning notes, caveats, links]
+
+## Playground
+
+import { Playground } from "/src/snippets/_playground.jsx";
+
+<Playground config={config} />
+```
+
+**Heading slugs matter** — the two widgets cross-link by scrolling to each
+other's section id (Mintlify auto-slugs headings: lowercase, spaces →
+hyphens, punctuation dropped). The engines look up:
+
+- the **Deploy** panel by id `deployment` (falls back to `deploy`) — used
+  by the Playground's "↑ Switch base" button and by deep-link scroll-on-
+  load. Title the section `## Deployment` (or `## Deploy`).
+- the **Playground** by id `playground` — used by `_deployment.jsx`'s
+  "Open the Playground →" link. Title the section `## Playground`.
+
+Avoid numbered headings like `## 3. Model Deployment` (slug
+`3-model-deployment`) for these two sections — the cross-links would break.
+The Playground reads the Deploy selection live via the URL hash + the
+`sglang-deploy-sel` custom event, so the two can live in different parts
+of the page.
+
+The `benchmarks` prop is **optional**. It points at a sibling
+`<model>-benchmarks.jsx` file (one entry per cell, keyed by the same
+`match` tuple) that renders an accuracy + speed sub-card under the command
+box; omit the import and the prop if the cookbook has no measured numbers
+yet. See the `_deployment.jsx` header and `deepseek-v4-benchmarks.jsx` for
+the full speed/accuracy schema.
+
+Each entry's `sglang_version` must be a **reproducible anchor** — a release
+tag/version (`v0.5.9`), a commit hash, or (for **Day-0 support**, before the
+enabling PR merges or a release is cut) a specific PR (`PR #27944`) or commit
+you can `gh pr checkout` / `git checkout`. Never a moving ref like `"main"` /
+`"main (2026-06-11)"` (not reproducible). A spec-decoding model whose cell
+carries `--speculative-algorithm` but no `--max-running-requests` auto-shows
+an amber Deploy + Playground callout (SGLang otherwise caps it at 48) — it is
+flag-driven, so no per-page prose is needed.
+
+To let users *reproduce* those numbers, add a `benchmarkCommands` block to
+the config (§2.1, next to `curl`). When present alongside `benchmarks`, the
+benchmark card grows a **"⚡ Reproduce"** button that opens a modal listing
+the runnable commands for the current cell — one `bench_serving` command for
+Speed (with concurrency chips that rewrite `--max-concurrency`) plus an
+Accuracy command with a chip per eval. No separate benchmark section needed.
+
+---
+
+## Pitfalls (authoring)
+
+**Stale URL hash hydration** — If a user shares a link from an old cell
+catalog and the hash names an impossible combination, `_deployment.jsx`'s
+`validateSelection` snaps to the nearest real cell. The Playground reads
+the hash too — make sure cookbook removals don't leave dangling shared
+links pointing at hardware/quant combos that no longer exist.
+
+**Mintlify constraints** — Module-level statements are stripped. The config
+MUST be a single `export const config = { ... }` literal — no function calls,
+spreads, fragment refs, or IIFE (Mintlify re-evaluates the export at hydration
+with module-level identifiers out of scope; any non-literal crashes with
+`ReferenceError`). In MDX, capitalized JSX tags get rebound — use the built-in
+Mintlify components (`<Accordion>`, `<Tabs>`, `<Card>`, ...) as documented.
+Avoid `!(x in y)` anywhere (Mintlify's AST walker crashes on it) — use
+`obj.key === undefined`.
+
+**Per-cell denormalization** — Cells repeat common flags on purpose. Do
+not factor them into a shared `commonFlags` array — Mintlify will fail
+to inline the reference. If you need to sweep a flag across cells, do it
+with a global find-replace in the config file.

.claude/skills/cookbook-add-model/references/engine-axis.md

@@ -0,0 +1,274 @@
+# Engine extension: add a new playground feature axis
+
+Loaded on demand by the `cookbook-add-model` skill. **Rare** — adding a model
+cookbook is data-only and never needs this. The current 7 built-in axes
+(`attention`, `moe`, `parsers`, `speculative`, `pdDisagg`, `hicache`,
+`hisparse`) already cover the SGLang feature surface most cookbooks need.
+
+**Model-specific features are config DATA, not engine code.** The axis
+handlers read options / flags / env / gating straight from
+`config.playgroundFeatures`, so a model-specific feature is added as data on
+an existing axis with NO engine edit — MegaMoE W4A4 is not its own axis, it's
+config data on `moe` (a `megamoe` backend option + a `megamoeQuant`
+sub-select). Reach for this file ONLY when a feature's *shape* — its
+title + the flag family it strips + its option/state model — is something no
+existing axis can express.
+
+**When you do extend, build a GENERIC primitive, never a model-named handler.**
+The right unit is a reusable shape (e.g. "a titled single-select that strips a
+configurable flag family and splices the picked option's flags" — exactly the
+`speculative` handler's shape, minus its hardcoded title + `--speculative-*`
+strip list). Parameterize title, strip-prefixes, and options from config so
+the next model of that shape is pure config. Do NOT add a `kvcache` /
+`<model-feature>` handler that hardcodes one model's flag — that's the
+model-specific-code-in-the-engine anti-pattern this architecture exists to
+avoid. (Precedent: Nemotron3's "KV Cache DType" and Qwen3's mamba-cache select
+are the SAME single-select shape → one generic primitive serves both, then
+both are config.)
+
+**A new axis is backward-compatible — zero churn on existing configs.** The
+runtime is opt-in per key: the apply/render loop does `const fc =
+pgFeatures[axisId]; if (!fc) continue;`, so any config that doesn't declare
+the key never sees the axis. And a model-specific axis does NOT join the
+opt-out "general axes ship on every cookbook" set (that set is an authoring
+convention for NEW configs, not a runtime default) — so review-pr won't flag
+existing pages for lacking it, and you never touch a merged config. Only the
+models that expose the control declare it. Touches `_playground.jsx` only.
+
+For the per-model config/cells/MDX reference see [authoring-reference.md](authoring-reference.md).
+
+---
+
+## 3.1 Decide
+
+Before touching the engine, confirm:
+
+- The feature cannot be expressed as data on an existing axis (a new MoE
+  backend belongs in `moe.backend.options`; a new parser in `parsers.items`;
+  a new spec preset in `speculative.options`). This is the common case —
+  most "new features" are new options, not new shapes.
+- The *shape* is genuinely new (state model + strip pattern), AND you are
+  adding it as a GENERIC config-parameterized axis (title / strip-prefixes /
+  options all from config), not a one-model handler. If you'd hardcode a
+  specific flag like `--kv-cache-dtype`, stop — generalize the shape instead.
+- The shape has a clean strip-prefix → emit-flag pattern.
+
+If unsure, add it as data first (in one cookbook's config under an
+existing axis) before promoting it to a built-in axis.
+
+## 3.2 Pick the axis id and state shape
+
+The axis id is the key in both `config.playgroundFeatures` and the
+internal `deltas` object. Use camelCase, descriptive but short:
+`mambaCache`, `attentionBackend`, `kvCacheDtype`.
+
+The state shape is whatever `initState` returns. Common shapes:
+
+- Single-select: a string sentinel (e.g. `"disabled"` / `"current"` / an
+  option id).
+- Multi-toggle: `{[itemId]: bool}`.
+- Sub-knobs: `{[knobId]: value | null}`.
+- Compound (axis with its own internal sub-state, like PD-Disagg's
+  `{mode, ibDevice}`): a plain object.
+
+Pick ONE "inherit base" sentinel and document it in the handler comment.
+
+## 3.3 Implement the handler
+
+Add one entry to `AXIS_HANDLERS` in `_playground.jsx`.
+The handler owns everything: state init, apply (strip+insert), hidden-revert,
+AND the JSX render. Engine main loop iterates `AXIS_HANDLERS` and calls each
+method by name — adding a new axis is genuinely a one-place change.
+
+Template:
+
+```js
+// ---- Axis: <Title> ----------------------------------------------------
+// <one-paragraph description of what this axis controls and why it
+//  exists. Mention the SGLang feature it wraps and the strip/insert
+//  policy.>
+<axisId>: {
+  initState: (fc) => /* initial state value */,
+
+  // Called when base cell changes. Return new value if the picked option
+  // is now hidden by a constraint; otherwise return value unchanged.
+  // Disabled picks are intentionally NOT auto-reverted (soft warning).
+  revertHidden: (value, fc, base, h) => {
+    // ... return value or a new value
+    return value;
+  },
+
+  // Pure function. Receives the current (flags, env) and returns the next
+  // (flags, env). Do NOT mutate inputs. The `value` argument is whatever
+  // initState returned. The `fc` argument is config.playgroundFeatures[axisId].
+  // The `sel` argument is the current base cell selection. The `h`
+  // argument is the helpers bundle (strip/insert primitives + anchors).
+  apply: ({ flags, env, value, fc, sel, h, derived }) => {
+    if (/* value is the inherit-base sentinel */) return { flags, env };
+    flags = h.stripFlagsByFirstToken(flags, [/* prefixes this axis owns */]);
+    if (/* an option is picked */) {
+      flags = h.insertAfter(flags, h.ANCHOR_NEAR_<X>, [/* new flags */]);
+      // or: flags = h.insertBeforeTail(flags, [/* new flags */]);
+      // if the axis mutates env:
+      // env = h.stripEnvByPrefix(env, fc.stripEnv || []);
+      // env = [...env, /* additional env vars */];
+    }
+    return { flags, env };
+  },
+
+  // Optional: read the base cell's flag array back into the same shape
+  // initState/apply use. Render shows this as the default selection
+  // (dropdown option or checked chip) when the state slot is the inherit
+  // sentinel — so the user sees the cell's actual --tp / MoE backend /
+  // spec preset instead of an opaque "Auto." When derive returns a real
+  // value, the inherit-sentinel option is hidden from the control. Apply
+  // also receives the derived
+  // value (as `derived`) and may use it as a no-op shortcut when the
+  // user's pick matches base. Skip when your axis owns flags that never
+  // appear in base cells (PD-Disagg / HiCache).
+  // deriveFromBase: (cell, fc, h) => ({ ... }) | null,
+
+  // Optional: hints for the renderer. Currently only pdDisagg uses this
+  // to report its role banner. Omit if not needed.
+  // getRenderHints: (value, fc) => ({ pdMode: ... }) | null,
+
+  // Returns the axis card JSX. The outer div MUST have key={axisId} so
+  // React can track it in the engine's map loop. Return null for
+  // axis-level gating (e.g. HiSparse when the live PD mode isn't `decode`). Lay out as a single
+  // compact horizontal row: title on the left, fields after.
+  render: ({ axisId, value, setValue, fc, base, s, h, renderChip, renderSelect, derived }) => {
+    if (/* axis-level gating fails */) return null;
+    return (
+      <div key={axisId} style={s.card}>
+        <div style={s.compactRow}>
+          <span style={s.axisTitle}>Axis Title</span>
+          {/* For multi-option fields, use renderSelect(...) — the default.
+              For on/off toggles or single-select chip groups, use
+              renderChip instead (see "Control choice" in the conventions
+              below). Read state from `value`; write via `setValue(next)`
+              (replaces the whole axis slot). */}
+          <span style={s.field}>
+            <span style={s.fieldLabel}>Field</span>
+            {renderSelect(value.slot, fc.entries, (v) =>
+              setValue({ ...value, slot: v }), base)}
+          </span>
+        </div>
+      </div>
+    );
+  },
+},
+```
+
+**Important conventions**:
+
+- Insert the entry in the position you want it rendered. `AXIS_HANDLERS`
+  is iterated in insertion order for both render and apply.
+- Use `h.ANCHOR_NEAR_*` constants for insertion. Add a new anchor to the
+  helpers bundle if your axis needs to land somewhere new in the flag
+  block.
+- Use lowercase HTML JSX tags only. Capitalized tags get rebound by
+  Mintlify.
+- Inside `render`, read state via `value` (the slice for this axis).
+  Write state via `setValue(next)` (replaces the whole slice). For
+  compound axes, do `setValue({ ...value, [k]: nextK })`.
+- Layout: one `s.compactRow` per axis card, `s.axisTitle` for the
+  leading label, one `s.field` per (label + input) pair.
+- Control choice — `renderSelect` vs `renderChip`:
+  - `renderSelect(current, entries, onPick, base, labelFor?, opts?)` is
+    the **default** compact control (a `<select>` dropdown). It filters
+    hidden chips and disables greyed-out ones internally — no per-chip
+    `evaluateChip` loop needed in the render body. Most axes use it
+    (attention, moe, pdDisagg, hisparse, hicache). Pass
+    `{ hideValues: [<sentinel>] }` when your `deriveFromBase` resolved to
+    a real value, so the inherit-sentinel ("Auto" / "Inherited" /
+    "current") doesn't clutter the dropdown.
+  - `renderChip(label, current, value, onPick, { disabled?, disabledReason? })`
+    renders a **button** instead of a dropdown row. Use it for a chip
+    group when you want the options laid out as buttons. It serves two
+    shapes:
+    - **Multi-toggle** (Parsers) — one independent on/off chip per item;
+      `current` is that item's effective bool, `value` is `true`, so the
+      chip is "checked" when the item is on.
+    - **Single-select** (Speculative) — a radio-style group; pass the
+      group's effective value as `current` and each option's id as
+      `value`, so exactly one chip is checked (`current === value`).
+    Chip groups own their visibility/disable filtering: loop
+    `h.evaluateChip(opt, base)` in the render body, skip `c.hidden`,
+    filter the inherit-sentinel yourself when `deriveFromBase` resolved
+    to a real value, and forward `c.disabled` / `c.disableReason` into
+    `renderChip`'s opts (this is what surfaces a disabled chip's tooltip,
+    e.g. a "Coming soon" entry).
+- Selected chips use the same terracotta (`#D45D44`) as the Deploy
+  panel's selected button, so both widgets read as one
+  visual system. Don't introduce a per-axis accent color.
+- Default-from-base: if your axis can be read out of base cells'
+  flags, implement `deriveFromBase` and have your render show the
+  derived value when state is the sentinel (e.g.
+  `const eff = value.tp !== null ? value.tp : (derived && derived.tp)`).
+  This is what makes a fresh playground load show the user's actual
+  recipe instead of "auto." Flag-parsing helpers on `h`:
+  `parseIntFlag`, `hasFlag`, `findFlagArg`.
+- **Avoid the `in` operator wrapped in unary** (`!(x in y)`). Mintlify's
+  AST walker crashes on it (`TypeError: this[e] is not a function`). Use
+  `obj.key === undefined` or `obj.id !== undefined` instead. Bare
+  `if (key in obj)` (no surrounding `!`) is fine.
+
+## 3.4 Document the per-cookbook schema
+
+Edit the file header in `_playground.jsx` to add your new axis to the
+"Recognised keys" list, with a one-line description of its schema.
+Optionally add a paragraph below explaining its strip/insert policy.
+
+Update the §2.3 axis table in [authoring-reference.md](authoring-reference.md) to list the new axis.
+
+## 3.5 Migrate cookbooks that need it
+
+For each cookbook that should expose this axis, add a
+`playgroundFeatures.<axisId>` entry to its config. Verify the chip group
+renders, options apply correctly, and the diff matches expectations.
+
+---
+
+## Pitfalls (engine work)
+
+**Insertion anchor misses** — `insertAfter` falls back to right-after
+`--model-path` if none of its anchor prefixes are present. If your axis
+emits flags that should land somewhere specific, include the most likely
+anchor prefixes in your call. Order doesn't matter (set semantics).
+
+**Conditional strips** — Some axes strip ONLY when overridden
+(`attention.tp`, `moe.backend` (incl. the MegaMoE quant env), `speculative`).
+Others strip
+UNCONDITIONALLY whenever declared (`parsers`, `pdDisagg`, `hicache`). The
+header comment in `AXIS_HANDLERS` documents which policy each axis uses;
+follow the same pattern when adding a new axis. If unsure, prefer
+conditional strip — it preserves base behavior when the user does not
+opt in.
+
+**Closure of `AXIS_HANDLERS`** — Inside a handler method, you can
+reference `AXIS_HANDLERS.<otherAxis>` for cross-handler calls (no built-in
+axis currently needs this, but it works because `AXIS_HANDLERS` is in
+lexical scope). Do NOT use this for general logic — it tightly couples
+handlers. Reserve it for one handler's helpers shared between its own
+`render` and `revertHidden`.
+
+---
+
+## Review checklist for a new-axis PR
+
+- [ ] `AXIS_HANDLERS` is the ONLY place that mentions the new axis id
+      (apart from per-cookbook config). No `if (axisId === '<new>')`
+      branches anywhere in the engine.
+- [ ] `initState` is deterministic and idempotent (does not depend on
+      the base cell).
+- [ ] `apply` is pure — does not mutate inputs.
+- [ ] `revertHidden` returns the same reference when nothing changed
+      (avoids unnecessary re-renders).
+- [ ] `render` returns `null` when axis-level gating fails (whole card
+      hidden) — does not render an empty placeholder.
+- [ ] `render` sets `key={axisId}` on its outer element.
+- [ ] No `!(x in y)` patterns introduced (Mintlify AST walker crashes).
+- [ ] File header lists the new axis in "Recognised keys".
+- [ ] The §2.3 table in `authoring-reference.md` lists the new axis.
+- [ ] One existing cookbook config is updated to consume the new axis,
+      and visual verification shows the diff is correct.

.claude/skills/cookbook-add-model/references/mintlify-authoring.md

@@ -0,0 +1,116 @@
+# MDX authoring rules (Mintlify) + invocation-example patterns
+
+Loaded on demand by the `cookbook-add-model` skill (Phase 5, writing the page prose).
+These are model-agnostic Mintlify hygiene rules — the most common review findings.
+The cookbook is **Mintlify**, not Docusaurus.
+
+## Mintlify syntax
+
+**Allowed components**: `<Card>`, `<CardGroup>`, `<Note>`, `<Tip>`, `<Warning>`,
+`<Info>`, `<Accordion>`, `<AccordionGroup>`, `<Steps>`, `<Step>`, `<Tabs>`, `<Tab>`,
+`<CodeGroup>`, `<Frame>`, `<Icon>`.
+
+**Forbidden** (flag every occurrence):
+- Docusaurus admonitions (`:::note` / `:::warning` / …) — use `<Note>` / `<Warning>`.
+- `@site/...` / `@theme/...` imports — use absolute `/src/snippets/...`.
+- GitHub alert blocks (`> [!NOTE]`, `> [!WARNING]`).
+- **Markdown pipe tables** on new pages — use JSX `<table>` (see below).
+- Inline `<details>` / `<summary>` — use `<Accordion>`.
+- Unknown / non-Mintlify components.
+- `<CardGroup>` / `<Card>` on individual model pages — those are for category `intro.mdx` only.
+
+**Code fences**: always labeled — ` ```python Example `, ` ```bash Command `,
+` ```shell Command `, ` ```text Output `. When nesting a fenced block inside another,
+the **outer** fence uses four backticks.
+
+**Internal links**: root-relative, no extension (`/cookbook/<category>/<Vendor>/<Model>`);
+`docs.sglang.io` is canonical. Flag `.md`/`.mdx` extensions and `../`-relative page links
+in body prose. (Existing cookbook pages do use `../../../docs/...` for cross-links into
+the non-cookbook docs tree — that's the established exception; don't introduce new ones.)
+
+## JSX tables (required for all tables on new pages)
+
+```jsx
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Col</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr><td style={{padding: "9px 12px"}}>cell</td></tr>
+  </tbody>
+</table>
+```
+
+Alternate column background colors (`rgba(255,255,255,0.02)` / `0.05`) for readability;
+adjust `<colgroup>` widths for 3- or 5-column tables. The DeepSeek-V4 page §1 variants
+table is a live reference.
+
+## Invocation-example patterns (§3 Advanced Usage)
+
+- **Reasoning-parser output shape must match the example**:
+  - *Separate-field* parsers (most qwen/glm, `kimi_k2`, `deepseek-v4`): thinking lands
+    in `message.reasoning_content`, answer in `message.content` — print both.
+  - *Inline-tag* parsers (e.g. `minimax-append-think`): thinking is wrapped in
+    `<think>...</think>` **inside** `message.content` — the client parses the tags; for
+    streaming, buffer and split on the markers.
+  Pick the pattern from the model card / SGLang docs for that specific parser.
+- **Hybrid reasoning models**: show both thinking-on (default) and thinking-off
+  (`extra_body={"chat_template_kwargs": {"thinking": False}}` or `enable_thinking: False`).
+- **Tool-call follow-up on thinking models**: the final assistant turn may put text in
+  `reasoning_content` instead of (or with) `content` — print both so the output isn't a
+  misleading `None`.
+- **§3 commands and outputs are collapsible (required)**: every runnable example
+  lives in an `<Accordion title="… (Python)">` and its **real** server output
+  (verbatim, not paraphrased) in an immediately following
+  `<Accordion title="Example Output">` — match the DeepSeek-V4 §3 pattern. No
+  inline `**Output Example:**` headings / bare blocks. `Pending update...` is
+  acceptable only with the user's explicit acknowledgement.
+- **Do not hardcode sampling params** (`temperature`, `top_p`) in sample code — SGLang
+  uses `generation_config.json` defaults. Listing "Recommended generation" in §1 is fine.
+- Format raw API objects (`ChatCompletionMessage(...)`) into readable Reasoning /
+  Content / Tool Calls blocks.
+
+## Frontmatter
+
+- **Top-level `description:`** is the canonical field — it sets the page's SEO meta
+  description (`og:`/`twitter:description` fall back to it) AND renders as the visible
+  **subtitle** under the title, filling the header band before the first heading. Give every
+  page a one-line top-level `description` (a lede / value prop) — without it, a page that
+  opens straight into `## Deployment` shows an empty gap under the title (the title and
+  `## Deployment` are the same size, so they read as two bare headings). Do **not** put the
+  description inside a `metatags` block — `metatags` is for other/custom tags, and
+  `metatags.description` is redundant with (and non-canonical vs) the top-level field.
+  - **Write it for SEO** (it doubles as the search-result snippet): front-load the exact
+    model name + intent — e.g. `Deploy <Model> with SGLang — …` — aim for ~150–160 chars, and
+    pack secondary keywords (variants + sizes, `Mixture-of-Experts` / architecture, target
+    GPUs). Phrase it as a value prop, not a generic "`<Model>` is a … model" intro.
+- **No `mode:` on a model page.** Leave it unset so Mintlify renders the default layout
+  *with* the right-hand "On this page" table of contents — every model page relies on this.
+  `mode: wide` drops that ToC; it's only for the category `intro.mdx` card-grid landing pages
+  (which have no ToC by design). The Deploy/Playground panels don't need the extra width —
+  they self-cap at `maxWidth: 900px` and center, which fits the default column fine. (Symptom
+  of a stray `mode: wide`: the page loses its right-hand ToC while its siblings keep theirs.)
+- Frontmatter MUST be the first thing in the file — no comment or blank line before the
+  opening `---`.
+
+## Commands & ports
+
+- **Deploy/launch** commands use `sglang serve --model-path …` — never
+  `python -m sglang.launch_server` / `python3 -m sglang.launch_server` (deprecated).
+- **Benchmark workload** commands use `python3 -m sglang.bench_serving …` (never bare
+  `python -m`); built-in accuracy scripts use `python3 benchmark/...`.
+- Port **30000** everywhere on a page — launch, curl, client `base_url`, and bench must
+  agree. Keep one canonical deploy command (the Deploy widget) and don't re-paste launch
+  commands across sections; the documented command must match the widget's output for the
+  same selection (doc ↔ config parity).
+
+## Factual hygiene
+
+- License must match the actual HuggingFace license (don't copy from another model).
+- HF URLs resolve to a real model; Docker images from `lmsysorg/sglang`.
+- No Google-Drive image links (they don't render); host images in the repo.
+- Shell placeholders are `export VAR=<value>`, not `export VAR=${VAR}` (a bash no-op).
+- `tag: NEW` is sparing — at most one per `<category>/<Vendor>/` dir (the newest); strip
+  it from siblings when adding a new NEW page.

.claude/skills/cookbook-add-model/references/vendor-logo.md

@@ -0,0 +1,76 @@
+# Vendor card logo (new brand only)
+
+A new vendor/brand in the cookbook landing grid needs a card logo at
+`docs_new/cards/logos/<org-slug>.png`. **Ask the user for the brand's logo, then generate
+the conforming PNG** — never invent, copy, or hallucinate one, and never ship a
+non-conforming file. Reference: PR #27400 (added `tencent.png` + `poolside.png`).
+
+If the org already has a card/logo, do nothing here — only update the `<Card href>`.
+
+## Spec (match the existing logos exactly)
+
+| Property | Value |
+|---|---|
+| Path | `docs_new/cards/logos/<org-slug>.png` — lowercase, matches the `img=` in the `<Card>` |
+| Canvas | **940 × 525** px |
+| Mode | **RGBA**, fully **transparent** background |
+| Content | **Icon-only** — the brand glyph/mark (the "swirl"), **no wordmark text** |
+| Placement | centered; glyph ≈ 0.33 × width and ≈ 0.50 × height of the canvas |
+
+Why icon-only + transparent: cards render on both light and dark backgrounds, so a baked-in
+(usually black) wordmark vanishes on dark. `deepseek.png` / `ernie.png` are 940×525 RGBA
+exemplars — eyeball your output against them.
+
+## 1. Get the source
+
+Ask the user for the brand logo (SVG preferred → crisp + already transparent; else a high-res
+transparent PNG, or a link to the official press/brand asset). Prefer an **icon-only** source;
+if they only have a full lockup, ask them to crop the glyph, or crop it yourself.
+
+If the user **pasted** an image inline, it may not be on disk — recover the base64 `image`
+block from the session transcript (`~/.claude/projects/<slug>/*.jsonl`) and decode it to a file.
+
+## 2. Generate (Pillow)
+
+There's no system Pillow — use a venv:
+
+```bash
+python3 -m venv /tmp/logo-venv && /tmp/logo-venv/bin/pip install -q Pillow
+```
+
+```python
+from PIL import Image
+src = Image.open("SOURCE").convert("RGBA")        # icon-only, already transparent
+W, H = 940, 525
+target_h = round(H * 0.50)                         # glyph ≈ half the canvas height
+scale = target_h / src.height
+glyph = src.resize((round(src.width * scale), target_h), Image.LANCZOS)
+canvas = Image.new("RGBA", (W, H), (0, 0, 0, 0))   # transparent
+canvas.paste(glyph, ((W - glyph.width) // 2, (H - glyph.height) // 2), glyph)
+canvas.save("docs_new/cards/logos/<org-slug>.png")
+```
+
+Notes:
+- **Wordmark present?** Crop to the glyph first (or ask the user for an icon-only asset). Don't
+  ship text in the logo.
+- **Solid background?** Don't auto-strip it (risky) — ask the user for a transparent source.
+- **SVG source?** Rasterize at high res first (`cairosvg` / `rsvg-convert`), then run the above.
+- If the glyph is much wider than tall, cap by width instead (≈ 0.33 × W) so it doesn't overflow.
+
+## 3. Verify
+
+```bash
+sips -g pixelWidth -g pixelHeight -g hasAlpha docs_new/cards/logos/<org-slug>.png
+# → pixelWidth: 940   pixelHeight: 525   hasAlpha: yes
+```
+
+## 4. Wire + track + validate
+
+```bash
+# Card in the landing grid (keep card order aligned with the docs.json nav order):
+#   <Card title="<NavGroup>" mode="card"
+#         href="/cookbook/<category>/<Vendor>/<Model>"
+#         img="/cards/logos/<org-slug>.png" />
+git add -f docs_new/cards/logos/<org-slug>.png      # root .gitignore ignores *.png repo-wide
+cd docs_new && mint validate && mint broken-links    # confirms the card href + img resolve
+```

.claude/skills/cookbook-add-model/templates/benchmarks.jsx.tmpl

@@ -0,0 +1,32 @@
+// TEMPLATE — instantiate via the cookbook-add-model skill. NOT a live cookbook.
+// Copy to docs_new/src/snippets/configs/<hf-org>/<model-slug>-benchmarks.jsx and
+// fill measured numbers — OR delete this file entirely if you have none yet (the
+// MDX simply omits the `benchmarks` import/prop).
+//
+// One entry per cell `match` tuple (same 5 keys as config cells). The card stays
+// "pending" until an entry has a non-null speed metric or accuracy. Speed shape:
+//   speed: [{ workload: {dataset, isl, osl, max_concurrency}, ttft_ms, tpot_ms,
+//             tokens_per_sec_per_gpu }, ...]
+//   - ttft_ms/tpot_ms are P50 (median); set config.latencyPercentile ("P50" default, or "Mean").
+//   - tokens_per_sec_per_gpu = total (in+out) tok/s/GPU
+//     (= output tok/s ÷ GPUs × (isl+osl)/osl). interactivity is derived = 1000/TPOT (tokens/s/user).
+// Per-cell `accuracy: { <key>: <pct> }` overrides the config's defaultAccuracy.
+
+export const benchmarks = [
+  // EXAMPLE — one filled entry showing the shape; replace numbers, add one per cell.
+  {
+    match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
+    sglang_version: "0.0.0", // TODO: ASK the user for the sglang version these numbers were measured on — don't invent one
+    speed: [
+      { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 1 },
+        ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: null },
+      { workload: { dataset: "random", isl: 8192, osl: 1024, max_concurrency: 16 },
+        ttft_ms: null, tpot_ms: null, tokens_per_sec_per_gpu: null },
+    ],
+  },
+  // Bare-match stubs (no data yet) are fine — the card shows "pending" for these.
+  { match: { hw: "h200",   variant: "default", quant: "fp8",  strategy: "balanced",        nodes: "single"  } },
+  { match: { hw: "h100",   variant: "default", quant: "fp4",  strategy: "high-throughput", nodes: "single"  } },
+  { match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "balanced",        nodes: "single"  } },
+  { match: { hw: "b200",   variant: "default", quant: "fp4",  strategy: "high-throughput", nodes: "multi-2" } },
+];

.claude/skills/cookbook-add-model/templates/config.jsx.tmpl

@@ -0,0 +1,407 @@
+// TEMPLATE — instantiate via the cookbook-add-model skill. NOT a live cookbook.
+// Copy to docs_new/src/snippets/configs/<hf-org>/<model-slug>.jsx, then:
+//   1. replace every __TOKEN__,
+//   2. fill cells[] with your verified recipes (the examples below show the shape),
+//   3. DELETE the hardware / playground axes / quantizations your model lacks.
+//
+// Instantiation tokens (skill fills these; distinct from the engine's runtime
+// {{PLACEHOLDER}} which MUST survive verbatim into the output):
+//   __MODEL_DISPLAY__   display name, e.g. "DeepSeek-V4"
+//   __MODEL_SLUG__      file slug, e.g. "deepseek-v4"
+//   __HF_ORG__          HuggingFace org, e.g. "deepseek-ai"
+//   __HF_REPO__         HuggingFace repo, e.g. "DeepSeek-V4-Flash"
+//   __REASONING_PARSER__  e.g. "deepseek-v4"  (delete the parsers axis if none)
+//   __TOOLCALL_PARSER__   e.g. "deepseekv4"   (delete the parsers axis if none)
+//
+// Mintlify: single `export const config = {...}` literal — no spreads/calls/IIFE,
+// no `!(x in y)`. Cells are denormalized: no --nnodes/--node-rank/--dist-init-addr/
+// --host/--port literals (the engine injects them).
+
+export const config = {
+  modelName: "__MODEL_DISPLAY__",
+
+  // List ONLY hardware you ship a cell for; unlisted ids auto-grey-out. The full
+  // catalog is below — delete the families your model doesn't support (e.g. drop
+  // every `mi*` if there's no AMD recipe).
+  supportedHardware: [
+    "h100", "h200", "b200", "b300", "gb200", "gb300",
+    "mi300x", "mi325x", "mi350x", "mi355x",
+  ],
+
+  // OPTIONAL — declare GPUs the shared HARDWARE_CATALOG (in _deployment.jsx) doesn't
+  // carry (workstation / desktop / future chips). The engine merges these in, so a
+  // model-specific GPU is config data, never an engine-catalog edit. Add the id to
+  // supportedHardware above too. Delete if you only use catalog GPUs.
+  // hardware: [
+  //   { id: "rtx6000", label: "RTX PRO 6000", vram: "96GB", vendor: "nvidia" },
+  // ],
+
+  // 2nd dim. Single-element `default` if the model has no variant axis; else list
+  // real variants (e.g. {id:"flash",...},{id:"pro",...}) and key modelNames/
+  // defaultAccuracy by them.
+  variants: [
+    { id: "default", label: "Default" },
+  ],
+  // 3rd dim. Keep only what your model ships (BF16 / FP8 / FP4 / …).
+  quantizations: [
+    { id: "bf16", label: "BF16" },
+    { id: "fp8",  label: "FP8"  },
+    { id: "fp4",  label: "FP4"  },
+  ],
+  // 4th dim. The count follows the model's operating points: 1 recipe → a
+  // single "balanced"; 2 → low-latency + high-throughput; 3 → the full trio
+  // (the ideal). Per-combination: a single-recipe combination (e.g. a CPU
+  // platform) parks under its semantically honest tier — no slant → balanced;
+  // the page's list is the union and the engine greys unused chips. Never
+  // invent a recipe just to fill chips.
+  strategies: [
+    { id: "low-latency",     label: "Low-Latency"     },
+    { id: "balanced",        label: "Balanced"        },
+    { id: "high-throughput", label: "High-Throughput" },
+  ],
+  // `multi-N` id carries the node count for `--nnodes N`.
+  nodesOptions: [
+    { id: "single",  label: "Single Node" },
+    { id: "multi-2", label: "Multi-Nodes" },
+  ],
+
+  // HF slug lookup. Key by `variant|quant` (or `hw|variant|quant` for a per-hw
+  // repackaging, e.g. an FP8 conversion only valid on one platform).
+  modelNames: {
+    "default|bf16": "__HF_ORG__/__HF_REPO__",
+    "default|fp8":  "__HF_ORG__/__HF_REPO__",
+    "default|fp4":  "__HF_ORG__/__HF_REPO__",
+  },
+
+  placeholders: {
+    HOST_IP:   { target: "command", label: "Bind host",        default: "0.0.0.0"        },
+    PORT:      { target: "command", label: "Bind port",        default: "30000"          },
+    NODE0_IP:  { target: "command", label: "Head node IP",     default: "<node0-ip>"     },
+    NODE_RANK: { target: "command", label: "This node rank",   default: "<node-rank>"    },
+    HF_TOKEN:  { target: "command", label: "HF token (Docker)", default: "<your-hf-token>" },
+    CURL_HOST: { target: "curl",    label: "Server host",      default: "localhost"      },
+    CURL_PORT: { target: "curl",    label: "Server port",      default: "30000"          },
+  },
+
+  curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
+-H 'Content-Type: application/json' \\
+-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
+
+  // OPTIONAL — powers the benchmark card's "⚡ Reproduce" modal. Delete the whole
+  // block (and the benchmarks file) if you have no measured numbers yet.
+  benchmarkCommands: {
+    speed:
+`python3 -m sglang.bench_serving \\
+  --backend sglang \\
+  --host {{CURL_HOST}} --port {{CURL_PORT}} \\
+  --model {{MODEL_NAME}} \\
+  --dataset-name {{DATASET}} \\
+  --random-input-len {{ISL}} --random-output-len {{OSL}} \\
+  --num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}}`,
+    // One entry per accuracy field. A value is a string, OR a {[variant]: string}
+    // object when the command differs per variant. Keys must match ACCURACY_LABELS
+    // in _deployment.jsx + the per-cell/defaultAccuracy keys.
+    accuracy: {
+      gsm8k_pct:
+`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval
+sgl-eval run gsm8k \\
+  --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\
+  --num-threads 32`,
+    },
+    // {{NUM_PROMPTS}} fallback per concurrency (else max(c*2, 200)).
+    numPromptsByConc: { 1: 8, 16: 32, 64: 128, 256: 512, 1024: 2048, 4096: 4096 },
+  },
+
+  // OPTIONAL — per-variant accuracy applied to EVERY cell of a variant (hardware-
+  // independent, e.g. GPQA/AIME). Per-cell `accuracy` overrides. Keys must match
+  // the effective accuracy labels + benchmarkCommands.accuracy. Delete if no numbers yet.
+  defaultAccuracy: {
+    default: { gsm8k_pct: null },
+  },
+
+  // The eval set rendered in the benchmark card + "⚡ Reproduce" — the engine
+  // ships NO default; required whenever the benchmarks carry accuracy data
+  // (without it the accuracy rows silently don't render). [key, label, unit]
+  // tuples; keys must match benchmarks[].accuracy + defaultAccuracy +
+  // benchmarkCommands.accuracy. Delete only if there are no accuracy numbers.
+  accuracyLabels: [
+    ["gsm8k_pct", "GSM8K", "%"],
+  ],
+
+  // OPTIONAL — `# ...` hint lines prepended to multi-node commands, ONLY for hw
+  // whose fabric needs manual NIC env (e.g. gb200 NVL72/MNNVL). NOT every multi-N
+  // hw needs this — standard-IB DeepEP / Marlin multi-node don't. Delete if unused.
+  multiNodeHints: {
+    gb200: [
+      "The following env vars may be needed depending on your cluster:",
+      "  GLOO_SOCKET_IFNAME=<your-nic>",
+      "  NVSHMEM_ENABLE_NIC_PE_MAPPING=1",
+      "  NVSHMEM_HCA_LIST=<your-hca-list>",
+    ],
+  },
+
+  // Image for `docker run` framing, keyed by `hw` (or `hw|quant`, resolved first, when one
+  // quant on a shared GPU needs its own image — e.g. an FP4 dev build while FP8/BF16 use the
+  // release tag). ASK the user which sglang build the recipes ran on; don't guess a supporting
+  // release. Default below is :dev (nightly) — replace the tag with the user's release if they
+  // give one. NVIDIA share one image; AMD uses ROCm tags. GB200/GB300/B300 may need a
+  // `-cu130` (CUDA 13) tag — confirm per release.
+  dockerImages: {
+    h100:  "lmsysorg/sglang:dev",
+    h200:  "lmsysorg/sglang:dev",
+    b200:  "lmsysorg/sglang:dev",
+    b300:  "lmsysorg/sglang:dev",
+    gb200: "lmsysorg/sglang:dev",
+    gb300: "lmsysorg/sglang:dev",
+    mi300x: "lmsysorg/sglang:dev-rocm720-mi30x",
+    mi325x: "lmsysorg/sglang:dev-rocm720-mi30x",
+    mi350x: "lmsysorg/sglang:dev-rocm720-mi35x",
+    mi355x: "lmsysorg/sglang:dev-rocm720-mi35x",
+  },
+
+  // Prefills the issue template's free-form `model` field on "Submit verified cell".
+  // Use the HF id (`<hf-org>/<model-slug>`). Do NOT delete this block when pruning —
+  // without it the engine falls back to "deepseek-ai/deepseek-v4" and mislabels submissions.
+  github: {
+    cookbookModel: "__HF_ORG__/__MODEL_SLUG__",
+  },
+
+  // Opt-OUT per axis: general axes (attention / moe-for-MoE / parsers /
+  // speculative / pdDisagg / hicache) ship on every cookbook — DELETE only the
+  // axes this model genuinely cannot use (e.g. hisparse on non-DSA models, moe
+  // on pure-dense models); prefer disable+disableReason for variant/hw subsets.
+  playgroundFeatures: {
+
+    // ----- Card: "Attention Parallelism" -----  KEEP if the model exposes TP/CP/DP
+    // knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
+    attention: {
+      knobs: [
+        { id: "tp", label: "TP", values: [
+          null, 1, 2, 4, 8,
+          { value: 16, disable: { nodes: ["single"] },
+            disableReason: "TP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
+        ]},
+        { id: "cp",     label: "CP", values: [null, 1, 2, 4] },
+        { id: "dpAttn", label: "DP-Attention",
+          values: [
+            null, false, 1, 2, 4, 8,
+            { value: 16, disable: { nodes: ["single"] },
+              disableReason: "DP-Attention=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
+          ],
+          labels: { "auto": "Auto", "false": "Off" } },
+      ],
+    },
+
+    // ----- Card: "MoE Parallelism" -----  KEEP if MoE + multiple `--moe-*-backend`
+    // choices. DELETE for dense models.
+    moe: {
+      backend: {
+        options: [
+          { id: null,               label: "Inherited" },
+          { id: "deepep",           label: "DeepEP",            flags: ["--moe-a2a-backend deepep"] },
+          // KEEP the MegaMoE option + the megamoeQuant block below ONLY for Blackwell
+          // MoE kernel-fusion models; DELETE both otherwise. requiresHw gates it to
+          // Blackwell (the engine hides it elsewhere); add excludesStrategy: [...] too
+          // for a strategy gate. Selecting MegaMoE reveals the Quantization sub-select.
+          { id: "megamoe",          label: "MegaMoE",           flags: ["--moe-a2a-backend megamoe"],
+            requiresHw: ["b200", "b300", "gb200", "gb300"] },
+          { id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)", flags: ["--moe-runner-backend flashinfer_mxfp4"] },
+          { id: "marlin",           label: "Marlin (W4A16)",    flags: ["--moe-runner-backend marlin"] },
+        ],
+      },
+      // MegaMoE quantization sub-select — shown only when backend === "megamoe".
+      // W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env.
+      // DELETE this block if there's no MegaMoE backend option above.
+      megamoeQuant: {
+        stripEnv: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK"],
+        options: [
+          { id: "w4a8", label: "W4A8",
+            env: ["SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320"] },
+          { id: "w4a4", label: "W4A4",
+            env: [
+              "SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320",
+              "SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1",
+              "SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1",
+            ] },
+        ],
+      },
+      ep: { label: "EP", values: [
+        null, 1, 2, 4, 8,
+        { value: 16, disable: { nodes: ["single"] },
+          disableReason: "EP=16 requires 16 ranks — switch the Deploy panel's Nodes to Multi-Nodes first." },
+      ]},
+    },
+
+    // ----- Card: "Parsers" -----  KEEP if the model has reasoning / tool-call
+    // parsers (set the slugs below). DELETE the axis if neither applies.
+    parsers: {
+      items: [
+        { id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser __REASONING_PARSER__" },
+        { id: "toolCall",  label: "Tool Call Parser", flag: "--tool-call-parser __TOOLCALL_PARSER__" },
+      ],
+    },
+
+    // ----- Card: "Speculative Decoding" -----  KEEP if the model has spec-decoding
+    // presets. Drop options the model doesn't support.
+    speculative: {
+      options: [
+        { id: "current", label: "Inherited from base" },
+        { id: "off",     label: "Off (greedy)" },
+        { id: "mtp",     label: "EAGLE / MTP",
+          flags: ["--speculative-algorithm EAGLE", "--speculative-num-steps 3",
+                  "--speculative-eagle-topk 1", "--speculative-num-draft-tokens 4"] },
+        { id: "ngram",   label: "NGRAM",
+          flags: ["--speculative-algorithm NGRAM",
+                  "--speculative-num-draft-tokens 16",
+                  "--speculative-ngram-max-bfs-breadth 10"],
+          disable: { dpAttnOn: [true] },
+          disableReason: "NGRAM is incompatible with DP-Attention. Turn DP-Attention off in the Attention card above to use NGRAM." },
+      ],
+    },
+
+    // ----- Card: "PD Disaggregation" -----  KEEP if the model supports prefill/
+    // decode disaggregation. Delete `router` if you have no router topology.
+    pdDisagg: {
+      modes: [
+        { id: "off",     label: "Off" },
+        { id: "prefill", label: "Prefill role" },
+        { id: "decode",  label: "Decode role" },
+      ],
+      transferBackends: [
+        { id: "mooncake", label: "Mooncake",
+          env: ["NCCL_MNNVL_ENABLE=1", "NCCL_CUMEM_ENABLE=1"],
+          envWhen: { hw: ["gb200", "gb300"] } },
+        { id: "nixl", label: "NiXL" },
+      ],
+      // `auto` is a sentinel (emits no --disaggregation-ib-device flag).
+      ibDevices: [{ id: "auto", label: "Auto" }, "mlx5_0", "mlx5_7"],
+      // Router fronting prefill + decode; substitute <prefill-host>/<decode-host>.
+      router: {
+        port: 8000,
+        command:
+`python3 -m sglang_router.launch_router \\
+  --pd-disaggregation \\
+  --prefill http://<prefill-host>:30000 \\
+  --decode http://<decode-host>:30001 \\
+  --host 0.0.0.0 --port 8000 \\
+  --disable-circuit-breaker \\
+  --health-check-interval-secs 999999`,
+      },
+    },
+
+    // ----- Card: "Hierarchical KV Cache" -----  KEEP if the model is large enough
+    // that hierarchical KV caching matters.
+    hicache: {
+      backends: [
+        { id: null,       label: "Auto" },
+        { id: "file",     label: "File" },
+        { id: "mooncake", label: "Mooncake" },
+        { id: "hf3fs",    label: "HF3FS" },
+        { id: "nixl",     label: "NiXL" },
+      ],
+      writePolicies: [
+        { id: "auto",                    label: "Auto" },
+        { id: "write_through",           label: "Write-through" },
+        { id: "write_back",              label: "Write-back" },
+        { id: "write_through_selective", label: "Write-through (selective)" },
+      ],
+    },
+
+    // ----- Card: "HiSparse" -----  KEEP only for DSA-style sparse-attention models
+    // (DeepSeek-V3.2/V4, GLM-5). Decode-only: shown when live PD-Disagg mode is `decode`.
+    hisparse: {
+      requiredFlags: ["--disable-radix-cache"],
+      config: { top_k: 2048, device_buffer_size: 6144 },
+      hostRatios: [
+        { id: 5,  label: "5 (~1TB host)" },
+        { id: 10, label: "10 (~2TB host)" },
+      ],
+      defaultHostRatio: 10,
+    },
+  },
+
+  // EXAMPLE cells — one per hardware family to show the shape. REPLACE each with
+  // your model's verified recipe, or DELETE families you don't support. `match`
+  // MUST have exactly the 5 keys; env/flags are flat literals.
+  // Accuracy-degrading flags (W4A4-style runtime quant, lossy --kv-cache-dtype)
+  // default to Playground/tips — putting one in a cell needs explicit user
+  // confirmation (authoring-reference §2.2).
+  cells: [
+    // ==== NVIDIA Blackwell + FP4 (single node) ====
+    {
+      match: { hw: "b200", variant: "default", quant: "fp4", strategy: "low-latency", nodes: "single" },
+      verified: true, // EXAMPLE — set false / replace with your verified recipe
+      env: [],
+      flags: [
+        "--trust-remote-code",
+        "--model-path {{MODEL_NAME}}",
+        "--tp 4",
+        "--moe-runner-backend flashinfer_mxfp4",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    // ==== NVIDIA Hopper + FP8 (single node, DP-attention + DeepEP) ====
+    {
+      match: { hw: "h200", variant: "default", quant: "fp8", strategy: "balanced", nodes: "single" },
+      verified: true, // EXAMPLE
+      env: ["SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256"],
+      flags: [
+        "--trust-remote-code",
+        "--model-path {{MODEL_NAME}}",
+        "--tp 4",
+        "--dp 4",
+        "--enable-dp-attention",
+        "--moe-a2a-backend deepep",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    // ==== NVIDIA Hopper + FP4 (single node, Marlin W4A16 — Hopper has no FP4 runner) ====
+    {
+      match: { hw: "h100", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "single" },
+      verified: true, // EXAMPLE
+      env: [],
+      flags: [
+        "--trust-remote-code",
+        "--model-path {{MODEL_NAME}}",
+        "--tp 8",
+        "--moe-runner-backend marlin",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    // ==== AMD + BF16 (single node) — Triton attention + AITER; EP == TP for MoE ====
+    {
+      match: { hw: "mi300x", variant: "default", quant: "bf16", strategy: "balanced", nodes: "single" },
+      verified: true, // EXAMPLE
+      env: ["SGLANG_USE_AITER=1", "SGLANG_ROCM_FUSED_DECODE_MLA=0"],
+      flags: [
+        "--trust-remote-code",
+        "--model-path {{MODEL_NAME}}",
+        "--tp 8",
+        "--ep 8",
+        "--attention-backend triton",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+    // ==== Multi-node example (2 nodes, TP=16) — engine injects --nnodes/--node-rank/
+    // --dist-init-addr from match.nodes; do NOT add them here. ====
+    {
+      match: { hw: "b200", variant: "default", quant: "fp4", strategy: "high-throughput", nodes: "multi-2" },
+      verified: true, // EXAMPLE
+      env: [],
+      flags: [
+        "--trust-remote-code",
+        "--model-path {{MODEL_NAME}}",
+        "--tp 16",
+        "--dp 16",
+        "--enable-dp-attention",
+        "--moe-a2a-backend deepep",
+        "--host {{HOST_IP}}",
+        "--port {{PORT}}",
+      ],
+    },
+  ],
+};

.claude/skills/cookbook-add-model/templates/page.mdx.tmpl

@@ -0,0 +1,155 @@
+---
+title: __MODEL_DISPLAY__
+description: "__ONE_LINER__"
+tag: NEW
+---
+
+{/* TEMPLATE — instantiate via the cookbook-add-model skill, then DELETE this banner.
+    (Frontmatter MUST stay the first thing in the file, so this note lives below it.)
+    Replace every __TOKEN__, fill the TODO prose, delete the §3 subsections your model
+    lacks. Tokens: __MODEL_DISPLAY__ __ONE_LINER__ __HF_ORG__ __MODEL_SLUG__ __HF_REPO__
+    __REASONING_PARSER__ __TOOLCALL_PARSER__.  MDX rules (JSX tables, labeled fences, no
+    Docusaurus/@site/GitHub-alert/pipe-tables):
+    .claude/skills/cookbook-add-model/references/mintlify-authoring.md */}
+
+## Deployment
+
+<a id="install" />
+
+<Accordion title="Install SGLang">
+
+For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
+
+<Tabs>
+
+<Tab title="Python (pip / uv)">
+
+```bash Command
+pip install --upgrade pip
+pip install uv
+uv pip install sglang
+```
+
+Then run the **Python** output of the command panel below in that environment.
+
+</Tab>
+
+<Tab title="Docker">
+
+```bash Command
+docker pull lmsysorg/sglang:latest
+```
+
+For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
+
+</Tab>
+
+</Tabs>
+
+</Accordion>
+
+Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
+
+- **Low-Latency** — fastest reply for a single user. Pick for chat.
+- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
+- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
+
+import { Deployment } from "/src/snippets/_deployment.jsx";
+import { config }     from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__.jsx";
+import { benchmarks } from "/src/snippets/configs/__HF_ORG__/__MODEL_SLUG__-benchmarks.jsx";
+
+<Deployment config={config} benchmarks={benchmarks} />
+
+## Playground
+
+The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
+
+import { Playground } from "/src/snippets/_playground.jsx";
+
+<Playground config={config} />
+
+## 1. Model Introduction
+
+{/* TODO: 1-2 paragraph intro from the HF card — what the model is, release date,
+    license, architecture highlights, context length. Keep it lean. */}
+**__MODEL_DISPLAY__** is __ONE_LINER__.
+
+{/* TODO: variants table (JSX, NOT a markdown pipe table). Drop the table if there's
+    a single variant and inline the HF link in the intro paragraph above instead. */}
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left",  padding: "10px 12px", fontWeight: 700}}>Variant</th>
+      <th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700}}>Total params</th>
+      <th style={{textAlign: "left",  padding: "10px 12px", fontWeight: 700}}>Use</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/__HF_ORG__/__HF_REPO__">__MODEL_DISPLAY__</a></strong></td>
+      <td style={{padding: "9px 12px", textAlign: "right"}}>TODO</td>
+      <td style={{padding: "9px 12px"}}>TODO</td>
+    </tr>
+  </tbody>
+</table>
+
+**Recommended generation:** {/* TODO e.g. `temperature=1.0`, `top_p=1.0` (informational; do NOT hardcode in sample code) */}
+
+**Resources:** [HuggingFace](https://huggingface.co/__HF_ORG__/__HF_REPO__).
+
+## 2. Configuration Tips
+
+{/* TODO: model/hardware-specific tuning notes, caveats, known issues. Delete if none. */}
+
+## 3. Advanced Usage
+
+{/* Keep only the subsections that apply. Commands and outputs in this section are
+    COLLAPSIBLE (required — match DeepSeek-V4 §3): each runnable example lives in an
+    <Accordion>, its REAL server output in a following <Accordion title="Example Output">. */}
+
+### 3.1 Reasoning
+
+Enable the `__REASONING_PARSER__` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer.
+
+{/* This example assumes a SEPARATE-FIELD parser (thinking → `reasoning_content`,
+    answer → `content`). If your parser emits inline `<think>...</think>` tags inside
+    `content`, parse the tags from `content` instead. */}
+
+<Accordion title="Reasoning Example (Python)">
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+resp = client.chat.completions.create(
+    model="__HF_ORG__/__HF_REPO__",
+    messages=[{"role": "user", "content": "What is 15% of 240?"}],
+    extra_body={"chat_template_kwargs": {"thinking": True}},
+)
+msg = resp.choices[0].message
+print("Reasoning:", getattr(msg, "reasoning_content", None))
+print("Answer:", msg.content)
+```
+
+</Accordion>
+
+<Accordion title="Example Output">
+
+```text Output
+TODO: paste real server output here.
+```
+
+</Accordion>
+
+### 3.2 Tool Calling
+
+Enable the `__TOOLCALL_PARSER__` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
+
+{/* TODO: tool-calling example in an <Accordion> + an <Accordion title="Example Output">.
+    On thinking-mode models the follow-up may put text in `reasoning_content`;
+    print both that and `content`. */}
+
+### 3.3 HiCache (Hierarchical KV Caching)
+
+{/* TODO: keep only if the model is large enough for hierarchical KV caching; link
+    the HiCache card in the Playground. Otherwise delete this subsection. */}

.claude/skills/cookbook-migrate-model/SKILL.md

@@ -0,0 +1,240 @@
+---
+name: cookbook-migrate-model
+description: Migrate a legacy-template SGLang cookbook page (monolithic per-model generator under docs_new/src/snippets/autoregressive/) onto the config-driven template (shared _deployment.jsx / _playground.jsx engines + per-model config). Use when asked to migrate, convert, or port an existing cookbook page — NOT for brand-new models (use cookbook-add-model for those). Run with /cookbook-migrate-model <Model page name, e.g. GLM-5.1>.
+---
+
+# Cookbook Migrate Model
+
+Convert one legacy cookbook page to the config-driven format, faithfully. The
+legacy page — its generator widget and its measured benchmark blocks — is the
+**single source of truth**. You are transcribing it into the new data model,
+not improving it.
+
+Reuses the `cookbook-add-model` skill's assets (read them on demand):
+- `../cookbook-add-model/templates/config.jsx.tmpl`, `page.mdx.tmpl`, `benchmarks.jsx.tmpl`
+- `../cookbook-add-model/references/authoring-reference.md` (config/cells/playground contract)
+- `../cookbook-add-model/references/mintlify-authoring.md` (MDX rules)
+
+Migration-specific references in this skill:
+- [references/dimension-mapping.md](references/dimension-mapping.md) — legacy-control → new-dimension mapping rules, command rewrite table, per-family strategy sets, and the Qwen3.5 pilot as a worked example (PR #27848).
+
+The round's per-model inventory (scope, batch order, quirks, measured-data
+survey) is tracked by the migration maintainer outside the repo — expect it in
+your dispatch prompt, or ask for it.
+
+## Hard rules (non-negotiable)
+
+1. **Never modernize.** Env vars, flags, TP values, docker tags, version strings
+   are copied verbatim from the legacy page — even when today's defaults differ
+   (e.g. `SGLANG_ENABLE_SPEC_V2=1` is now default; keep it anyway). The recipe
+   that was verified is the recipe as written. Allowed normalizations are ONLY
+   the five alias rewrites in dimension-mapping.md §2 (`launch_server`→`sglang
+   serve`, `--model`→`--model-path`, `--tp-size`→`--tp`, abbreviated
+   `--speculative-algo`→`--speculative-algorithm`,
+   `--expert-parallel-size`→`--ep`). **Accuracy-degrading flags**
+   (`--kv-cache-dtype fp8_e4m3`, W4A4-style runtime quant) follow a
+   deterministic rule — enforced in migration, no asking: offered as a
+   legacy **selectable option** → never select it (cells mirror the
+   accuracy-safe side), and the option **survives as a Playground axis** —
+   an existing one where it fits, else add one via a separate prior engine
+   PR (rule 4); a legacy choice never degrades to a tips mention. Baked
+   into the recipe's **default/unconditional command** → keep it verbatim
+   (the recipe was measured with it, and fp8 KV halves KV memory —
+   stripping could OOM it). See dimension-mapping.md §2 caveats.
+2. **Never invent versions or numbers.** Benchmark numbers only from the
+   legacy page's measured blocks, and a result migrates ONLY when its
+   `sglang_version` is a **reproducible anchor** — the bar is reproducibility,
+   not "must be a release":
+   - ✅ release tag/version (`v0.5.9` / `0.5.9`), commit hash, OR — for
+     **Day-0 support** (the enabling PR isn't merged and no release is cut
+     yet) — a specific **PR (`PR #27944`) or commit** you can `gh pr checkout`
+     / `git checkout <sha>`. Commit is most precise; a PR pin is fine for day-0.
+   - ❌ a moving ref — `"main branch"`, `"main (2026-06-11)"`, open-ended
+     `"0.5.8+"` — is NOT reproducible: **drop the WHOLE result (speed AND
+     accuracy)**, not just speed. Keep `benchmarkCommands` so ⚡Reproduce still
+     guides re-measurement against a pinned build.
+   **Never inherit cross-model numbers** — measurements the legacy page
+   attributes to a *different model* (e.g. a K2.6 page carrying K2.5-measured
+   speed) are dropped regardless of version. When kept, `sglang_version` is the
+   legacy page's string verbatim. Docker tags only the ones the legacy page
+   pinned (unmapped hw falls back to `:dev`).
+3. **Verified policy (strictest tier).** `verified: true` ONLY when (a) the
+   legacy page has concrete measured data for that exact 5-dim combo AND
+   (b) the cell's flags equal the deployment command used for that measurement
+   — modulo `{{HOST_IP}}`/`{{PORT}}`, the five alias rewrites, and **parser
+   flags**: `--reasoning-parser`/`--tool-call-parser` are stripped from every
+   cell (Playground-only feature; when the measured run had them on, say so in
+   the benchmarks file header). When the measured command diverges from the
+   generator default, **the verified cell follows the measured command**; the
+   generator default stays as the sibling strategy/cell or a tips note. Everything else is unverified (yellow) —
+   including combos that look memory-infeasible; keep them verbatim and list
+   them in the PR body for the re-verification track.
+4. **Engines are read-only.** `_deployment.jsx` / `_playground.jsx` must not
+   change in a migration PR. Model-specific features are config DATA consumed
+   by generic axis handlers (MegaMoE precedent), so they need NO engine change.
+   A **titled single-select that strips a flag family** — KV Cache DType
+   (`--kv-cache-dtype`), mamba (`--mamba-scheduler-strategy`), … — is already
+   covered by the merged generic **`flagSelects`** axis: declare it in the
+   config (a list of `{ id, title, stripPrefixes, options }`; see the Qwen3.5
+   mamba example), **no engine PR**. Only a genuinely new control *shape* that
+   `flagSelects` can't express would need a one-time generic primitive (never a
+   model-named handler) on a separate prior PR (engine-axis.md).
+5. **`github.cookbookModel` must be set** (`<hf-org>/<page-slug>`, e.g.
+   `qwen/qwen3.5`) and the block never pruned — without it Submit ↗ mislabels
+   as deepseek-v4. The issue template itself needs NO edits (free-form input).
+6. **Strategy tiers are signal-driven.** A cell goes under `low-latency` /
+   `high-throughput` ONLY on a signal present in the legacy source (an
+   explicit performance toggle, a named recipe, or prose stating the
+   operating point); **no signal → `balanced`**. Never derive a slant from
+   your own hardware intuition — re-tiering on measured evidence is the
+   hardware owner's follow-up PR, not part of a migration
+   (dimension-mapping.md §4).
+
+## Workflow (one model = one PR)
+
+### 1. Inventory the legacy assets
+- Read the legacy generator (`docs_new/src/snippets/autoregressive/<slug>-deployment.jsx`)
+  end-to-end: every option dimension (radio vs checkbox vs dynamic), every
+  gate/SUPPORT matrix, the full emitted command per reachable combo (env
+  prefixes, `# Error` pseudo-commands included).
+- Read the legacy MDX: §2 install docker tags → `dockerImages` (pinned, not
+  upgraded); §3.2 tips → new §2; §4 invocation examples → new §3 (keep real
+  Output Examples verbatim); §5 benchmark blocks → transcribe each measured
+  block: deploy command used, bench command (dataset/isl/osl/num-prompts/
+  concurrency), P50 (median) TTFT/TPOT, output tok/s, hardware, version string.
+- Inbound-anchor sweep: `grep -rn "<PageName>" docs_new/ --include='*.mdx'` —
+  find links/`#fragments` into this page (`mint broken-links` does NOT check
+  fragments). Fix referrers or add `<a id="old-anchor" />` shims in the same PR.
+- Check the maintainer-provided inventory notes for this model's known quirks —
+  but treat them (and the §4 family table) as a **survey snapshot**: re-verify
+  every dimension against the live legacy files before mapping. Pages keep
+  receiving updates (precedent: Kimi-K2.6's live generator has a speculative
+  toggle the 2026-06-10 survey notes lack).
+
+### 2. Design the 5-dim mapping
+Apply [references/dimension-mapping.md](references/dimension-mapping.md). Key
+decision — which legacy toggle becomes the `strategies` dimension: a toggle
+that **changes other parts of the command** (TP, mem) must (the Playground
+can't do coupled changes), and so does a toggle the legacy page itself labels
+with operating-point words — e.g. a `dpattention` radio whose options are
+subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
+even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
+flag add). Any other toggle that only adds/removes its own flags becomes a
+Playground axis with the flags baked into cells when the legacy default was
+ON — EXCEPT parsers:
+`--reasoning-parser`/`--tool-call-parser` are NEVER baked into cells, they are
+Playground-only (DSv4 convention). **Every legacy control survives as an
+interactive control** — a dimension or a Playground axis, never a tips-only
+mention — and a model-specific control is **config data, not engine code**
+(MegaMoE W4A4 is all DSv4 config on the existing `moe` axis). It's pure
+config whenever it fits an existing axis's data schema. A **titled
+single-select that strips a flag family** (Nemotron3's "KV Cache DType",
+mamba `--mamba-scheduler-strategy`, …) fits the merged generic **`flagSelects`**
+axis — so it too is config-only (declare a `flagSelects` list). Only a control
+whose *shape* `flagSelects` still can't express would need a ONE-TIME generic
+primitive (never a model-named handler) on a separate PRIOR engine PR, keeping
+the migration PR data-only (hard rule 4, engine-axis.md). The strategy count follows the page's
+operating points: **one recipe → a single `balanced`; two → `low-latency` +
+`high-throughput`; three → the full trio (the ideal)**. The tiers apply per
+(hw × variant × quant) combination — a single-recipe combination on a
+multi-strategy page parks under its semantically honest tier (no
+latency/throughput slant → `balanced`; the page's list is the union). When the
+legacy toggle is MTP / speculative decoding, the direction is a deterministic
+default — apply without asking: **MTP on → `low-latency`, MTP off →
+`high-throughput`** (reversed only with maintainer confirmation). Tier
+placement is signal-driven (hard rule 6). Never invent a recipe just to fill
+strategy chips (see dimension-mapping.md §4). Record the outcome as a
+**strategy mapping table** for the PR body — one row per group of
+combinations sharing the same legacy signal (e.g. "all GPU combos: MTP
+toggle → low-latency / high-throughput"; "xeon: (none) → balanced"), with a
+one-line rationale each; don't enumerate 60 identical rows. The table is what
+hardware owners sign off on at review.
+
+### 3. Generate the config (codegen, then audit)
+- For >~30 cells, port the legacy `generateCommand()` into a throwaway Node
+  script that enumerates combos and emits the `cells:[...]` literal (output
+  must stay a pure literal — Mintlify forbids runtime spreads/calls). Apply the
+  verified-cell override in the script. See the pilot scripts embedded in the
+  worked example of dimension-mapping.md §5.
+- **Independent equivalence audit (required):** extract the ORIGINAL generator
+  from git (`git show main:<path>` — NOT `HEAD:`, the migration branch deletes
+  the file, see dimension-mapping.md §5 item 7), stub React hooks, run it for
+  every combo, and diff token-by-token against the new cells. Expected deltas
+  only: the appended `--host {{HOST_IP}}`/`--port {{PORT}}`, the
+  engine-injected multi-node trio, the §2 alias rewrites (the entrypoint
+  rewrite doesn't appear in cell tokens — cells hold flags only; the audit
+  script normalizes it on the legacy side), and the intentional verified-cell
+  override. Paste the PASS count + the audit script in the PR body
+  (collapsed `<details>`).
+- Hand-author the non-cells fields per authoring-reference.md. Structural
+  self-checks: every cell resolves a `modelNames` key; no `--nnodes/--node-rank/
+  --dist-init-addr/--host/--port` literals; every `{{KEY}}` declared; every
+  `supportedHardware` id has ≥1 cell.
+
+### 4. Benchmarks file
+One entry per measured block only (cells without entries already render
+"pending" — bare `{match}` stubs are unnecessary). `tokens_per_sec_per_gpu` =
+total (in+out) tok/s/GPU = `output tok/s ÷ (tp × nnodes) ×
+(isl+osl)/osl` — stored directly (the card shows it as-is). TTFT/TPOT
+take the P50 (median) rows; set `config.latencyPercentile` (default `"P50"`; use
+`"Mean"` only for legacy Mean-recorded data — temporary, being migrated to P50).
+Put the workload's
+`num_prompts` into `workload`. **`config.accuracyLabels` is required whenever
+the benchmarks carry accuracy data** — the engine ships no default eval set
+(#27842), so missing labels means the accuracy rows silently don't render;
+extra context (sample counts, suites that don't fit) goes in the entry's
+`notes`. Zero-measured-data pages: skip the file and the `benchmarks` prop
+entirely, but keep `benchmarkCommands` so ⚡Reproduce still guides users.
+
+### 5. Rewrite the MDX
+From `page.mdx.tmpl`: keep the original `title` (nav identity), write a fresh
+SEO `description` (top-level — delete any legacy `metatags.description`), **no
+`tag: NEW`** (a migration is not a launch), **no `mode:`**. Install accordion
+carries the legacy install content + pinned images. Keep the template's
+DSv4-style strategy bullets — serving semantics first (single-user chat /
+typical multi-user / batch throughput), trimmed to the strategies the page
+ships, plus at most a one-line note on what each strategy changes on this
+model; do NOT rewrite them as toggle-/migration-centric explanations.
+Legacy §5 benchmark prose is deleted (numbers → benchmark card, commands →
+⚡Reproduce); legacy prose deploy commands are deleted (doc↔config parity —
+fold their unique flags into §2 tips). Invocation examples + real outputs carry
+over verbatim, but **wrapped in Accordions** — §3 commands and outputs are
+collapsible (required, DeepSeek-V4 pattern): code in an
+`<Accordion title="… (Python)">`, output in a following
+`<Accordion title="Example Output">`; legacy pages kept them inline.
+
+### 6. Delete the legacy generator
+Remove `docs_new/src/snippets/autoregressive/<slug>-deployment.jsx` and its
+import. `grep -rn "<slug>-deployment" docs_new/` must return nothing (config
+provenance comments must not name the deleted path). Site wiring needs **no
+changes**: docs.json path/title unchanged, vendor card + logo already exist.
+
+### 7. Validate
+- `grep -rn '__[A-Z_]*__'` on the new files (no template tokens).
+- `cd docs_new && mint validate && mint broken-links` (pre-existing breaks on
+  main are not yours — say so in the PR).
+- `mint dev` browser smoke: initial selection = the verified cell (first in
+  `cells[]`) with green badge; multi-node cells show the injected trio +
+  header; AMD cells show env prefixes; Docker mode wraps with the pinned image
+  and passes cell env as `--env`; condition-hidden combos grey out; benchmark
+  card values; NO parser flags in any Deploy command; Playground parser
+  toggles ADD the parser flags (green additions) while spec toggles strike
+  the baked spec flags (red); Submit ↗ prefills this model. **Probe pitfall:** drive at most
+  ONE programmatic click per evaluation and wait for React to settle —
+  multiple clicks in one synchronous script batch and read stale DOM.
+- Token-level audit from step 3 passes.
+
+### 8. PR + review
+One PR per model. PR body: migration framing, verified policy applied, the
+strategy mapping table (step 2), the audit PASS count + script, any
+inherited-infeasible combos flagged for re-verification. Then run `/cookbook-review-pr <N>` and fix findings.
+FYI: docs previews only build for in-repo (`sgl-project/sglang`) branches —
+a fork-headed PR is perfectly fine but renders no preview; a maintainer can
+re-push the branch in-repo if a preview is wanted for review.
+
+### 9. Keep this skill current
+Any new convention, engine behavior, or pitfall you discover while migrating
+(naming decisions, audit-script gotchas, review-rule conflicts, …) MUST be fed
+back into this skill — same PR if it's skill-file-only, or an immediate
+follow-up commit on the skill's branch/PR. The next agent runs on what's
+written here, not on your session's context.

.claude/skills/cookbook-migrate-model/references/dimension-mapping.md

@@ -0,0 +1,248 @@
+# Legacy → config-driven dimension mapping
+
+Loaded on demand by the `cookbook-migrate-model` skill. How to translate a
+legacy generator's option space into the 5-dim matrix + Playground axes.
+Field schemas live in `../../cookbook-add-model/references/authoring-reference.md`;
+this file is about the *mapping decisions*.
+
+## 1. Legacy control → new home
+
+| Legacy control | New home | Rule |
+|---|---|---|
+| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
+| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
+| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
+| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
+| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
+| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
+| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
+
+## 2. Command rewrite table (the ONLY allowed normalizations)
+
+| Legacy | New |
+|---|---|
+| `python(3) -m sglang.launch_server` | (engine emits `sglang serve`; cells hold flags only) |
+| `--model X` / `--model-path X` | `--model-path {{MODEL_NAME}}` + `modelNames` key |
+| `--tp-size N` | `--tp N` |
+| `--speculative-algo X` (abbreviated) | `--speculative-algorithm X` — the Playground spec axis strips/derives by the full first token only; an abbreviated alias would survive toggles and double up |
+| `--speculative-algorithm NEXTN` | `--speculative-algorithm EAGLE` — **NEXTN is an alias of EAGLE** (same algorithm). Normalize cells + presets to EAGLE; never expose both NEXTN and EAGLE as separate `speculative` presets (they'd be duplicate chips). Keep a one-line "the bench reported NEXTN, an alias of EAGLE" provenance note where the measured command used it. |
+| `--expert-parallel-size N` | `--ep N` — the Playground EP knob recognizes/strips only `--ep`; the long form would survive toggles and double up |
+| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
+| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
+| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
+| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
+
+Caveats discovered in the pilot:
+- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
+  `--expert-parallel-size N` to `--ep N` (alias, see table above) so the knob
+  can recognize/strip it.
+- `multiNodeHints` only for hw whose fabric needs manual NIC env (gb200-class);
+  standard-IB H100 multi-node needs none.
+- `dockerImages`: only the tags the legacy page pinned. CPU/Xeon stays unmapped
+  (`:dev` fallback) with a "install from source" tip.
+- **Accuracy-degrading flags** (`--kv-cache-dtype fp8_e4m3`, W4A4-style
+  runtime quant) — deterministic rule, enforced in migration without
+  asking:
+  - offered as a legacy **selectable option/toggle** → never select it;
+    cells mirror the accuracy-safe side (even if the legacy default was the
+    lossy side). The option itself **must survive as a Playground control**
+    — the user's choice may not degrade to a tips mention. Express it as
+    config data on the fitting axis (DSv4 gates W4A4 behind `megamoeQuant`;
+    a single-select like Nemotron3-Ultra's "KV Cache DType" radio
+    None/fp8_e4m3/bf16 rides the merged generic **`flagSelects`** axis — declare
+    a `flagSelects` block, config-only, no engine PR);
+  - baked into the recipe's **unconditional/default command** → keep it
+    verbatim. The legacy measurements ran with it, and fp8 KV halves KV
+    memory — stripping could OOM the recipe. Expect this pattern: legacy
+    AMD recipes routinely append `--kv-cache-dtype fp8_e4m3` ("for memory
+    efficiency"), and GLM-5's NVFP4 path ships it too — all keep.
+
+  (Only migration gets this auto-keep — faithfulness wins here. On new
+  pages the same flags are flag-and-confirm with the maintainer:
+  authoring-reference §2.2 / review checklist.)
+
+## 2b. Playground axes: opt-out, not opt-in
+
+The legacy page's silence about a feature does NOT mean the axis is dropped.
+Every cookbook ships the **general axes** by default — `attention`
+(TP/CP/DP-Attn), `moe` (backend + EP) for MoE models, `parsers`,
+`speculative`, `pdDisagg`, `hicache` — then adds model-specific axes, and
+deletes ONLY axes the model genuinely cannot use (`hisparse` is DSA-only;
+MegaMoE is DeepSeek-V4 Blackwell-only). Knobs meaningless for a subset of
+variants/hw get `disable` + `disableReason` (per-chip constraints), not
+removal — e.g. MoE backend/EP greyed out on dense variants.
+
+`speculative` presets must include every algorithm that actually appears on
+the page (otherwise a stripped cell's baseline can't be re-applied) — but
+**collapse aliases**: NEXTN is an alias of EAGLE (§2 rewrite table), so a page
+benchmarked with NEXTN ships a single `eagle` preset, not both. (Pilot history:
+Qwen3.5 once shipped both; corrected to EAGLE-only.)
+
+**MTP `--max-running-requests` hint (engine, automatic):** when a cell's
+command turns speculative decoding on (`--speculative-algorithm` present)
+without `--max-running-requests`, the Deploy panel + Playground auto-render an
+amber callout (SGLang otherwise caps it at 48). It is FLAG-driven, not
+strategy-driven — nothing to author per page; do NOT duplicate it in §2 prose.
+
+The `parsers` axis is **add-only**: `--reasoning-parser` /
+`--tool-call-parser` are never part of any Deployment cell (see §1) — the
+axis adds them on top of the base command, so toggling a parser renders a
+green addition, never a strikethrough.
+
+## 3. Verified policy mechanics
+
+- Green requires measured data + flag equality with the measured command (see
+  SKILL.md hard rule 3). Order `cells[]` so the verified flagship cell is
+  **first** — `cells[0]` is the page's initial selection.
+- When the measured command and the generator default disagree (Qwen3.5: bench
+  ran `NEXTN` + `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, generator emitted `EAGLE` +
+  fusion flags), the verified cell mirrors the measurement; the generator
+  default lives on as the not-verified sibling cells. Offer BOTH as Playground
+  `speculative` presets and explain the split in §2 tips.
+- `config.accuracyLabels` is REQUIRED whenever benchmarks carry accuracy data —
+  the engine ships no default eval set (#27842); without it the accuracy rows
+  silently don't render. `defaultAccuracy` paints every *entry-bearing* cell of
+  a variant — under the strict policy prefer per-entry `accuracy` on the
+  measured cell only.
+
+## 4. Per-family strategy sets (survey sketches — re-derive from the live page)
+
+The family table below was sketched from the 2026-06-10 survey at PAGE level.
+At migration time **re-derive it from the live generator**: pages drift
+(precedent: Kimi-K2.6's live page has a speculative toggle the survey notes
+lack), and the per-combination rule means gated/hidden toggles — typically on
+Xeon, AMD, or a single-recipe quant like NVFP4 — produce `balanced` combos the
+page-level sketch doesn't show.
+
+**Strategy-set rule — the count follows the page's operating points** (ids
+always from the DeepSeek-V4 vocabulary, never model-specific ids like
+`mtp`/`no-mtp`):
+
+- **1 operating point** (a single recipe, no performance toggle) → a single
+  **`balanced`** strategy. Never invent a second recipe just to fill chips.
+- **2 operating points** → **`low-latency` + `high-throughput`**. When the
+  legacy toggle is MTP / speculative decoding, the mapping is a
+  **deterministic default — apply it without asking**: MTP on →
+  `low-latency`, MTP off → `high-throughput`. (Why it's near-certain:
+  speculative decoding cuts per-token latency at low concurrency, but at
+  saturation the draft+verify overhead costs more than it saves — DSv4's
+  high-throughput recipes disable MTP for the same reason.) Other toggles
+  map by the same serving semantics — the two recurring **high-throughput
+  markers** are **dp-attention ON** (MLA-attention models) and **EP / DP+EP
+  ON** (MoE models): both shard work across ranks for saturated throughput
+  at some per-request latency cost. These directions apply to the toggle
+  CHOSEN as the strategy dimension (§1); a flag-only spec toggle riding
+  alongside a named operating-point toggle stays a Playground axis and bakes
+  per its legacy default — GLM-5.1's spec defaults ON, so its flags bake
+  into BOTH tiers there. Only if a legacy page documents the OPPOSITE slant
+  (e.g. "enable MTP for high throughput") stop and confirm with the
+  maintainer.
+- **3 operating points** → the **full trio** (the ideal — e.g. GPU-budget
+  tiers 2/4/8).
+
+**Signal-driven tiers (hard rule).** A cell goes under `low-latency` /
+`high-throughput` ONLY on a signal present in the legacy source: an explicit
+performance toggle (MTP/speculative, dp-attention, EP, gpuCount, …), a named
+recipe/strategy checkbox, option subtitles ("Low Latency" / "High
+Throughput"), or prose stating the operating point. Reading such
+a signal is SGLang-level serving semantics (MTP favors latency on any
+vendor's silicon), so any migrator can tier any vendor's cells without
+hardware-specific judgment. **No signal → `balanced`** — legacy silence is
+itself information: the page offered that command as the hardware's
+general-purpose operating point, and `balanced` transcribes exactly that.
+Never derive a slant from your own hardware intuition ("this flag combo
+feels throughput-tuned"); re-tiering on measured evidence is the hardware
+owner's follow-up PR, not part of a migration. A toggle that maps to no
+dimension, or a suspected undocumented slant → stop and ask the maintainer.
+
+The tiers apply **per (hw × variant × quant) combination**, not just per page:
+a combination with fewer operating points than the page parks its cells in
+the semantically honest tier. A single recipe with a signal-evidenced slant
+goes to that tier (DSv4's RTX PRO 6000 → `low-latency`: workstation card,
+low-batch Marlin recipe — the recipe's own SGLang-legible content is the
+evidence); a general-purpose recipe with no latency/throughput slant goes to
+`balanced` (Qwen3.5's Xeon → `balanced`). Never park a no-slant recipe under
+`low-latency`/`high-throughput` just because the page's toggle mapping lands
+there — that reads as a semantic lie ("CPU = high-throughput?"). The page's
+`strategies` list is the union of tiers actually used (a mixed
+[low-latency, balanced, high-throughput] page where GPUs use the two ends and
+CPU uses the middle is fine); the engine greys unused chips per selection and
+auto-snaps, no extra config needed.
+
+Deviations (e.g. how to name pure GPU-budget tiers) need maintainer sign-off.
+The MDX strategy bullets describe serving semantics in the DSv4 style
+(single-user chat / typical multi-user / batch jobs), with at most a one-line
+model-specific note — never toggle-/migration-centric explanations.
+
+| Family | strategies | Notes |
+|---|---|---|
+| Gemma4 | `low-latency` (MTP on — the legacy toggle's own "Lower Latency" subtitle) / `high-throughput` (MTP off); mi300x hides the toggle → its single recipe → `balanced` (trio union, Qwen3.5 Xeon pattern) | variants = e2b/e4b/12b/31b/26b-a4b; checkpoint radio Standard(BF16)/QAT(q4_0) → quant ids via `modelNames`; §3.3 prose carries AMD recipes beyond the widget's mi300x — maintainer call on cells-from-prose vs tips; vision/audio invocation prose carries over (deployment matrix is text-standard); "gemma4 branch" version → speed drops, MMLU/GSM8K accuracy keeps (mind the few-shot vs run_eval harness footnote); dedicated multi-arch dev images verbatim |
+| Nemotron3-Ultra | dpattention carries "Low latency"/"High throughput" subtitles (naming rule) but THREE perf controls stack — multi-value DP-Attention (2/4/8) × MTP × EP — design the tier mapping via the step-2 table; maintainer sign-off required | NVIDIA-only (h100→gb300) with a per-quant verified-hw SUPPORT matrix → absent cells; "Model" radio = the quant dim (BF16 / NVFP4 Blackwell-only); TP radio 8/16 — TP=16 is 2-node → `nodes` dim; **kvcache radio (None/fp8_e4m3/bf16) → `flagSelects` axis, config-only** (the generic primitive merged in #28128 — NO engine PR); `launch_server` + spec-V2 env prefix verbatim; dedicated `dev-nemotron3-ultra(+cu13)` images verbatim ("not in any stable release"); **"main branch" version is non-reproducible → drop the WHOLE measured result (speed AND accuracy)** unless it can be pinned to the support PR/commit (day-0 rule, §hard-rule-2) |
+| GLM-4.5, GLM-4.6 | `low-latency` (TP, + MTP from the legacy checkbox) / `high-throughput` (TP+DP+EP) | |
+| GLM-4.7 | `low-latency`(2 GPUs) / `balanced`(4) / `high-throughput`(8) — gpus 2/4/8 + SUPPORT matrix; confirm naming, tiers are GPU budgets | measured-best B200 TP=2 NVFP4 → the verified cell |
+| GLM-4.7-Flash | `low-latency` (tp1 + MTP from the legacy checkbox) / `high-throughput` (DP) | derive from the legacy dp/mtp checkboxes |
+| GLM-5, GLM-5.1 | `low-latency` (dpattention off) / `high-throughput` (dpattention on) — the dpattention radio carries the page's own "Low Latency"/"High Throughput" subtitles (naming rule, §1) | spec is flag-only, default ON, hidden on AMD → bakes into both tiers on NVIDIA + `speculative` axis; NVFP4 hides all toggles → single no-signal recipe → `balanced` (page ships the trio union) |
+| Kimi-K2 | `low-latency` (tp8) / `high-throughput` (dp4+ep4) | variants = instruct/thinking; reasoning chip `hide` on instruct |
+| Kimi-K2.5, K2.6 | `low-latency` (dpattention off) / `high-throughput` (dpattention on) — same named-subtitle pattern as GLM-5.1 | K2.5 spec preset carries `--speculative-draft-model-path …eagle3-mla`, chip-gated to h200/b300; K2.6's live page has a NVIDIA-only spec toggle, default OFF → `speculative` axis only, no bake (missed by the survey) |
+| Qwen3.6, Qwen3-Next | `low-latency` (MTP on, the legacy speculative toggle) / `high-throughput` (MTP off); Xeon hides the toggle → its single recipe → `balanced` (page ships the trio) | same pattern as the Qwen3.5 pilot |
+| Kimi-Linear, MiniMax-M2, Qwen3, Qwen3-Coder, Qwen3-Coder-Next | single `balanced` — one recipe, no performance toggle (rule above: 1 operating point → `balanced`) | renders as one chip; Qwen3-Coder-Next has NO speculative dim on the live page (quant × toolcall × mambaCache only — an earlier sketch wrongly lumped it with Qwen3.6) |
+| MiniMax-M2.5, M2.7 | `low-latency`(2) / `balanced`(4) / `high-throughput`(8=tp8+ep8) — confirm naming, tiers are GPU budgets | Xeon (M2.7) is a single no-slant recipe (fixed TP=6) → `balanced` (per-combination rule, Qwen3.5 Xeon precedent) |
+| Qwen3.5 (DONE — pilot) | `low-latency` (MTP on) / `high-throughput` (MTP off); Xeon's single no-slant recipe → `balanced` (the page ships the full trio) | see §5 |
+
+Qwen3 variant fan-out: variants = deployable checkpoints size-ordered
+(`235b-instruct`, `235b-thinking`, `235b`, `30b-*`, `32b`, …); do NOT abuse
+strategies for the instruct/thinking category. Trim original-hybrid chips to
+the ones the legacy page actually measured.
+
+## 5. Worked example — the Qwen3.5 pilot (PR #27848)
+
+Decisions log, in the order they came up:
+
+1. **Strategy split over Playground toggle** because MTP couples with TP on
+   three H100 combos (35B/27B BF16: tp2↔tp1+mem0.88; 122B FP8: tp4↔tp2).
+   Canonical naming: `low-latency` = MTP on (legacy default), `high-throughput`
+   = MTP off. Xeon has a single operating point (the legacy widget hid the MTP
+   toggle there) and its recipe has no latency/throughput slant → its 12 cells
+   park under `balanced` (per-combination placement; parking them under
+   high-throughput as a toggle-mapping side effect read as a semantic lie).
+   Result: 186 cells = 87 low-latency + 87 high-throughput + 12 balanced; the
+   page ships the full trio and the engine greys unused chips per selection.
+2. **Verified cell follows the measurement**: H200/397B/BF16/low-latency =
+   `SGLANG_USE_CUDA_IPC_TRANSPORT=1` env + `--speculative-algorithm EAGLE`
+   (the bench reported NEXTN, an alias of EAGLE — normalized to EAGLE, §2) +
+   measured flag set **minus the parser flags** (the measured run had both
+   parsers on; cells never carry them — noted in the benchmarks header). All
+   other cells = the generator's parsers-OFF output verbatim. A single `eagle`
+   spec preset on the speculative axis (the duplicate NEXTN preset was dropped).
+3. **FP4 single quant id** with `hw|variant|quant` modelNames keys →
+   `nvidia/...NVFP4` (b200/b300) vs `amd/...MXFP4` (mi355x).
+4. **Xeon** as `config.hardware` `vendor:"intel"`; cells carry
+   `--device cpu --disable-overlap-schedule`; no docker mapping.
+5. **Playground axes**: the full general set per §2b — attention
+   (TP/CP/DP-Attn), moe (DeepEP backend + EP knob, `disable`+reason on the
+   dense variants), parsers, speculative (NEXTN + EAGLE — both algorithms
+   appear on the page), pdDisagg, hicache. Excluded as inapplicable:
+   hisparse (DSA-only), MegaMoE (DSv4 Blackwell-only). The legacy
+   `--expert-parallel-size 8` flag is normalized to `--ep 8` for the EP knob.
+6. **Benchmarks**: one entry (the measured cell) only — entry-less cells render
+   "pending" without stubs. The legacy speed numbers were DROPPED: they were
+   measured on a drifting "main branch" build, which is no version anchor
+   (speed migrates only under an exact release tag / commit hash — hard rule
+   2), so the entry carries accuracy only (GSM8K + MMMU via `accuracyLabels`,
+   sample counts in `notes`) and no `sglang_version`.
+7. **Codegen + audit scripts** (adapt per model): a generator-port script that
+   emits the cells literal, and an independent audit that `git show`s the
+   ORIGINAL generator, stubs `useState`/`useEffect`, calls its
+   `generateCommand(values)` per combo via indirect eval, and token-diffs
+   against the new cells (expected deltas only). Read the legacy source via
+   `git show main:<path>` — NOT `HEAD:` (the migration branch's HEAD has
+   already deleted the file, so the audit breaks after the deletion commit).
+   Re-run the audit after ANY later cells revision (renames included). Pilot
+   result: 185/185 identical + 1 intentional override. Scripts are archived
+   in PR #27848's description (collapsed details block).
+8. **Inherited-infeasible combos kept verbatim** (e.g. 122B BF16 tp1 on
+   mi325x: 244 GB weights vs 256 GB VRAM with mem-fraction 0.8) — they stay
+   yellow and are listed in the PR body for the re-verification track.
+9. **Browser-smoke probe pitfall**: multiple programmatic `.click()` calls in
+   one synchronous eval batch under React 18 — the DOM reads between them are
+   stale and look like snap-logic bugs. One click per eval, then settle.

.claude/skills/cookbook-review-pr/SKILL.md

@@ -0,0 +1,245 @@
+---
+name: cookbook-review-pr
+description: Review a pull request against the SGLang Cookbook (docs_new/, Mintlify) contribution checklist — the config-driven format (per-model config + benchmarks JSX consumed by the shared _deployment.jsx / _playground.jsx engines). Run with /cookbook-review-pr <PR number>.
+---
+
+# Cookbook Review PR
+
+Fetch the diff, run the checklist, report what you find. The cookbook is **config-driven**:
+shared engines (`_deployment.jsx`, `_playground.jsx`) with NO model-specific code; each
+model is a data `config` (+ optional `benchmarks`) under `src/snippets/configs/<vendor>/`
+plus an MDX page. This checklist targets that layout. Field-schema detail lives in
+`.claude/skills/cookbook-add-model/references/authoring-reference.md` — defer to it rather
+than restating.
+
+## Usage
+
+```
+/cookbook-review-pr <PR number>
+```
+
+## Steps
+
+1. `gh pr view <N> --repo sgl-project/sglang --json title,body,files,author,baseRefName,headRefName,commits,reviews`
+2. `gh pr diff <N> --repo sgl-project/sglang`
+3. `gh pr list --repo sgl-project/sglang --state open --search "<model name>"` (duplicate check)
+4. Run every checklist item against the diff.
+5. Output per-file verdicts + overall recommendation.
+
+## Checklist
+
+### 1. File hygiene
+- A cookbook PR should only touch: `docs_new/src/snippets/configs/<vendor>/*.jsx`
+  (config + benchmarks), `docs_new/cookbook/**/*.mdx`, `docs_new/docs.json`,
+  `docs_new/cookbook/<category>/intro.mdx` (vendor card), `docs_new/cards/logos/<vendor>.png`
+  (new vendor only). Flag stray files (`settings.local.json`, lockfiles, IDE configs).
+- Pages must be `.mdx`, not `.md`. Files end with a trailing newline. Check commit history
+  for unrelated commits accidentally included.
+- **Engines untouched**: `_deployment.jsx` / `_playground.jsx` should NOT change in a
+  model-add PR (adding a model is data-only). Engine edits = a separate axis/feature PR
+  (see `cookbook-add-model/references/engine-axis.md`); review them against that checklist.
+
+### 2. Config quality (the per-model `config`)
+- Single `export const config = { ... }` literal — **no** function calls, spreads,
+  fragment refs, or IIFE (Mintlify re-evals at hydration → `ReferenceError`).
+- No `!(x in y)` anywhere (Mintlify AST walker crashes) — use `obj.key === undefined`.
+- `supportedHardware` ⊆ `HARDWARE_CATALOG` (in `_deployment.jsx`) ∪ `config.hardware`. A
+  model-specific GPU the shared catalog lacks must be declared in `config.hardware`
+  (`{id,label,vram,vendor}`), **not** added to the engine catalog.
+- `placeholders` declares every `{{KEY}}` used in `curl` or any cell.
+- `modelNames` covers every cell (by `hw|variant|quant` triple or `variant|quant` pair).
+- `strategies` count matches the page's operating points — 1 recipe → a single `balanced`;
+  2 → `low-latency` + `high-throughput`; 3 → the full trio. Tiers apply per
+  (hw × variant × quant) combination: a single-recipe combination must park under its
+  semantically honest tier (clear slant → that tier, e.g. a workstation card under
+  `low-latency`; no slant → `balanced`, e.g. a CPU platform) — **flag a no-slant recipe
+  parked under low-latency/high-throughput**. Mixed unions
+  like [low-latency, balanced, high-throughput] with per-selection greying are fine. Also
+  flag model-specific ids (e.g. `mtp`), and flag an INVERTED speculative mapping — the
+  deterministic default is MTP/spec-decoding ON → `low-latency`, OFF → `high-throughput`
+  (at saturation the draft+verify overhead outweighs the speedup); the reverse needs an
+  explicit maintainer-confirmed justification in the PR. The MDX strategy bullets describe serving semantics
+  in the DSv4 style (single-user chat / typical multi-user / batch jobs), not internal
+  toggles.
+- `dockerImages` covers the hw ids that have cells (else users hit the `:dev` fallback); a
+  `hw|quant` key (resolved before the plain `hw`) is valid when one quant on a shared GPU needs
+  a different image (e.g. an FP4 dev build) — don't flag those.
+- `multiNodeHints` present ONLY for hw whose fabric needs manual NIC env (e.g. `gb200`
+  NVL72) — NOT every `multi-N` hw (standard-IB DeepEP / Marlin multi-node don't need it).
+- `github.cookbookModel` is set to the model's HF id (`<hf-org>/<model-slug>`). The issue
+  template's `model` field is a free-form input prefilled from this value; if the config
+  omits the `github` block, the engine falls back to `deepseek-ai/deepseek-v4` and the
+  page's submissions get mislabeled.
+- `playgroundFeatures` is opt-OUT: the **general axes ship on every cookbook by default**
+  (`attention` TP/CP/DP-Attn, `moe` backend+EP for MoE models, `parsers`, `speculative`,
+  `pdDisagg`, `hicache`) — flag a missing general axis unless the model genuinely cannot
+  use it. Model-specific axes only where applicable (MegaMoE backend + `megamoeQuant`
+  only on Blackwell MoE, gated by `requiresHw`; `hisparse` only DSA-style). Knobs that are
+  meaningless for a subset of variants/hw are `disable`d with a reason, not silently live
+  (e.g. MoE knobs greyed on dense variants). No empty/stub axes.
+- **No leftover `__TOKEN__`** — the config was stamped from the template and every
+  placeholder is filled (`grep -rn '__[A-Z_]*__'` on the new config/benchmarks/MDX returns
+  nothing).
+- **All-hardware considered**: every `supportedHardware` id (from the catalog or `config.hardware`) has ≥1 cell OR is a deliberate
+  greyed "coming soon"; AMD was pruned or kept on purpose (not a leftover template family).
+
+### 3. Cells / 5-dim matrix
+- Every cell `match` has EXACTLY the 5 keys (`hw`, `variant`, `quant`, `strategy`, `nodes`).
+- `env` / `flags` are flat literals (only `{{PLACEHOLDER}}` subst) — no shared
+  `commonFlags` reference (Mintlify won't inline it).
+- NO `--nnodes` / `--node-rank` / `--dist-init-addr` literals in multi-node cells
+  (the renderer injects them from `match.nodes`).
+- NO literal `--host` / `--port` — use `{{HOST_IP}}` / `{{PORT}}`.
+- NO `--reasoning-parser` / `--tool-call-parser` in any cell — parsers are a
+  Playground-only feature added on top of the base command (DSv4 convention);
+  flag any cell that bakes them in.
+- Accuracy-degrading flags in cells — runtime quant below the checkpoint
+  (e.g. MegaMoE W4A4 — DSv4 gates it behind the Playground's `megamoeQuant`)
+  and lossy `--kv-cache-dtype` (e.g. `fp8_e4m3` over a higher-precision-KV
+  checkpoint): **flag for explicit maintainer confirmation**. Output quality
+  should be exactly what the quant chip declares, so absent a recorded
+  sign-off in the PR (e.g. carried verbatim from a measured legacy recipe's
+  default command), request the flag move to Playground/tips.
+- Flag order: `--model-path` first (an optional `--trust-remote-code` may precede it —
+  the DSv4 cells do), then parallelism, then MoE, then tuning, `--host`/`--port`
+  last (the playground's insert anchors assume this).
+- TP/memory sanity: `model_weight_GB / (tp × gpu_mem)` fits with ~20–30% headroom
+  (BF16 ≈ params×2 GB, FP8 ≈ ×1, FP4 ≈ ×0.5; MoE uses **total** weight, not active params).
+
+### 4. Benchmarks
+- Each `benchmarks[]` entry's `match` tuple corresponds to a real cell.
+- `accuracyLabels` is present whenever the benchmarks carry accuracy data — the engine
+  ships NO default eval set; without it the accuracy rows silently don't render.
+  `defaultAccuracy` / per-cell `accuracy` / `benchmarkCommands.accuracy` keys all
+  ∈ `config.accuracyLabels`.
+- A benchmark's quantization must match a variant actually listed — `(BF16)` on a model
+  that only released FP8/FP4 is a factual bug.
+- `benchmarkCommands.speed` is `python3 -m sglang.bench_serving` (the workload), separate
+  from the `sglang serve` deploy command.
+- `sglang_version` is a real build the author ran (a release, or `dev`/nightly) — not a
+  guessed/placeholder value (no leftover `0.0.0`).
+- **Latency percentile**: `config.latencyPercentile` (default `"P50"`, or `"Mean"`) matches the
+  percentile the TTFT/TPOT values actually are — the card renders `TTFT (<pct>)`. (`"Mean"` is
+  temporary — legacy data is being re-measured to P50.)
+- **Throughput convention**: `tokens_per_sec_per_gpu` is stored as **total (in+out)/GPU**
+  = `output tok/s/GPU × (isl+osl)/osl`, shown by the card as-is. Flag output-only values.
+- **Consistent accuracy harness across entries**: every value under one `accuracyLabels`
+  column must be produced by the SAME harness — flag a page that, say, measures one
+  platform's GSM8K with `few_shot_gsm8k --num-questions 200` and another's with
+  `run_eval --eval-name gsm8k --num-examples 1319` and shows both as one "GSM8K %"
+  (the scores aren't comparable). Either standardize on one harness (matching
+  `benchmarkCommands.accuracy`) or require an explicit per-entry note. Common when folding
+  a second contributor's measurements (e.g. an AMD/ROCm PR) into the page.
+
+### 5. Doc ↔ config parity (the #1 finding)
+- Any `sglang serve` command shown in MDX prose (config tips, benchmark section) must
+  equal what the engine emits from the corresponding cell — same flags, same order. Drift
+  here is the most common review miss.
+
+### 6. Commands / port
+- Launch uses `sglang serve` — flag any `python -m sglang.launch_server` /
+  `python3 -m sglang.launch_server` (deprecated). The engine already emits `sglang serve`;
+  guard against prose/cells reintroducing the old launcher.
+- Port `30000` everywhere (launch, curl, client `base_url`, bench) — flag `8000`.
+  Launch port must match client/curl port on the same page.
+
+### 7. Frontmatter
+- Every new MDX page has `title:` and a **top-level** `description:` (a real one-line value
+  prop, not copied from another vendor) — NOT `metatags.description` (non-canonical; the
+  top-level field is what renders as the subtitle and SEO meta — see mintlify-authoring).
+- **No `mode: wide` on a model page** — it hides the right-hand "On this page" ToC that every
+  other model page has. Leave `mode` unset (the Deploy/Playground panels self-cap at 900px, so
+  the default column holds them fine). `mode: wide` belongs only on category `intro.mdx` grids.
+- `tag: NEW` only for genuine new launches; when one is added, stale `tag: NEW` on older
+  pages should be dropped in the same PR (`grep -RlE "^tag: NEW" docs_new/cookbook/`).
+- MDX imports BOTH `Deployment` and `Playground` from `/src/snippets/...` (absolute).
+- Deploy heading slugs to `deployment` (or `deploy`), Playground to `playground` — so
+  "↑ Switch base" and "Open the Playground →" scroll. No numbered headings for these two.
+
+### 8. Navigation & homepage
+- New page → `docs_new/docs.json` updated: under the right vendor group inside
+  `navigation` → Cookbook → Autoregressive Models, root-relative, **no `.mdx`**:
+  `cookbook/<category>/<Vendor>/<Model>`.
+- Homepage `<Card href>` in `docs_new/cookbook/<category>/intro.mdx` points to the vendor's
+  flagship; new vendors get a new `<Card>` + a logo at `docs_new/cards/logos/<vendor>.png` —
+  **940×525 RGBA transparent, icon-only (no wordmark)**, lowercase filename, tracked via
+  `git add -f` (`*.png` is gitignored repo-wide). Card order matches the `docs.json` nav order.
+- Don't change `docs_new/cookbook/intro.mdx` for individual model adds (top-level only).
+
+### 9. Links & factual
+- HuggingFace URLs resolve to a real model. License section matches the actual HF license
+  (don't copy from another model). Docker images from `lmsysorg/sglang`; no `sgl-project-dev`.
+  The image **tag** is a real build (a release the author ran, or `:dev`/nightly) — not a
+  guessed version.
+- Internal links root-relative, no extension (`/cookbook/.../<Model>`); flag `.md`/`.mdx`
+  or `../`-relative links. `docs.sglang.io` is canonical.
+- No Google-Drive image links (don't render). Shell placeholders are `export VAR=<value>`,
+  not `${VAR}` (a bash no-op).
+- **Parser ids must exist in the code registries** on the PR's target branch: every
+  `--reasoning-parser X` / `--tool-call-parser Y` named in prose or in
+  `playgroundFeatures.parsers` flags is a registered key in
+  `python/sglang/srt/parser/reasoning_parser.py` (DetectorMap) /
+  `python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
+  prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
+  factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
+  (`python/sglang/srt/managers/template_detection.py`) actually resolve THIS model's
+  chat template to the right parser — no rule match means auto silently disables the
+  parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
+
+### 9b. MDX authoring (Mintlify) — detail in `cookbook-add-model/references/mintlify-authoring.md`
+- **Forbidden syntax**: no Docusaurus admonitions (`:::`), `@site`/`@theme`, GitHub alert
+  blocks (`> [!NOTE]`), markdown **pipe tables** (use JSX `<table>`), inline `<details>`,
+  or unknown components. `<CardGroup>`/`<Card>` only on category `intro.mdx`, not model pages.
+- Code fences are **labeled** (e.g. `python Example` / `bash Command` / `text Output` after
+  the opening fence); a fenced block nested inside another uses four backticks outside.
+- §3 commands and outputs are **collapsible** (DeepSeek-V4 pattern): every runnable
+  example wrapped in an `<Accordion>`, its real output in a following
+  `<Accordion title="Example Output">` (`Pending update...` only with user
+  acknowledgement). Flag bare/inline example blocks and `**Output Example:**` headings.
+- Reasoning-parser example matches the parser's **output shape**: separate-field
+  (`reasoning_content` + `content`) vs inline `<think>` tags parsed out of `content`.
+- No hardcoded sampling params (`temperature` / `top_p`) in sample code (SGLang uses
+  `generation_config.json` defaults); listing them in §1 informationally is fine.
+
+### 10. Quantization rules
+- **NVFP4** checkpoints are Blackwell-only (B200/B300/GB300) — never AMD. An AMD FP4 cell
+  is legitimate ONLY when the vendor published an **MXFP4** checkpoint for it (e.g.
+  `amd/Qwen3.5-397B-A17B-MXFP4` on MI355X) — verify the HF repo resolves; otherwise the
+  AMD FP4 chip must be absent/`disabled`.
+- BF16 / FP8 work on NVIDIA and AMD. `--kv-cache-dtype fp8_e4m3` in a cell is an
+  accuracy-degrading flag — see §3 (needs explicit maintainer sign-off; default
+  home is Playground/tips).
+
+### 11. Scope
+- Changes match the PR title. Flag global changes hiding behind a platform-specific title
+  (e.g. an "H200 FP8" PR that adds a flag to ALL cells). Unmentioned side-fixes belong in
+  the PR body.
+
+### 12. Duplicate PRs
+- Another open PR for the same model? Flag it; compare completeness; note merge-conflict
+  risk on `docs.json` + the vendor card; flag a superseded older PR by the same author.
+
+### 13. Build / validate
+```bash
+cd docs_new
+mint validate
+mint broken-links
+```
+Optional: `mint dev` for a visual smoke test.
+
+### 14. Reviewer feedback
+- `gh api repos/sgl-project/sglang/pulls/<N>/comments` — have prior reviewer requests been
+  addressed? Unresolved requested-changes should be flagged.
+
+### 15. Grammar & spelling
+- Check added/changed prose for typos and grammar (e.g. "recommend" vs "recommended").
+  Flag each with the exact wrong text + correction.
+
+## Output
+
+Per file:
+- ✅ PASS
+- ⚠️ ISSUE: \<what\>
+- 🔴 BLOCK: \<what\>
+
+Overall: **APPROVE** / **REQUEST CHANGES** / **BLOCKED**

.claude/skills/debug-cuda-crash/SKILL.md

@@ -0,0 +1,657 @@
+---
+name: debug-cuda-crash
+description: Call this skill when you need to debug CUDA crashes in SGLang using kernel API logging
+---
+
+# Tutorial: Debugging CUDA Crashes with Kernel API Logging
+
+This tutorial shows you how to debug CUDA crashes and errors in SGLang using the `@debug_kernel_api` logging decorator.
+
+## Goal
+
+When your code crashes with CUDA errors such as illegal memory access, device-side assert, out-of-bounds, or NaN/Inf, use kernel API logging to:
+- Capture input tensors BEFORE the crash occurs
+- Understand what data caused the problem
+- Track tensor shapes, dtypes, and values through the call boundary that triggered the crash
+- Detect numerical issues such as NaN, Inf, or obviously wrong shapes
+
+## Why Use Kernel API Logging?
+
+**Problem**: CUDA errors often crash the program before normal debugging output is flushed.
+
+**Solution**: SGLang's `@debug_kernel_api` decorator logs inputs before execution, so you can still see what caused the crash even after the program aborts.
+
+## What Is Covered?
+
+The current logging coverage focuses on the highest-value kernel boundaries in SGLang:
+- Custom ops registered through `register_custom_op(...)`
+- External custom ops registered through `register_custom_op_from_extern(...)`
+- LLM attention, linear, quantization, and multi-platform wrapper entry points
+- Diffusion attention impl, linear, rotary, and custom-op wrapper entry points
+- Selected direct `torch.ops.sglang.*` hotspots and model-specific bypasses
+
+This means the logging is useful for both LLM and diffusion kernel debugging, but it does not automatically cover every pure PyTorch call in the repository.
+
+## Step 1: Enable Kernel API Logging
+
+### Basic Logging (Function Names Only)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=1
+export SGLANG_KERNEL_API_LOGDEST=stdout
+
+python my_script.py
+```
+
+Output:
+```
+================================================================================
+[2026-03-19 00:47:06] SGLang Kernel API Call: RMSNorm.forward
+================================================================================
+[2026-03-19 00:47:06] SGLang Kernel API Call: sglang.quant_method.UnquantizedLinearMethod.apply
+================================================================================
+[2026-03-19 00:47:06] SGLang Kernel API Call: sglang.custom_op.fused_inplace_qknorm
+```
+
+This is a real level-1 excerpt captured from `Qwen/Qwen3-0.6B`.
+
+### Detailed Logging (Inputs with Metadata)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+python my_script.py
+```
+
+Output in `debug.log`:
+```
+================================================================================
+[2026-03-19 00:47:30] SGLang Kernel API Call: sglang.quant_method.UnquantizedLinearMethod.apply
+Positional input arguments:
+  arg[0]=QKVParallelLinear(
+      repr=QKVParallelLinear(in_features=1024, output_features=4096, bias=False, tp_size=1, gather_output=False)
+    )
+  arg[1]=Tensor(
+      shape=(1, 1024)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+    )
+  arg[2]=None
+Output:
+  return=Tensor(
+      shape=(1, 4096)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+    )
+```
+
+This is a real level-3 excerpt captured from `Qwen/Qwen3-0.6B`.
+
+### Full Logging (With Tensor Statistics)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=5
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+python my_script.py
+```
+
+Additional output:
+```
+================================================================================
+[2026-03-19 01:00:42] SGLang Kernel API Call: diffusion.quant_method.UnquantizedLinearMethod.apply
+Positional input arguments:
+  arg[1]=Tensor(
+      shape=(1, 77, 768)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+      min=-27.250000
+      max=28.500000
+      mean=0.011723
+      nan_count=0
+      inf_count=0
+    )
+Output:
+  return=Tensor(
+      shape=(1, 77, 2304)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+      min=-8.937500
+      max=9.375000
+      mean=0.009460
+      nan_count=0
+      inf_count=0
+    )
+```
+
+This is a real level-5 excerpt captured from `black-forest-labs/FLUX.1-dev`.
+
+### Crash-Safe Dumps (Inputs Saved Before Execution)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+export SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_kernel_api_dumps
+
+python my_script.py
+```
+
+At level 10, SGLang saves the inputs before execution. If the kernel crashes, the dump directory still contains the inputs and exception metadata.
+
+If CUDA graph capture is active, tensor dumps are skipped automatically to avoid capture-time CUDA errors. In that case, you still get the kernel API call log, but not `inputs.pt` / `outputs.pt`.
+
+Level-10 dumps are best understood as crash-safe call snapshots. They always preserve the observed call boundary. They do not guarantee one-click replay for every method, because some methods depend on module state that is not serialized into the dump.
+
+Real level-10 dump layout from `Qwen/Qwen3-0.6B`:
+
+```text
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001/inputs.pt
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001/metadata.json
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001/outputs.pt
+```
+
+Real `metadata.json` excerpt:
+
+```json
+{
+  "function_name": "RotaryEmbedding.forward",
+  "timestamp": "20260319_004821_182",
+  "process_id": 919286,
+  "execution_status": "completed",
+  "input_tensor_keys": ["arg_0", "arg_1", "arg_2"],
+  "output_tensor_keys": ["result_0", "result_1"]
+}
+```
+
+## Step 2: Reproduce an LLM CUDA Crash
+
+Create a temporary reproducer:
+
+```bash
+python3 - <<'PY'
+from pathlib import Path
+Path("/tmp/sglang_llm_crash.py").write_text(
+    "import torch\\n"
+    "import torch.nn.functional as F\\n"
+    "from sglang.srt.utils.custom_op import register_custom_op\\n\\n"
+    "def _fake_embedding(indices, table):\\n"
+    "    return torch.empty((*indices.shape, table.shape[-1]), device=table.device, dtype=table.dtype)\\n\\n"
+    "@register_custom_op(op_name='mock_llm_cuda_crash', fake_impl=_fake_embedding)\\n"
+    "def mock_llm_cuda_crash(indices, table):\\n"
+    "    out = F.embedding(indices, table)\\n"
+    "    torch.cuda.synchronize()\\n"
+    "    return out\\n\\n"
+    "table = torch.randn(4, 8, device='cuda', dtype=torch.float16)\\n"
+    "indices = torch.tensor([0, 7], device='cuda', dtype=torch.long)\\n"
+    "mock_llm_cuda_crash(indices, table)\\n"
+)
+PY
+
+SGLANG_KERNEL_API_LOGLEVEL=1 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_llm_level1.log \
+python3 /tmp/sglang_llm_crash.py
+```
+
+What to expect:
+- The script exits with a CUDA `device-side assert`
+- The log still contains the last API boundary before the crash
+
+Try the same example at level 3:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=3 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_llm_level3.log \
+python3 /tmp/sglang_llm_crash.py
+```
+
+Now the log shows tensor metadata before the crash.
+
+Try level 10:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=10 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_llm_level10.log \
+SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_llm_level10_dumps \
+python3 /tmp/sglang_llm_crash.py
+```
+
+Now you should see:
+- A log entry for `sglang.custom_op.mock_llm_cuda_crash`
+- A dump directory with `inputs.pt`
+- `metadata.json` showing `execution_status: "exception"`
+- No `outputs.pt`, because the kernel crashed before producing output
+
+For real-model success-path level-10 dumps, it is often easier to temporarily disable CUDA graph and piecewise CUDA graph for the debug run.
+
+## Step 3: Reproduce a Diffusion CUDA Crash
+
+Create a temporary diffusion-side reproducer:
+
+```bash
+python3 - <<'PY'
+from pathlib import Path
+Path("/tmp/sglang_diffusion_crash.py").write_text(
+    "import torch\\n"
+    "import torch.nn.functional as F\\n"
+    "from sglang.multimodal_gen.runtime.layers.utils import register_custom_op\\n\\n"
+    "def _fake_embedding(positions, cache):\\n"
+    "    return torch.empty((*positions.shape, cache.shape[-1]), device=cache.device, dtype=cache.dtype)\\n\\n"
+    "@register_custom_op(op_name='mock_diffusion_cuda_crash', fake_impl=_fake_embedding)\\n"
+    "def mock_diffusion_cuda_crash(positions, cache):\\n"
+    "    out = F.embedding(positions, cache)\\n"
+    "    torch.cuda.synchronize()\\n"
+    "    return out\\n\\n"
+    "cache = torch.randn(4, 64, device='cuda', dtype=torch.float16)\\n"
+    "positions = torch.tensor([0, 9], device='cuda', dtype=torch.long)\\n"
+    "mock_diffusion_cuda_crash(positions, cache)\\n"
+)
+PY
+
+SGLANG_KERNEL_API_LOGLEVEL=1 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_diffusion_level1.log \
+python3 /tmp/sglang_diffusion_crash.py
+```
+
+Try level 3:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=3 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_diffusion_level3.log \
+python3 /tmp/sglang_diffusion_crash.py
+```
+
+Try level 10:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=10 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_diffusion_level10.log \
+SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_diffusion_level10_dumps \
+python3 /tmp/sglang_diffusion_crash.py
+```
+
+If your local environment has unrelated FlashInfer import issues, resolve them in the shell before running the example. The example itself does not set any `FLASHINFER_*` environment variable.
+
+## Step 4: Multi-Process Debugging
+
+When running with multiple GPUs or worker processes, use `%i` in the log path:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug_rank_%i.log
+
+torchrun --nproc_per_node=4 my_script.py
+```
+
+This creates separate logs such as:
+- `debug_rank_12345.log`
+- `debug_rank_12346.log`
+- `debug_rank_12347.log`
+- `debug_rank_12348.log`
+
+Real multi-process example from a 2-GPU `Qwen/Qwen2.5-0.5B-Instruct` run:
+
+```text
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950201.log
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950349.log
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950350.log
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950351.log
+```
+
+You should usually do the same for level-10 dump directories:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+export SGLANG_KERNEL_API_LOGDEST=debug_rank_%i.log
+export SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_kernel_api_dumps_%i
+```
+
+This avoids multiple ranks writing into the same dump directory tree.
+
+## Step 5: Filter Level-10 Dumps
+
+If level 10 is too noisy, restrict dumps to specific APIs:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+export SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_kernel_api_dumps
+export SGLANG_KERNEL_API_DUMP_INCLUDE='sglang.custom_op.*'
+export SGLANG_KERNEL_API_DUMP_EXCLUDE='*.fake_impl'
+```
+
+`SGLANG_KERNEL_API_DUMP_INCLUDE` and `SGLANG_KERNEL_API_DUMP_EXCLUDE` use shell-style wildcard matching.
+
+## Step 6: Common CUDA Errors and What to Check
+
+### Illegal Memory Access or Device-Side Assert
+
+**Typical errors**:
+```
+RuntimeError: CUDA error: an illegal memory access was encountered
+torch.AcceleratorError: CUDA error: device-side assert triggered
+```
+
+Use:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+Check in the logs:
+- ✅ Tensor shapes
+- ✅ Tensor dtypes
+- ✅ CUDA vs CPU device placement
+- ✅ Tensor stride / contiguity
+- ✅ Whether the failing call has inputs logged but no outputs logged
+
+Typical shape-mismatch pattern:
+
+```text
+SGLang Kernel API Call: ...
+arg[0]=Tensor(shape=(..., 128), ...)   # ✅ expected dimension
+arg[1]=Tensor(shape=(..., 64), ...)    # ❌ mismatch
+```
+
+This often points to head-dim, hidden-dim, or cache-layout mismatch rather than a random CUDA failure.
+
+### NaN or Inf
+
+Use:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=5
+```
+
+Check:
+- `min`
+- `max`
+- `mean`
+- `nan_count`
+- `inf_count`
+
+Typical bad pattern:
+
+```text
+Tensor(
+  ...
+  min=-1234567.000000   # ❌ suspiciously large
+  max=9876543.000000    # ❌ suspiciously large
+  mean=nan              # ❌ bad
+  nan_count=128         # ❌ found NaNs
+  inf_count=0           # ✅ no Infs here
+)
+```
+
+This usually means the bad values were already present before the crashing kernel.
+
+### Out of Memory
+
+Use:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+Check:
+- Unexpectedly large tensor shapes
+- Batch size
+- Sequence length
+- Frame count or image resolution in diffusion workloads
+
+Also check whether a supposedly per-token or per-frame tensor accidentally became full-sequence or full-image sized.
+
+Typical bad pattern:
+
+```text
+Tensor(
+  shape=(1024, 8192, 128, 128)   # ❌ way too large
+  ...
+)
+```
+
+### Example: Spot a Shape Bug from the Log
+
+Suppose the failing API log looks like this:
+
+```text
+[2026-03-19 00:47:30] SGLang Kernel API Call: RotaryEmbedding.forward
+Positional input arguments:
+  arg[0]=Tensor(shape=(1, 8), dtype=torch.int64, ...)
+  arg[1]=Tensor(shape=(1, 8, 8, 256), dtype=torch.bfloat16, ...)    # ✅ query
+  arg[2]=Tensor(shape=(1, 8, 4, 64), dtype=torch.bfloat16, ...)     # ❌ key head_dim mismatch
+```
+
+What this tells you:
+- ✅ positions look reasonable
+- ✅ query looks plausible
+- ❌ key last dimension is inconsistent with the expected rotary/head dimension
+
+That usually means the bug is in projection layout, head packing, or cache format rather than in the rotary kernel itself.
+
+## Step 7: Combine with compute-sanitizer
+
+For harder bugs, combine kernel API logging with CUDA memory checking:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+compute-sanitizer --tool memcheck python3 /tmp/sglang_llm_crash.py
+```
+
+Use `debug.log` to see the exact inputs that reached the crashing API boundary.
+
+Typical `compute-sanitizer` output:
+
+```text
+========= COMPUTE-SANITIZER
+========= Invalid __global__ write of size 4 bytes
+=========     at 0x1234 in SomeKernel
+=========     by thread (256,0,0) in block (10,0,0)
+=========     Address 0x... is out of bounds
+```
+
+Use the sanitizer output to identify the failing kernel and use `debug.log` to identify the exact tensors that reached the API boundary right before it.
+
+If you need more synchronous host-side error reporting, you can try `CUDA_LAUNCH_BLOCKING=1` as a separate follow-up experiment. It is not part of the default workflow because it changes execution timing and can hide concurrency-related behavior.
+
+## Step 8: Combine with cuda-gdb
+
+For crashes that need a stack trace instead of only memory diagnostics:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+cuda-gdb --args python3 /tmp/sglang_llm_crash.py
+```
+
+Inside `cuda-gdb`:
+
+```text
+(cuda-gdb) run
+(cuda-gdb) where
+```
+
+Then correlate the backtrace with `debug.log`.
+
+## Step 9: Kernel-Level Debugging with printf()
+
+When you own the CUDA kernel, `printf()` is still useful for narrowing down bad indices, bad launch geometry, or broken state propagation.
+
+Basic pattern:
+
+```cpp
+__global__ void MyKernel(const float* input, float* output, int n) {
+  int idx = blockIdx.x * blockDim.x + threadIdx.x;
+
+  if (threadIdx.x == 0 && blockIdx.x == 0) {
+    printf("n=%d input0=%f\n", n, input[0]);
+  }
+
+  if (idx < n) {
+    output[idx] = input[idx] * 2.0f;
+  }
+}
+```
+
+After launch, force the output to flush:
+
+```python
+my_kernel(...)
+torch.cuda.synchronize()
+```
+
+For warp-specialized kernels, do not blindly print only on `threadIdx.x == 0`. Pick one representative thread per warp or per specialization group instead.
+
+### Warp-Specialized Kernels: Choosing the Right Print Thread
+
+Problem:
+- `threadIdx.x == 0` only prints from the first warp in the block
+- for warp-specialized kernels, that often misses the warp or group that is actually wrong
+
+Better pattern:
+
+```cpp
+__global__ void WarpSpecializedKernel(...) {
+  // Example: first lane of each warp
+  if ((threadIdx.x % 32) == 0) {
+    printf("warp=%d\n", threadIdx.x / 32);
+  }
+}
+```
+
+Or, if the kernel is organized in larger specialization groups, print once per group instead of once per block.
+
+Common mistake:
+
+```cpp
+// Only warp 0 prints
+if (threadIdx.x == 0) {
+  printf("warp=%d\n", threadIdx.x / 32);
+}
+```
+
+### Quick Reference
+
+| Kernel Type | Print Condition | Notes |
+|----------|----------|-------------|
+| Simple kernel | `threadIdx.x == 0` | One thread per block is usually enough |
+| Warp-specialized kernel | one representative lane per warp | e.g. `threadIdx.x % 32 == 0` |
+| Group-specialized kernel | one representative lane per group | choose based on the kernel's scheduling layout |
+
+### Other Kernel Debugging Tools
+
+```cpp
+assert(value >= 0.0f && "value must be non-negative");
+static_assert(BLOCK_SIZE % 32 == 0, "BLOCK_SIZE must be warp aligned");
+```
+
+## Environment Variables Reference
+
+| Variable | Values | Description |
+|----------|--------|-------------|
+| `SGLANG_KERNEL_API_LOGLEVEL` | `0` | No logging (default) |
+|  | `1` | Function names only |
+|  | `3` | Inputs and outputs with metadata |
+|  | `5` | Level 3 plus tensor statistics |
+|  | `10` | Level 5 plus crash-safe tensor dumps |
+| `SGLANG_KERNEL_API_LOGDEST` | `stdout` | Log to stdout |
+|  | `stderr` | Log to stderr |
+|  | `<path>` | Log to file |
+|  | `log_%i.txt` | `%i` expands to process ID |
+| `SGLANG_KERNEL_API_DUMP_DIR` | `<path>` | Directory for level-10 dumps |
+| `SGLANG_KERNEL_API_DUMP_INCLUDE` | wildcard list | Only dump matching API names |
+| `SGLANG_KERNEL_API_DUMP_EXCLUDE` | wildcard list | Skip matching API names |
+
+## Best Practices
+
+### 1. Start with Level 3
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+Level 3 is usually enough to catch wrong shapes, wrong dtypes, and wrong devices.
+
+### 2. Use Level 5 for Numerical Issues
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=5
+```
+
+Use it when you suspect NaN or Inf values.
+
+### 3. Use Level 10 for Crash Reproduction
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+```
+
+This is the most useful mode when the process crashes before you can inspect live tensors.
+
+If you need successful input/output dumps from a real model run, temporarily disable CUDA graph for that debug session.
+
+When level 10 is too noisy, pair it with `SGLANG_KERNEL_API_DUMP_INCLUDE` / `SGLANG_KERNEL_API_DUMP_EXCLUDE` instead of dumping every covered API.
+
+### 4. Log to File for Crashes
+
+```bash
+export SGLANG_KERNEL_API_LOGDEST=crash.log
+```
+
+File logs are safer than stdout when the process aborts.
+
+### 5. Disable Logging in Production
+
+```bash
+unset SGLANG_KERNEL_API_LOGLEVEL
+```
+
+When disabled, the decorator returns the original callable and adds no runtime logging overhead.
+
+## Troubleshooting
+
+### No Logs Appear
+
+Check:
+1. `echo $SGLANG_KERNEL_API_LOGLEVEL`
+2. `echo $SGLANG_KERNEL_API_LOGDEST`
+3. Whether the failing path goes through a covered API boundary
+
+### Too Much Output
+
+Reduce the level:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+### Statistics Are Skipped During CUDA Graph Capture
+
+If you see:
+```text
+statistics=[skipped: CUDA graph capture in progress]
+```
+
+That is expected. Level-5 statistics are intentionally skipped during CUDA graph capture to avoid synchronization side effects.
+
+### Tensor Dumps Are Skipped During CUDA Graph Capture
+
+If you see:
+```text
+Tensor dump skipped: CUDA graph capture in progress
+```
+
+That is also expected. Level-10 dumps require copying tensors to CPU, which is not allowed during CUDA graph capture.

.claude/skills/debug-distributed-hang/SKILL.md

@@ -0,0 +1,248 @@
+---
+name: debug-distributed-hang
+description: Debug hanging issues in SGLang distributed inference (TP/PP/DP/EP). Covers identifying hang locations via py-spy/watchdog/cuda coredump, per-rank logging to find state divergence, binary-search methodology for locating the first diverge point, and fix patterns. Use when a multi-GPU SGLang run hangs, freezes, or times out during collective operations.
+---
+
+# Debugging Distributed Hangs in SGLang
+
+## Overview
+
+Hangs in distributed inference happen when ranks diverge in state, causing collective operations (AllGather, AllReduce, Broadcast, Barrier) to deadlock. Common causes:
+
+- **Size mismatch**: ranks pass different tensor sizes to a collective
+- **Branch divergence**: one rank enters a collective, another skips it
+- **Cascading state drift**: a small non-determinism (e.g., floating-point) propagates into different batch structures
+- **Resource exhaustion**: one rank OOMs or crashes, others wait forever
+
+## Prerequisites
+
+- **py-spy**: `pip install py-spy` or system package. Requires root or `CAP_SYS_PTRACE` to attach to running processes.
+- **cuda-gdb**: Ships with the CUDA toolkit. Ensure it's on your `PATH`.
+
+## Step 1: Confirm and Locate the Hang
+
+### 1a. Watchdog / py-spy
+
+SGLang's watchdog automatically dumps py-spy traces on timeout. Look for:
+
+```
+Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=False)
+```
+
+The py-spy dump shows the stack trace of each thread. The hanging thread is typically blocked in a CUDA synchronize or NCCL collective:
+
+```
+Thread (active): "MainThread"
+    cuStreamSynchronize (libcuda.so)
+    ...
+    forward_extend (model_runner.py)
+```
+
+SGLang has two watchdog modes (see `python/sglang/srt/utils/watchdog.py`):
+- **Hard watchdog** (`soft=False`, default): dumps py-spy traces then sends `SIGQUIT` to kill the parent process.
+- **Soft watchdog** (`soft=True`): only logs the timeout without killing the process, giving you more time to manually attach debuggers or collect coredumps.
+
+If the watchdog doesn't trigger, manually dump:
+
+```bash
+py-spy dump --pid <scheduler_pid>
+```
+
+### 1b. NCCL Debug Logging
+
+```bash
+export NCCL_DEBUG=INFO
+export NCCL_DEBUG_SUBSYS=COLL
+```
+
+Look for the last collective logged before the hang. Mismatched sizes show up as one rank waiting and another never entering.
+
+### 1c. CUDA Coredump
+
+When a process hangs, you can trigger a GPU coredump on demand to see which kernel is stuck. Set these env vars before launching:
+
+```bash
+export CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1
+export CUDA_COREDUMP_PIPE="/tmp/cuda_pipe_%h_%p"
+export CUDA_COREDUMP_FILE="/tmp/cuda_coredump_%h_%p"
+export CUDA_COREDUMP_SHOW_PROGRESS=1
+export CUDA_COREDUMP_GENERATION_FLAGS='skip_nonrelocated_elf_images,skip_global_memory,skip_shared_memory,skip_local_memory,skip_constbank_memory'
+```
+
+While the process is hanging, find the pipe via `/proc/<pid>/fd/` and write to it to trigger the dump:
+
+```bash
+ls /proc/<pid>/fd/ -la 2>/dev/null | grep cuda_pipe
+dd if=/dev/zero bs=1M count=1 > /tmp/cuda_pipe_<hostname>_<pid>
+```
+
+Alternatively, if you don't need to keep the process alive, `kill -SIGABRT <pid>` also triggers a CUDA coredump (but terminates the process).
+
+Then open with `cuda-gdb --batch -ex "target cudacore <coredump_file>"`. On load, it immediately shows which kernel is stuck. For example:
+
+```
+Opening GPU coredump: <coredump_file>
+[Current focus set to CUDA kernel 0, grid 622721, cluster (4,0,0), block (16,0,0), thread (64,0,0), device 0, sm 0, warp 0, lane 0]
+#0  0x00007f8029b2b040 in ncclDevKernel_AllGather_RING_LL(ncclDevKernelArgsStorage<4096ul>)<<<(24,1,1),(512,1,1)>>> ()
+```
+
+This told us the hang was in an NCCL AllGather — not a compute kernel. Combined with the py-spy stack pointing to `LogitsProcessor.forward` → `tensor_model_parallel_all_gather`, we knew it was an AllGather size mismatch between TP ranks.
+
+
+### 1d. Identify the Collective
+
+From the stack traces and logs, identify:
+- Which collective hangs (AllGather, AllReduce, Broadcast)
+- Which code path invokes it (e.g., `LogitsProcessor`, `tensor_model_parallel_all_gather`)
+- Whether it's a size mismatch or a missing participant
+
+## Step 2: Per-Rank Logging
+
+The key technique: each rank writes its own log file so you can diff them.
+
+### Setup Pattern
+
+```python
+import os
+
+_debug_files = {}
+
+def get_debug_file(rank):
+    key = f"rank{rank}"
+    if key not in _debug_files:
+        _debug_files[key] = open(f"/tmp/debug_rank{rank}.log", "w")
+    return _debug_files[key]
+```
+
+Gate logging behind an env var to avoid overhead in production. `SGLANG_DEBUG_HANG` is not a built-in SGLang env var — you need to add this check yourself in the code you're instrumenting:
+
+```python
+if os.environ.get("SGLANG_DEBUG_HANG"):
+    f = get_debug_file(rank)
+    f.write(f"EVENT_NAME key1={val1} key2={val2}\n")
+    f.flush()
+```
+
+### What to Log
+
+Log structured events at key state-mutation points:
+
+```python
+f.write(f"SCHED_BATCH step={step} num_reqs={n} extend_lens={lens}\n")
+f.write(f"VERIFY predict_hash={hash} accept_len={alen}\n")
+f.write(f"CACHE_INSERT rid={rid} num_tokens={n}\n")
+```
+
+Use consistent event names (uppercase prefix) for easy grep/diff.
+
+### Hash Large Tensors
+
+For tensor values, compute a hash instead of dumping raw data:
+
+```python
+import hashlib
+h = hashlib.md5(tensor.cpu().numpy().tobytes()).hexdigest()[:8]
+f.write(f"LOGITS logits_hash={h}\n")
+```
+
+For token ID lists, `str(list).encode()` works:
+
+```python
+h = hashlib.md5(str(tensor.tolist()).encode()).hexdigest()[:8]
+```
+
+### Avoid Implicit Synchronization
+
+`tensor.cpu()`, `tensor.tolist()`, and `tensor.numpy()` all trigger CUDA synchronization. This can:
+- Change timing and mask or move the hang
+- Deadlock if the log point is between two collectives that must run back-to-back
+
+Prefer logging values that are already on CPU (e.g., Python ints, list lengths, request IDs). When you must hash a GPU tensor, do it at a point where the GPU is already idle (e.g., between scheduler steps, not inside a model forward pass).
+
+## Step 3: Diff to Find the Diverge Point
+
+### Basic Diff
+
+```bash
+# Extract specific event type
+grep "^VERIFY" /tmp/debug_rank0.log > /tmp/v_r0.txt
+grep "^VERIFY" /tmp/debug_rank1.log > /tmp/v_r1.txt
+diff /tmp/v_r0.txt /tmp/v_r1.txt | head -20
+```
+
+### Count Events
+
+```bash
+grep -c "^VERIFY" /tmp/debug_rank*.log
+```
+
+If counts differ, one rank executed more iterations — that's already a diverge signal.
+
+### Find First Diverge
+
+The first diff line tells you the exact step where ranks diverge. All lines before it are identical — the root cause is at or before this step.
+
+## Step 4: Binary-Search the Root Cause
+
+Once you find the diverging event, trace backwards:
+
+### 4a. Identify Inputs
+
+For the diverging operation, list all its inputs. Add hash logging for each:
+
+```python
+f.write(
+    f"OP_INPUTS input_a_hash={h_a} input_b_hash={h_b} "
+    f"input_c_hash={h_c} input_d_hash={h_d}\n"
+)
+```
+
+### 4b. Diff Inputs Across Ranks
+
+Compare the hashes. Some inputs will match, some won't. The non-matching input is where divergence entered.
+
+### 4c. Recurse
+
+For the non-matching input, trace where it was produced and repeat: hash its inputs, diff across ranks, find the divergent one. Continue until you reach the root cause.
+
+## Step 5: Common Root Causes and Fixes
+
+### Floating-Point Non-Determinism
+
+**Symptom**: All "logical" inputs are identical (same logits after all-gather), but derived floating-point values (softmax, probabilities) differ across GPUs.
+
+**Example**: EAGLE speculative decoding — `F.softmax` → `top_k_renorm_prob` → `top_p_renorm_prob` produces slightly different `target_probs` on each GPU. The sampling kernel then picks different tokens. These flow into `output_ids` → radix cache → different prefix match depths → different `extend_seq_lens` → AllGather size mismatch → hang.
+
+### Random Number Divergence
+
+**Symptom**: Operations using `torch.rand` produce different values on each rank.
+
+**Fix**: Generate on rank 0 and broadcast, or use a shared seed.
+
+### Conditional Code Paths
+
+**Symptom**: A condition (e.g., memory check, queue length) evaluates differently on different ranks, causing one rank to enter a collective while another skips it.
+
+**Fix**: Synchronize the condition value before branching, or restructure to ensure all ranks take the same path.
+
+### Pipeline Parallel (PP) Send/Recv Mismatch
+
+**Symptom**: In PP setups, one stage issues a `send` that the next stage never `recv`s (or vice versa), causing both to block indefinitely. Unlike TP hangs (collective mismatches), PP hangs typically involve point-to-point operations.
+
+**Fix**: Ensure all stages agree on the number of microbatches and the sequence of send/recv calls for each microbatch.
+
+## Step 6: Verify the Fix
+
+Run the failing test multiple times to confirm the fix is stable. Intermittent hangs require many runs. A test that hung ~30% of the time needs at least 10 clean passes to be confident.
+
+## Quick Reference
+
+| Technique | When to Use |
+|-----------|-------------|
+| py-spy dump | First step — see where each rank is stuck |
+| `NCCL_DEBUG=INFO` | Identify which collective and sizes |
+| CUDA coredump + `cuda-gdb` | See which GPU kernel is blocked |
+| Per-rank log files | Compare rank states over time |
+| Hash of tensors | Efficiently compare large tensors across ranks |
+| `diff` on extracted events | Find the exact step of divergence |
+| `broadcast(result, src=0)` | Fix floating-point or sampling non-determinism |

.claude/skills/env-var-conventions/SKILL.md

@@ -0,0 +1,203 @@
+---
+name: env-var-conventions
+description: Conventions for SGLang environment variables — where to define, how to access, how to name, and how to deprecate. Use when adding, renaming, or reviewing any `SGLANG_*` environment variable (or migrating a legacy `SGL_*` alias), or when touching `python/sglang/srt/environ.py`.
+---
+
+# Environment Variables — Conventions
+
+Apply this skill when adding, renaming, or reviewing any sglang-owned environment variable (`SGLANG_*`, or a legacy `SGL_*` alias being phased out), or when touching `python/sglang/srt/environ.py`.
+
+## Rule 1 — Define in the `Envs` class in `python/sglang/srt/environ.py`
+
+All sglang-owned env vars live as `EnvField` descriptors on the `Envs` class. Never add a new `os.getenv("SGLANG_...")`, `get_bool_env_var("SGLANG_...")`, or `get_int_env_var("SGLANG_...")` call site — the helpers in `python/sglang/srt/utils/common.py` carry an explicit `FIXME: move your environment variable to sglang.srt.environ` and exist only for pre-existing call sites.
+
+Group the new entry under an existing section comment (e.g. `# Logging Options`, `# Scheduler: recv interval`, `# Flashinfer`). Add a new section comment only when none fits — never drop a new entry at the bottom of an unrelated block.
+
+### Decision table: register in `Envs` or use `os.getenv`?
+
+| Variable | Owner | Goes through `Envs`? |
+|---|---|---|
+| `SGLANG_*` | sglang | **Always.** The canonical prefix for all new entries. |
+| `MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` | Upstream/vendor alias that sglang wants to centralize | **Yes** — register in `Envs` so `.get()` / `.override()` work uniformly. Keep the upstream prefix. |
+| `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `HF_HUB_*` (raw upstream) | External tooling | **No.** Read with `os.getenv` — they're set by the launcher / driver, not by sglang. |
+| `RANK`, `LOCAL_RANK`, `WORLD_SIZE`, `MASTER_ADDR`, `MASTER_PORT`, `HOME`, `PATH` | Distributed launcher / OS | **No.** `os.getenv` only. |
+| Test runner internals (`PYTEST_CURRENT_TEST`, etc.) | Test framework | **No.** `os.getenv` only. |
+
+`SGL_*` is **not** a parallel valid prefix — it's a deprecated legacy alias. `_convert_SGL_to_SGLANG` rewrites `SGL_*` to `SGLANG_*` at import time with a `DeprecationWarning`. Never define a new `SGL_*` descriptor or `os.getenv("SGL_...")` call site; if you see one in code, it's tech debt to migrate.
+
+The rule of thumb: if the value's lifecycle is owned by sglang code (we read it, we may want to override it in tests, we may want to rename it), put it in `Envs`. If the value is set by something outside sglang and we only consume it as-is, use `os.getenv`.
+
+## Rule 2 — Pick the typed descriptor
+
+| Type | Use for |
+|---|---|
+| `EnvBool(default)` | boolean flag |
+| `EnvInt(default)` | integer |
+| `EnvFloat(default)` | float |
+| `EnvStr(default)` | string |
+| `EnvTuple(())` | comma-separated list, parsed via `s.split(",")` and stripped |
+
+Default value:
+- For a knob whose "unset" state must be distinguishable from any concrete value, use `None` as the default (e.g. `EnvStr(None)`, `EnvInt(None)`). The descriptor handles set-to-None correctly via `_set_to_none`.
+- For a feature flag, the default encodes the production behavior. `ENABLE_FOO = EnvBool(False)` means foo is off in prod; `DISABLE_FOO = EnvBool(False)` means foo is on in prod. See Rule 4 on picking the verb.
+
+### IntEnum for multi-state knobs
+
+When a knob has more than two discrete states (e.g. an off / soft / strict ladder), define an `IntEnum` next to `Envs` and pass the enum member as the `EnvInt` default. Callers compare against the enum, not raw integers:
+
+```python
+class ToolStrictLevel(IntEnum):
+    OFF = 0
+    FUNCTION = 1
+    PARAMETER = 2
+
+SGLANG_TOOL_STRICT_LEVEL = EnvInt(ToolStrictLevel.OFF)
+
+# At the call site:
+if envs.SGLANG_TOOL_STRICT_LEVEL.get() >= ToolStrictLevel.PARAMETER:
+    ...
+```
+
+Don't pile `SGLANG_ENABLE_FOO_STRICT` / `SGLANG_ENABLE_FOO_OFF` boolean knobs that fight each other — one ordered integer is cleaner.
+
+## Rule 3 — Access via the `EnvField` API, never raw `os.environ`
+
+```python
+from sglang.srt.environ import envs
+
+if envs.SGLANG_FOO.get():
+    ...
+```
+
+`envs.SGLANG_FOO` is the descriptor itself; its `__bool__` and `__len__` raise on purpose so that `if envs.SGLANG_FOO:` fails loudly instead of silently reading as truthy. The `.get()` is mandatory at every read site.
+
+### Full API surface
+
+| Method | Use |
+|---|---|
+| `.get()` | Read value (parsed). Returns `default` if unset, or `None` if explicitly set to None. |
+| `.set(value)` | Set value. `set(None)` flips the internal `_set_to_none` flag so the next `.get()` returns `None`, not `default`. |
+| `.clear()` | Unset entirely. Next `.get()` returns `default`. |
+| `.is_set()` | True iff the key is present in `os.environ` (regardless of value, including the explicit-None case). |
+| `.override(value)` | Context manager — set on enter, restore exactly what was there on exit. Use this in tests. |
+
+The `_set_to_none` distinction matters: `clear()` → `is_set()=False, get()=default`; `set(None)` → `is_set()=True, get()=None`. A few descriptors (e.g. `SGLANG_TEST_MAX_RETRY = EnvInt(None)`, `SGLANG_DISAGGREGATION_THREAD_POOL_SIZE = EnvInt(None)`) rely on this to distinguish "user opted out" from "default applies".
+
+### Test overrides
+
+```python
+with envs.SGLANG_TEST_RETRACT.override(True):
+    ...
+```
+
+Don't mutate `os.environ` directly in tests — `override` restores the original cleanly, including the explicit-None state, even if the block raises.
+
+For multiple overrides composed dynamically, use `ExitStack`:
+
+```python
+with ExitStack() as stack:
+    for name, value in test_envs.items():
+        stack.enter_context(getattr(envs, name).override(value))
+    run_test()
+```
+
+### Subprocess inheritance
+
+`override` mutates the real `os.environ`, so child processes spawned **inside** the `with` block inherit the override. This is the supported way to seed a `subprocess.Popen`:
+
+```python
+with envs.SGLANG_TEST_RETRACT.override(True):
+    subprocess.Popen([...]).wait()
+```
+
+A child started outside the `with` block sees the original value.
+
+### `temp_set_env` is for non-sglang keys only
+
+The module-level `temp_set_env(**env_vars)` helper exists for overriding **non-sglang** env vars (e.g. `CUDA_LAUNCH_BLOCKING`, `NCCL_DEBUG`) in tests. It explicitly rejects `SGLANG_*` / `SGL_*` keys:
+
+```python
+# Wrong — raises ValueError
+with temp_set_env(SGLANG_TEST_RETRACT="true"): ...
+
+# Right — sglang keys go through the descriptor
+with envs.SGLANG_TEST_RETRACT.override(True): ...
+
+# Right — non-sglang keys go through temp_set_env
+with temp_set_env(CUDA_LAUNCH_BLOCKING="1"): ...
+```
+
+The `allow_sglang=True` escape hatch exists for the rare case where you must bypass `Envs` (e.g. setting an env var **name** that's only constructed at runtime); don't use it just to skip writing a descriptor.
+
+## Rule 4 — Naming: `SGLANG_` prefix + verb category
+
+Prefix is always `SGLANG_` for new entries. `SGL_*` is auto-translated to `SGLANG_*` with a `DeprecationWarning` in `_convert_SGL_to_SGLANG`; never add a new `SGL_*` key.
+
+The second token signals intent. Pick the right verb up front — renames require an alias entry (Rule 5).
+
+| Verb | Meaning | Example |
+|---|---|---|
+| `ENABLE_FOO` | Knob that turns feature foo on/off. Default in the `EnvBool` encodes prod behavior. | `SGLANG_ENABLE_TORCH_COMPILE`, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM` |
+| `DISABLE_FOO` | Kill-switch. `DISABLE_FOO=True` turns foo off. | `SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP` |
+| `USE_FOO` | Selects which implementation / backend | `SGLANG_USE_AITER`, `SGLANG_USE_DEEPGEMM_BMM` |
+| `FORCE_FOO` | Overrides autodetection | `SGLANG_FORCE_FP8_MARLIN`, `SGLANG_FORCE_STREAM_INTERVAL` |
+| `LOG_FOO` | Logging-only knob | `SGLANG_LOG_GC`, `SGLANG_LOG_MS` |
+| `TEST_FOO` | Test-only hook | `SGLANG_TEST_RETRACT`, `SGLANG_TEST_MAX_RETRY` |
+| `DEBUG_FOO` | Debug-only instrumentation | `SGLANG_DEBUG_MEMORY_POOL`, `SGLANG_DEBUG_SYMM_MEM` |
+| `OPT_FOO` | Perf-optimization toggle (heavily used by DSV4 work) | `SGLANG_OPT_USE_FUSED_HASH_TOPK`, `SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2` |
+
+Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only forbidden combination is `DISABLE_FOO = EnvBool(True)`, because it produces a true double-negative at the call site (`if not envs.SGLANG_DISABLE_FOO.get():` reads as "if not disabled"). All other combinations are fine:
+
+| Pattern | Call site | Verdict |
+|---|---|---|
+| `ENABLE_FOO = EnvBool(False)` | `if envs.SGLANG_ENABLE_FOO.get():` | OK — opt-in feature |
+| `ENABLE_FOO = EnvBool(True)`  | `if envs.SGLANG_ENABLE_FOO.get():` | OK — on in prod, user opts out via `False` |
+| `DISABLE_FOO = EnvBool(False)` | `if not envs.SGLANG_DISABLE_FOO.get():` | OK — single negation, reads as "if enabled" |
+| `DISABLE_FOO = EnvBool(True)`  | `if not envs.SGLANG_DISABLE_FOO.get():` | **Forbidden** — true double-negative |
+
+`SGLANG_*` is the canonical sglang prefix. Vendor-integration keys (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`) keep their upstream prefix and live in the same `Envs` class — these are integration aliases, not sglang-owned feature flags.
+
+## Rule 5 — Renames go through `*WithAlias` or `_print_deprecated_env`
+
+For a rename where the old key must keep working with a warning:
+
+```python
+SGLANG_DSA_FUSE_TOPK = EnvBoolWithAlias(True, deprecated_name="SGLANG_NSA_FUSE_TOPK")
+```
+
+Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over.
+
+For a full removal where the env var is going away, add to `_convert_SGL_to_SGLANG`:
+
+```python
+_print_deprecated_env("SGLANG_OLD_NAME", "SGLANG_NEW_NAME")  # mapped to a replacement
+_print_deprecated_env("SGLANG_OLD_NAME")                     # no replacement, gone
+```
+
+For env-var to CLI-flag migration, add at module top-level:
+
+```python
+_warn_deprecated_env_to_cli_flag(
+    "SGLANG_FOO",
+    "Please use '--foo' instead.",
+)
+```
+
+Don't silently flip a default during a rename. If the new default disagrees with the old one, that's a behavior change — call it out in the PR body separately from the rename.
+
+## Rule 6 — Env var vs CLI flag
+
+| If the knob is… | Goes in |
+|---|---|
+| User-facing (documented, expected to flip per deployment) | `server_args.py` CLI flag |
+| Expert toggle, A-B kill-switch, vendor integration | `environ.py` env var |
+| Test / debug hook | `environ.py` env var with `TEST_` / `DEBUG_` prefix |
+| Temporary env var rolling out to a CLI flag | env var first, then migrate via `_warn_deprecated_env_to_cli_flag` |
+
+Don't add a CLI flag that just forwards to an env var, and don't add an env var that duplicates an existing CLI flag. Pick one surface.
+
+## Out of scope
+
+- **External / vendor env vars consumed raw** (`HF_HUB_*`, `CUDA_*`, `NCCL_*`, `TORCH_*`, `OMP_*`, `RANK`, `MASTER_ADDR`, etc.): see the decision table in Rule 1 — `os.getenv` is correct, don't pull them into `Envs`.
+- **Pre-existing `get_bool_env_var(...)` / `get_int_env_var(...)` call sites**: leave them as is; new code shouldn't add more, but mass-migration is out of scope for a feature PR.
+- **Upstream-aliased keys already in `Envs`** (`MOONCAKE_*`, `ASCEND_*`, `DEEP_NORMAL_*`, `IS_H200`, `USE_TRITON_W8A8_FP8_KERNEL`, `HF_HUB_DISABLE_XET`, `DISABLE_OPENAPI_DOC` — see Rule 1 decision table): the `SGLANG_` prefix rules in Rule 4 don't apply — the upstream prefix is the canonical name.

.claude/skills/generate-profile/SKILL.md

@@ -0,0 +1,143 @@
+---
+name: generate-profile
+description: Generate an e2e profiling trace of an SGLang server run. Launches a server, validates accuracy, captures a Chrome-compatible trace, and returns the profile path.
+---
+
+# Generate an E2E Profile of an SGLang Server Run
+
+This skill launches an SGLang server, validates it with a quick accuracy test, generates a profiling trace, and returns the profile file path.
+
+## Prerequisites
+
+- A working SGLang installation (`pip install -e .` or equivalent)
+- At least one available CUDA GPU
+
+## Step-by-step Workflow
+
+### Step 1: Launch the server
+
+```bash
+CUDA_VISIBLE_DEVICES=<gpu_id> sglang serve --model-path <model> --port <port> &
+```
+
+- Default model: `Qwen/Qwen3-8B` (good balance of speed and quality)
+- Default port: `30000`
+- The server runs in the background. Save the PID for cleanup.
+- Use the GPU specified by the user's preferences (check memory files for GPU preferences).
+
+### Step 2: Wait for server readiness
+
+Poll the health endpoint until the server is ready:
+
+```bash
+for i in $(seq 1 120); do
+  if curl -s http://127.0.0.1:<port>/health 2>/dev/null | grep -q "ok\|healthy"; then
+    echo "Server ready"
+    break
+  fi
+  sleep 5
+done
+```
+
+The server prints **"The server is fired up and ready to roll!"** to stdout when ready. The health endpoint returns 200 once the server can accept requests.
+
+Typical startup time: 30-90 seconds depending on model size and whether CUDA graphs are being compiled.
+
+### Step 3: Validate accuracy (sanity check)
+
+```bash
+python3 -m sglang.test.run_eval --host 127.0.0.1 --port <port> --eval-name gsm8k --num-examples 20
+```
+
+- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
+- This is a quick sanity check, not a rigorous benchmark.
+- `sglang.test.few_shot_gsm8k` is deprecated; use the unified `run_eval` entrypoint.
+- If you intentionally need the old completion-style GSM8K path, add `--api completion`.
+- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
+
+### Step 4: Generate the profile
+
+```bash
+python3 -m sglang.test.send_one --profile
+```
+
+This command:
+1. Sends a request to the server
+2. Triggers the profiler for 5 steps (default)
+3. Generates a trace file under `/tmp/<timestamp>/`
+4. The trace directory contains:
+   - `<timestamp>-TP-0.trace.json.gz` — Chrome trace format (open in `chrome://tracing` or Perfetto)
+   - `server_args.json` — the server configuration used
+
+**Output format:**
+```
+Dump profiling traces to /tmp/<timestamp>
+```
+
+The profile path is printed to stdout. Parse it from the output.
+
+**Optional flags:**
+- `--profile-steps N` — number of profiling steps (default: 5)
+- `--profile-by-stage` — profile by stage (prefill/decode separately)
+- `--profile-prefix <path>` — custom output prefix
+
+### Step 5: Kill the server
+
+```bash
+pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
+```
+
+Wait a moment and verify no sglang processes remain:
+```bash
+sleep 2 && pgrep -af "sglang serve" || echo "Server killed"
+```
+
+### Step 6: Report the profile path
+
+Return the profile directory path (e.g., `/tmp/1773999986.4769795`) and list its contents so the user knows what files were generated.
+
+## Example Full Run
+
+```bash
+# 1. Launch server
+source cleanup/bin/activate
+CUDA_VISIBLE_DEVICES=1 sglang serve --model-path Qwen/Qwen3-8B --port 30000 &
+
+# 2. Wait for ready
+for i in $(seq 1 120); do
+  curl -s http://127.0.0.1:30000/health | grep -q "ok" && break
+  sleep 5
+done
+
+# 3. Accuracy check
+python3 -m sglang.test.run_eval --host 127.0.0.1 --port 30000 --eval-name gsm8k --num-examples 20
+# Expected: Accuracy > 0.8
+
+# 4. Profile
+python3 -m sglang.test.send_one --profile
+# Output: "Dump profiling traces to /tmp/1773999986.4769795"
+
+# 5. Cleanup
+pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
+sleep 2
+
+# 6. Check output
+ls -la /tmp/1773999986.4769795/
+# 1773999986.4851577-TP-0.trace.json.gz  (Chrome trace)
+# server_args.json                        (server config)
+```
+
+## Customization
+
+- **Different port**: Pass `--port <port>` and use `--host 127.0.0.1 --port <port>` for test commands
+- **Multi-GPU**: Use `--tp <N>` for tensor parallelism; trace files will be generated per TP rank
+- **Longer profile**: Use `--profile-steps 10` for more steps in the trace
+- **Stage profiling**: Use `--profile-by-stage` to separate prefill and decode phases
+
+## Viewing the Profile
+
+Open the `.trace.json.gz` file in:
+- **Perfetto UI**: https://ui.perfetto.dev/ (drag and drop the file)
+- **Chrome tracing**: `chrome://tracing` (load the file)
+
+Both support the gzipped Chrome trace format natively.

.claude/skills/large-class-style/SKILL.md

@@ -0,0 +1,139 @@
+---
+name: large-class-style
+description: 'Code style for SGLang large classes `Scheduler`, `TokenizerManager`, and `ModelRunner`: frozen-code conventions and `__init__` orchestration style. Use when modifying any of these three classes or reviewing changes to them.'
+---
+
+# Code Style for Scheduler / TokenizerManager / ModelRunner
+
+Conventions for SGLang's three large classes:
+
+- `Scheduler` — `python/sglang/srt/managers/scheduler.py`
+- `TokenizerManager` — `python/sglang/srt/managers/tokenizer_manager.py`
+- `ModelRunner` — `python/sglang/srt/model_executor/model_runner.py`
+
+## 1. Frozen Code
+
+- Some core files are **frozen**: *orchestration-only* — a thin composition root that constructs collaborators, wires them, delegates to them, and coordinates the calls. They must stay that way.
+- **Domain logic does not belong in a frozen file**; it lives in a collaborator class in its own module.
+
+### 1.1 Why
+
+- The file is a thin orchestrator over collaborator classes; freezing keeps it that way and stops it growing back into a god class.
+- Keeping domain logic in collaborators (their own files) is what makes per-file code ownership, single responsibility, and unit testing possible.
+- The orchestrator is the composition root: it may know about every collaborator, because wiring and sequencing them is its job. Coordination stays here — domain logic does not.
+
+### 1.2 Frozen files
+
+- `python/sglang/srt/model_executor/model_runner.py`
+
+### 1.3 Allowed: orchestration
+
+Every statement refers to a collaborator and is one of:
+
+1. **Construct** — a short `init_<thing>` helper whose body is essentially a single construction (follows §2); use `maybe_init_<thing>` with a one-line gate when conditional.
+2. **Wire** — a short call that runs the helper from the orchestrator (e.g. in `__init__`).
+3. **Delegate** — calls to a collaborator's methods at the necessary call sites (`self.foo.run(...)`).
+4. **Coordinate** — the minimal control flow that *selects or orders* the above: an `if` choosing whether / which collaborator to wire or call, the order of calls, threading one call's result into the next.
+
+- Heuristic: a statement is allowed only if it constructs, wires, delegates, or selects/orders those — never if it *computes or transforms* a value beyond passing arguments and results through.
+
+```python
+# model_runner.py — orchestration only.
+def init_foo(self):                        # construct
+    self.foo = FooManager(server_args=self.server_args, device=self.device)
+
+self.init_foo()                            # wire (in __init__)
+
+if self.server_args.enable_bar:            # coordinate: select
+    self.bar.prepare(forward_batch)        # delegate
+out = self.foo.run(forward_batch)          # delegate
+self.baz.consume(out)                      # coordinate: thread result into next delegate
+```
+
+### 1.4 Not allowed: domain logic
+
+- Config building, data transformation, algorithm bodies, math, post-processing — any branch or loop that *computes* rather than *coordinates*.
+- It belongs in the collaborator.
+
+```python
+# NOT allowed in a frozen file: domain logic inlined.
+self.foo = None
+if self.server_args.enable_foo:
+    config = build_foo_config(self.model_config, self.device)   # config logic in frozen file
+    self.foo = FooManager(config)                               # inline construction, not via (maybe_)init_foo
+    out = [step(x) for x in batch]                              # computation, not coordination
+```
+
+- Fix: move that body into `FooManager` (its `__init__` or a factory) plus a `(maybe_)init_foo` helper.
+
+### 1.5 Where coordination logic goes
+
+1. **Default: extract.** Pull cohesive coordination into a low-coupling collaborator (an initializer, a forward pipeline) and delegate to it.
+2. **Residue stays.** Coordination that can't be cohesively extracted may remain — but only the minimal **Coordinate** form above, kept pseudocode-readable. This is the explicit exception, not a fallback; note why it stays.
+
+- When the residue outgrows pseudocode, that is the signal to extract a dedicated coordinator — not to keep inlining.
+
+### 1.6 Pass what the collaborator needs, not the god object
+
+- When you extract domain logic into a collaborator (a factory, an initializer, a pipeline), give it the **specific values** it needs — `model_config`, `device`, the sizes — not the whole frozen object (`ModelRunner`, `Scheduler`).
+- Passing the god object back re-creates the coupling the split was meant to remove: the module still reads dozens of attributes off it, can't be unit-tested without building the whole class, and every field rename ripples back in.
+
+- Default to **narrow, keyword args**. Reference shape: `layer_setup.resolve_layer_indices(*, model, model_config, is_draft_worker, spec_algorithm)`.
+- Return a small **frozen struct** and let the orchestrator assign it onto its own fields. The collaborator should not reach back in and mutate the god object.
+- If a leaf genuinely needs the live object — its constructor contract already takes the runner, or it reads state that mutates after init — confine that dependency to the **smallest leaf** and pass narrow args everywhere above it. Note why it can't be narrowed.
+
+### 1.7 If you do pass the god object, keep it read-only
+
+- A callee that genuinely takes the live object should **read** fields off it and **return** results; it writes fields back only when there is genuinely no other way.
+- The orchestrator owns the assignment onto its own fields.
+- Why: a callee that mutates the god object scatters its writes across other modules — you can no longer see what `ModelRunner` owns by reading `model_runner.py`, the hidden writes race with the orchestrator's own ordering, and the callee silently depends on being invoked at exactly the right moment.
+
+```python
+# Good — callee reads the runner and returns a small frozen struct; the orchestrator
+# owns the writes.
+# model_runner.py
+class ModelRunner:
+    def bar(self):
+        self.foo_result = foo(self)
+
+# another_file.py
+def foo(model_runner) -> FooResult:
+    return FooResult(a=xx, b=yy, c=zz)
+
+# Avoid — callee reaches back in and writes the runner's fields.
+# model_runner.py
+class ModelRunner:
+    def bar(self):
+        foo(self)
+
+# another_file.py
+def foo(model_runner):
+    model_runner.a = xx
+    model_runner.b = yy
+    model_runner.c = zz
+```
+
+## 2. `__init__` style
+
+Apply when modifying the `__init__` of the three classes above.
+
+### 2.1 Why
+
+- Downstream forks override one piece (tokenizer, KV cache, IPC, …).
+- Inline logic forces them to copy the whole `__init__`, which rots against upstream.
+- Splitting into `init_*` helpers lets them override exactly what they need.
+- Reference shape: `TokenizerManager.__init__` in `python/sglang/srt/managers/tokenizer_manager.py`.
+
+### 2.2 Rules
+
+- **`__init__` is an orchestrator.** Sequence of `self.init_*(...)` calls + minimal glue. No non-trivial construction inlined.
+- **One helper per overridable unit.** Each `init_*` = one concern a subclass might swap. Don't lump.
+- **Naming:** `init_<thing>` (snake_case, names the component). Conditional construction → `maybe_init_<thing>`, gate inside the helper.
+- **No silent state coupling.** A helper only reads `self.*` set by earlier helpers. Ordering lives in `__init__`. Shared intermediates → pass as args, not via `self.*`.
+- **New logic = new helper.** Default to adding `init_<thing>`, not another inline block. One-line `self.foo = server_args.foo` is fine; structured logic is not.
+- **Preserve override points.** Prefer additive changes to existing `init_*` signatures. Breaking changes → call out in PR.
+
+### 2.3 Scope
+
+- Only the three classes listed above.
+- Not other manager-style classes, not small dataclass/utility constructors.

.claude/skills/llm-torch-profiler-analysis/SKILL.md

@@ -0,0 +1,562 @@
+---
+name: llm-torch-profiler-analysis
+description: "Unified LLM torch-profiler triage skill for `sglang`, `vllm`, `TensorRT-LLM`, and `TokenSpeed`. Use it to inspect an existing `trace.json(.gz)` or profile directory, or to drive live profiling against a running server when supported and return one three-table report with kernel, overlap-opportunity, and fuse-pattern tables."
+---
+
+# Unified LLM Torch Profiler Analysis
+
+## Overview
+
+Use this skill for `torch.profiler` analysis across:
+
+- `sglang`
+- `vllm`
+- `TensorRT-LLM`
+- `TokenSpeed`
+
+There is only one public workflow:
+
+- `triage`
+
+Preferred unified entrypoint:
+
+- [scripts/analyze_llm_torch_profile.py](scripts/analyze_llm_torch_profile.py)
+
+Backwards-compatibility shim (kept so older `docker exec ... analyze_sglang_torch_profile.py ...` calls keep working; it just forwards to the unified entrypoint):
+
+- [scripts/analyze_sglang_torch_profile.py](scripts/analyze_sglang_torch_profile.py)
+
+Markdown bundling helper:
+
+- [scripts/render_triage_markdown_bundle.py](scripts/render_triage_markdown_bundle.py)
+
+`triage` always prints the same three tables:
+
+- kernel table
+- overlap-opportunity table
+- fuse-pattern table
+
+By default, all three tables only render rows at or above `1.0%` cumulative GPU-time share.
+Rows below that are hidden by default unless the user asks for a lower cutoff.
+
+Keep the fuse-pattern table source-backed and deterministic.
+Do not turn it into a fuzzy matcher.
+
+If exact source-backed matching is weak but a kernel cluster is still close to a known family,
+add one short note after the tables with exactly one of:
+
+- `high`
+- `medium`
+- `low`
+
+## Capability Matrix
+
+| Capability | SGLang | vLLM | TensorRT-LLM | TokenSpeed |
+| --- | --- | --- | --- | --- |
+| Existing trace triage | yes | yes | yes | yes |
+| Single-trace live capture | yes | yes, if torch profiler is enabled on server | requires profiler control endpoints | yes, if `/start_profile` and `/stop_profile` are exposed |
+| Two-trace mapping+formal triage | yes | yes | yes | yes |
+| Stage-separated live workload | yes | yes | yes, with a writable shared trace dir or per-stage host runner | yes, via workload-separated HTTP capture |
+| `--profile-by-stage` capture | yes | no | no | no |
+| `--profile-prefix` control | yes | usually ignored on HTTP profiler route | usually ignored on HTTP profiler route | yes, mapped to `profile_id` |
+
+For TensorRT-LLM, live capture only works when the server exposes `/start_profile` and
+`/stop_profile`, and when the deployment already provides a shared trace path plus the
+required env vars.
+
+For TokenSpeed, this skill supports both existing trace triage and live capture
+against current servers that expose `/start_profile` and `/stop_profile`.
+The live helper sends `output_dir`, `activities`, `with_stack`,
+`record_shapes`, and `profile_id` in the start payload. TokenSpeed also has its
+own native `profile_by_stage` field for manual capture, but the unified helper
+uses workload-separated `prefill/` and `decode/` directories by default so the
+tables stay comparable across frameworks.
+
+## Real H100 Validation
+
+The current reference run is the `4x H100` matrix captured on `2026-04-23` on
+`h100_sglang` under:
+
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3`
+
+Rendered markdown bundle:
+
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md`
+
+Validated model directories:
+
+- `mixtral_8x7b_instruct`
+- `qwen2_5_32b_instruct`
+- `qwen3_32b`
+
+Each model directory contains:
+
+- `analysis_sglang.txt`
+- `analysis_vllm.txt`
+- `analysis_trtllm.txt`
+- framework-specific trace roots and probe artifacts
+
+Validated matrix:
+
+| Model | SGLang | vLLM | TensorRT-LLM | Result |
+| --- | --- | --- | --- | --- |
+| `mistralai/Mixtral-8x7B-Instruct-v0.1` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; benchmark probes returned direct, non-empty text |
+| `Qwen/Qwen2.5-32B-Instruct` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; benchmark probes returned direct, non-empty text |
+| `Qwen/Qwen3-32B` | `4x H100` | `4x H100` | `4x H100` | three tables rendered correctly on all three frameworks; vLLM and TensorRT-LLM chat probes often emitted `<think>` prefixes |
+
+Use this run as the main H100 reference.
+The older `2026-04-22` single-card Qwen3 matrix is still useful for bring-up, but it is
+not the default reference anymore.
+TokenSpeed support was added later and is covered by existing-trace triage and
+HTTP profiler-control support, but it is not part of this older H100 validation
+matrix yet.
+
+Stage-separated workload validation captured on `2026-05-01` on `h100_sglang`:
+
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation`
+- `/data/bbuf/validate/unified_llm_profiler_skill/runs/20260501_stage_split_validation_large`
+
+Validated models:
+
+| Model | GPU | Workloads | Result |
+| --- | --- | --- | --- |
+| `Qwen/Qwen2.5-0.5B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate `prefill/*.trace.json.gz` and `decode/*.trace.json.gz`; kernel, overlap, and fuse tables rendered with separate `extend/prefill` and `decode` sections |
+| `Qwen/Qwen2.5-1.5B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate `prefill/*.trace.json.gz` and `decode/*.trace.json.gz`; kernel, overlap, and fuse tables rendered with separate `extend/prefill` and `decode` sections |
+| `Qwen/Qwen2.5-7B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate traces; prefill kernel table captured 28-layer GEMM/FA3/RMSNorm work, decode captured 5-step graph launches, and fuse rows were split by stage |
+| `Qwen/Qwen2.5-14B-Instruct` | `1x H100` | prefill `4090->1`, decode `1->2048` | generated separate traces; prefill kernel table captured 48-layer GEMM/FA3/RMSNorm work, decode captured 5-step graph launches, and fuse rows were split by stage |
+| `Qwen/Qwen3-8B` | `2x H100`, TP=2 | prefill `4090->1`, decode `1->2048`, warmup 10/capture 5 | generated separate prefill/decode traces and all three tables; unique probe prompts avoided prefix-cache pollution in the prefill table |
+| `mistralai/Mistral-7B-Instruct-v0.3` | `2x H100`, TP=2 | prefill `4090->1`, decode `1->2048`, warmup 10/capture 5 | generated separate prefill/decode traces and all three tables; server logs showed no repeated-prompt prefix-cache shortcut during the active prefill window |
+
+This validation also covers the compatibility fix for older SGLang profiler
+state machines: workload-separated live capture labels stages by output
+directory and avoids nesting SGLang's internal `profile_by_stage` state machine
+inside each workload. The helper
+adds one internal scheduler guard step because SGLang increments `forward_ct`
+before checking whether the profiler should stop; without that guard, a
+`num_steps=1` prefill capture can stop just before the actual prefill forward.
+The 2026-05-01 two-card validation artifacts for the additional models are:
+
+- `/data/bbuf/validate/core_skill_validation_20260501/qwen3_8b/profiler`
+- `/data/bbuf/validate/core_skill_validation_20260501/mistral_7b_instruct_v03/profiler`
+
+To render a validated run into one markdown document:
+
+```bash
+python3 scripts/render_triage_markdown_bundle.py \
+  --analysis-root /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3 \
+  --output /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix_v3/h100_large_model_matrix_v3_bundle.md
+```
+
+The bundle groups by model and keeps the three tables for each framework.
+
+H100 notes:
+
+- all three frameworks now render kernel, overlap, and fuse tables with separate `extend/prefill` and `decode` sections when the trace contains a clean stage split
+- SGLang live capture is validated and calls the server profiler API directly instead of shelling out to `sglang.profiler`
+- SGLang trace flush can lag well beyond a few seconds, so the runner waits longer for artifacts than the earlier implementation
+- SGLang kernel-site reconstruction keeps sampling disabled in the mapping path so the optimized parser does not perturb SGLang table output; equality rechecks matched for `Mixtral-8x7B-Instruct-v0.1`, `Qwen3-32B`, and `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`
+- vLLM live capture requires `--output-dir` to match the server `torch_profiler_dir`; the validated H100 flow uses `--profiler-config {"profiler":"torch","torch_profiler_dir":"..."}` and then drives `/start_profile` and `/stop_profile`
+- TensorRT-LLM validation stays on `--backend pytorch`; the H100 flow writes the trace with `TLLM_TORCH_PROFILE_TRACE` and then analyzes the saved trace
+- TensorRT-LLM current mainline was rechecked at `0722c5f47d2cae69ac1a237da51e550dd214532c` on 2026-06-26; the latest delta affects KV eviction / block-offset staging rather than profiler trace controls, so the `b9e1945` profiler evidence still applies: PyTorch profiling uses `record_shapes=True` and `with_modules=True`, but not `with_stack=True`; keep the override path for table-quality Python locations unless the target image proves otherwise
+- TokenSpeed trace analysis has first-class registry rows for native TokenSpeed CuTe DSL MLA, MLA KV pack + FP8 quantize, fused top-k/top-p sampling, persistent lm_head GEMM, and NVFP4 GEMM + SwiGLU + quant; live capture still requires an existing torch-profiler trace until the target TokenSpeed image exposes a supported profiler API
+- on this host, keep all trace roots under `/data/...`, not `/home/...`
+
+## When To Use It
+
+- inspect a `torch.profiler` trace or profile directory from `sglang`, `vllm`,
+  `TensorRT-LLM`, or `TokenSpeed`
+- profile a live serving endpoint and analyze the result
+- summarize which kernel families dominate prefill or decode
+- map kernels back to Python code paths
+- judge whether a code path still leaves overlap opportunity
+- check whether an already-known fusion or overlap path should have applied
+
+## Diffusion Backend Gate
+
+For diffusion benchmark or profiling work, only analyze traces produced by the native
+SGLang diffusion backend.
+
+If the run that generated the trace logs any of:
+
+- `Falling back to diffusers backend`
+- `Using diffusers backend`
+- `Loaded diffusers pipeline`
+
+stop the workflow instead of analyzing the trace.
+Handle it as a backend-selection issue, not as native-kernel profiler evidence.
+
+## Main Flows
+
+## Stage-Separated Live Capture Contract
+
+Live capture must not use one mixed prompt as the default.
+By default, `analyze_llm_torch_profile.py --url ...` captures two labeled
+workloads and then renders the same three tables with separate stage sections:
+
+- prefill: synthetic input length `4090`, output length `1`
+- decode: synthetic input length `1`, output length `2048`
+
+Every live profiler path warms up `10` steps before arming the profiler and then
+captures `5` active steps by default. Keep this warmup/active split aligned
+across SGLang, vLLM, and TensorRT-LLM before comparing kernel tables.
+
+Use these options to override the contract when the benchmark workload is known:
+
+```bash
+--profile-workload both \
+--warmup-steps 10 --num-steps 5 \
+--prefill-input-len 4090 --prefill-output-len 1 \
+--decode-input-len 1 --decode-output-len 2048
+```
+
+Allowed `--profile-workload` values:
+
+- `both`: default; capture prefill and decode separately
+- `prefill`: capture only the long-input / one-token workload
+- `decode`: capture only the one-input / long-output workload
+- `legacy`: keep the old `--probe-prompt` / `--probe-max-new-tokens` behavior
+
+For `sglang-sota-humanize-loop`, do not use the defaults if the slow SGLang
+benchmark scenario has a known input/output distribution.
+Set the profiler lengths from that slow scenario instead: prefill uses the slow
+input length with output `1`, and decode uses input `1` with the slow output
+length. For a mixed dataset, profile the slowest representative bucket such as
+the p50 or p95 input/output pair used in the benchmark report, and record the
+bucket in the artifact notes.
+
+### 1. Single-trace triage from an existing profile dir or trace
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --input /path/to/profile_dir_or_trace.json.gz
+```
+
+Use this when one trace is enough.
+The overlap table stays conservative in single-trace mode and will tell you when a
+mapping/formal pair is needed.
+
+### 2. Single-trace live capture from SGLang
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --framework sglang \
+  --url http://127.0.0.1:30000 \
+  --output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/sglang_profile_live \
+  --num-steps 5 \
+  --warmup-steps 10 \
+  --profile-by-stage \
+  --profile-workload both
+```
+
+The script sends `POST /start_profile` to the SGLang server directly.
+Keep `--output-dir` under `/data/...` so later analysis and docs can see the trace.
+The script writes `server_args.json`, warms up with the same workload shape,
+sends the active probe requests after profiling is armed, captures separate
+`prefill/` and `decode/` profile roots by default, and waits longer for trace
+flush than the earlier implementation.
+For the default workload-separated capture, the directory name labels the stage
+and the SGLang internal `profile_by_stage` mode is not used inside each
+workload. This avoids mixing a one-token prefill probe with a separate decode
+profile. The helper still adds one internal guard step because older SGLang
+profilers check the target counter before running the next forward.
+
+### 3. Single-trace live capture from vLLM
+
+Launch vLLM with torch profiler enabled, for example:
+
+```bash
+vllm serve meta-llama/Llama-3.1-8B-Instruct \
+  --profiler-config '{"profiler":"torch","torch_profiler_dir":"/data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile"}'
+```
+
+Then run:
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --framework vllm \
+  --url http://127.0.0.1:8000 \
+  --output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/vllm_profile \
+  --num-steps 5 \
+  --warmup-steps 10 \
+  --no-profile-by-stage \
+  --profile-workload both
+```
+
+For vLLM, `--output-dir` must point to the same `torch_profiler_dir` the server uses.
+The current vLLM profiler config already defaults `torch_profiler_with_stack=true`,
+so the runner only needs to set `torch_profiler_dir`.
+On `h100_sglang`, external vLLM containers should mount both:
+
+- `/data/.cache/huggingface:/root/.cache/huggingface`
+- `/data/bbuf/validate/unified_llm_profiler_skill:/data/bbuf/validate/unified_llm_profiler_skill`
+
+### 4. Single-trace live capture from TensorRT-LLM
+
+Use this only when the server exposes `POST /start_profile` and `POST /stop_profile`,
+and the trace path is shared with the current machine.
+
+Typical env expectations are:
+
+- `TLLM_PROFILE_START_STOP=<start>-<stop>` such as `10-20`
+- `TLLM_TORCH_PROFILE_TRACE=/shared/path/trace.json` or `.json.gz`
+
+Then run:
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --framework trtllm \
+  --url http://127.0.0.1:8000 \
+  --output-dir /shared/path \
+  --num-steps 5 \
+  --no-profile-by-stage \
+  --profile-workload both
+```
+
+If the deployment does not expose the profiler control endpoints, fall back to analyzing
+an existing trace instead of trying live capture.
+If the TensorRT-LLM trace output is configured as one fixed file path, use
+`scripts/run_trtllm_pytorch_profile_host.sh --stage prefill` and `--stage decode`
+instead of direct `--profile-workload both`, so each stage gets its own trace file.
+
+On the current TensorRT-LLM mainline path, `py_executor.py` creates the torch profiler
+with `record_shapes=True` and `with_modules=True` but not `with_stack=True`.
+For table-quality validation, use the override generator:
+
+```bash
+python3 scripts/make_trtllm_py_executor_override.py \
+  --source /path/to/original/py_executor.py \
+  --output /data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm/py_executor_with_stack.py
+```
+
+The matrix runner does this automatically on H100 before TensorRT-LLM capture starts.
+
+This is the validated TensorRT-LLM flow on `h100_sglang`:
+
+1. launch `trtllm-serve` with `TLLM_PROFILE_START_STOP=<start>-<stop>` and `TLLM_TORCH_PROFILE_TRACE=/data/.../trace.json`
+2. run a few benchmark requests
+3. analyze the emitted trace with `--input /data/.../trace.json`
+
+### 5. Single-trace live capture or triage from TokenSpeed
+
+For a running TokenSpeed server that exposes the profiler routes, the unified
+helper can drive live capture:
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --framework tokenspeed \
+  --url http://127.0.0.1:8000 \
+  --output-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example/tokenspeed_profile \
+  --num-steps 5 \
+  --warmup-steps 10 \
+  --no-profile-by-stage \
+  --profile-workload both \
+  --profile-prefix ts-triage
+```
+
+The helper sends `POST /start_profile` with:
+
+- `output_dir`: the `--output-dir` path
+- `activities`: `["CPU", "GPU"]`
+- `with_stack`: `true`
+- `record_shapes`: `false`
+- `profile_id`: `--profile-prefix`, with `-prefill` or `-decode` appended during workload-separated capture
+
+It then sends OpenAI-compatible probe requests and calls `POST /stop_profile`.
+TokenSpeed writes files such as `ts-triage-prefill-TP-0.trace.json.gz` under the
+output directory. If the server was launched with multiple TP ranks, expect one
+trace per rank.
+
+Existing TokenSpeed torch-profiler traces can still be analyzed directly:
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --framework tokenspeed \
+  --input /path/to/tokenspeed_profile_dir_or_trace.json.gz
+```
+
+TokenSpeed's own manual profiler control surface can also be used:
+
+```bash
+curl -X POST http://127.0.0.1:8000/start_profile \
+  -H 'Content-Type: application/json' \
+  -d '{"output_dir":"/data/bbuf/profiles/tokenspeed","activities":["CPU","GPU"],"with_stack":true,"record_shapes":false,"profile_id":"ts-manual"}'
+
+# send representative workload here
+
+curl -X POST http://127.0.0.1:8000/stop_profile
+```
+
+For server-side automatic stop, pass `num_steps`. For TokenSpeed-native
+EXTEND/DECODE split, pass `profile_by_stage: true`; this produces files with
+stage suffixes such as `-EXTEND` and `-DECODE`.
+
+TokenSpeed's benchmark driver can capture traces too:
+
+```bash
+tokenspeed bench serve \
+  --base-url http://127.0.0.1:8000 \
+  --model <model> \
+  --dataset-name random \
+  --random-input-len 4090 \
+  --random-output-len 1 \
+  --num-prompts 64 \
+  --profile \
+  --profile-num-steps 5 \
+  --extra-body '{"output_dir":"/data/bbuf/profiles/tokenspeed","activities":["CPU","GPU"],"with_stack":true,"profile_id":"ts-bench"}'
+```
+
+If `output_dir` is omitted, TokenSpeed falls back to `TOKENSPEED_PROFILER_DIR`
+and then `/tmp`.
+
+Use [scripts/probe_llm_server.py](scripts/probe_llm_server.py) with
+`--framework tokenspeed` for a small OpenAI-compatible endpoint probe before or
+after trace collection:
+
+```bash
+python3 scripts/probe_llm_server.py \
+  --framework tokenspeed \
+  --url http://127.0.0.1:8000 \
+  --requests 6 \
+  --max-tokens 48
+```
+
+For `sglang-sota-humanize-loop`, keep TokenSpeed profiler evidence aligned to
+the same slow scenario bucket as the benchmark result. Prefer the unified
+workload-separated live capture when possible; if only a mixed agentic trace is
+available, label that limitation in `analysis/root-cause.md` before comparing
+it to SGLang prefill/decode traces.
+
+### 6. Two-trace triage from existing profile dirs or traces
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --mapping-input /path/to/graph_off_profile_dir \
+  --formal-input /path/to/graph_on_profile_dir
+```
+
+Use this when you need stronger overlap attribution and kernel-to-source mapping.
+
+### 7. Two-trace triage from running servers
+
+```bash
+python3 scripts/analyze_llm_torch_profile.py \
+  --framework sglang \
+  --mapping-url http://127.0.0.1:31025 \
+  --formal-url http://127.0.0.1:31026 \
+  --num-steps 5 \
+  --profile-by-stage
+```
+
+For `vllm` or `TensorRT-LLM`, use the same shape but pass:
+
+- `--framework vllm` or `--framework trtllm`
+- `--mapping-output-dir ...`
+- `--formal-output-dir ...`
+- `--no-profile-by-stage`
+
+For TokenSpeed, either use `--mapping-url` and `--formal-url` against servers
+that expose `/start_profile` and `/stop_profile`, or pass two existing trace
+directories with `--mapping-input` and `--formal-input`.
+
+## `profile_by_stage`
+
+`--profile-by-stage` is only meaningful on the SGLang live-capture path.
+
+- With `--profile-workload both` / `prefill` / `decode`, workload directories
+  are the stage labels; the live-capture helper disables SGLang's internal
+  stage profiler per workload, warms up first, and captures the requested
+  active step count for the selected workload.
+- On legacy or hand-captured SGLang serving, internal `profile_by_stage` is
+  still useful because prefill and decode usually have very different
+  bottlenecks.
+- On the current profile-v2 path inside SGLang, stage-based profiling is effectively the normal path.
+- PD-disaggregated serving adds one extra rule: prefill workers and decode workers must be profiled separately. That is stricter than ordinary `profile_by_stage`.
+- For `vllm`, `TensorRT-LLM`, and `TokenSpeed`, disable it with
+  `--no-profile-by-stage`.
+
+## How To Choose The Triage Shape
+
+### Single-trace triage
+
+Use when you want the lowest-friction report:
+
+- one trace is already available
+- you mainly want kernel share and fusion clues
+- you are comparing two runs side by side by running triage once per trace
+
+Prefer this by default.
+
+### Two-trace triage
+
+Use when you need:
+
+- a stronger overlap answer
+- graph-off source mapping plus graph-on final behavior
+- more trustworthy overlap recommendations in the middle table
+
+1. mapping trace with graph disabled or with the lower-fusion / more-readable config
+2. formal trace with the real serving optimizations enabled
+
+Do not call the mapping pass a "fast profile".
+It exists to recover `kernel -> cpu_op -> python scope`.
+
+## Workflow
+
+### Single-trace workflow
+
+1. If the user only wants a diagnosis, one trace is enough.
+2. Prefer one-rank traces over merged traces whenever the profiler emitted both.
+3. For a live server, let the script drive the profiler only when the framework-specific prerequisites are already met.
+4. Prefer `--profile-workload both`; use `legacy` only when reproducing an old trace contract.
+5. Prefer workload-separated SGLang capture; use internal `--profile-by-stage`
+   mainly for `legacy` or manually collected traces.
+6. When on `h100_sglang`, create or clean the target trace directory through `docker exec sglang_bbuf ...` so the path is definitely writable under `/data`.
+
+### Two-trace workflow
+
+1. Produce a mapping trace first with graph disabled or the lower-fusion configuration.
+2. Produce a formal trace second with the real serving optimizations enabled.
+3. Run `triage` for the three-table report.
+4. Read the results in this order:
+   - kernel table
+   - overlap-opportunity table
+   - fuse-pattern table
+5. Before calling something a "new" optimization idea, compare the top rows against both [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md) and [references/overlap-catalog.md](references/overlap-catalog.md). Check mainline rows first, then the `PR-backed / in-flight` sections. Prefer reporting:
+   - an existing fused or overlap path that should already apply here
+   - an existing path that appears disabled, unsupported, or regressed in this trace
+   - an upstream pattern that is mainline elsewhere but missing locally, or still open upstream
+   - a truly new opportunity only when no catalog entry fits
+6. If no exact pattern fully matches but the trace is still close to a known family, add one flat similarity note after the tables.
+   Use `high`, `medium`, or `low` only.
+   Base that note on the full pattern shape, not on one kernel name alone.
+   Prefer semantic cues such as producer-consumer chain, source locations, CPU op names, TP context, and model-specific structure.
+   Do not rewrite the script table itself to include these heuristic judgments.
+
+## References
+
+Load these only when needed:
+
+- [references/source-map.md](references/source-map.md)
+  - upstream SGLang profiler entrypoints and trace-writing paths; still most useful for SGLang-specific source follow-up
+- [references/heuristics.md](references/heuristics.md)
+  - overlap labels, dependency-risk interpretation, and limits
+- [references/fuse-overlap-catalog.md](references/fuse-overlap-catalog.md)
+  - mixed source-backed catalog of existing fuse and overlap patterns, including mainline rows plus PR-backed / in-flight rows
+- [references/vllm-torch-compile-fusions.md](references/vllm-torch-compile-fusions.md)
+  - current vLLM torch.compile fusion passes and the source patterns they target
+- [references/overlap-catalog.md](references/overlap-catalog.md)
+  - overlap-only lookup table across LLM, VLM, diffusion, disaggregation, HiSparse, and speculative scheduling
+
+## Output Contract
+
+Return:
+
+- trace path or generated profile path
+- framework
+- model/server args when available
+- kernel table
+- overlap-opportunity table
+- fuse-pattern table
+- optional similarity note with `high` / `medium` / `low` when exact matching is inconclusive
+- one short summary of what dominates the run
+- whether the overlap read came from single-trace triage or mapping/formal two-trace triage

.claude/skills/llm-torch-profiler-analysis/references/fuse-overlap-catalog.md

@@ -0,0 +1,399 @@
+# Fuse And Overlap Catalog
+
+This catalog is the source-backed lookup table that the profiler skill should
+consult before labeling a fuse or overlap opportunity as novel.
+
+For overlap-only triage, also load `references/overlap-catalog.md`.
+
+This revision is intentionally kernel-scoped. Keep rows here only when they map
+to one fused GPU/NPU kernel family, one fused collective-plus-kernel family, or
+one profiler-visible stream overlap among GPU kernels / collective kernels.
+Host-only scheduler, event-loop, executor, offload, and load-path patterns are
+intentionally excluded.
+
+Use it like this:
+
+1. Start from the three `triage` tables.
+2. Match top rows against the `Trace keywords` and `Primary code` columns below.
+3. If a finding matches an existing row, report it as:
+   - an existing optimization path that is missing, disabled, regressed, or unsupported for the current backend, or
+   - an already-known family that should be re-applied to the current model shape.
+4. Check the mainline comparison sections and the `PR-backed / in-flight` sections too. If a match exists there, do not call it novel; call it an upstream or in-flight pattern instead.
+5. Only call a finding "new" when it does not fit any mainline or PR-backed row in this catalog.
+
+The `vLLM-origin` sections below are comparative references. They are not
+necessarily present in the checked-out `sglang` tree, but they should still be
+treated as upstream or analogous kernel families before labeling a fuse or
+overlap opportunity as novel.
+
+The catalog is grouped by reusable optimization family, not by one specific model.
+
+Refresh note `2026-06-26`: rechecked official main heads for SGLang
+`8524678889485801e7a4a12d62015be0c68f7a90`, vLLM
+`abc71548ef029132c3316b902207f254a246d593`, TensorRT-LLM
+`0722c5f47d2cae69ac1a237da51e550dd214532c`, and TokenSpeed
+`5aedf69d6b476baa65571011de6ea60fd5a238a8`. The vLLM torch.compile pass
+inventory is split out in
+[`vllm-torch-compile-fusions.md`](vllm-torch-compile-fusions.md). Stable
+current-code families remain folded into the mainline rows below. This refresh
+adds first-class TokenSpeed-origin rows for CuTe DSL MLA, MLA KV pack+FP8
+quantize, sampling, lm_head GEMM, and NVFP4 GEMM+SwiGLU+quant, plus the latest
+SGLang LTX2 Ada-value diffusion fusion. Recheck PR state before treating an
+in-flight row as shipped.
+
+## 1. LLM / SRT fused-kernel families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Fused residual add + RMSNorm | `fused_add_rmsnorm*`<br>`npu_add_rms_norm`<br>`add_rmsnorm_bias`<br>`gemma_fused_add_rmsnorm`<br>`gemma_rmsnorm_residual_scalar`<br>`_gemma_rmsnorm_residual_kernel`<br>residual add right before norm | `python/sglang/srt/layers/layernorm.py`<br>`python/sglang/srt/layers/gemma4_fused_ops.py`<br>`python/sglang/srt/layers/quantization/modelslim/modelslim.py` | Shared CUDA / ROCm / CPU / NPU fused add-RMSNorm implementations, including Gemma, Gemma4 scalar-residual, and NPU-bias variants | Treat split residual add + RMSNorm as an existing cross-backend fusion first, not a new idea. |
+| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/communicator.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
+| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/communicator.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
+| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
+| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
+| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/jit_kernel/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
+| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
+| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
+| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/jit_kernel/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
+| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
+| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/jit_kernel/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
+| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
+| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/jit_kernel/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
+| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
+| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
+| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
+| NPU fused split-QKV + RMSNorm + RoPE | `split_qkv_rmsnorm_rope` | `python/sglang/srt/models/llama.py`<br>`python/sglang/srt/models/qwen3.py`<br>`python/sglang/srt/models/qwen3_moe.py`<br>`python/sglang/srt/models/glm4_moe.py` | Ascend path fuses QKV split, Q / K RMSNorm, and RoPE in one op | On NPU traces, separate split / norm / rope kernels usually mean the fused path is unavailable or bypassed. |
+| Fused FP8 quantize + paged KV cache write | `trtllm_fp8_kv_kernel`<br>`fp8 kv cache write`<br>`paged KV cache write` | `python/sglang/srt/layers/attention/triton_ops/trtllm_fp8_kv_kernel.py` | TRTLLM MHA path fuses FP8 quantization, scale computation, and paged K / V cache write | If FP8 KV cache traces show standalone quant plus write kernels, first compare against this existing Triton fuse. |
+| Fused MLA KV cache write + FP8 quant | `set_mla_kv_buffer_fp8_quant*`<br>`set_mla_kv_buffer_triton_fp8_quant` | `python/sglang/srt/mem_cache/utils.py`<br>`python/sglang/srt/mem_cache/memory_pool.py` | MLA / NSA KV pool path can quantize K and write directly into KV storage without a separate concat-and-quant chain | Treat standalone quant + KV-buffer write on MLA paths as missing existing fusion first. |
+| Fused MoE router / top-k / softcapping | `FusedMoeRouter`<br>`fused_moe_router*`<br>router GEMM + `topk` + `tanh` | `python/sglang/srt/layers/moe/router.py` | Single fused router kernel covers router matmul, softcapping, and top-k selection | Treat exposed router matmul + softcap + top-k chains as an existing MoE fusion family. |
+| Fused MoE grouped-topk / gate kernels | `fused_topk_deepseek`<br>`moe_fused_gate`<br>`aiter_fused_topk`<br>`kimi_k2_moe_fused_gate` | `python/sglang/srt/layers/moe/topk.py` | CUDA / ROCm / FlashInfer kernels fuse bias, grouped-topk, renorm, and routed scaling into one gate op | Check backend / model eligibility before proposing a novel router-gate fusion. |
+| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
+| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
+| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
+| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/jit_kernel/nvfp4.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/fused_moe_method_npu.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
+| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
+| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. |
+| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/jit_kernel/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
+| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
+| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
+| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/jit_kernel/triton/gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
+| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/srt/layers/attention/fla/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
+| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
+| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
+| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
+| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/srt/layers/attention/fla/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/srt/layers/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
+| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
+| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
+
+## 2. LLM / SRT kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Single-batch overlap (SBO) | MoE combine, down-gemm, shared-expert work in nearby two-stream windows | `python/sglang/srt/batch_overlap/single_batch_overlap.py` | combine vs down-gemm overlap, combine vs shared-expert overlap, one-stream dispatch+shared overlap, explicit SM partitioning and events | If exposed MoE combine sits near neighboring compute, classify it against SBO before calling it new overlap. |
+| Q and K normalization on different streams | Q-side norm and K-side norm on different streams | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/srt/models/qwen3.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py` | Q stays on current stream, K can run on `alt_stream` in capture mode | Treat split Q / K norm as an existing overlap family when `alt_stream` is already wired. |
+| DeepSeek shared-expert / routed-expert overlap | shared-expert GEMMs near DeepEP dispatch / combine | `python/sglang/srt/models/deepseek_v2.py`<br>`python/sglang/srt/batch_overlap/single_batch_overlap.py` | shared experts on `alt_stream`, overlap with dispatch / combine and down-gemm, Blackwell-specific env gating | This is an established routed-vs-shared branch overlap pattern, not a novel idea. |
+| Llama4 shared branch vs routed branch overlap | shared expert branch plus routed MoE branch as adjacent windows | `python/sglang/srt/models/llama4.py` | shared expert on current stream, router + topk + routed experts on `alt_stream` | Use Llama4 as the first precedent for branch-level overlap in similar sparse models. |
+| ExaoneMoE shared experts vs router experts overlap | shared expert output and router-expert output form a two-branch window | `python/sglang/srt/models/exaone_moe.py::forward_normal_dual_stream` | shared experts on current stream, router + routed experts on `alt_stream`, explicit join before combine | This is an existing dual-stream MoE overlap family. |
+| Grok residual-MoE branch overlap | dense MLP and block-sparse MoE branches in parallel | `python/sglang/srt/models/grok.py::moe_with_rmoe` | dense MLP on current stream, MoE on `alt_stream`, fused dual residual RMSNorm around boundaries | Treat exposed Grok branch overlap as an existing pattern. |
+| NSA dual-stream overlap | Q-proj, K-proj, RoPE, cache-store, quantization in tight two-stream windows | `python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Q / K projection split, RoPE split, cache-store vs quantization overlap | NSA already contains several dual-stream overlap precedents. |
+| MoriEP async dispatch / combine comm stream | `MoriEP`<br>`_comm_stream`<br>`dispatch`<br>`combine`<br>`done_event` | `python/sglang/srt/layers/moe/token_dispatcher/moriep.py` | MoriEP can submit dispatch and combine onto a dedicated communication stream and synchronize only through events | Treat MoriEP comm / compute interleave as an existing MoE overlap family. |
+| Heterogeneous-TP staging scatter overlap | `scatter_stream`<br>`_scatter_stream`<br>`staging` | `python/sglang/srt/disaggregation/common/staging_handler.py`<br>`python/sglang/srt/disaggregation/common/staging_buffer.py` | decode-side staging scatter kernels can run on a dedicated stream while forward continues on the main stream | If decode traces show staging scatter kernels adjacent to forward kernels, classify them against this existing overlap family first. |
+| Generic `alt_stream` overlap families | `alt_stream` plus explicit `wait_stream` / `with torch.cuda.stream(...)` | `qwen2_moe.py`<br>`qwen3_moe.py`<br>`glm4_moe.py`<br>`bailing_moe.py`<br>`llada2.py`<br>`grok.py`<br>`olmo2.py`<br>`step3p5.py`<br>`longcat_flash.py`<br>`falcon_h1.py` | model-specific overlap on attention prep, MoE branches, or cache-store | Search these families before designing a new overlap scheme from scratch. |
+
+## 3. VLM-specific kernel families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Vision QK norm with aux stream | vision-side QK norm or norm-like kernels before attention | `python/sglang/srt/layers/attention/vision.py` | vision QK normalization can call shared `apply_qk_norm(...)`, with K-side work on `aux_stream` | If vision QK prep is split, first check this existing aux-stream path. |
+| ViT CUDA graph disables vision aux stream | expected vision overlap is absent under ViT graph | `python/sglang/srt/models/internvl.py`<br>`python/sglang/srt/layers/attention/vision.py`<br>`python/sglang/srt/environ.py::SGLANG_VIT_ENABLE_CUDA_GRAPH` | vision `aux_stream` is intentionally disabled when ViT CUDA graph is on | Missing vision overlap may be intentional, not a regression. |
+| Fused multimodal RoPE kernel | `triton_mrope_fused`<br>`multimodal_rotary_embedding_cpu`<br>`npu_mrope`<br>`MRotaryEmbedding` | `python/sglang/srt/layers/rotary_embedding/mrope.py`<br>`python/sglang/srt/layers/rotary_embedding/triton_kernels.py`<br>`python/sglang/srt/models/qwen3.py` | CUDA Triton, CPU `sgl_kernel`, and NPU paths already fuse multimodal t / h / w position lookup plus in-place Q / K rotary application | If VLM traces show separate mRoPE gather / shuffle / apply steps, first classify them as a missing existing mRoPE fusion. |
+
+## 4. Diffusion fused-kernel families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Fused residual + norm + scale + shift | residual add, norm, scale, shift, gate around DiT blocks | `python/sglang/jit_kernel/diffusion/cutedsl/scale_residual_norm_scale_shift.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py` | `fused_scale_residual_norm_scale_shift(...)` | Treat split residual + norm + modulation as a missing existing diffusion fusion first. |
+| Fused norm + scale + shift | norm followed by scale / shift elementwise kernels | `python/sglang/jit_kernel/diffusion/cutedsl/scale_residual_norm_scale_shift.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py` | `fused_norm_scale_shift(...)` | Existing modulation fusion already covers this family. |
+| Triton scale / shift and gate-select kernels | tiny scale / shift or gate-select kernels dominate modulation blocks | `python/sglang/jit_kernel/diffusion/triton/scale_shift.py`<br>`python/sglang/multimodal_gen/runtime/layers/elementwise.py` | `fuse_scale_shift_kernel(...)` and `fuse_layernorm_scale_shift_gate_select01_kernel(...)` | Check whether the runtime is missing these existing Triton fusions. |
+| Fused add-RMSNorm and one-pass RMSNorm | residual add plus RMSNorm still split on short hidden sizes | `python/sglang/multimodal_gen/runtime/layers/layernorm.py`<br>`python/sglang/jit_kernel/diffusion/triton/rmsnorm_onepass.py` | `fused_add_rmsnorm(...)` and `triton_one_pass_rms_norm(...)` | For short hidden-size diffusion blocks, this is already an established fusion family. |
+| Fused diffusion QK norm + RoPE | split QK norm and RoPE in diffusion attention blocks | `python/sglang/jit_kernel/diffusion/qknorm_rope.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py::apply_qk_norm_rope` | `fused_inplace_qknorm_rope(...)`, with fallback to QK norm plus `apply_flashinfer_rope_qk_inplace(...)` | Distinguish between missing fused qknorm + rope and the existing FlashInfer RoPE fallback. |
+| Z-Image fused `norm(x) * tanh(scale) + shift` | `fused_norm_tanh_mul_add`<br>`tanh(gate) * rmsnorm(x)` | `python/sglang/jit_kernel/diffusion/cutedsl/norm_tanh_mul_add_norm_scale.py`<br>`python/sglang/multimodal_gen/runtime/layers/layernorm.py` | CuTeDSL kernel plus runtime helper for Z-Image residual-form modulation | Treat split Z-Image residual-form modulation as a missing existing diffusion fusion, not a novel idea. |
+| Z-Image fused residual modulation + next norm-scale | `fused_norm_tanh_mul_add_norm_scale`<br>`residual + tanh(gate) * rmsnorm(x)`<br>`ffn_norm1(x) * scale_mlp` | `python/sglang/jit_kernel/diffusion/cutedsl/norm_tanh_mul_add_norm_scale.py`<br>`python/sglang/multimodal_gen/runtime/models/dits/zimage.py` | One CuTeDSL kernel fuses the first residual-form modulation and the next normalization / scale stage | If you see this chain split in Z-Image traces, report it as a missing existing mainline fusion family. |
+| LTX2 fused Ada values | `ltx2_ada_values9`<br>`get_ada_values`<br>`scale_shift_table + timestep.reshape` | `python/sglang/jit_kernel/diffusion/triton/ltx2_ada_values.py`<br>`python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py` | PR `#29390` fuses LTX-2.3 Ada value materialization for video/audio streams and reuses the 9 Ada tensors across self-attention, MLP, and prompt-cross-attention blocks | Treat repeated Ada add/reshape/slice ladders in LTX2 traces as a missing shipped SGLang fusion first. |
+| LTX2 residual-gate add | `diffusion_residual_gate_add`<br>`residual_gate_add`<br>`residual + update * gate` | `python/sglang/jit_kernel/diffusion/residual_gate_add.py`<br>`python/sglang/jit_kernel/csrc/diffusion/residual_gate_add.cuh`<br>`python/sglang/multimodal_gen/runtime/models/dits/ltx_2.py` | PR `#29361` fuses LTX2 `residual + update * gate` sites for attention, cross-attention, and feed-forward updates into one CUDA custom op when dtype, shape, device, and contiguity guards pass | Treat split add/mul gate ladders in LTX2 traces as a missing shipped SGLang fusion first. |
+| Nunchaku fused GELU MLP | `_fused_gelu_mlp`<br>`fused_gelu_mlp` | `python/sglang/multimodal_gen/runtime/models/dits/flux.py` | Nunchaku path fuses `fc1 GEMM + GELU + shift + re-quant + fc2.lora_down` before the second GEMM | Treat split GELU-MLP on Nunchaku checkpoints as an existing fused family, not a new discovery. |
+
+## 5. Diffusion kernel-overlap and async-communication families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Ulysses sequence-parallel attention | exposed `all_to_all` around attention blocks | `python/sglang/multimodal_gen/runtime/layers/attention/layer.py`<br>`python/sglang/multimodal_gen/runtime/distributed/communication_op.py` | head / sequence redistribution before and after attention | Treat sequence-parallel all-to-all as an existing distributed attention family. |
+| USP attention with all-to-all and ring attention | `all_to_all`, ring-attention comm, head / sequence reshards | `python/sglang/multimodal_gen/runtime/layers/attention/layer.py` | `_usp_input_all_to_all(...)`, `_usp_output_all_to_all(...)`, `ring_attn(...)` | This is the primary existing overlap / comm family for many diffusion models. |
+| Turbo-layer async all-to-all pipelining | pipelined A2A windows with explicit waits on a comm stream | `python/sglang/multimodal_gen/runtime/layers/attention/turbo_layer.py` | looped `all_to_all_single(..., async_op=True)` plus staged postprocess on a comm stream | Treat exposed turbo A2A windows as an existing pipelined overlap pattern. |
+| TorchInductor compute / communication reorder | compiled traces with compute and comm partially interleaved | `python/sglang/multimodal_gen/runtime/pipelines_core/stages/denoising.py`<br>`python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/mova.py` | `torch._inductor.config.reorder_for_compute_comm_overlap = True` | Existing compile-time reordering may already explain partial overlap in diffusion traces. |
+| Dual-stream diffusion models | two nearby compute branches inside one DiT / UNet block | `python/sglang/multimodal_gen/runtime/models/dits/hunyuan3d.py` | `use_dual_stream = True` | Treat dual-branch diffusion execution as an existing overlap family. |
+
+## 6. PR-backed / in-flight fused-kernel families
+
+These rows track still-open upstream work or status-sensitive PR families.
+Stable entries should be folded into the mainline family rows above.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#21877` fused grouped down-GEMM + combine | `grouped_gemm_nt_masked`<br>`combine`<br>`fused grouped gemm combine` | `PR #21877`<br>`python/sglang/srt/layers/moe/ep_moe/flashinfer_cutedsl_moe.py`<br>`python/sglang/srt/layers/moe/token_dispatcher/deepep.py` | FlashInfer CuTeDSL kernel fuses the second expert GEMM with DeepEP low-latency combine | Treat this as a concrete upstream MoE fuse / overlap family, not a new thought experiment. |
+| PR `#21889` fused BF16 to FP4 quant + paged KV write | `set_mla_kv_buffer_fp4_quant_kernel`<br>`fp4 kv cache` | `PR #21889`<br>`python/sglang/srt/mem_cache/utils.py` | Triton kernel writes FP4 NSA KV pages directly while quantizing BF16 input | If NSA FP4 KV paths are split into quant plus store, classify them as an in-flight upstream fuse family. |
+| PR `#21889` fused FP4 paged dequant to FP8 + page-table remap | `_dequant_fp4_to_fp8_paged_kernel`<br>`WRITE_PT`<br>`dequant_fp4_paged_decode` | `PR #21889`<br>`python/sglang/srt/layers/attention/nsa/dequant_fp4_to_fp8.py` | Triton kernel reads FP4 pages, writes FP8 directly, and can fuse decode-side page-table remap | Treat this as an upstream in-flight decode-prep fusion family. |
+| PR `#21491` FlashInfer TRTLLM FP8 MoE with fused shared experts | `num_fused_shared_experts`<br>`trtllm_fp8_block_scale_moe` | `PR #21491`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/models/deepseek_v2.py` | FlashInfer TRTLLM FP8 MoE path can fuse shared experts inside the routed MoE kernel | On FP8 TRTLLM MoE discussions, treat fused shared experts as an upstream pattern that already has a concrete PR. |
+| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/jit_kernel/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/jit_kernel/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
+| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
+| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`sgl-kernel/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
+| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/jit_kernel/nvfp4.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
+| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
+| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
+| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
+| PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`<br>`virtual experts`<br>`MoE LoRA`<br>`fused_moe_lora` | `PR #24007`<br>`python/sglang/srt/layers/lora_backend.py`<br>`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. |
+| PR `#24150` torch.compile local decode support | `enable_torch_compile`<br>`local compile`<br>`decode compile`<br>`torchinductor` | `PR #24150`<br>`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. |
+
+## 7. PR-backed / in-flight kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#21877` fused down-GEMM + combine superseding SBO | `enable_fused_grouped_gemm_combine`<br>`combine`<br>`down_gemm` | `PR #21877`<br>`python/sglang/srt/server_args.py`<br>`python/sglang/srt/layers/moe/token_dispatcher/deepep.py` | Fused combine eliminates the standalone combine window, so SBO is intentionally disabled when this path is on | If the trace discussion is about combine overlap, first classify it as this upstream fused-overlap family. |
+| PR `#23965` PDL for DSV32 / GLM5 kernels | `enable_pdl`<br>`TRTLLM_ENABLE_PDL`<br>`cudaGridDependencySynchronize`<br>`DSV32`<br>`GLM5` | `PR #23965`<br>`python/sglang/srt/layers`<br>`sgl-kernel` | Enables programmatic dependent launch on selected DeepSeek / GLM kernels so dependent decode kernels can overlap launch-to-start gaps | Treat tight same-stream decode windows around DSV32 / GLM5 as an in-flight PDL overlap family. |
+| PR `#21878` TTFT / TPOT torch.compile optimization | `enable_torch_compile`<br>`decode graph`<br>`piecewise cudagraph` | `PR #21878`<br>`python/sglang/srt` | Uses compiler and graph capture changes to shave TTFT / TPOT rather than adding one handwritten kernel | If the trace shows many small compiler-visible decode ops, compare against this compile-overlap / graph-capture family first. |
+| PR `#24168` batched GPU-to-CPU sync for logprobs / embeddings | `logprobs`<br>`embeddings`<br>`GPU->CPU sync`<br>`batch sync` | `PR #24168`<br>`python/sglang/srt` | Batches per-request synchronization work that can otherwise serialize decode progress around logprob or embedding outputs | Treat per-request CPU sync stalls in logprob / embedding traces as a concrete in-flight SGLang scheduler/data-movement family. |
+
+## 8. FlashInfer mainline fused-kernel families
+
+These rows are comparative references from `flashinfer`. Use them when a trace
+looks like an upstream FlashInfer family even if the current `sglang` checkout
+only consumes a subset of that implementation.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| FlashInfer activation / gate epilogues | `silu_and_mul`<br>`gelu_tanh_and_mul`<br>`gelu_and_mul`<br>`silu_and_mul_scaled_nvfp4_experts_quantize` | `flashinfer/activation.py`<br>`flashinfer/quantization/fp4_quantization.py` | FlashInfer covers both the plain activation-plus-mul epilogues and the NVFP4 expert-quantized extension used on MoE expert paths | Treat standalone activation, multiply, and expert-side quant ladders as one existing FlashInfer epilogue family first. |
+| FlashInfer norm / residual / quant epilogues | `rmsnorm_quant`<br>`fused_add_rmsnorm`<br>`fused_add_rmsnorm_quant`<br>`gemma_rmsnorm`<br>`gemma_fused_add_rmsnorm`<br>`fused_rmsnorm_silu`<br>`rmsnorm_fp4quant`<br>`add_rmsnorm_fp4quant` | `flashinfer/norm/__init__.py`<br>`flashinfer/cute_dsl/rmsnorm_fp4quant.py`<br>`flashinfer/cute_dsl/add_rmsnorm_fp4quant.py` | The norm family spans plain RMSNorm derivatives, residual-add epilogues, norm+activation, and direct FP8 / NVFP4 output variants instead of materializing each intermediate | Treat split residual add, norm, activation, and quant chains as one existing FlashInfer epilogue family first. |
+| FlashInfer allreduce + post-op fusion family | `allreduce_fusion`<br>`AllReduceFusionPattern`<br>`kARResidualRMSNorm`<br>`kARResidualRMSNormFP8Quant`<br>`kARResidualRMSNormFP4Quant`<br>`trtllm_mnnvl_allreduce_fusion` | `flashinfer/comm/allreduce.py`<br>`flashinfer/comm/trtllm_ar.py`<br>`flashinfer/comm/trtllm_mnnvl_ar.py` | TRTLLM and MNNVL backends fuse all-reduce with residual add, RMSNorm, and backend-appropriate quant / norm-output variants | Treat TP collective + norm (+ quant) ladders as an existing FlashInfer fused-collective family first. |
+| FlashInfer RoPE + FP8 quant / cache-update family | `rope_quantize_fp8`<br>`mla_rope_quantize_fp8`<br>`rope_quantize_fp8_append_paged_kv_cache`<br>`seqlen=0`<br>`batch_indices < 0` | `flashinfer/rope.py` | The RoPE family covers both RoPE+FP8 output and the larger decode / prefill-prep path that writes K / V directly into paged KV cache, including padding-token / zero-length sequence handling | Treat split RoPE, quant, cache-write, and padding-token ladders as one existing FlashInfer attention-prep family first. |
+| FlashInfer fused DeepSeek grouped-topk routing | `fused_topk_deepseek`<br>`NoAuxTc` | `flashinfer/fused_moe/fused_routing_dsv3.py` | One kernel performs sigmoid+bias, grouped score reduction, group top-k, expert top-k, and routed renorm for DeepSeek-V3-style routing | Treat router score activation -> grouped top-k -> renorm ladders as an existing FlashInfer router family first. |
+| FlashInfer fused MoE expert execution | `cutlass_fused_moe`<br>`trtllm_bf16_moe`<br>`trtllm_fp8_per_tensor_scale_moe`<br>`trtllm_fp8_block_scale_moe`<br>`trtllm_fp4_block_scale_moe`<br>`trtllm_mxint4_block_scale_moe`<br>`non-gated` | `flashinfer/fused_moe/core.py` | CUTLASS and TRTLLM backends collapse expert execution, routed combine, and quantized expert variants into fused MoE runners, including gated and non-gated FP8 per-tensor cases | Treat exposed expert-side tiny GEMM or non-gated FP8 ladders as matching an existing FlashInfer fused-MoE family. |
+| FlashInfer CuTeDSL two-stage MoE fusion | `blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion_nvfp4`<br>`blockscaled_contiguous_grouped_gemm_finalize_fusion_nvfp4`<br>`moe_permute`<br>`moe_unpermute` | `flashinfer/fused_moe/cute_dsl/blockscaled_contiguous_gather_grouped_gemm_swiglu_fusion.py`<br>`flashinfer/fused_moe/cute_dsl/blockscaled_contiguous_grouped_gemm_finalize_fusion.py` | The CuTeDSL path fuses gather+GEMM1+SwiGLU in the first stage and finalize+unpermute+scatter-reduce in the second stage, removing standalone `moe_permute` and `moe_unpermute` kernels | Treat multi-kernel MoE ladders around permute / finalize as one existing FlashInfer CuTeDSL family first. |
+| FlashInfer SM120 FP4 / groupwise GEMM heuristics | `cutlass_fp4_gemm_sm120`<br>`CutlassTileConfigSM120`<br>`group_gemm_nvfp4_nt_groupwise`<br>`group_gemm_mxfp4_nt_groupwise` | `flashinfer/gemm/gemm_base.py`<br>`include/flashinfer/gemm/fp4_gemm_cutlass_template_sm120.h`<br>`include/flashinfer/gemm/group_gemm_nvfp4_groupwise_sm120.cuh`<br>`csrc/nv_internal/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp` | FlashInfer mainline adds SM120-oriented FP4 GEMM selection and b12x CuTeDSL fused-MoE kernels | Treat SM120 FP4 MoE/GEMM tile selection and Blackwell-lite shape restrictions as an upstream FlashInfer kernel family before inventing a local heuristic. |
+| FlashInfer MoE `routing_replay_out` support | `routing_replay_out`<br>`mPtrRoutingReplayOut`<br>`trtllm_fp8_block_scale_moe` | `flashinfer/fused_moe/core.py`<br>`csrc/trtllm_fused_moe_kernel_launcher.cu`<br>`csrc/fused_moe/noAuxTcKernels.cu` | TRTLLM-gen MoE kernels can optionally emit compact routing replay metadata without a separate routing-side reconstruction pass | Treat routing-replay writes in MoE traces as part of the upstream FlashInfer TRTLLM MoE family, not a separate postprocess opportunity. |
+
+## 9. FlashInfer mainline kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| FlashInfer PDL launch-overlap family | `enable_pdl`<br>`launch_with_pdl`<br>`cudaGridDependencySynchronize`<br>`cudaTriggerProgrammaticLaunchCompletion`<br>`trigger_completion_at_end=False`<br>`allreduce_fusion` | `flashinfer/norm/__init__.py`<br>`flashinfer/activation.py`<br>`flashinfer/rope.py`<br>`flashinfer/comm/allreduce.py`<br>`flashinfer/comm/trtllm_ar.py` | FlashInfer uses Programmatic Dependent Launch broadly, and the allreduce path can further advance completion so the next PDL-aware kernel overlaps on the same stream | Treat tight same-stream dependent windows and allreduce-followed-by-kernel windows as one existing FlashInfer launch-overlap family first. |
+| FlashInfer CuTeDSL MoE aux-stream async-memset overlap | `aux_stream`<br>`main_event`<br>`memset_event`<br>`use_async_memset` | `flashinfer/fused_moe/cute_dsl/fused_moe.py` | Preallocated MoE output is zeroed on an auxiliary CUDA stream while GEMM1 runs on the main stream, then both streams join before finalize | Treat GEMM1 vs output-zero windows as an existing FlashInfer multi-stream overlap family. |
+| FlashInfer green-context SM partition overlap | `split_device_green_ctx`<br>`split_device_green_ctx_by_sm_count`<br>`green_ctx` | `flashinfer/green_ctx.py` | CUDA green contexts partition SMs and create dedicated streams for concurrent kernel families on separate SM slices | Treat full-device two-stream traces and SM-partitioned traces as different manifestations of an existing FlashInfer overlap mechanism. |
+
+## 10. FlashInfer PR-backed / in-flight fused-kernel and kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#2720` PDL runtime-API migration | `cudaGridDependencySynchronize`<br>`cudaTriggerProgrammaticLaunchCompletion`<br>`inline PTX` | `PR #2720`<br>`include/flashinfer/comm/trtllm_allreduce_fusion.cuh`<br>`include/flashinfer/pos_enc.cuh` | Repo-wide migration preserves the existing PDL overlap family while replacing inline PTX with CUDA runtime APIs across norm, RoPE, attention, and MoE codepaths | Treat PDL-looking launch groups as an upstream FlashInfer overlap family even when implementation details differ across revisions. |
+
+## 11. TensorRT-LLM-origin fused-kernel families
+
+These rows are comparative references from `TensorRT-LLM`. Use them when a
+trace looks like a TensorRT-LLM or TensorRT-LLM-plus-FlashInfer family even if
+the current `sglang` checkout only carries an analogous implementation.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| TensorRT-LLM FlashInfer activation / gate epilogues | `flashinfer_silu_and_mul`<br>`flashinfer_gelu_tanh_and_mul`<br>`auto_deploy::silu_and_mul`<br>post-GEMM `silu` + `mul` | `tensorrt_llm/_torch/custom_ops/flashinfer_custom_ops.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/fuse_silu_mul.py`<br>`tensorrt_llm/_torch/models/modeling_gemma3.py` | Runtime custom ops and AutoDeploy rewrite `split/getitem + activation + mul` MLP epilogues into one FlashInfer op, including Gemma3 `gelu_tanh_and_mul` | Treat split gate activation + multiply as an existing TensorRT-LLM/FlashInfer epilogue family first. |
+| TensorRT-LLM FlashInfer RMSNorm family | `flashinfer_rmsnorm`<br>`flashinfer_gemma_rmsnorm`<br>`auto_deploy::flashinfer_rms_norm` | `tensorrt_llm/_torch/custom_ops/flashinfer_custom_ops.py`<br>`tensorrt_llm/_torch/modules/rms_norm.py`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/normalization/rms_norm.py` | Runtime modules and AutoDeploy can lower plain RMSNorm and Gemma RMSNorm directly to FlashInfer kernels | Treat split RMSNorm ladders as an existing TensorRT-LLM norm family before calling them novel. |
+| TensorRT-LLM FlashInfer residual add + RMSNorm | `flashinfer_fused_add_rmsnorm`<br>`flashinfer_gemma_fused_add_rmsnorm`<br>`auto_deploy::flashinfer_fused_add_rms_norm_inplace` | `tensorrt_llm/_torch/custom_ops/flashinfer_custom_ops.py`<br>`tensorrt_llm/_torch/modules/rms_norm.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/fused_add_rms_norm.py` | Residual add immediately before RMSNorm can collapse to one in-place FlashInfer op, with Gemma variant support | Treat residual add + RMSNorm chains as an existing TensorRT-LLM fused epilogue family first. |
+| TensorRT-LLM Triton fused residual add + RMSNorm + FP8 quant | `triton_fused_add_rms_norm_quant_fp8`<br>`fuse_rmsnorm_quant_fp8`<br>`fp8 static quant` | `tensorrt_llm/_torch/auto_deploy/custom_ops/normalization/triton_fused_add_rms_norm_quant_fp8.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/fuse_rmsnorm_quant_fp8.py` | Mainline AutoDeploy can rewrite residual-add plus RMSNorm plus FP8 static quant into one Triton op that emits BF16 norm output, FP8 quant output, and residual-add output together | Treat split add + norm + FP8 quant ladders as an existing TensorRT-LLM mainline family first. |
+| TensorRT-LLM FlashInfer RoPE with shared cos/sin cache | `flashinfer_apply_rope_with_cos_sin_cache_inplace`<br>`flashinfer_rope`<br>`cos_sin_cache` | `tensorrt_llm/_torch/modules/rotary_embedding.py`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/rope/flashinfer_rope.py`<br>`tensorrt_llm/_torch/auto_deploy/transform/library/rope.py` | Runtime path applies in-place RoPE from a shared cos/sin cache, while AutoDeploy can prebuild the full cache and lower diverse RoPE graphs to `flashinfer_rope` | Treat separate cos/sin gather + RoPE application ladders as an existing TensorRT-LLM attention-prep family. |
+| TensorRT-LLM FlashInfer cached paged attention | `append_paged_kv_cache`<br>`BatchPrefillWithPagedKVCacheWrapper`<br>`BatchDecodeWithPagedKVCacheWrapper`<br>`auto_deploy::flashinfer_attention_mha_with_cache`<br>`read_cache_only` | `tensorrt_llm/_torch/attention_backend/flashinfer.py`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/attention/flashinfer_attention.py`<br>`docs/source/features/attention.md` | FlashInfer attention backend fuses metadata setup, optional paged-KV append, and prefill/decode wrapper execution, including shared-KV and read-cache-only variants in AutoDeploy | Treat metadata + KV-append + cached-attention ladders as one existing TensorRT-LLM cached-attention family first. |
+| TensorRT-LLM FlashInfer MLA regular prefill | `append_paged_mla_kv_cache`<br>`BatchPrefillWithRaggedKVCacheWrapper`<br>`flashinfer_mla`<br>`rank 256`<br>`gpu append kernel` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Regular MLA prefill writes compressed KV pages and runs FlashInfer ragged prefill instead of a split append-plus-prefill ladder, with rank-256 paged-KV setups using the GPU append path | Treat MLA regular-prefill prep as an existing TensorRT-LLM FlashInfer family first. |
+| TensorRT-LLM FlashInfer MLA chunked prefill with absorbed `W_kn` | `BatchMLAPagedAttentionWrapper`<br>`chunked prefill`<br>`W_kn`<br>`W_v` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Chunked prefill absorbs `W_kn` into the query-side projection, runs paged MLA attention in compressed space, then projects back with `W_v` | Treat split absorbed-proj + MLA + output-proj ladders as an existing TensorRT-LLM MLA family first. |
+| TensorRT-LLM FlashInfer MLA decode with absorbed `W_kn` + `W_v` | `plan_decode`<br>`BatchMLAPagedAttentionWrapper`<br>`decode`<br>`W_kn`<br>`W_v` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mla/flashinfer_mla.py` | Decode path reuses the absorbed-query MLA family and projects the compressed attention output back with `W_v` | Treat similar decode-time absorbed MLA ladders as an existing TensorRT-LLM family, not a new idea. |
+| TensorRT-LLM FlashInfer fused MoE backend | `flashinfer.fused_moe`<br>`trtllm_bf16_moe`<br>`trtllm_fp8_block_scale_moe`<br>`trtllm_fp4_block_scale_moe`<br>`TRTLLM_GEN_FUSED_MOE_USE_FLASHINFER` | `tensorrt_llm/_torch/modules/fused_moe/moe_op_backend.py`<br>`tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py` | TRTLLM-gen MoE can route expert execution and quant helpers through FlashInfer instead of exposing per-expert eager ladders | Treat expert-side tiny GEMM ladders as matching an existing TensorRT-LLM FlashInfer MoE family first. |
+| TensorRT-LLM FlashInfer cached SSM / Mamba update | `flashinfer_cached_ssm`<br>`selective_state_update`<br>`flashinfer_ssm` | `tensorrt_llm/_torch/auto_deploy/custom_ops/mamba/flashinfer_backend_mamba.py`<br>`tensorrt_llm/_torch/modules/mamba/mamba2_mixer.py` | Mamba2 paths can lower cached SSM state updates to FlashInfer selective-state-update kernels instead of many smaller state ops | Treat split cached-SSM state update ladders as an existing TensorRT-LLM FlashInfer family first. |
+
+## 12. TensorRT-LLM-origin kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| TensorRT-LLM multi-stream MLA attention | `multi_stream_mla_attn`<br>`record_event_passthrough`<br>`_aux`<br>`wait_event` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_attn.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | AutoDeploy rewrites MLA Q/KV forks so the KV projection runs on an auxiliary stream while the Q path stays on the caller stream | Treat exposed Q-branch vs KV-branch overlap as an existing TensorRT-LLM multi-stream family first. |
+| TensorRT-LLM multi-stream MoE shared-vs-routed overlap | `multi_stream_moe`<br>`begin_aux_stream_passthrough`<br>`end_aux_stream_passthrough`<br>`wait_aux_stream_passthrough`<br>`mlir_elementwise_fusion`<br>`piecewise cudagraph`<br>`caller_stream.synchronize()` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Shared-expert work is moved to an auxiliary stream while routed-expert MoE work remains on the main stream and rejoins at the merge node; the same family includes synchronization rules for MLIR-fused kernels and piecewise cudagraph replay | Treat shared-expert vs routed-expert windows, including altered `multi_stream_moe` behavior under MLIR / piecewise graph modes, as an existing TensorRT-LLM branch-overlap family. |
+| TensorRT-LLM multi-stream FP8 GEMM fork parallelism | `multi_stream_gemm`<br>`trtllm_finegrained_fp8_linear`<br>`record_event_passthrough`<br>`_aux` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_gemm.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Compiler pass identifies fork points with multiple FP8 linears and moves the largest GEMM to the auxiliary stream so sibling GEMMs overlap | Treat sibling FP8 linear branches as an existing TensorRT-LLM overlap family before designing a new stream split. |
+
+## 13. TensorRT-LLM-origin PR-backed / in-flight fused-kernel and kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#12525` FlashInfer TRTLLM-gen FMHA paged-index / buffer rework | `shared paged index`<br>`trtllm-gen attention`<br>`flashinfer`<br>`kv cache buffer` | `PR #12525`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/attention/flashinfer_attention.py` | Open PR refines the existing FlashInfer TRTLLM-gen cached-attention family by disabling shared paged index and unifying KV-buffer construction | Treat these attention-prep changes as an in-flight implementation evolution of an existing family first. |
+| PR `#12544` NVFP4 KV cache support in TRTLLM-gen attention | `NVFP4 KV cache`<br>`trtllm-gen attention`<br>`flashinfer` | `PR #12544`<br>`tensorrt_llm/_torch/auto_deploy/custom_ops/attention/flashinfer_attention.py` | Open PR extends the cached-attention family so the FlashInfer-backed TRTLLM-gen path can build and consume NVFP4 KV buffers directly | Treat split KV-cache quant + buffer-build ladders as an in-flight TensorRT-LLM attention family first. |
+| PR `#12738` / `#12557` BF16 TRTLLM-gen MoE through FlashInfer | `bf16 trtllm-gen moe`<br>`flashinfer`<br>`trtllm_bf16_moe` | `PR #12738`<br>`PR #12557`<br>`tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py` | Open PRs extend the TRTLLM-gen MoE family so BF16 expert execution can route through FlashInfer instead of only CUTLASS-like paths | Treat BF16 expert ladders as an in-flight TensorRT-LLM FlashInfer MoE family. |
+
+## 14. vLLM-origin fused-kernel families
+
+These rows are comparative references from `vllm`. Use them when a trace looks
+similar to an upstream family even if the current `sglang` checkout does not
+contain the same implementation.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| vLLM-origin fused residual add + RMSNorm | `fused_add_rms_norm*`<br>residual add right before RMSNorm | `vllm/model_executor/layers/layernorm.py`<br>`vllm/_custom_ops.py`<br>`csrc/layernorm_kernels.cu`<br>`csrc/cpu/layernorm.cpp` | Custom CUDA / CPU fused add-RMSNorm op reused directly and as a building block for later compile-time fusions | Treat split residual add + RMSNorm as a long-standing vLLM-origin precedent before calling the opportunity novel in sglang. |
+| vLLM-origin AllReduce + RMSNorm (+ residual / quant) | `fuse_allreduce_rms`<br>`AllReduceFusionPass`<br>`allreduce + rmsnorm` | `vllm/compilation/passes/fusion/allreduce_rms_fusion.py`<br>`docs/design/fusions.md` | Compile-time patterns cover `AllReduce -> RMSNorm(+residual_add)` and optional FP8 / NVFP4 quant suffixes | Treat TP collective + norm (+ quant) ladders as a known vLLM-origin fusion family first. |
+| vLLM-origin RMSNorm (+ residual add) + quant | `RMSNormQuantFusionPass`<br>`fused_add_rms_norm_static_fp8_quant`<br>`per_token_quant`<br>`per_group_quant` | `vllm/compilation/passes/fusion/rms_quant_fusion.py`<br>`vllm/compilation/passes/fusion/rocm_aiter_fusion.py` | Compile-time and ROCm AITER paths fuse RMSNorm or fused-add-RMSNorm with FP8 / FP4 quant output | Treat split norm/add + quant as an upstream fused family, not an unexplored direction. |
+| vLLM-origin SiLU+Mul + quant | `ActivationQuantFusionPass`<br>`SiluMulFp8*`<br>`Nvfp4`<br>`rocm_aiter` | `vllm/compilation/passes/fusion/act_quant_fusion.py`<br>`vllm/compilation/passes/fusion/rocm_aiter_fusion.py` | Activation epilogues fuse `SiLU+Mul` with FP8 / NVFP4 / AITER group quant instead of materializing the BF16 activation first | Treat standalone activation then quant kernels as matching a vLLM-origin precedent. |
+| vLLM-origin add + RMSNorm + pad | `fuse_act_padding`<br>`RocmAiterTritonAddRMSNormPadFusionPass`<br>`add_rmsnorm_pad` | `vllm/compilation/passes/fusion/rocm_aiter_fusion.py`<br>`docs/design/fusions.md` | ROCm / AITER path fuses residual add + RMSNorm directly into the padded layout expected by the next kernel | Treat norm-plus-padding ladders as an existing backend-specific fuse family first. |
+| vLLM-origin attention + output quant | `fuse_attn_quant`<br>`AttnQuantFusionPass`<br>`merge_attn_states`<br>`output_scale`<br>`output_group_scale`<br>`output_block_scale` | `vllm/compilation/passes/fusion/attn_quant_fusion.py`<br>`vllm/v1/attention/ops/merge_attn_states.py`<br>`csrc/attention/merge_attn_states.cu`<br>`docs/design/fusions.md` | Compile-time fusion pushes FP8 / NVFP4 quantization into the attention epilogue on supported Triton / FlashInfer / ROCm / AITER backends, and mainline `merge_attn_states` kernels already support FP8 output when `output_scale` is provided | Treat attention-output quant and merged-attention quant epilogues as a known upstream family before calling them novel. |
+| vLLM-origin fused QK RMSNorm + RoPE | `fused_qk_norm_rope`<br>`QKNormRoPEFusionPass`<br>`qk norm + rope` | `vllm/compilation/passes/fusion/qk_norm_rope_fusion.py`<br>`vllm/_custom_ops.py`<br>`csrc/fused_qknorm_rope_kernel.cu` | Compile-time and direct custom-op paths fuse per-head Q / K RMSNorm with RoPE | Treat split QK norm + RoPE as a clear vLLM-origin precedent. |
+| vLLM-origin fused reshape + KV cache write | `reshape_and_cache`<br>`triton_reshape_and_cache_flash`<br>`kv cache write` | `vllm/v1/attention/ops/triton_reshape_and_cache_flash.py`<br>`vllm/v1/attention/backends/triton_attn.py` | Triton cache-update kernels reshape K / V into paged-cache layout and can include FP8 KV-cache scale/write logic | Treat reshape / transpose / cache-write ladders as an existing cache-store fusion family. |
+| vLLM-origin fused RoPE + KV cache update | `fuse_rope_kvcache`<br>`RopeKVCacheFusionPass`<br>`triton_rope_and_cache` | `vllm/compilation/passes/fusion/rope_kvcache_fusion.py`<br>`vllm/_aiter_ops.py`<br>`docs/design/fusions.md` | ROCm / AITER compile-time fusion combines RoPE with paged KV cache update instead of launching them separately | Treat split RoPE + cache-store as a known upstream family, especially on ROCm-like paths. |
+| vLLM-origin fused MLA RoPE + unified KV-cache update | `fused_rope_unified_mla_kv_cache_update`<br>`concat_and_cache_mla_rope_fused`<br>`unified_mla_kv_cache_update` | `vllm/compilation/passes/fusion/mla_rope_kvcache_cat_fusion.py`<br>`vllm/_custom_ops.py`<br>`csrc/cache_kernels_fused.cu` | Current vLLM compile pass fuses MLA-oriented RoPE on `q_pe` / `k_pe`, concat, and unified MLA KV-cache update into a direct paged-store path | Treat MLA RoPE + concat + cache-write ladders as a vLLM-origin mainline precedent before calling them novel. |
+| vLLM-origin fused grouped top-k / biased grouped top-k router | `grouped_topk`<br>`biased_grouped_topk`<br>`grouped_topk_fused_kernel` | `vllm/_custom_ops.py`<br>`vllm/_aiter_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/grouped_topk_router.py`<br>`csrc/moe/grouped_topk_kernels.cu` | CUDA / ROCm router kernels fuse grouped score processing, top-k selection, and routed renorm / bias handling | Treat MoE router ladders as matching an upstream grouped-topk family first. |
+| vLLM-origin fused top-k softmax / sigmoid router | `topk_softmax`<br>`topk_sigmoid`<br>`topkGating`<br>`fused_topk` | `vllm/_custom_ops.py`<br>`vllm/_aiter_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/fused_topk_router.py`<br>`vllm/model_executor/layers/fused_moe/router/fused_topk_bias_router.py`<br>`csrc/moe/topk_softmax_kernels.cu` | CUDA and ROCm / AITER router kernels fuse score activation (`softmax` / `sigmoid`), top-k selection, optional bias correction, and routed renorm into one op instead of routing through grouped-topk or eager softmax-plus-topk ladders | Treat standalone score activation -> top-k -> bias / renorm chains as a known upstream fused router family first. |
+| vLLM-origin DSV3 router GEMM | `dsv3_router_gemm`<br>`allow_dsv3_router_gemm`<br>`router logits` | `vllm/_custom_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/dsv3_router_gemm_entry.cu`<br>`csrc/moe/dsv3_router_gemm_float_out.cu` | Hopper-class CUDA kernel specializes the DeepSeek router linear for small decode batches and can emit FP32 logits directly without a generic GEMM chain | Treat DeepSeek-style router linear paths as an existing upstream specialized fuse, distinct from grouped-topk itself. |
+| vLLM-origin DeepSeek-V4 fused norm + router GEMM | `dsv4_norm_router_gemm`<br>`norm_gate_linear`<br>`router_gemm`<br>`DeepseekV4ForCausalLM` | `vllm/model_executor/layers/fused_moe/router/norm_gate_linear.py`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`csrc/moe/dsv4_norm_router_gemm*` | Current vLLM mainline has a low-latency DeepSeek-V4 path that fuses norm/router-adjacent work into specialized DSV4 router GEMM kernels | Treat DSV4 norm + router ladders as an upstream mainline fused-router family. |
+| vLLM-origin DeepSeek-V4 MHC fused kernels | `mhc_post_pre`<br>`head_compute_mix_kernel`<br>`aiter mhc`<br>`MHC` | `vllm/model_executor/layers/mhc.py`<br>`vllm/model_executor/kernels/mhc/aiter.py`<br>`vllm/model_executor/models/deepseek_v4.py` | Current vLLM mainline includes Tile head-compute kernels, fused `mhc_post_pre`, and ROCm AITER MHC support for DSV4 | Treat DSV4 MHC/head-compute ladders as known upstream kernel families before proposing a new head-compute kernel. |
+| vLLM-origin GPT-OSS router GEMM | `gpt_oss_router_gemm`<br>`router gemm` | `vllm/_custom_ops.py`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/gpt_oss_router_gemm.cu` | Model-specific CUDA kernel replaces the router linear plus bias path with one specialized GEMM op | Treat GPT-OSS-style router linear chains as an existing upstream specialized fuse. |
+| vLLM-origin DeepSeek min-latency fused QKV-A projection | `dsv3_fused_a_gemm`<br>`fused_qkv_a_proj`<br>`q_a_proj` | `vllm/model_executor/models/deepseek_v2.py`<br>`vllm/_custom_ops.py`<br>`csrc/dsv3_fused_a_gemm.cu` | Hopper-class CUDA kernel replaces the tiny-batch DeepSeek QKV-A projection path with one specialized min-latency GEMM instead of a generic linear launch | Treat small-batch DeepSeek QKV-A projection ladders as a known upstream fused kernel family first. |
+| vLLM-origin DSV3.2 fused indexer projections | `wk_weights_proj`<br>`MergedColumnParallelLinear`<br>`weights_proj` | `vllm/model_executor/models/deepseek_v2.py`<br>`vllm/model_executor/models/deepseek_mtp.py` | DSV3.2 indexer paths can fuse the `wk` and `weights_proj` projections into one GEMM and carry the matching MTP weight-loading path | Treat paired indexer projection chains as a known upstream fused linear family before calling the opportunity novel. |
+| vLLM-origin ROCm AITER sparse-MLA paged MQA logits | `rocm_aiter_mla_sparse`<br>`paged_mqa_logits`<br>`gluon`<br>`gfx950` | `vllm/v1/attention/ops/rocm_aiter_mla_sparse.py` | Current vLLM enables the AITER/Gluon paged-MQA logits path on gfx950 / MI355X sparse MLA shapes | On AMD sparse-MLA traces, compare against the AITER paged-MQA logits path before proposing a new logits kernel. |
+| vLLM-origin ROCm DSV4 sparse MLA Triton kernels | `rocm_aiter_mla_sparse_dsv4`<br>`flashmla_sparse`<br>`sparse_swa` | `vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse_dsv4.py`<br>`vllm/v1/attention/backends/mla/flashmla_sparse.py`<br>`vllm/v1/attention/backends/mla/sparse_swa.py` | Current vLLM mainline has ROCm DSV4 sparse-MLA Triton backend coverage | On DSV4 ROCm sparse-MLA traces, compare backend selection and sparse-SWA paths before writing a new MLA backend. |
+| vLLM-origin DSV4 dequant gather K cache | `dequant_gather_k_cutedsl`<br>`fused_indexer_q_cutedsl`<br>`cache_utils` | `vllm/v1/attention/ops/deepseek_v4_ops/dequant_gather_k_cutedsl.py`<br>`vllm/v1/attention/ops/deepseek_v4_ops/cache_utils.py` | Current vLLM mainline has a newer CuTe DSL dequant-gather K cache path for DSV4 | Treat K-cache dequant/gather ladders as a known upstream DSV4 cache-kernel family. |
+| vLLM-origin TokenSpeed MLA backend | `TOKENSPEED_MLA`<br>`tokenspeed_mla_decode`<br>`tokenspeed_mla_prefill` | `vllm/v1/attention/backends/mla/tokenspeed_mla.py`<br>`vllm/v1/attention/backends/mla/prefill/tokenspeed_mla.py`<br>`vllm/model_executor/layers/attention/mla_attention.py` | Current vLLM can select TokenSpeed MLA prefill/decode kernels for Blackwell FP8-KV DeepSeek/Kimi MLA shapes when the package and platform gates pass | On Blackwell MLA traces, compare against TokenSpeed MLA backend selection before proposing a new attention kernel. |
+| vLLM-origin MiniMax allreduce_rms kernels | `minimax_allreduce_rms`<br>`minimax_allreduce_rmsnorm`<br>`MiniMax-M2.5`<br>`allreduce_rms` | `vllm/model_executor/models/minimax_m2.py` | TensorRT-LLM-derived MiniMax allreduce-plus-RMSNorm kernels are a concrete upstream TP decode family | Treat MiniMax TP norm + collective ladders as an upstream specialized fusion family. |
+| vLLM-origin CUTLASS scaled MM with scale / bias epilogue | `cutlass_scaled_mm`<br>`cutlass_scaled_mm_azp`<br>`scaled mm` | `vllm/_custom_ops.py`<br>`vllm/model_executor/kernels/linear/scaled_mm/cutlass.py`<br>`csrc/libtorch_stable/quantization/w8a8/cutlass/scaled_mm_entry.cu` | CUTLASS kernels fuse activation scales, weight scales, matmul, and optional bias / AZP epilogues | Treat separate scale-mul + GEMM + bias ladders as a vLLM-origin fused linear family first. |
+| vLLM-origin fused MoE expert execution | `cpu_fused_moe`<br>`rocm_aiter_fused_moe`<br>`FusedMoE` | `vllm/model_executor/layers/fused_moe/layer.py`<br>`vllm/model_executor/layers/fused_moe/cpu_fused_moe.py`<br>`vllm/model_executor/layers/fused_moe/rocm_aiter_fused_moe.py`<br>`vllm/_aiter_ops.py` | MoE backends on CUDA / ROCm / CPU already collapse packed expert execution into fused expert kernels rather than per-expert eager GEMMs | Treat exposed expert-side tiny GEMM ladders as matching an upstream fused-MoE family. |
+| vLLM-origin fused MoE LoRA | `fused_moe_lora`<br>`fused_moe_lora_fp8`<br>`w13_shrink`<br>`w2_expand` | `vllm/lora/ops/triton_ops/fused_moe_lora_op.py`<br>`vllm/lora/ops/triton_ops/fused_moe_lora_fp8_op.py`<br>`vllm/lora/layers/fused_moe.py` | Triton kernels fuse LoRA shrink / expand work into MoE expert execution, including FP8 variants | Treat MoE-LoRA adapter work as an upstream fused family before proposing a brand new kernel. |
+| vLLM-origin ViT fused bilinear position-embedding interpolation | `triton_pos_embed_interpolate`<br>`bilinear_pos_embed`<br>`pos_embed_interpolate_native` | `vllm/model_executor/models/qwen3_vl.py` | Triton kernel fuses bilinear interpolation and spatial-merge reorder for Qwen3-VL ViT position embeddings, replacing many tiny eager kernels | Treat VLM position-embedding ladders as an existing vLLM-origin Triton fusion family. |
+
+## 15. TokenSpeed-origin fused-kernel families
+
+These rows are direct TokenSpeed families from `lightseekorg/tokenspeed`, not
+only vLLM references to the TokenSpeed package.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| TokenSpeed CuTe DSL MLA prefill / decode | `tokenspeed_mla_decode`<br>`tokenspeed_mla_prefill`<br>`BlackwellMultiHeadLatentAttentionForward` | `python/tokenspeed/runtime/layers/attention/backends/tokenspeed_mla.py`<br>`tokenspeed-mla/python/tokenspeed_mla/mla_decode.py`<br>`tokenspeed-mla/python/tokenspeed_mla/mla_prefill.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/ops/attention/tokenspeed_mla/__init__.py` | Blackwell SM100 CuTe DSL MLA kernels cover FP8-KV prefill/decode/verify paths through the `tokenspeed_mla` backend | On TokenSpeed or vLLM+TokenSpeed MLA traces, compare backend selection before proposing a new MLA attention kernel. |
+| TokenSpeed MLA KV pack + FP8 quantize | `_mla_kv_pack_quantize_fp8_kernel`<br>`mla_kv_pack_quantize_fp8`<br>`k_nope` / `k_pe` | `tokenspeed-mla/python/tokenspeed_mla/mla_kv_pack_quantize_fp8.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/ops/attention/tokenspeed_mla/__init__.py` | One Triton kernel packs `k_nope`, broadcast `k_pe`, and `v`, then writes FP8 K/V for MLA chunked prefill | Treat split K/V concat + FP8 cast ladders as a known TokenSpeed fusion family. |
+| TokenSpeed fused top-k + top-p sampling | `fused_topk_topp`<br>`fused_topk_topp_renorm` | `tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/fused_topk_topp.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/csrc/fused_topk_topp/fused_topk_topp.cu` | CUDA extension fuses top-k, top-p, and renormalization for decode sampling | Treat top-k/top-p/renorm chains in TokenSpeed traces as an existing sampling fusion first. <!-- codespell:ignore thirdparty --> |
+| TokenSpeed persistent lm_head GEMM | `lm_head_gemm`<br>`should_use_fused`<br>`persistent` | `tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/lm_head_gemm.py`<br>`tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cuda/csrc/lm_head_gemm.cu` | Shape-gated persistent GEMM replaces `torch.matmul` for selected lm_head / router-like projection shapes | Treat visible lm_head matmul ladders as a candidate for this existing TokenSpeed path before inventing a new logits GEMM. <!-- codespell:ignore thirdparty --> |
+| TokenSpeed NVFP4 GEMM + SwiGLU + quant | `nvfp4_gemm_swiglu_nvfp4_quant`<br>`SwiGLU`<br>`SFC` | `tokenspeed-kernel/python/tokenspeed_kernel/thirdparty/cute_dsl/nvfp4_gemm_swiglu_nvfp4_quant.py` | CuTe DSL kernel fuses block-scaled NVFP4 GEMM, SwiGLU, and optional output quantization | Treat split expert GEMM + activation + FP4 quant chains as matching an upstream TokenSpeed kernel family. <!-- codespell:ignore thirdparty --> |
+
+## 16. vLLM-origin kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| vLLM-origin AsyncTP GEMM + collective overlap | `fuse_gemm_comms`<br>`fused_matmul_reduce_scatter`<br>`fused_all_gather_matmul` | `vllm/compilation/passes/fusion/collective_fusion.py`<br>`docs/design/fusions.md` | AsyncTP overlaps GEMM with reduce-scatter / all-gather via symmetric-memory collectives | Treat GEMM+comm windows as a clear vLLM-origin overlap precedent first. |
+| vLLM-origin Sequence Parallelism staging | `enable_sp`<br>`ReduceScatter`<br>`AllGather`<br>`SequenceParallelismPass` | `vllm/compilation/passes/fusion/sequence_parallelism.py`<br>`docs/design/fusions.md` | Sequence-parallel rewrites all-reduce into RS -> local norm -> AG so later passes can overlap comm and compute | Treat RS / AG staging around norm blocks as an upstream overlap-enabling family. |
+| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py`<br>`vllm/model_executor/layers/fused_moe/runner/moe_runner_base.py` | MoE shared experts can record the cloned input on `shared_experts_stream`, wait on the caller stream, run in parallel with router-side work, and rejoin before merge | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
+| vLLM-origin DCP async all-to-all overlap | `dcp_alltoall`<br>`all_to_all_single`<br>`async_op=True` | `vllm/v1/attention/ops/dcp_alltoall.py` | Output / LSE exchange uses async all-to-all handles instead of serializing collective completion on the main path | Treat DCP all-to-all windows as an upstream async-collective family. |
+
+## 17. vLLM-origin PR-backed / in-flight fused-kernel and kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#35968` DSV3.2 multi-stream indexer overlap | `weights_proj`<br>`wk`<br>`k_norm`<br>`aux_stream` | `PR #35968`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/utils/torch_utils.py` | Closed PR explored overlapping the small `weights_proj` GEMM with `wk + k_norm` on a secondary CUDA stream for decode batches instead of serializing both on the default stream | Treat this as a concrete upstream decode-time kernel-overlap family when traces show underutilized projection overlap opportunities. |
+| PR `#37110` Triton attention + per-group FP8 dynamic quant | `group_size=128`<br>`group_size=64`<br>`output_group_scale`<br>`per-group FP8` | `PR #37110`<br>`vllm/compilation/passes/fusion/attn_quant_fusion.py`<br>`vllm/v1/attention/ops/triton_unified_attention.py` | In-flight Triton attention epilogue computes per-group FP8 scales and quantizes output directly instead of launching a separate group-quant kernel | Treat attention + per-group FP8 quant as a concrete upstream vLLM family, not a novel idea. |
+| PR `#38445` MiniMax-M2 FP32 gate kernel | `fp32_router_gemm`<br>`MiniMax-M2`<br>`gate kernel` | `PR #38445`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/model_executor/models/minimax_m2.py` | Draft CUDA kernel fuses BF16->FP32 conversion and low-batch router GEMM for MiniMax-M2, replacing up to three kernels on the gate path | Treat MiniMax-M2 gate ladders as an in-flight upstream fused router family first. |
+| PR `#38621` fused QK norm + RoPE + cache + quant | `fused_qk_norm_rope_cache_quant`<br>`QK Norm + RoPE + Cache + Quant` | `PR #38621`<br>`csrc/fused_qk_norm_rope_cache_quant.cu`<br>`vllm/compilation/passes/fusion/qk_norm_rope_cache_quant_fusion.py` | Draft CUDA kernel and compile-time pass try to fuse QK RMSNorm, RoPE, KV cache write, and optional FP8 quant for small-batch decode | Treat this as an in-flight upstream fusion family before calling a similar idea novel. |
+| PR `#37646` ROCm AITER fused allreduce + RMSNorm | `rocm_aiter_fused_allreduce_rmsnorm`<br>`custom_fused_ar_rms`<br>`RocmAiterAllReduceFusionPass` | `PR #37646`<br>`vllm/_aiter_ops.py`<br>`vllm/compilation/passes/pass_manager.py` | ROCm-specific compile-time path swaps the generic all-reduce fusion pass for an AITER fused allreduce-plus-RMSNorm kernel family | Treat ROCm TP all-reduce + RMSNorm ladders as an in-flight upstream fused-collective family first. |
+| PR `#36413` FlashInfer RMSNorm + FP4 quant fusion | `fuse_norm_quant`<br>`flashinfer`<br>`NVFP4`<br>`rmsnorm + fp4 quant` | `PR #36413`<br>`vllm/compilation/passes/fusion/rms_quant_fusion.py`<br>`vllm/docs/design/fusions.md` | FlashInfer-backed norm-plus-FP4 quant fusion extends the existing RMSNorm+quant family to NVFP4 flows | Treat split RMSNorm + FP4 quant ladders as an upstream in-flight family, not a fresh idea. |
+| PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`csrc/moe/dsv3_router_gemm_utils.h` | Extends the specialized router GEMM family to GLM5 hidden size and uses PDL to overlap the router launch with the preceding fused allreduce-plus-RMS block | Treat this as an in-flight upstream router-kernel plus launch-overlap family before calling it novel. |
+| PR `#41455` ROCm WMMA paged prefill and split-K decode | `wmma`<br>`paged prefill`<br>`split-K decode`<br>`ROCm attention` | `PR #41455`<br>`vllm/v1/attention`<br>`vllm/_aiter_ops.py` | Adds ROCm WMMA attention kernels for paged prefill and split-K decode shapes | Treat split attention support kernels on AMD as an in-flight vLLM attention-kernel family before calling them novel. |
+| PR `#41263` DeepSeek-V4 fused norm / router low-latency path | `DSV4`<br>`fuse norm router`<br>`low latency`<br>`router` | `PR #41263`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`vllm/model_executor/layers/fused_moe/router/norm_gate_linear.py` | Merged into current mainline as the DSV4 fused norm + router GEMM family above | Treat this row as provenance for the shipped mainline family, not as merely in-flight. |
+| PR `#41428` DSV4 fused indexer Q quant kernel | `DSV4`<br>`fused Indexer Q quant`<br>`indexer q`<br>`fp4` | `PR #41428`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`vllm/v1/attention/ops/deepseek_v4_ops/fused_indexer_q.py`<br>`vllm/v1/attention/ops/deepseek_v4_ops/fused_indexer_q_cutedsl.py` | Merged current-main improvement to the fused DeepSeek-V4 indexer Q quant kernel instead of materializing Q then quantizing separately | Treat DSV4 indexer-Q quant ladders as an upstream mainline fused quant family. |
+| PR `#41255` DeepSeek-V4 Tile kernels / `head_compute_mix_kernel` | `head_compute_mix_kernel`<br>`Tile kernel`<br>`DSV4`<br>`MLA` | `PR #41255`<br>`vllm/model_executor/models/deepseek_v4.py`<br>`vllm/model_executor/layers/mhc.py`<br>`vllm/model_executor/kernels/mhc/tilelang.py` | Merged current-main DeepSeek-V4 Tile kernels that mix head compute work in one specialized kernel | Treat DSV4 MLA head-compute ladders as a known upstream mainline specialized-kernel family. |
+| PR `#41441` DSV4 all-reduce plus `mhc_post` fusion | `DSV4`<br>`AR+mhc_post`<br>`allreduce`<br>`mhc_post` | `PR #41441`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/compilation/passes/fusion` | Fuses or overlaps DSV4 all-reduce with post-MLA head-compute work | Treat all-reduce followed by `mhc_post` in DSV4 traces as an in-flight vLLM overlap/fusion family. |
+| PR `#41446` AMD GatedDeltaNet FLA prefill kernels | `GatedDeltaNet`<br>`FLA prefill`<br>`AMD`<br>`Qwen3-Next` | `PR #41446`<br>`vllm/model_executor/models/qwen3_next.py`<br>`vllm/v1/attention` | Optimizes GatedDeltaNet / FLA prefill kernels on AMD linear-attention models | Treat split GDN prefill kernels on ROCm as an in-flight upstream family. |
+| PR `#39748` dual-stream GDN input projection | `dual-stream`<br>`input projection`<br>`GatedDeltaNet`<br>`Qwen3.5` | `PR #39748`<br>`vllm/model_executor/models/qwen3_next.py` | Overlaps sibling input-projection branches for Qwen3 / Qwen3.5 GDN-style blocks | Treat serial GDN input projections as a known in-flight overlap opportunity. |
+| PRs `#41433` / `#41434` / `#41429` / `#40561` GPU/CPU sync removal | `GPU->CPU sync`<br>`cpu sync`<br>`item()`<br>`non_blocking` | `PR #41433`<br>`PR #41434`<br>`PR #41429`<br>`PR #40561` | Removes or gates accidental GPU-to-CPU synchronization points and adds sync-detection coverage | Treat CPU gaps next to small GPU kernels as an upstream vLLM sync-removal family before proposing a kernel-only fix. |
+| PR `#36823` vLLM IR `fused_add_rms_norm` overload | `vllm_ir`<br>`fused_add_rms_norm`<br>`maybe_inplace` | `PR #36823`<br>`vllm/compilation/passes/ir`<br>`vllm/compilation/passes/fusion/rms_quant_fusion.py` | Extends vLLM IR lowering so fused-add-RMSNorm variants remain visible to later compile-time fusions | Treat missing norm/quant compile fusion as potentially an IR-lowering visibility issue. |
+
+## 18. Important toggles and caveats
+
+| Toggle / env | Location | Effect on trace interpretation |
+| --- | --- | --- |
+| `enable_flashinfer_allreduce_fusion` | `python/sglang/srt/server_args.py` | Enables the FlashInfer TP allreduce fusion family. |
+| `enable_aiter_allreduce_fusion` | `python/sglang/srt/server_args.py` | Enables ROCm AITER TP allreduce fusion. |
+| `enable_deterministic_inference` | `python/sglang/srt/server_args.py` | Can intentionally disable or change some fast fusion paths, especially AITER allreduce fusion and some sampling / router choices, so split kernels may be expected. |
+| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
+| `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. |
+| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
+| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. |
+| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
+| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
+| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
+| `SGLANG_ENABLE_FUSED_QKNORM_ROPE` | `python/sglang/multimodal_gen/runtime/layers/layernorm.py` | Gates the diffusion fused qknorm+rope path. |
+| `enable_pdl` / `launch_with_pdl` | `flashinfer/norm/__init__.py`<br>`flashinfer/activation.py`<br>`flashinfer/rope.py`<br>`flashinfer/fused_moe/core.py`<br>`flashinfer/comm/allreduce.py` | Enables FlashInfer PDL across many kernels; launch grouping and same-stream overlap can change substantially when it is on. |
+| `trigger_completion_at_end` | `flashinfer/comm/allreduce.py` | `False` enables downstream PDL-aware overlap after FlashInfer allreduce fusion; `True` delays completion to kernel end and removes that overlap window. |
+| `use_cuda_graph` | `flashinfer/fused_moe/cute_dsl/fused_moe.py` | Enables the preallocated-buffer path and the safe aux-stream async-memset overlap in FlashInfer CuTeDSL MoE. |
+| `split_device_green_ctx*` | `flashinfer/green_ctx.py` | Changes trace shape by partitioning SMs into separate green contexts instead of overlapping full-device streams on the default context. |
+| `rmsnorm_backend` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Chooses whether AutoDeploy lowers RMSNorm to FlashInfer, so split norm ladders may reflect backend selection rather than a missing fuse. |
+| `insert_cached_attention.backend` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Selects the cached-attention backend; `flashinfer` enables the paged-KV cached-attention family. |
+| `insert_cached_mla_attention.backend` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Selects the cached MLA backend; `flashinfer_mla` enables the MLA prefill / decode family. |
+| `TRTLLM_GEN_FUSED_MOE_USE_FLASHINFER` | `tensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.py` | Forces or guards the FlashInfer-backed TRTLLM-gen MoE family, so expert-kernel shape can change substantially when it is set. |
+| `multi_stream_moe` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Enables the TensorRT-LLM shared-expert vs routed-expert overlap family. |
+| `multi_stream_mla_attn` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Enables the TensorRT-LLM MLA Q-vs-KV branch overlap family. |
+| `multi_stream_gemm` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Enables generalized FP8 GEMM fork overlap in TensorRT-LLM AutoDeploy. |
+| `mlir_elementwise_fusion` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Can absorb merge adds into larger fused kernels, so missing explicit merge nodes in multi-stream traces may be intentional. |
+| `enable_torch_compile` | `python/sglang/srt/server_args.py`<br>`python/sglang/multimodal_gen/runtime/server_args.py` | Compiler-generated fusion / reordering can hide handwritten kernel names; absence of a custom kernel does not always mean absence of fusion. |
+| `enable_fused_grouped_gemm_combine` | `PR #21877` | In-flight path that intentionally disables SBO because combine is folded into down-GEMM. |
+| `PassConfig.fuse_allreduce_rms` | `vllm/config/compilation.py` | Enables vLLM's AllReduce -> RMSNorm (+ residual / quant) compile-time fusion family; on ROCm, AITER variants and add-RMSNorm-pad ordering can change the visible kernel split. |
+| `PassConfig.fuse_norm_quant` | `vllm/config/compilation.py` | Enables vLLM's RMSNorm(+residual add) -> FP8 / FP4 quant compile-time fusion family. |
+| `PassConfig.fuse_act_quant` | `vllm/config/compilation.py` | Enables vLLM's `SiLU+Mul -> quant` fusion family, plus ROCm AITER variants where applicable. |
+| `PassConfig.fuse_attn_quant` | `vllm/config/compilation.py` | Enables attention-epilogue quant fusion; requires the right backend / graph visibility, so split kernels may still be expected. |
+| `PassConfig.fuse_mla_dual_rms_norm` | `vllm/config/compilation.py` | Enables the AITER-backed MLA paired-Q/KV RMSNorm fusion family on ROCm. |
+| `PassConfig.enable_qk_norm_rope_fusion` | `vllm/config/compilation.py` | Enables the compile-time QK RMSNorm + RoPE family on CUDA-like backends. |
+| `PassConfig.fuse_rope_kvcache` | `vllm/config/compilation.py` | Enables ROCm / AITER RoPE + KV-cache update fusion and is range-limited by token count. |
+| `PassConfig.fuse_rope_kvcache_cat_mla` | `vllm/config/compilation.py` | Enables the MLA RoPE + unified MLA KV-cache update compile-time fusion family. |
+| `AttentionBackendEnum.TOKENSPEED_MLA` | `vllm/model_executor/layers/attention/mla_attention.py`<br>`vllm/platforms/cuda.py` | Selects the TokenSpeed MLA backend on supported Blackwell FP8-KV MLA shapes when `tokenspeed-mla` is installed. |
+| `rocm_aiter_mla_sparse_dsv4` | `vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse_dsv4.py` | Selects the ROCm DSV4 sparse-MLA backend path on supported AMD shapes. |
+| `PassConfig.fuse_minimax_qk_norm` | `vllm/config/compilation.py` | Enables the MiniMax decode Q/K allreduce-plus-RMSNorm compile-time fusion family. |
+| `PassConfig.fuse_act_padding` | `vllm/config/compilation.py` | Enables the ROCm AITER add-RMSNorm-plus-pad fusion family when AITER is available. |
+| `PassConfig.enable_sp` | `vllm/config/compilation.py` | Rewrites all-reduce into sequence-parallel staging; this is often a prerequisite for the overlap family, not just a pure fuse toggle. |
+| `PassConfig.fuse_gemm_comms` | `vllm/config/compilation.py` | Enables AsyncTP GEMM + collective overlap and auto-enables `enable_sp` when valid. |
+| vLLM PR `#46735` Triton MoE CUDA graph capture fix | `vllm/model_executor/layers/fused_moe/experts/triton_moe.py`<br>`vllm/model_executor/layers/fused_moe/experts/nvfp4_emulation_moe.py` | Latest vLLM mainline fixes CUDA graph capture around Triton / NVFP4-emulation MoE; stale target images may show graph-capture failures or eager fallbacks that are not SGLang kernel wins. |
+| `TRTLLM_ENABLE_PDL` | `csrc/libtorch_stable/dsv3_fused_a_gemm.cu`<br>`csrc/moe/dsv3_router_gemm_utils.h` | Enables programmatic dependent launch for the DSV3 specialized CUDA kernels, which can change launch grouping and trace shape for router / QKV-A paths. |
+| TokenSpeed `--attention-backend tokenspeed_mla` | `python/tokenspeed/runtime/layers/attention/backends/tokenspeed_mla.py` | Selects TokenSpeed's native CuTe DSL MLA backend; requires compatible Blackwell FP8-KV MLA shapes, so split MLA support kernels may indicate backend gating rather than a missing kernel. |
+| TokenSpeed `TOKENSPEED_MLA_PREFILL_BACKEND` | `tokenspeed-mla/python/tokenspeed_mla/mla_prefill.py` | Chooses CuTe DSL JIT vs binary prefill backend; trace kernel names can differ even when the same MLA fused family applies. |
+| TokenSpeed `--comm-fusion-max-num-tokens` / `--enable-allreduce-fusion` | `docs/configuration/server.md`<br>`python/tokenspeed/runtime/distributed/comm_backend` | Gates TokenSpeed communication-fusion behavior; inspect these before treating all-reduce + compute separation as a novel overlap gap. |
+
+## 19. Suggested refresh commands
+
+These commands are only for maintainers refreshing this catalog by rescanning
+the local source trees. They are not used by the triage scripts at runtime.
+
+```bash
+# Optional sibling checkouts used for comparative scanning:
+FLASHINFER_REPO=${FLASHINFER_REPO:-../flashinfer}
+TRTLLM_REPO=${TRTLLM_REPO:-../TensorRT-LLM}
+VLLM_REPO=${VLLM_REPO:-../vllm}
+
+rg -n "fused_add_rmsnorm|gemma_fused_add_rmsnorm|silu_and_mul|gelu_and_mul|fused_qk_rope_reshape_and_cache|fused_set_kv_buffer|fused_metadata_copy|normal_decode_set_metadata|_append_shared_to_topk_output|fused_append_shared_experts_with_weights" python/sglang
+rg -n "MiniMaxM2RMSNormTP|fused_qknorm_rope|fused_qk_rope_cat_and_cache_mla|fused_qk_norm_mrope_3d_cache_pts_quant_shuffle|split_qkv_rmsnorm_rope|trtllm_fp8_kv_kernel|set_mla_kv_buffer_fp8_quant" python/sglang
+rg -n "FusedMoeRouter|fused_topk_deepseek|moe_fused_gate|aiter_fused_topk|fused_rms_fp8_group_quant|fast_topk_transform_fused|fused_store_index_k_cache|fused_temperature_softmax|fused_softcap" python/sglang
+rg -n "fused_qkvzba_split_reshape_cat|fused_gdn_gating|rms_norm_gated|layer_norm_gated|chunk_gated_delta_rule_fwd_kkt_solve_kernel|fused_recurrent_gated_delta_rule_update|fused_mamba_state_scatter_with_mask|_fused_gather_to_staging_kernel|_fused_scatter_from_staging_kernel" python/sglang
+rg -n "single_batch_overlap|alt_stream|shared_expert|_comm_stream|scatter_stream|triton_mrope_fused|ring_attn|all_to_all_single|reorder_for_compute_comm_overlap|use_dual_stream" python/sglang
+git log --all --format='%h %s' | rg -i 'fused|fusion|overlap|cutedsl|triton|cuda|rope|topk|quant|combine|allreduce|all_to_all'
+rg -n "silu_and_mul|gelu_tanh_and_mul|gelu_and_mul|silu_and_mul_scaled_nvfp4_experts_quantize|rmsnorm_quant|fused_add_rmsnorm|fused_add_rmsnorm_quant|fused_rmsnorm_silu" "$FLASHINFER_REPO/flashinfer"
+rg -n "AllReduceFusionPattern|allreduce_fusion|trigger_completion_at_end|rope_quantize_fp8|rope_quantize_fp8_append_paged_kv_cache|fused_topk_deepseek|cutlass_fused_moe|trtllm_.*_moe" "$FLASHINFER_REPO/flashinfer"
+rg -n "aux_stream|use_async_memset|split_device_green_ctx|split_device_green_ctx_by_sm_count|enable_pdl|launch_with_pdl" "$FLASHINFER_REPO/flashinfer" "$FLASHINFER_REPO/include"
+git -C "$FLASHINFER_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|pdl|stream|rope|kv|quant|topk|moe'
+rg -n "flashinfer_silu_and_mul|flashinfer_gelu_tanh_and_mul|flashinfer_rmsnorm|flashinfer_gemma_rmsnorm|flashinfer_fused_add_rmsnorm|flashinfer_apply_rope_with_cos_sin_cache_inplace|triton_fused_add_rms_norm_quant_fp8|fuse_rmsnorm_quant_fp8" "$TRTLLM_REPO/tensorrt_llm/_torch"
+rg -n "flashinfer_attention_mha_with_cache|append_paged_kv_cache|flashinfer_mla|append_paged_mla_kv_cache|flashinfer_cached_ssm|selective_state_update|flashinfer.fused_moe" "$TRTLLM_REPO/tensorrt_llm/_torch" "$TRTLLM_REPO/docs/source"
+rg -n "multi_stream_moe|multi_stream_mla_attn|multi_stream_gemm|record_event_passthrough|begin_aux_stream_passthrough|end_aux_stream_passthrough|wait_aux_stream_passthrough" "$TRTLLM_REPO/tensorrt_llm/_torch"
+git -C "$TRTLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|flashinfer|mla|kv cache|multi-stream|stream|rope|rmsnorm|moe'
+rg -n "fused_add_rms_norm|merge_attn_states|fused_qk_norm_rope|grouped_topk|topk_softmax|topk_sigmoid|dsv3_router_gemm|dsv3_fused_a_gemm|concat_and_cache_mla_rope_fused|gpt_oss_router_gemm|cutlass_scaled_mm|cpu_fused_moe|fused_moe_lora|triton_pos_embed_interpolate" "$VLLM_REPO/vllm" "$VLLM_REPO/csrc"
+rg -n "fuse_allreduce_rms|fuse_norm_quant|fuse_act_quant|fuse_attn_quant|enable_qk_norm_rope_fusion|fuse_rope_kvcache|enable_sp|fuse_gemm_comms|RocmAiter|dcp_alltoall|shared_experts_stream|TRTLLM_ENABLE_PDL|wk_weights_proj" "$VLLM_REPO/vllm" "$VLLM_REPO/docs/design/fusions.md" "$VLLM_REPO/csrc"
+git -C "$VLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|triton|cuda|rope|kv cache|topk|router|allreduce|reduce-scatter|all-gather|all_to_all|quant'
+# GitHub PR scan terms for the connector or web UI:
+#   "fused OR overlap repo:sgl-project/sglang"
+#   "triton OR cutedsl OR cuda fused repo:sgl-project/sglang"
+#   "fused OR overlap repo:flashinfer-ai/flashinfer"
+#   "pdl OR aux_stream OR green_ctx repo:flashinfer-ai/flashinfer"
+#   "fused OR overlap repo:NVIDIA/TensorRT-LLM"
+#   "flashinfer OR mla OR moe OR rmsnorm repo:NVIDIA/TensorRT-LLM"
+#   "multi-stream OR aux_stream OR cudagraph repo:NVIDIA/TensorRT-LLM"
+#   "fused OR overlap repo:vllm-project/vllm"
+#   "triton OR cuda fused repo:vllm-project/vllm"
+```

.claude/skills/llm-torch-profiler-analysis/references/heuristics.md

@@ -0,0 +1,119 @@
+# Overlap Heuristics
+
+This analyzer is intentionally conservative.
+
+## What Comes From Which Trace
+
+### Mapping trace
+
+Used for:
+
+- `kernel -> cpu_op -> python scope`
+- launch-site call chains
+
+This trace should be easier to read, even if it is not the exact final serving schedule.
+
+### Formal trace
+
+Used for:
+
+- hidden ratio
+- exclusive ratio
+- overlap headroom
+- ASCII timelines
+
+This trace should reflect the real serving shape.
+
+## What It Treats As Hidden
+
+A kernel is treated as hidden for a segment if:
+
+- it is active during that segment
+- at least one kernel on a different stream is also active
+
+If the overlapping kernel is compute-like, the analyzer separately records that it is hidden under compute.
+
+## Category Heuristics
+
+The analyzer classifies kernels by name:
+
+- `compute`: GEMM, attention, cutlass, cublas, Triton matmul-like kernels
+- `communication`: NCCL, all-reduce, reduce-scatter, all-gather, DeepEP dispatch/combine
+- `elementwise`: sigmoid, top-k, gate, rmsnorm, layernorm, rope, casts
+- `memory`: memcpy, memset, fill, copy
+- `other`: everything else
+
+These categories are for prioritization only.
+
+## How To Read The Action Table
+
+The overlap-opportunity table is intentionally not a full kernel dump.
+
+It only keeps rows that already have an action-oriented label:
+
+- `headroom`
+- `low-roi-hidden`
+
+It also prunes very small `headroom` rows after prioritization.
+
+- if a `headroom` row would end up as `P5` because it is below the default `1%` share bar, it is omitted from the table
+- `low-roi-hidden` rows can still remain even when they are small, because they are useful as "do not chase this first" signals
+
+### `headroom`
+
+Interpretation:
+
+- the kernel still spends meaningful time exposed in the formal trace
+- the mapped Python scope is a good place to inspect scheduling or fusion opportunities
+- the dependency signal should still be checked before treating it as a serious overlap candidate
+
+### `low-roi-hidden`
+
+Interpretation:
+
+- the kernel is already mostly hidden by another stream
+- optimizing it in isolation is less likely to move end-to-end latency
+- focus on fusion, launch reduction, or the surrounding schedule instead
+
+## Dependency Signal
+
+The table includes a dependency-oriented adjacency signal from the formal trace.
+
+It is built from the nearest previous and next kernels on the same stream plus the mapping-trace source attribution.
+
+Communication kernels are treated more conservatively than before:
+
+- if a tight adjacent kernel looks like a likely producer or consumer, the table will raise the dependency risk even when the Python scope names differ
+- this avoids over-claiming that an all-reduce-like kernel is a clean overlap candidate just because its neighbors map to different functions
+
+Typical labels:
+
+- `serial risk low`: adjacent kernels do not look like a tight same-code serial chain
+- `prev-side serial risk`: the previous adjacent kernel looks tightly tied to the same code path
+- `next-side serial risk`: the next adjacent kernel looks tightly tied to the same code path
+- `both-side serial risk`: both sides look like a tight serial chain
+- `adjacency unclear`: the timing is tight but source attribution is too weak to trust a stronger claim
+
+Treat this as a strong heuristic, not proof of dataflow.
+
+The readable table compresses those into shorter labels:
+
+- `low`
+- `high`
+- `unclear`
+
+The recommendation labels are also intentionally short:
+
+- `try overlap`
+- `try fusion`
+- `check deps`
+- `skip overlap`
+- `manual check`
+- `observe later`
+
+## Important Limits
+
+- A trace shows what overlapped, not what could legally overlap.
+- Two kernels on different streams do not prove they are dependency-free.
+- A mapped Python scope is a launch-site clue, not the only relevant code location.
+- A hidden kernel can still matter if it changes occupancy, launch count, or surrounding schedule.

.claude/skills/llm-torch-profiler-analysis/references/overlap-catalog.md

@@ -0,0 +1,197 @@
+# Overlap Catalog
+
+This catalog is the overlap-only companion to
+`references/fuse-overlap-catalog.md`.
+
+This revision is intentionally kernel-scoped. Keep rows here only when the
+overlap is visible in a profiler as GPU kernels, collective kernels, or
+streamed kernel families. Host-only scheduler, event-loop, executor, offload,
+and load-path overlaps are intentionally excluded.
+
+Use it like this:
+
+1. Start from the `overlap-opportunity table`.
+2. Match visible kernel windows, collective windows, or stream-level overlap
+   against the rows below.
+3. If a match exists in the mainline sections, report it as an existing
+   overlap family that is missing, disabled, regressed, or unsupported on the
+   current backend.
+4. If a match exists only in the `PR-backed / in-flight`
+   section, report it as an upstream overlap pattern, not a novel idea.
+5. Only call an overlap opportunity "new" when no row in this file or
+   `fuse-overlap-catalog.md` fits.
+
+The `vLLM-origin` sections below are comparative references. They are not
+necessarily present in the checked-out `sglang` tree, but they should still be
+treated as upstream or analogous kernel-overlap families before labeling an
+overlap opportunity as novel.
+
+Refresh note `2026-06-26`: rechecked official main heads for SGLang
+`8524678889485801e7a4a12d62015be0c68f7a90`, vLLM
+`abc71548ef029132c3316b902207f254a246d593`, TensorRT-LLM
+`0722c5f47d2cae69ac1a237da51e550dd214532c`, and TokenSpeed
+`5aedf69d6b476baa65571011de6ea60fd5a238a8`, then added the first
+TokenSpeed-origin communication-fusion row. Closed-unmerged SGLang
+[#22410](https://github.com/sgl-project/sglang/pull/22410) and FlashInfer
+[#2840](https://github.com/flashinfer-ai/flashinfer/pull/2840) were removed
+from the PR-backed sections. SGLang
+[#21877](https://github.com/sgl-project/sglang/pull/21877), FlashInfer
+[#2720](https://github.com/flashinfer-ai/flashinfer/pull/2720), and vLLM
+[#35968](https://github.com/vllm-project/vllm/pull/35968) /
+[#39301](https://github.com/vllm-project/vllm/pull/39301) remain useful
+upstream overlap references as of this refresh.
+
+## 1. LLM / SRT kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Single-batch overlap (SBO) | MoE combine, down-gemm, shared-expert work in nearby two-stream windows | `python/sglang/srt/batch_overlap/single_batch_overlap.py` | combine vs down-gemm overlap, combine vs shared-expert overlap, one-stream dispatch+shared overlap, explicit SM partitioning and events | If exposed MoE combine sits near neighboring compute, classify it against SBO before calling it new overlap. |
+| Q and K normalization on different streams | Q-side norm and K-side norm on different streams | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/srt/models/qwen3.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py` | Q stays on current stream, K can run on `alt_stream` in capture mode | Treat split Q / K norm as an existing overlap family when `alt_stream` is already wired. |
+| DeepSeek shared-expert / routed-expert overlap | shared-expert GEMMs near DeepEP dispatch / combine | `python/sglang/srt/models/deepseek_v2.py`<br>`python/sglang/srt/batch_overlap/single_batch_overlap.py` | shared experts on `alt_stream`, overlap with dispatch / combine and down-gemm, Blackwell-specific env gating | This is an established routed-vs-shared branch overlap pattern, not a novel idea. |
+| Llama4 shared branch vs routed branch overlap | shared expert branch plus routed MoE branch as adjacent windows | `python/sglang/srt/models/llama4.py` | shared expert on current stream, router + topk + routed experts on `alt_stream` | Use Llama4 as the first precedent for branch-level overlap in similar sparse models. |
+| ExaoneMoE shared experts vs router experts overlap | shared expert output and router-expert output form a two-branch window | `python/sglang/srt/models/exaone_moe.py::forward_normal_dual_stream` | shared experts on current stream, router + routed experts on `alt_stream`, explicit join before combine | This is an existing dual-stream MoE overlap family. |
+| Grok residual-MoE branch overlap | dense MLP and block-sparse MoE branches in parallel | `python/sglang/srt/models/grok.py::moe_with_rmoe` | dense MLP on current stream, MoE on `alt_stream`, fused dual residual RMSNorm around boundaries | Treat exposed Grok branch overlap as an existing pattern. |
+| NSA dual-stream overlap | Q-proj, K-proj, RoPE, cache-store, quantization in tight two-stream windows | `python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Q / K projection split, RoPE split, cache-store vs quantization overlap | NSA already contains several dual-stream overlap precedents. |
+| MoriEP async dispatch / combine comm stream | `MoriEP`<br>`_comm_stream`<br>`dispatch`<br>`combine`<br>`done_event` | `python/sglang/srt/layers/moe/token_dispatcher/moriep.py` | MoriEP can submit dispatch and combine onto a dedicated communication stream and synchronize only through events | Treat MoriEP comm / compute interleave as an existing MoE overlap family. |
+| Generic `alt_stream` overlap families | `alt_stream` plus explicit `wait_stream` / `with torch.cuda.stream(...)` | `qwen2_moe.py`<br>`qwen3_moe.py`<br>`glm4_moe.py`<br>`bailing_moe.py`<br>`llada2.py`<br>`grok.py`<br>`olmo2.py`<br>`step3p5.py`<br>`longcat_flash.py`<br>`falcon_h1.py` | model-specific overlap on attention prep, MoE branches, or cache-store | Search these families before designing a new overlap scheme from scratch. |
+
+## 2. Staging / communication kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Decode scatter on dedicated `scatter_stream` | `scatter_stream`<br>`_scatter_stream` | `python/sglang/srt/disaggregation/common/staging_handler.py` | staging scatter kernels are submitted to a dedicated stream so the decode thread does not block on the main forward stream | Treat decode-side staging scatter windows as an existing overlap pattern. |
+| Staging-buffer fused gather / scatter kernels | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather KV slices into contiguous staging memory and scatter them back to KV cache | If heterogeneous-TP staging shows many small copy kernels, compare against this existing fused-plus-overlap family first. |
+
+## 3. VLM / diffusion kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| Vision QK norm with aux stream | vision-side QK norm or norm-like kernels before attention | `python/sglang/srt/layers/attention/vision.py` | vision QK normalization can call shared `apply_qk_norm(...)`, with K-side work on `aux_stream` | If vision QK prep is split, first check this existing aux-stream path. |
+| ViT CUDA graph disables vision aux stream | expected vision overlap is absent under ViT graph | `python/sglang/srt/models/internvl.py`<br>`python/sglang/srt/layers/attention/vision.py`<br>`python/sglang/srt/environ.py::SGLANG_VIT_ENABLE_CUDA_GRAPH` | vision `aux_stream` is intentionally disabled when ViT CUDA graph is on | Missing vision overlap may be intentional, not a regression. |
+| Ulysses sequence-parallel attention | exposed `all_to_all` around attention blocks | `python/sglang/multimodal_gen/runtime/layers/attention/layer.py`<br>`python/sglang/multimodal_gen/runtime/distributed/communication_op.py` | head / sequence redistribution before and after attention | Treat sequence-parallel all-to-all as an existing distributed attention family. |
+| USP attention with all-to-all and ring attention | `all_to_all`, ring-attention comm, head / sequence reshards | `python/sglang/multimodal_gen/runtime/layers/attention/layer.py` | `_usp_input_all_to_all(...)`, `_usp_output_all_to_all(...)`, `ring_attn(...)` | This is the primary existing overlap / comm family for many diffusion models. |
+| Turbo-layer async all-to-all pipelining | pipelined A2A windows with explicit waits on a comm stream | `python/sglang/multimodal_gen/runtime/layers/attention/turbo_layer.py` | looped `all_to_all_single(..., async_op=True)` plus staged postprocess on a comm stream | Treat exposed turbo A2A windows as an existing pipelined overlap pattern. |
+| TorchInductor compute / communication reorder | compiled traces with compute and comm partially interleaved | `python/sglang/multimodal_gen/runtime/pipelines_core/stages/denoising.py`<br>`python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/mova.py` | `torch._inductor.config.reorder_for_compute_comm_overlap = True` | Existing compile-time reordering may already explain partial overlap in diffusion traces. |
+| Dual-stream diffusion models | two nearby compute branches inside one DiT / UNet block | `python/sglang/multimodal_gen/runtime/models/dits/hunyuan3d.py` | `use_dual_stream = True` | Treat dual-branch diffusion execution as an existing overlap family. |
+
+## 4. PR-backed / in-flight kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#21877` fused down-GEMM + combine superseding SBO | `enable_fused_grouped_gemm_combine`<br>`combine`<br>`down_gemm` | `PR #21877`<br>`python/sglang/srt/server_args.py`<br>`python/sglang/srt/layers/moe/token_dispatcher/deepep.py` | Fused combine eliminates the standalone combine window, so SBO is intentionally disabled when this path is on | If the trace discussion is about combine overlap, first classify it as this upstream fused-overlap family. |
+
+## 5. FlashInfer kernel-overlap families
+
+These rows are comparative references from `flashinfer`. Use them when a trace
+looks like an upstream FlashInfer overlap family even if the current `sglang`
+checkout only calls part of that implementation.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| FlashInfer PDL launch-overlap family | `enable_pdl`<br>`launch_with_pdl`<br>`cudaGridDependencySynchronize`<br>`cudaTriggerProgrammaticLaunchCompletion`<br>`trigger_completion_at_end=False`<br>`allreduce_fusion` | `flashinfer/norm/__init__.py`<br>`flashinfer/activation.py`<br>`flashinfer/rope.py`<br>`flashinfer/comm/allreduce.py`<br>`flashinfer/comm/trtllm_ar.py` | FlashInfer uses Programmatic Dependent Launch broadly, and the allreduce path can further advance completion so the next PDL-aware kernel overlaps on the same stream | Treat tight same-stream dependent windows and allreduce-followed-by-kernel windows as one existing FlashInfer launch-overlap family first. |
+| FlashInfer CuTeDSL MoE aux-stream async-memset overlap | `aux_stream`<br>`main_event`<br>`memset_event`<br>`use_async_memset` | `flashinfer/fused_moe/cute_dsl/fused_moe.py` | Preallocated MoE output is zeroed on an auxiliary CUDA stream while GEMM1 runs on the main stream, then both streams join before finalize | Treat GEMM1 vs output-zero windows as an existing FlashInfer multi-stream overlap family. |
+| FlashInfer green-context SM partition overlap | `split_device_green_ctx`<br>`split_device_green_ctx_by_sm_count`<br>`green_ctx` | `flashinfer/green_ctx.py` | CUDA green contexts partition SMs and create dedicated streams for concurrent kernel families on separate SM slices | Treat SM-partitioned concurrency as an existing FlashInfer overlap mechanism, not a novel scheduler idea. |
+
+## 6. FlashInfer PR-backed / in-flight kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#2720` PDL runtime-API migration | `cudaGridDependencySynchronize`<br>`cudaTriggerProgrammaticLaunchCompletion`<br>`inline PTX` | `PR #2720`<br>`include/flashinfer/comm/trtllm_allreduce_fusion.cuh`<br>`include/flashinfer/pos_enc.cuh` | Repo-wide migration preserves the existing PDL overlap family while replacing inline PTX with CUDA runtime APIs across norm, RoPE, attention, and MoE codepaths | Treat PDL-looking launch groups as an upstream FlashInfer overlap family even when implementation details differ across revisions. |
+
+## 7. TensorRT-LLM-origin kernel-overlap families
+
+These rows are comparative references from `TensorRT-LLM`. Current mainline
+TensorRT-LLM overlap rows are mostly explicit auxiliary-stream rewrites in
+AutoDeploy rather than same-stream PDL windows.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| TensorRT-LLM multi-stream MLA attention | `multi_stream_mla_attn`<br>`record_event_passthrough`<br>`_aux`<br>`wait_event` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_attn.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | AutoDeploy rewrites MLA Q/KV forks so the KV projection runs on an auxiliary stream while the Q path stays on the caller stream | Treat exposed Q-branch vs KV-branch overlap as an existing TensorRT-LLM multi-stream family first. |
+| TensorRT-LLM multi-stream MoE shared-vs-routed overlap | `multi_stream_moe`<br>`begin_aux_stream_passthrough`<br>`end_aux_stream_passthrough`<br>`wait_aux_stream_passthrough`<br>`mlir_elementwise_fusion`<br>`piecewise cudagraph`<br>`caller_stream.synchronize()` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_moe.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Shared-expert work is moved to an auxiliary stream while routed-expert MoE work remains on the main stream and rejoins at the merge node; the same family includes synchronization rules for MLIR-fused kernels and piecewise cudagraph replay | Treat shared-expert vs routed-expert windows, including altered behavior under MLIR / piecewise graph modes, as an existing TensorRT-LLM branch-overlap family. |
+| TensorRT-LLM multi-stream FP8 GEMM fork parallelism | `multi_stream_gemm`<br>`trtllm_finegrained_fp8_linear`<br>`record_event_passthrough`<br>`_aux` | `tensorrt_llm/_torch/auto_deploy/transform/library/multi_stream_gemm.py`<br>`tensorrt_llm/_torch/auto_deploy/utils/multi_stream_utils.py` | Compiler pass identifies fork points with multiple FP8 linears and moves the largest GEMM to the auxiliary stream so sibling GEMMs overlap | Treat sibling FP8 linear branches as an existing TensorRT-LLM overlap family before designing a new stream split. |
+
+## 8. TokenSpeed-origin kernel-overlap families
+
+These rows are comparative references from `lightseekorg/tokenspeed`. Use them
+when the trace is from TokenSpeed or from a vLLM/TokenSpeed hybrid deployment.
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| TokenSpeed allreduce / communication fusion | `enable_allreduce_fusion`<br>`comm_fusion`<br>`comm_fusion_max_num_tokens`<br>`allreduce` | `docs/configuration/server.md`<br>`python/tokenspeed/runtime/distributed/comm_backend` | TokenSpeed exposes runtime knobs for communication fusion and token-count gating, so all-reduce windows may be a disabled or shape-gated TokenSpeed path | Treat split all-reduce + compute windows in TokenSpeed traces as a comm-fusion eligibility question before calling it a novel overlap opportunity. |
+
+## 9. vLLM-origin kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| vLLM-origin AsyncTP GEMM + collective overlap | `fuse_gemm_comms`<br>`fused_matmul_reduce_scatter`<br>`fused_all_gather_matmul` | `vllm/compilation/passes/fusion/collective_fusion.py`<br>`docs/design/fusions.md` | AsyncTP overlaps GEMM with reduce-scatter / all-gather via symmetric-memory collectives | Treat GEMM+comm windows as a clear vLLM-origin overlap precedent first. |
+| vLLM-origin Sequence Parallelism staging | `enable_sp`<br>`ReduceScatter`<br>`AllGather`<br>`SequenceParallelismPass` | `vllm/compilation/passes/fusion/sequence_parallelism.py`<br>`docs/design/fusions.md` | Sequence-parallel rewrites all-reduce into RS -> local norm -> AG so later passes can overlap comm and compute | Treat RS / AG staging around norm blocks as an upstream overlap-enabling family. |
+| vLLM-origin shared-expert aux-stream overlap | `aux_stream`<br>`shared_experts_stream`<br>shared expert near router | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py`<br>`vllm/model_executor/layers/fused_moe/runner/moe_runner_base.py` | MoE shared experts can record the cloned input on `shared_experts_stream`, wait on the caller stream, run in parallel with router-side work, and rejoin before merge | Treat shared-expert vs router overlap as an existing upstream sparse-model family. |
+| vLLM-origin DCP async all-to-all overlap | `dcp_alltoall`<br>`all_to_all_single`<br>`async_op=True` | `vllm/v1/attention/ops/dcp_alltoall.py` | Output / LSE exchange uses async all-to-all handles instead of serializing collective completion on the main path | Treat DCP all-to-all windows as an upstream async-collective family. |
+
+## 10. vLLM-origin PR-backed / in-flight kernel-overlap families
+
+| Pattern | Trace keywords | Primary code | Existing path | Skill should conclude |
+| --- | --- | --- | --- | --- |
+| PR `#35968` DSV3.2 multi-stream indexer overlap | `weights_proj`<br>`wk`<br>`k_norm`<br>`aux_stream` | `PR #35968`<br>`vllm/model_executor/models/deepseek_v2.py`<br>`vllm/utils/torch_utils.py` | Closed PR explored overlapping the small `weights_proj` GEMM with `wk + k_norm` on a secondary CUDA stream for decode batches instead of serializing both on the default stream | Treat this as a concrete upstream decode-time kernel-overlap family when traces show underutilized projection overlap opportunities. |
+| PR `#39301` GLM5 router GEMM with PDL overlap | `TRTLLM_ENABLE_PDL`<br>`router_gemm`<br>`GLM5`<br>`FI AR RMS fusion` | `PR #39301`<br>`vllm/model_executor/layers/fused_moe/router/gate_linear.py`<br>`vllm/csrc/moe/dsv3_router_gemm_utils.h` | The GLM5 router GEMM path explicitly uses PDL so the router kernel can overlap with the preceding fused allreduce-plus-RMS block on supported GPUs | Treat router-GEMM launch overlap on GLM5-like traces as an in-flight upstream family first. |
+
+## 11. Important toggles and caveats
+
+| Toggle / env | Location | Effect on trace interpretation |
+| --- | --- | --- |
+| `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. |
+| `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. |
+| `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. |
+| `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. |
+| `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. |
+| `enable_pdl` / `launch_with_pdl` | `flashinfer/norm/__init__.py`<br>`flashinfer/activation.py`<br>`flashinfer/rope.py`<br>`flashinfer/fused_moe/core.py`<br>`flashinfer/comm/allreduce.py` | Enables FlashInfer PDL across many kernels; launch grouping and same-stream overlap can change substantially when it is on. |
+| `trigger_completion_at_end` | `flashinfer/comm/allreduce.py` | `False` enables downstream PDL-aware overlap after FlashInfer allreduce fusion; `True` delays completion to kernel end and removes that overlap window. |
+| `use_cuda_graph` | `flashinfer/fused_moe/cute_dsl/fused_moe.py` | Enables the preallocated-buffer path and the safe aux-stream async-memset overlap in FlashInfer CuTeDSL MoE. |
+| `split_device_green_ctx*` | `flashinfer/green_ctx.py` | Changes trace shape by partitioning SMs into separate green contexts instead of overlapping full-device streams on the default context. |
+| `multi_stream_moe` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Enables the TensorRT-LLM shared-expert vs routed-expert overlap family. |
+| `multi_stream_mla_attn` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Enables the TensorRT-LLM MLA Q-vs-KV branch overlap family. |
+| `multi_stream_gemm` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Enables generalized FP8 GEMM fork overlap in TensorRT-LLM AutoDeploy. |
+| `mlir_elementwise_fusion` | `tensorrt_llm/_torch/auto_deploy/config/default.yaml` | Can absorb merge adds into larger fused kernels, so missing explicit merge nodes in TensorRT-LLM multi-stream traces may be intentional. |
+| `enable_torch_compile` | `python/sglang/srt/server_args.py`<br>`python/sglang/multimodal_gen/runtime/server_args.py` | Compiler-generated reordering can hide or rename overlap windows. |
+| `enable_fused_grouped_gemm_combine` | `PR #21877` | In-flight path that intentionally disables SBO because combine is folded into down-GEMM. |
+| `PassConfig.enable_sp` | `vllm/config/compilation.py` | Enables vLLM's sequence-parallel staging family that creates RS / AG overlap opportunities. |
+| `PassConfig.fuse_gemm_comms` | `vllm/config/compilation.py` | Enables AsyncTP GEMM + collective overlap and auto-enables `enable_sp` when valid. |
+| TokenSpeed `--comm-fusion-max-num-tokens` / `--enable-allreduce-fusion` | `docs/configuration/server.md` | Gates TokenSpeed communication fusion; inspect it before treating all-reduce + compute separation as a new overlap gap. |
+
+## 12. Suggested refresh commands
+
+These commands are only for maintainers refreshing this catalog by rescanning
+the local source trees. They are not used by the triage scripts at runtime.
+
+```bash
+# Optional sibling checkouts used for comparative scanning:
+FLASHINFER_REPO=${FLASHINFER_REPO:-../flashinfer}
+TRTLLM_REPO=${TRTLLM_REPO:-../TensorRT-LLM}
+VLLM_REPO=${VLLM_REPO:-../vllm}
+TOKENSPEED_REPO=${TOKENSPEED_REPO:-../tokenspeed}
+
+rg -n "single_batch_overlap|alt_stream|shared_expert|scatter_stream|_fused_gather_to_staging_kernel|_fused_scatter_from_staging_kernel|async_op=True" python/sglang
+rg -n "apply_qk_norm|vision.py|ring_attn|all_to_all_single|reorder_for_compute_comm_overlap|use_dual_stream" python/sglang/multimodal_gen python/sglang/srt
+git log --all --format='%h %s' | rg -i 'fused|fusion|overlap|combine|all_to_all|ring attn|stream|triton|cutedsl|cuda'
+rg -n "enable_pdl|launch_with_pdl|trigger_completion_at_end|aux_stream|use_async_memset|split_device_green_ctx|split_device_green_ctx_by_sm_count" "$FLASHINFER_REPO/flashinfer" "$FLASHINFER_REPO/include"
+git -C "$FLASHINFER_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|pdl|stream|rope|kv|quant|topk|moe'
+rg -n "multi_stream_moe|multi_stream_mla_attn|multi_stream_gemm|record_event_passthrough|begin_aux_stream_passthrough|end_aux_stream_passthrough|wait_aux_stream_passthrough" "$TRTLLM_REPO/tensorrt_llm/_torch"
+rg -n "mlir_elementwise_fusion|piecewise|cudagraph|caller_stream.synchronize" "$TRTLLM_REPO/tensorrt_llm/_torch"
+git -C "$TRTLLM_REPO" log --all --format='%h %s' | rg -i 'overlap|multi-stream|aux stream|cudagraph|mlir|stream|flashinfer|moe|mla'
+rg -n "fuse_gemm_comms|enable_sp|fused_matmul_reduce_scatter|fused_all_gather_matmul|shared_experts_stream|maybe_sync_shared_experts_stream|dcp_alltoall|async_op=True|aux_stream|maybe_execute_in_parallel" "$VLLM_REPO/vllm" "$VLLM_REPO/docs/design/fusions.md"
+git -C "$VLLM_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|allreduce|reduce-scatter|all-gather|all_to_all|stream|multi-stream|triton|cuda|router'
+rg -n "enable_allreduce_fusion|comm_fusion|comm_fusion_max_num_tokens|allreduce|reduce_scatter" "$TOKENSPEED_REPO/python" "$TOKENSPEED_REPO/docs"
+git -C "$TOKENSPEED_REPO" log --all --format='%h %s' | rg -i 'fused|fusion|overlap|allreduce|stream|comm|mla|tokenspeed_mla'
+# GitHub PR scan terms for the connector or web UI:
+#   "fused OR overlap repo:sgl-project/sglang"
+#   "triton OR cutedsl OR cuda overlap repo:sgl-project/sglang"
+#   "fused OR overlap repo:flashinfer-ai/flashinfer"
+#   "pdl OR aux_stream OR green_ctx repo:flashinfer-ai/flashinfer"
+#   "fused OR overlap repo:NVIDIA/TensorRT-LLM"
+#   "multi-stream OR aux_stream OR cudagraph repo:NVIDIA/TensorRT-LLM"
+#   "mlir OR piecewise OR flashinfer repo:NVIDIA/TensorRT-LLM"
+#   "fused OR overlap repo:vllm-project/vllm"
+#   "triton OR cuda overlap repo:vllm-project/vllm"
+#   "multi-stream OR aux_stream overlap repo:vllm-project/vllm"
+#   "fused OR overlap OR comm_fusion repo:lightseekorg/tokenspeed"
+```

.claude/skills/llm-torch-profiler-analysis/references/source-map.md

@@ -0,0 +1,42 @@
+# Source Map
+
+Use these upstream files when the workflow or behavior needs to be justified from SGLang source.
+
+## Profiler entrypoints
+
+- `python/sglang/profiler.py`
+  - live profiler CLI
+  - writes `server_args.json`
+  - forwards `num_steps`, `profile_by_stage`, `merge_profiles`, and `profile_prefix`
+
+- `python/sglang/test/send_one.py`
+  - minimal request path that can trigger profiling from a single command
+
+- `python/sglang/bench_serving.py`
+  - profile-capable serving benchmark path
+  - forwards `profile_activities`, `profile_by_stage`, `profile_stages`, and `profile_prefix`
+
+## Scheduler-side trace writing
+
+- `python/sglang/srt/managers/scheduler_profiler_mixin.py`
+  - actual trace start/stop behavior
+  - filename pattern for `TP/DP/PP/EP` and optional stage suffixes
+  - `CUDA_PROFILER` and torch profiler handling
+
+- `python/sglang/srt/utils/profile_merger.py`
+  - merged distributed trace behavior
+  - why merged traces should be treated differently from rank-local traces
+
+- `python/sglang/srt/utils/profile_utils.py`
+  - newer profile v2 manager path used for stage-based traces
+
+## Documentation and tests
+
+- `docs/developer_guide/benchmark_and_profiling.md`
+  - canonical profiling docs
+
+- `test/registered/profiling/test_start_profile.py`
+  - validates `/start_profile` behavior, including `CUDA_PROFILER`
+
+- `test/registered/profiling/test_profile_v2.py`
+  - validates stage-scoped trace outputs under `SGLANG_PROFILE_V2`

.claude/skills/llm-torch-profiler-analysis/references/vllm-torch-compile-fusions.md

@@ -0,0 +1,74 @@
+# vLLM Torch Compile Fusion Patterns
+
+Refresh: `2026-06-26`.
+Source tree: vLLM `origin/main` at
+`abc71548ef029132c3316b902207f254a246d593`; no new LLM compile-fusion pass was
+added after `2317682f9` in this refresh. The mainline `#40392` MLA RoPE +
+KV-cache cat fusion is already included below. Recent post-`#46735` vLLM
+changes include runtime / frontend work such as `#44800` and `#46799`, but they
+do not add a new LLM compile-fusion pass to this inventory.
+
+Use this file when the fuse-pattern table reports split kernels in a trace and
+you need to decide whether the shape is already covered by vLLM's
+`torch.compile` pattern matcher. Treat every row here as an upstream precedent
+before calling a similar SGLang opportunity novel.
+
+## Pass Registration
+
+vLLM registers these passes from
+`vllm/compilation/passes/pass_manager.py` through `PassConfig`.
+
+| Toggle | Pass | Target shape |
+| --- | --- | --- |
+| `enable_sp` | `SequenceParallelismPass` | all-reduce around residual/norm blocks becomes reduce-scatter, local work, and all-gather |
+| `fuse_gemm_comms` | `AsyncTPPass` | GEMM plus reduce-scatter / all-gather overlap through symmetric-memory collectives |
+| `fuse_allreduce_rms` | `AllReduceFusionPass` or ROCm AITER variant | all-reduce followed by RMSNorm, optional residual add, optional FP8 / NVFP4 quant; current pass ordering runs AITER add-RMSNorm-pad before this fusion when available |
+| `fuse_minimax_qk_norm` | `MiniMaxQKNormPass` | MiniMax Q/K all-reduce plus RMSNorm decode path |
+| `fuse_norm_quant` | `RMSNormQuantFusionPass` | RMSNorm or fused-add-RMSNorm followed by FP8 / FP4 quant |
+| `fuse_norm_quant` + AITER | `RocmAiterRMSNormQuantFusionPass` | ROCm AITER RMSNorm / fused-add-RMSNorm followed by AITER or vLLM quant |
+| `fuse_act_quant` | `ActivationQuantFusionPass` | SiLU-and-mul followed by FP8 / NVFP4 / block quant |
+| `fuse_act_quant` + AITER | `RocmAiterSiluMulFp8GroupQuantFusionPass` | AITER SiLU-and-mul followed by FP8 group quant |
+| `fuse_act_padding` + AITER | `RocmAiterTritonAddRMSNormPadFusionPass` | AITER fused-add-RMSNorm followed by padding into the next layout |
+| `fuse_mla_dual_rms_norm` + AITER | `MLADualRMSNormFusionPass` | MLA paired Q and KV RMSNorms become `fused_mla_dual_rms_norm` |
+| `fuse_rope_kvcache` | `RopeKVCacheFusionPass` | RoPE plus paged KV-cache update, after split cleanup passes |
+| `fuse_rope_kvcache_cat_mla` | `MLARoPEKVCacheCatFusionPass` | MLA RoPE on `q_pe` / `k_pe` plus unified MLA KV-cache update through a fused concat/cache op |
+| `fuse_attn_quant` | `AttnQuantFusionPass` | attention output followed by FP8 / NVFP4 quant |
+| `fuse_attn_quant` | `MLAAttnQuantFusionPass` | MLA attention output followed by FP8 / NVFP4 / FP8 group quant |
+| `enable_qk_norm_rope_fusion` | `QKNormRoPEFusionPass` | Q/K RMSNorm plus RoPE on packed QKV tensors |
+
+## Pattern Inventory
+
+| Source file | Pattern classes | Trace clue | Replacement |
+| --- | --- | --- | --- |
+| `fusion/allreduce_rms_fusion.py` | `AllReduceRMSNormPattern`, `AllReduceFusedAddRMSNormPattern`, `AllReduceFusedRMSNormStaticQuantFP8Pattern`, `AllReduceFusedAddRMSNormStaticQuantFP8Pattern`, `AllReduceFusedRMSNormStaticQuantNVFP4Pattern`, `AllReduceFusedAddRMSNormStaticQuantNVFP4Pattern` | TP all-reduce directly before RMSNorm, residual-add RMSNorm, or quant | `flashinfer_trtllm_fused_allreduce_norm` with FlashInfer allreduce fusion pattern codes |
+| `fusion/rms_quant_fusion.py` | `RMSNormStaticQuantPattern`, `FusedAddRMSNormStaticQuantPattern`, `RMSNormDynamicQuantPattern`, `FusedAddRMSNormDynamicQuantPattern`, `RMSNormGroupQuantPattern`, `FusedAddRMSNormGroupQuantPattern` | RMSNorm or fused-add-RMSNorm followed by static FP8, dynamic per-token FP8, FP8 group quant, or NVFP4 quant | `_C.rms_norm_*_quant`, `_C.fused_add_rms_norm_*_quant`, or per-block quant custom op |
+| `fusion/rocm_aiter_fusion.py` | `AiterRMSNormDynamicQuantPattern`, `AiterFusedAddRMSNormDynamicQuantPattern`, `AiterRMSFp8GroupQuantPattern`, `AiterFusedAddRMSFp8GroupQuantPattern` | AITER RMSNorm/fused-add-RMSNorm followed by AITER or vLLM FP8 quant | AITER fused RMSNorm-quant custom ops |
+| `fusion/act_quant_fusion.py` | `SiluMulFp8StaticQuantPattern`, `SiluMulNvfp4QuantPattern`, `SiluMulBlockQuantPattern` | SiLU-and-mul activation output immediately quantized | fused activation-plus-quant custom op |
+| `fusion/rocm_aiter_fusion.py` | `AiterSiluMulFp8GroupQuantPattern` | AITER SiLU-and-mul followed by FP8 group quant | AITER `act_mul_fused_fp8_group_quant` |
+| `fusion/rocm_aiter_fusion.py` | `AddAiterRMSNormPadPattern` | AITER fused-add-RMSNorm output padded before the next op | AITER add-RMSNorm-pad op |
+| `fusion/rocm_aiter_fusion.py` | `MLADualRMSNormPattern` | MLA Q branch and KV branch each run RMSNorm | `torch.ops.vllm.fused_mla_dual_rms_norm` backed by AITER fused QK RMSNorm |
+| `fusion/qk_norm_rope_fusion.py` | `QkNormRopePattern` | Q/K RMSNorm, split/getitem reshapes, then RoPE | `_C.fused_qk_norm_rope` |
+| `fusion/rope_kvcache_fusion.py` | `RopeReshapeKVCachePattern` | RoPE output followed by reshape/cache update | `vllm.fused_rope_and_unified_kv_cache_update` |
+| `fusion/mla_rope_kvcache_cat_fusion.py` | `MLARoPEKVCacheCatPattern` | MLA RoPE on `q_pe` and `k_pe` flows into `unified_mla_kv_cache_update` | `vllm.fused_rope_unified_mla_kv_cache_update`, backed by `concat_and_cache_mla_rope_fused` |
+| `fusion/attn_quant_fusion.py` | `AttnFp8StaticQuantPattern`, `AttnNvfp4QuantPattern` | attention output followed by FP8 static quant or NVFP4 quant | backend attention op with fused output quant when supported |
+| `fusion/mla_attn_quant_fusion.py` | `MLAAttnFp8StaticQuantPattern`, `MLAAttnNvfp4QuantPattern`, `MLAAttnFp8GroupQuantPattern` | MLA attention output followed by static FP8, NVFP4, or FP8 group quant | MLA attention op with fused output quant when supported |
+| `fusion/minimax_qk_norm_fusion.py` | `MiniMaxQKNormPattern` | MiniMax `forward_qk`: Q/K variance all-reduce divided by TP world size, then RMS apply | `vllm.minimax_qk_norm_fused` / Lamport fused kernel |
+| `fusion/sequence_parallelism.py` | `FirstAllReduceRMSNormPattern`, `MiddleAllReduceRMSNormPattern`, `FirstAllReduceRMSNormStaticFP8Pattern`, `MiddleAllReduceRMSNormStaticFP8Pattern` | all-reduce plus norm block in a full-graph TP model | sequence-parallel reduce-scatter, local norm, all-gather staging |
+| `fusion/collective_fusion.py` | `GEMMReduceScatterPattern`, `AllGatherGEMMPattern`, `ScaledMMReduceScatterPattern`, `AllGatherScaledMMPattern`, `CutlassScaledMMReduceScatterPattern`, `AllGatherCutlassScaledMMPattern`, `FlashInferBMMFP8ReduceScatterPattern`, `FlashInferAllGatherBMMFP8Pattern` | matmul / scaled-mm / FlashInfer BMM adjacent to TP collectives | symmetric-memory fused matmul+reduce-scatter or all-gather+matmul |
+
+## Triage Rules
+
+- If the trace shows split norm/add/quant, compare first against
+  `RMSNormQuantFusionPass`, AITER variants, and `AllReduceFusionPass`.
+- If the trace shows attention output followed by quant kernels, compare against
+  `AttnQuantFusionPass` or `MLAAttnQuantFusionPass`, not only handwritten
+  attention kernels.
+- If the trace shows Q/K norm followed by RoPE or cache update, compare
+  `QKNormRoPEFusionPass`, `RopeKVCacheFusionPass`, and the MLA-specific
+  `MLARoPEKVCacheCatFusionPass`; they are separate passes.
+- If the trace is a TP decode trace with visible collectives, check whether
+  `enable_sp` and `fuse_gemm_comms` would transform the same region into
+  sequence-parallel or AsyncTP overlap.
+- A missing vLLM compile fusion may be intentional when the graph range, backend
+  support check, dtype, token count, or AITER / FlashInfer availability does not
+  satisfy the pass-specific guard.

.claude/skills/llm-torch-profiler-analysis/scripts/analyze_llm_torch_profile.py

@@ -0,0 +1,872 @@
+"""Compact triage entrypoint for unified LLM torch-profiler analysis."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from collections import defaultdict
+from pathlib import Path
+from typing import Dict, List, Optional, Sequence, Tuple
+
+import triage_kernel_helpers as kernel_helpers
+import triage_overlap_helpers as overlap_helpers
+from profile_common import (
+    DEFAULT_DECODE_INPUT_LEN,
+    DEFAULT_DECODE_OUTPUT_LEN,
+    DEFAULT_PREFILL_INPUT_LEN,
+    DEFAULT_PREFILL_OUTPUT_LEN,
+    DEFAULT_WARMUP_STEPS,
+    PROFILE_WORKLOAD_CHOICES,
+    discover_trace_targets,
+    framework_display_name,
+    load_server_args,
+    load_trace_json,
+    parse_stage,
+    resolve_framework,
+    run_profiler,
+)
+
+MIN_RENDER_SHARE_PCT = 1.0
+MAPPING_KERNEL_SAMPLE_LIMIT_PER_NAME = 16
+
+
+def build_triage_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="analyze_llm_torch_profile.py",
+        description=(
+            "Compact LLM torch-profiler triage entrypoint for SGLang, vLLM, "
+            "TensorRT-LLM, and TokenSpeed. "
+            "This prints three tables: kernel mapping, overlap opportunities, "
+            "and fuse opportunities. "
+            "Use either a single trace/profile input or a mapping+formal two-trace pair."
+        ),
+    )
+    parser.add_argument(
+        "--framework",
+        type=str,
+        default="auto",
+        choices=[
+            "auto",
+            "sglang",
+            "vllm",
+            "trtllm",
+            "tllm",
+            "tensorrt-llm",
+            "tokenspeed",
+            "token-speed",
+            "ts",
+        ],
+        help=(
+            "Serving framework. Use auto to detect from trace contents, path hints, "
+            "or URL features."
+        ),
+    )
+    parser.add_argument(
+        "--input",
+        type=str,
+        default=None,
+        help="Single trace file or profile directory to triage.",
+    )
+    parser.add_argument(
+        "--url",
+        type=str,
+        default=None,
+        help=(
+            "Running server URL for single-trace triage. SGLang supports direct "
+            "capture via sglang.profiler. vLLM and TensorRT-LLM require a server-side "
+            "torch-profiler output path exposed via --output-dir. TokenSpeed live "
+            "capture uses the server's /start_profile and /stop_profile endpoints "
+            "when they are available."
+        ),
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=str,
+        default=None,
+        help=(
+            "Trace output dir when using --url. For vLLM this should match the "
+            "server's torch_profiler_dir. For TensorRT-LLM it should match the "
+            "directory or file path configured by TLLM_TORCH_PROFILE_TRACE. "
+            "For TokenSpeed this is passed as start_profile.output_dir."
+        ),
+    )
+    parser.add_argument(
+        "--profile-prefix",
+        type=str,
+        default="triage-trace",
+        help=(
+            "Profile prefix when generating a trace from --url. SGLang uses it "
+            "directly; TokenSpeed maps it to profile_id; vLLM and TensorRT-LLM may "
+            "ignore it on the HTTP profiler path."
+        ),
+    )
+    parser.add_argument(
+        "--mapping-input",
+        type=str,
+        default=None,
+        help="Graph-off mapping trace file or directory.",
+    )
+    parser.add_argument(
+        "--mapping-url",
+        type=str,
+        default=None,
+        help="Running graph-off server URL for the mapping trace.",
+    )
+    parser.add_argument(
+        "--formal-input",
+        type=str,
+        default=None,
+        help="Formal graph-on trace file or directory.",
+    )
+    parser.add_argument(
+        "--formal-url",
+        type=str,
+        default=None,
+        help="Running graph-on server URL for the formal trace.",
+    )
+    parser.add_argument(
+        "--mapping-output-dir",
+        type=str,
+        default=None,
+        help="Trace output dir when using --mapping-url.",
+    )
+    parser.add_argument(
+        "--formal-output-dir",
+        type=str,
+        default=None,
+        help="Trace output dir when using --formal-url.",
+    )
+    parser.add_argument(
+        "--mapping-profile-prefix",
+        type=str,
+        default="mapping-trace",
+        help="Profile prefix for the mapping trace.",
+    )
+    parser.add_argument(
+        "--formal-profile-prefix",
+        type=str,
+        default="formal-trace",
+        help="Profile prefix for the formal trace.",
+    )
+    parser.add_argument(
+        "--num-steps",
+        type=int,
+        default=5,
+        help="Active profiler steps when generating traces from URLs.",
+    )
+    parser.add_argument(
+        "--warmup-steps",
+        type=int,
+        default=DEFAULT_WARMUP_STEPS,
+        help="Warmup steps to run before arming the profiler for URL capture.",
+    )
+    parser.add_argument(
+        "--profile-by-stage", action=argparse.BooleanOptionalAction, default=True
+    )
+    parser.add_argument(
+        "--merge-profiles", action=argparse.BooleanOptionalAction, default=False
+    )
+    parser.add_argument("--probe-requests", type=int, default=1)
+    parser.add_argument(
+        "--probe-prompt",
+        type=str,
+        default=(
+            "Repeat the word profiler many times with spaces so the server performs several decode steps. "
+            "Do not add explanations."
+        ),
+    )
+    parser.add_argument("--probe-max-new-tokens", type=int, default=None)
+    parser.add_argument("--probe-delay", type=float, default=0.5)
+    parser.add_argument(
+        "--profile-workload",
+        choices=PROFILE_WORKLOAD_CHOICES,
+        default="both",
+        help=(
+            "Live-capture workload shape. Default 'both' captures separate "
+            "prefill and decode profiles instead of one mixed request. Use "
+            "'legacy' to keep the old --probe-prompt behavior."
+        ),
+    )
+    parser.add_argument(
+        "--prefill-input-len",
+        type=int,
+        default=DEFAULT_PREFILL_INPUT_LEN,
+        help="Synthetic input length for the prefill profile workload.",
+    )
+    parser.add_argument(
+        "--prefill-output-len",
+        type=int,
+        default=DEFAULT_PREFILL_OUTPUT_LEN,
+        help="Output length for the prefill profile workload.",
+    )
+    parser.add_argument(
+        "--decode-input-len",
+        type=int,
+        default=DEFAULT_DECODE_INPUT_LEN,
+        help="Synthetic input length for the decode profile workload.",
+    )
+    parser.add_argument(
+        "--decode-output-len",
+        type=int,
+        default=DEFAULT_DECODE_OUTPUT_LEN,
+        help="Output length for the decode profile workload.",
+    )
+    parser.add_argument(
+        "--start-step",
+        type=int,
+        default=None,
+        help="Pass through to sglang.profiler when generating traces from URLs.",
+    )
+    parser.add_argument(
+        "--pid-substring",
+        type=str,
+        default=None,
+        help="Restrict overlap analysis to PIDs containing this substring.",
+    )
+    parser.add_argument(
+        "--kernel-table-limit",
+        type=int,
+        default=0,
+        help="How many kernel rows to print per stage. Use 0 for all kernels.",
+    )
+    parser.add_argument(
+        "--overlap-table-limit",
+        type=int,
+        default=0,
+        help="How many overlap rows to print per stage. Use 0 for all kernels.",
+    )
+    return parser
+
+
+def parse_triage_args(argv: Sequence[str]) -> argparse.Namespace:
+    parser = build_triage_parser()
+    args = parser.parse_args(argv)
+
+    single_trace_mode = bool(args.input) or bool(args.url)
+    dual_trace_mode = any(
+        [
+            args.mapping_input,
+            args.mapping_url,
+            args.formal_input,
+            args.formal_url,
+        ]
+    )
+
+    if single_trace_mode and dual_trace_mode:
+        parser.error(
+            "Use either single-trace mode (--input/--url) or two-trace mode "
+            "(--mapping-* plus --formal-*), not both."
+        )
+
+    if single_trace_mode:
+        if bool(args.input) == bool(args.url):
+            parser.error("Provide exactly one of --input or --url.")
+        return args
+
+    if bool(args.mapping_input) == bool(args.mapping_url):
+        parser.error("Provide exactly one of --mapping-input or --mapping-url.")
+    if bool(args.formal_input) == bool(args.formal_url):
+        parser.error("Provide exactly one of --formal-input or --formal-url.")
+    return args
+
+
+def resolve_profile_targets(
+    *,
+    label: str,
+    input_path: Optional[str],
+    url: Optional[str],
+    output_dir: Optional[str],
+    profile_prefix: Optional[str],
+    args: argparse.Namespace,
+) -> Tuple[List[Path], Optional[dict], str]:
+    if bool(input_path) == bool(url):
+        raise ValueError(f"{label} trace requires exactly one of input path or URL.")
+
+    if url:
+        framework = resolve_framework(
+            args.framework,
+            input_path=Path(output_dir).resolve() if output_dir else None,
+            url=url,
+        )
+        target_dir = run_profiler(
+            url=url,
+            output_dir=output_dir,
+            num_steps=args.num_steps,
+            profile_by_stage=args.profile_by_stage,
+            merge_profiles=args.merge_profiles,
+            profile_prefix=profile_prefix,
+            probe_requests=max(0, args.probe_requests),
+            probe_prompt=args.probe_prompt,
+            probe_max_new_tokens=args.probe_max_new_tokens,
+            probe_delay=args.probe_delay,
+            warmup_steps=args.warmup_steps,
+            start_step=args.start_step,
+            framework=framework,
+            framework_hint_path=output_dir,
+            profile_workload=args.profile_workload,
+            prefill_input_len=args.prefill_input_len,
+            prefill_output_len=args.prefill_output_len,
+            decode_input_len=args.decode_input_len,
+            decode_output_len=args.decode_output_len,
+        )
+        traces, server_args = discover_trace_targets(target_dir, all_traces=False)
+        resolved_framework = resolve_framework(
+            args.framework,
+            input_path=target_dir,
+            url=url,
+            server_args=server_args,
+        )
+        return traces, server_args, resolved_framework
+
+    resolved = Path(input_path).resolve()
+    traces, server_args = discover_trace_targets(resolved, all_traces=False)
+    if server_args is None:
+        server_args = load_server_args(resolved)
+    framework = resolve_framework(
+        args.framework, input_path=resolved, server_args=server_args
+    )
+    return traces, server_args, framework
+
+
+def build_mapping_kernel_map(trace_paths: Sequence[Path], framework: str) -> dict:
+    stage_site_stats = defaultdict(
+        lambda: defaultdict(lambda: defaultdict(kernel_helpers.MappingSiteAggregate))
+    )
+    stage_kernel_categories: Dict[str, Dict[str, str]] = defaultdict(dict)
+    global_site_stats = defaultdict(
+        lambda: defaultdict(kernel_helpers.MappingSiteAggregate)
+    )
+    global_kernel_categories: Dict[str, str] = {}
+
+    for trace_path in trace_paths:
+        trace = load_trace_json(trace_path)
+        kernels, cpu_ops, python_frames, launch_events, _, _ = (
+            kernel_helpers.extract_trace_data(trace)
+        )
+        if not kernels:
+            continue
+        cpu_ops_by_external_id = kernel_helpers.build_cpu_op_index(cpu_ops)
+        launches_by_correlation = kernel_helpers.build_launch_index(launch_events)
+        site_context_cache = {}
+        default_stage = parse_stage(trace_path)
+        for stage, stage_kernels in kernel_helpers.group_kernels_by_stage(
+            kernels, default_stage
+        ).items():
+            sampled_stage_kernels = (
+                stage_kernels
+                if framework == "sglang"
+                else sample_kernels_for_mapping(stage_kernels)
+            )
+            local_site_stats = kernel_helpers.aggregate_kernel_sites(
+                sampled_stage_kernels,
+                cpu_ops_by_external_id,
+                python_frames,
+                launches_by_correlation=launches_by_correlation,
+                site_context_cache=site_context_cache,
+            )
+            kernel_categories = {
+                kernel.canonical_name: kernel.category for kernel in stage_kernels
+            }
+            kernel_helpers.merge_site_stats(stage_site_stats[stage], local_site_stats)
+            kernel_helpers.merge_site_stats(global_site_stats, local_site_stats)
+            stage_kernel_categories[stage].update(kernel_categories)
+            global_kernel_categories.update(kernel_categories)
+
+    stage_payloads = {
+        stage: kernel_helpers.build_stage_payload(
+            dict(site_stats), stage_kernel_categories.get(stage, {})
+        )
+        for stage, site_stats in stage_site_stats.items()
+    }
+    global_payload = kernel_helpers.build_stage_payload(
+        dict(global_site_stats), global_kernel_categories
+    )
+    return {"stages": stage_payloads, "global": global_payload}
+
+
+def stage_index(stage: str) -> int:
+    return {"extend": 0, "prefill": 0, "decode": 1, "all": 2}.get(stage, 99)
+
+
+def sample_kernels_for_mapping(
+    kernels: Sequence[kernel_helpers.KernelEvent],
+    per_name_limit: int = MAPPING_KERNEL_SAMPLE_LIMIT_PER_NAME,
+) -> List[kernel_helpers.KernelEvent]:
+    if per_name_limit <= 0:
+        return list(kernels)
+
+    grouped: Dict[str, List[kernel_helpers.KernelEvent]] = defaultdict(list)
+    for kernel in kernels:
+        grouped[kernel.canonical_name].append(kernel)
+
+    sampled: List[kernel_helpers.KernelEvent] = []
+    for kernel_name in sorted(grouped):
+        items = grouped[kernel_name]
+        if len(items) <= per_name_limit:
+            sampled.extend(items)
+            continue
+        for sample_idx in range(per_name_limit):
+            pos = round(sample_idx * (len(items) - 1) / (per_name_limit - 1))
+            sampled.append(items[pos])
+    sampled.sort(key=lambda kernel: (kernel.ts, kernel.name))
+    return sampled
+
+
+def stage_display(stage: str) -> str:
+    return kernel_helpers.stage_label(stage)
+
+
+def pick_stage_value(stage_to_value: Dict[str, object], stage: str) -> Optional[object]:
+    if stage in stage_to_value:
+        return stage_to_value[stage]
+    if "all" in stage_to_value:
+        return stage_to_value["all"]
+    if len(stage_to_value) == 1:
+        return next(iter(stage_to_value.values()))
+    return None
+
+
+def render_stages(stage_to_value: Dict[str, object]) -> List[str]:
+    stages = set(stage_to_value)
+    if any(stage != "all" for stage in stages):
+        stages.discard("all")
+    return sorted(stages, key=stage_index)
+
+
+def build_overlap_stage_bundle_map(
+    trace_paths: Sequence[Path],
+    *,
+    label_prefix: str,
+    server_args: Optional[dict],
+    pid_substring: Optional[str],
+) -> Dict[str, overlap_helpers.TraceBundle]:
+    stage_bundles: Dict[str, overlap_helpers.TraceBundle] = {}
+    for trace_path in sorted(
+        trace_paths, key=lambda item: (stage_index(parse_stage(item)), item.name)
+    ):
+        trace_json = load_trace_json(trace_path)
+        raw_events = trace_json.get(
+            "traceEvents",
+            trace_json if isinstance(trace_json, list) else [],
+        )
+        events, pid = overlap_helpers.extract_kernel_events(trace_json, pid_substring)
+        if not events:
+            continue
+        default_stage = parse_stage(trace_path)
+        stage_groups = overlap_helpers.group_events_by_stage(events, default_stage)
+        for stage in render_stages(stage_groups):
+            if stage in stage_bundles:
+                continue
+            stage_bundles[stage] = overlap_helpers.TraceBundle(
+                label=f"{label_prefix}-{stage}",
+                trace_path=trace_path,
+                server_args=server_args,
+                raw_events=raw_events,
+                events=stage_groups[stage],
+                pid=pid,
+            )
+        if "all" in stage_groups and not stage_bundles:
+            stage_bundles["all"] = overlap_helpers.TraceBundle(
+                label=f"{label_prefix}-all",
+                trace_path=trace_path,
+                server_args=server_args,
+                raw_events=raw_events,
+                events=stage_groups["all"],
+                pid=pid,
+            )
+    return stage_bundles
+
+
+def group_rows_by_stage(rows: Sequence[dict]) -> List[Tuple[str, List[dict]]]:
+    grouped: Dict[str, List[dict]] = defaultdict(list)
+    for row in rows:
+        grouped[str(row.get("stage") or "all")].append(row)
+    return [
+        (stage, grouped[stage]) for stage in sorted(grouped.keys(), key=stage_index)
+    ]
+
+
+def render_kernel_table_for_stage(rows: Sequence[dict]) -> List[str]:
+    lines = [
+        "| Kernel | Category | GPU time | Share | Launches | Python location (site share) | CPU op |",
+        "| --- | --- | ---: | ---: | ---: | --- | --- |",
+    ]
+    if not rows:
+        lines.append(
+            "| No kernel rows at or above 1.0% share. | - | - | - | - | - | - |"
+        )
+        return lines
+    for row in rows:
+        lines.append(
+            "| {kernel} | {category} | {gpu_time} | {share:.1f}% | {launches} | {location} | {cpu_op} |".format(
+                kernel=kernel_helpers.escape_md_cell(row["kernel"]),
+                category=kernel_helpers.escape_md_cell(row["category"]),
+                gpu_time=kernel_helpers.format_ms(row["total_us"]),
+                share=row["share_pct"],
+                launches=row["launches"],
+                location=kernel_helpers.escape_md_cell(row["location"]),
+                cpu_op=kernel_helpers.escape_md_cell(row["cpu_op"]),
+            )
+        )
+    return lines
+
+
+def render_stage_section_tables(
+    rows: Sequence[dict],
+    *,
+    render_stage_fn,
+    stage_label_prefix: str = "#####",
+) -> List[str]:
+    if not rows:
+        return render_stage_fn([])
+    stage_groups = group_rows_by_stage(rows)
+    if len(stage_groups) == 1 and stage_groups[0][0] == "all":
+        return render_stage_fn(stage_groups[0][1])
+
+    lines: List[str] = []
+    for index, (stage, stage_rows) in enumerate(stage_groups):
+        lines.append(f"{stage_label_prefix} {stage_display(stage)}")
+        lines.extend(render_stage_fn(stage_rows))
+        if index != len(stage_groups) - 1:
+            lines.append("")
+    return lines
+
+
+def render_kernel_tables(rows: Sequence[dict]) -> List[str]:
+    return render_stage_section_tables(
+        rows, render_stage_fn=render_kernel_table_for_stage
+    )
+
+
+def render_overlap_table_for_stage(rows: Sequence[dict]) -> List[str]:
+    lines = [
+        "| Priority | Verdict | Kernel | Python scope | Formal signal | Dep risk | Recommendation |",
+        "| --- | --- | --- | --- | --- | --- | --- |",
+    ]
+    if not rows:
+        lines.append(
+            "| - | - | No rows cleared the 1.0% reporting bar. Use mapping/formal mode for overlap attribution. | - | - | - | - |"
+        )
+        return lines
+    for row in rows:
+        formal_signal = (
+            f"{row['total_us']:.1f} us, share {row['share_pct']:.1f}%, "
+            f"excl {row['exclusive_ratio'] * 100:.1f}% / hid {row['hidden_ratio'] * 100:.1f}%"
+        )
+        lines.append(
+            "| "
+            + " | ".join(
+                [
+                    row["priority"],
+                    row["verdict"],
+                    kernel_helpers.escape_md_cell(row["kernel"]),
+                    kernel_helpers.escape_md_cell(row["python_scope"]),
+                    kernel_helpers.escape_md_cell(formal_signal),
+                    overlap_helpers.dependency_risk_label(row["dependency_signal"]),
+                    row["recommendation"],
+                ]
+            )
+            + " |"
+        )
+    return lines
+
+
+def render_overlap_tables(rows: Sequence[dict]) -> List[str]:
+    return render_stage_section_tables(
+        rows,
+        render_stage_fn=render_overlap_table_for_stage,
+    )
+
+
+def render_fuse_table_for_stage(rows: Sequence[dict]) -> List[str]:
+    lines = [
+        "| Pattern | Confidence | Related GPU time | Share | Evidence kernels | Current kernel Python location | Candidate fused Python path | Rationale |",
+        "| --- | --- | ---: | ---: | --- | --- | --- | --- |",
+    ]
+    if not rows:
+        lines.append(
+            "| No medium-confidence source-backed fusion opportunity matched this trace. | - | - | - | - | - | - | - |"
+        )
+        return lines
+    for row in rows:
+        lines.append(
+            "| {pattern} | {confidence} | {gpu_time} | {share:.1f}% | {evidence} | {current_locations} | {candidate_path} | {rationale} |".format(
+                pattern=kernel_helpers.escape_md_cell(row["pattern"]),
+                confidence=kernel_helpers.escape_md_cell(row["confidence"]),
+                gpu_time=kernel_helpers.format_ms(row["related_us"]),
+                share=row["share_pct"],
+                evidence=kernel_helpers.escape_md_cell(row["evidence"]),
+                current_locations=kernel_helpers.escape_md_cell(
+                    row["current_locations"]
+                ),
+                candidate_path=kernel_helpers.escape_md_cell(row["candidate_path"]),
+                rationale=kernel_helpers.escape_md_cell(row["rationale"]),
+            )
+        )
+    return lines
+
+
+def render_fuse_tables(rows: Sequence[dict]) -> List[str]:
+    return render_stage_section_tables(
+        rows,
+        render_stage_fn=render_fuse_table_for_stage,
+    )
+
+
+def run_triage(args: argparse.Namespace) -> int:
+    single_trace_mode = bool(args.input) or bool(args.url)
+    if single_trace_mode:
+        formal_traces, formal_server_args, formal_framework = resolve_profile_targets(
+            label="input",
+            input_path=args.input,
+            url=args.url,
+            output_dir=args.output_dir,
+            profile_prefix=args.profile_prefix,
+            args=args,
+        )
+        mapping_traces = formal_traces
+        mapping_server_args = formal_server_args
+        mapping_framework = formal_framework
+    else:
+        mapping_traces, mapping_server_args, mapping_framework = (
+            resolve_profile_targets(
+                label="mapping",
+                input_path=args.mapping_input,
+                url=args.mapping_url,
+                output_dir=args.mapping_output_dir,
+                profile_prefix=args.mapping_profile_prefix,
+                args=args,
+            )
+        )
+        formal_traces, formal_server_args, formal_framework = resolve_profile_targets(
+            label="formal",
+            input_path=args.formal_input,
+            url=args.formal_url,
+            output_dir=args.formal_output_dir,
+            profile_prefix=args.formal_profile_prefix,
+            args=args,
+        )
+
+    mapping_kernel_map = build_mapping_kernel_map(mapping_traces, mapping_framework)
+
+    kernel_rows_rendered: List[dict] = []
+    fuse_rows_rendered: List[dict] = []
+    formal_stage_payloads: Dict[str, dict] = {}
+
+    for formal_trace in formal_traces:
+        trace = load_trace_json(formal_trace)
+        kernels, cpu_ops, python_frames, launch_events, _, _ = (
+            kernel_helpers.extract_trace_data(trace)
+        )
+        if not kernels:
+            continue
+        default_stage = parse_stage(formal_trace)
+        stage_groups = kernel_helpers.group_kernels_by_stage(kernels, default_stage)
+        formal_cpu_ops_by_external_id = kernel_helpers.build_cpu_op_index(cpu_ops)
+        formal_launches_by_correlation = kernel_helpers.build_launch_index(
+            launch_events
+        )
+        formal_site_context_cache = {}
+        for stage_name, stage_kernels in stage_groups.items():
+            local_site_stats = kernel_helpers.aggregate_kernel_sites(
+                stage_kernels,
+                formal_cpu_ops_by_external_id,
+                python_frames,
+                launches_by_correlation=formal_launches_by_correlation,
+                site_context_cache=formal_site_context_cache,
+            )
+            formal_stage_payloads[stage_name] = kernel_helpers.build_stage_payload(
+                local_site_stats,
+                {kernel.canonical_name: kernel.category for kernel in stage_kernels},
+            )
+        trace_total_us = sum(kernel.dur for kernel in kernels)
+        for stage in sorted(stage_groups, key=stage_index):
+            stage_kernels = stage_groups[stage]
+            if not stage_kernels:
+                continue
+            total_us = sum(kernel.dur for kernel in stage_kernels)
+            if (
+                stage == "all"
+                and default_stage == "all"
+                and kernel_helpers.pct(total_us, trace_total_us) < MIN_RENDER_SHARE_PCT
+            ):
+                continue
+            kernel_stats = kernel_helpers.aggregate(
+                stage_kernels, key_fn=lambda item: item.canonical_name
+            )
+            kernel_categories = {
+                kernel.canonical_name: kernel.category for kernel in stage_kernels
+            }
+            full_kernel_rows = kernel_helpers.build_kernel_rows(
+                stage=stage,
+                kernel_stats=kernel_stats,
+                kernel_categories=kernel_categories,
+                local_stage_payload=formal_stage_payloads.get(stage, {"kernels": {}}),
+                external_kernel_map=mapping_kernel_map,
+            )
+            visible_kernel_rows = kernel_helpers.limit_kernel_rows(
+                full_kernel_rows, args.kernel_table_limit
+            )
+            for row in visible_kernel_rows:
+                share_pct = kernel_helpers.pct(row.total_us, total_us)
+                if share_pct < MIN_RENDER_SHARE_PCT:
+                    continue
+                kernel_rows_rendered.append(
+                    {
+                        "stage": stage,
+                        "kernel": row.name,
+                        "category": row.category,
+                        "total_us": row.total_us,
+                        "share_pct": share_pct,
+                        "launches": row.aggregate.count,
+                        "location": row.location,
+                        "cpu_op": row.cpu_op,
+                    }
+                )
+            for item in kernel_helpers.detect_fusion_opportunities(
+                kernel_rows=full_kernel_rows,
+                total_us=total_us,
+                server_args=formal_server_args or mapping_server_args,
+                framework=formal_framework,
+            ):
+                share_pct = kernel_helpers.pct(item.related_us, total_us)
+                if share_pct < MIN_RENDER_SHARE_PCT:
+                    continue
+                fuse_rows_rendered.append(
+                    {
+                        "stage": stage,
+                        "pattern": item.pattern,
+                        "confidence": item.confidence,
+                        "related_us": item.related_us,
+                        "share_pct": share_pct,
+                        "evidence": item.evidence,
+                        "current_locations": item.current_locations,
+                        "candidate_path": item.candidate_path,
+                        "rationale": item.rationale,
+                    }
+                )
+
+    overlap_rows_rendered: List[dict] = []
+    if not single_trace_mode:
+        mapping_overlap_bundles = build_overlap_stage_bundle_map(
+            mapping_traces,
+            label_prefix="mapping",
+            server_args=mapping_server_args,
+            pid_substring=args.pid_substring,
+        )
+        formal_overlap_bundles = build_overlap_stage_bundle_map(
+            formal_traces,
+            label_prefix="formal",
+            server_args=formal_server_args,
+            pid_substring=args.pid_substring,
+        )
+        for stage in render_stages(formal_overlap_bundles):
+            formal_bundle = pick_stage_value(formal_overlap_bundles, stage)
+            mapping_bundle = pick_stage_value(mapping_overlap_bundles, stage)
+            if formal_bundle is None or mapping_bundle is None:
+                continue
+            formal_bundle.overlap_stats = overlap_helpers.analyze_overlap(
+                formal_bundle.events
+            )
+            aggregates = overlap_helpers.aggregate_events(formal_bundle.events)
+            source_map = overlap_helpers.build_kernel_source_map(
+                mapping_bundle,
+                kernel_map_entry_lookup=lambda stage_name, kernel_name: (
+                    kernel_helpers.lookup_kernel_map_entry(
+                        mapping_kernel_map, stage_name, kernel_name
+                    )
+                    if mapping_kernel_map
+                    else None
+                ),
+                stage=stage,
+            )
+            source_map = overlap_helpers.merge_source_map_from_kernel_payload(
+                source_map,
+                pick_stage_value(formal_stage_payloads, stage),
+            )
+            stage_rows = overlap_helpers.build_action_rows(
+                aggregates,
+                source_map,
+                formal_bundle.events,
+                formal_bundle.overlap_stats["total_busy_us"],
+                table_limit=max(0, args.overlap_table_limit),
+            )
+            for row in stage_rows:
+                if row.share_pct < MIN_RENDER_SHARE_PCT:
+                    continue
+                overlap_rows_rendered.append(
+                    {
+                        "stage": stage,
+                        "priority": row.priority,
+                        "verdict": row.verdict,
+                        "kernel": row.kernel,
+                        "python_scope": row.python_scope,
+                        "total_us": row.total_us,
+                        "share_pct": row.share_pct,
+                        "exclusive_ratio": row.exclusive_ratio,
+                        "hidden_ratio": row.hidden_ratio,
+                        "dependency_signal": row.dependency_signal,
+                        "recommendation": row.recommendation,
+                    }
+                )
+
+    lines: List[str] = []
+    lines.append("Triage View")
+    lines.append(f"Mode: {'single-trace' if single_trace_mode else 'mapping-formal'}")
+    if single_trace_mode:
+        lines.append(f"Framework: {framework_display_name(formal_framework)}")
+        lines.append(f"Input traces: {', '.join(str(path) for path in formal_traces)}")
+    else:
+        if mapping_framework == formal_framework:
+            lines.append(f"Framework: {framework_display_name(formal_framework)}")
+        else:
+            lines.append(
+                f"Mapping framework: {framework_display_name(mapping_framework)}"
+            )
+            lines.append(
+                f"Formal framework: {framework_display_name(formal_framework)}"
+            )
+        lines.append(
+            f"Mapping traces: {', '.join(str(path) for path in mapping_traces)}"
+        )
+        lines.append(f"Formal traces: {', '.join(str(path) for path in formal_traces)}")
+    if formal_server_args or mapping_server_args:
+        server_args = formal_server_args or mapping_server_args
+        model = server_args.get("model_path") or server_args.get("model")
+        if model:
+            lines.append(f"Model: {model}")
+    lines.append("")
+    lines.append("Kernel Table")
+    lines.extend(render_kernel_tables(kernel_rows_rendered))
+    lines.append("")
+    lines.append("Overlap Opportunity Table")
+    lines.extend(render_overlap_tables(overlap_rows_rendered))
+    lines.append("")
+    lines.append("Fuse Opportunity Table")
+    lines.extend(render_fuse_tables(fuse_rows_rendered))
+    print("\n".join(lines).rstrip())
+    return 0
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+    argv = list(argv or sys.argv[1:])
+    triage_parser = build_triage_parser()
+
+    if not argv or argv[0] in {"-h", "--help"}:
+        triage_parser.print_help()
+        return 0
+
+    if argv[0] == "triage":
+        argv = argv[1:]
+    elif not argv[0].startswith("-"):
+        triage_parser.error(
+            "This skill exposes only the triage workflow. "
+            "Use single-trace mode (--input/--url) or mapping+formal two-trace mode."
+        )
+        return 2
+
+    return run_triage(parse_triage_args(argv))
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

.claude/skills/llm-torch-profiler-analysis/scripts/analyze_sglang_torch_profile.py

@@ -0,0 +1,16 @@
+"""Backwards-compatibility shim for the unified LLM torch-profiler entrypoint.
+
+The real implementation now lives in ``analyze_llm_torch_profile`` because this
+skill covers SGLang, vLLM, TensorRT-LLM, and TokenSpeed. Older scripts and runbooks that
+still invoke ``analyze_sglang_torch_profile.py`` keep working by forwarding to
+that module.
+"""
+
+from __future__ import annotations
+
+import sys
+
+from analyze_llm_torch_profile import main
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv[1:]))

.claude/skills/llm-torch-profiler-analysis/scripts/make_trtllm_py_executor_override.py

@@ -0,0 +1,132 @@
+"""Generate a TensorRT-LLM py_executor override for stable torch-profiler capture."""
+
+from __future__ import annotations
+
+import argparse
+from dataclasses import dataclass
+from pathlib import Path
+
+START_MARKER = "torch_profiler = torch.profiler.profile("
+
+
+@dataclass
+class ProfileCallSpan:
+    start: int
+    end: int
+    block: str
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Create a py_executor.py override that enables with_stack=True for "
+            "TensorRT-LLM torch-profiler traces."
+        )
+    )
+    parser.add_argument("--source", required=True, help="Original py_executor.py path.")
+    parser.add_argument("--output", required=True, help="Override file path to write.")
+    return parser.parse_args()
+
+
+def find_profile_call_span(text: str) -> ProfileCallSpan:
+    start = text.find(START_MARKER)
+    if start == -1:
+        raise SystemExit("Could not find torch profiler setup in source file.")
+
+    open_paren = text.find("(", start)
+    if open_paren == -1:
+        raise SystemExit("Malformed torch profiler setup in source file.")
+
+    depth = 0
+    for index in range(open_paren, len(text)):
+        char = text[index]
+        if char == "(":
+            depth += 1
+        elif char == ")":
+            depth -= 1
+            if depth == 0:
+                return ProfileCallSpan(
+                    start=start,
+                    end=index + 1,
+                    block=text[start : index + 1],
+                )
+    raise SystemExit("Could not find the end of the torch profiler call.")
+
+
+def inject_with_stack(block: str) -> str:
+    if "with_stack=" in block:
+        return block
+
+    lines = block.splitlines()
+    if not lines:
+        raise SystemExit("Unexpected torch profiler block format.")
+
+    last_line = lines[-1]
+    if not last_line.strip():
+        raise SystemExit("Unexpected torch profiler block terminator.")
+
+    if last_line.strip() == ")":
+        if len(lines) < 2:
+            raise SystemExit("Could not find the last torch profiler argument line.")
+        last_arg_index = len(lines) - 2
+        last_arg_line = lines[last_arg_index]
+        indent = last_arg_line[: len(last_arg_line) - len(last_arg_line.lstrip())]
+        if not last_arg_line.rstrip().endswith(","):
+            lines[last_arg_index] = last_arg_line.rstrip() + ","
+        lines.insert(len(lines) - 1, f"{indent}with_stack=True")
+        return "\n".join(lines)
+
+    if not last_line.rstrip().endswith(")"):
+        raise SystemExit("Unexpected torch profiler block terminator.")
+
+    indent = last_line[: len(last_line) - len(last_line.lstrip())]
+    last_arg_text = last_line.rstrip()[:-1].rstrip()
+    if not last_arg_text.endswith(","):
+        last_arg_text += ","
+    lines[-1] = last_arg_text
+    lines.append(f"{indent}with_stack=True)")
+    return "\n".join(lines)
+
+
+def inject_rank0_trace_guard(text: str) -> str:
+    needle = (
+        "        enable_torch_trace = bool(torch_trace_path and profile_start_stop)\n"
+    )
+    replacement = (
+        "        # Multi-rank PyTorch backend workers race on the same chrome-trace "
+        "path.\n"
+        "        # Keep the full torch-profiler trace on rank 0 and let the other "
+        "ranks\n"
+        "        # continue with CUDA-profiler gating only.\n"
+        "        enable_torch_trace = bool(\n"
+        "            torch_trace_path and profile_start_stop and self.dist.rank == 0\n"
+        "        )\n"
+    )
+    if replacement in text:
+        return text
+    if needle not in text:
+        raise SystemExit("Could not find enable_torch_trace assignment in source file.")
+    return text.replace(needle, replacement, 1)
+
+
+def main() -> int:
+    args = parse_args()
+    source = Path(args.source).expanduser().resolve()
+    output = Path(args.output).expanduser().resolve()
+    text = source.read_text(encoding="utf-8")
+    span = find_profile_call_span(text)
+    patched_block = inject_with_stack(span.block)
+    patched = (
+        text
+        if patched_block == span.block
+        else (text[: span.start] + patched_block + text[span.end :])
+    )
+    patched = inject_rank0_trace_guard(patched)
+    output.parent.mkdir(parents=True, exist_ok=True)
+    output.write_text(patched, encoding="utf-8")
+    print(output)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.claude/skills/llm-torch-profiler-analysis/scripts/probe_llm_server.py

@@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+"""Run a small correctness and latency probe against an LLM server."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import statistics
+import time
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+from urllib import request
+
+from profile_common import extract_openai_chat_text
+
+DEFAULT_PROMPTS = [
+    "Introduce Shanghai in one short sentence.",
+    "What is 2+2? Answer briefly.",
+    "Write one short haiku about GPUs.",
+]
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Send a few short requests to an LLM server and record latency plus "
+            "sample outputs."
+        )
+    )
+    parser.add_argument(
+        "--framework",
+        required=True,
+        choices=("sglang", "vllm", "trtllm", "tokenspeed"),
+        help="Serving framework.",
+    )
+    parser.add_argument(
+        "--url",
+        required=True,
+        help="Server base URL, for example http://127.0.0.1:30000.",
+    )
+    parser.add_argument(
+        "--model",
+        default=None,
+        help="OpenAI model id. Auto-discovered for vLLM, TensorRT-LLM, and TokenSpeed when omitted.",
+    )
+    parser.add_argument(
+        "--requests",
+        type=int,
+        default=6,
+        help="How many probe requests to send.",
+    )
+    parser.add_argument(
+        "--max-tokens",
+        type=int,
+        default=48,
+        help="Generation length for each request.",
+    )
+    parser.add_argument(
+        "--timeout",
+        type=float,
+        default=180.0,
+        help="Per-request timeout in seconds.",
+    )
+    parser.add_argument(
+        "--prompt",
+        action="append",
+        default=[],
+        help="Optional prompt override. Repeat to add more prompts.",
+    )
+    parser.add_argument(
+        "--output",
+        default=None,
+        help="Optional JSON output path.",
+    )
+    return parser.parse_args()
+
+
+def post_json(url: str, payload: Dict[str, Any], timeout: float) -> Dict[str, Any]:
+    req = request.Request(
+        url=url,
+        data=json.dumps(payload).encode("utf-8"),
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    with request.urlopen(req, timeout=timeout) as resp:
+        raw = resp.read()
+    return json.loads(raw.decode("utf-8")) if raw else {}
+
+
+def get_json(url: str, timeout: float) -> Dict[str, Any]:
+    req = request.Request(url=url, method="GET")
+    with request.urlopen(req, timeout=timeout) as resp:
+        raw = resp.read()
+    return json.loads(raw.decode("utf-8")) if raw else {}
+
+
+def discover_openai_model(base_url: str, timeout: float) -> str:
+    payload = get_json(base_url.rstrip("/") + "/v1/models", timeout=timeout)
+    data = payload.get("data")
+    if not isinstance(data, list) or not data:
+        raise RuntimeError(f"No models returned by {base_url.rstrip('/')}/v1/models")
+    first = data[0]
+    if isinstance(first, dict) and first.get("id"):
+        return str(first["id"])
+    raise RuntimeError(f"Malformed /v1/models payload from {base_url.rstrip('/')}")
+
+
+def p95(values: List[float]) -> Optional[float]:
+    if not values:
+        return None
+    ordered = sorted(values)
+    index = max(0, math.ceil(len(ordered) * 0.95) - 1)
+    return ordered[index]
+
+
+def sglang_request(base_url: str, prompt: str, max_tokens: int, timeout: float) -> str:
+    payload = {
+        "text": prompt,
+        "sampling_params": {
+            "temperature": 0.0,
+            "max_new_tokens": max_tokens,
+        },
+        "stream": False,
+    }
+    body = post_json(base_url.rstrip("/") + "/generate", payload, timeout=timeout)
+    return str(body.get("text", ""))
+
+
+def openai_request(
+    base_url: str,
+    model: str,
+    prompt: str,
+    max_tokens: int,
+    timeout: float,
+) -> Dict[str, str]:
+    payload = {
+        "model": model,
+        "messages": [{"role": "user", "content": prompt}],
+        "temperature": 0.0,
+        "max_tokens": max_tokens,
+        "stream": False,
+    }
+    body = post_json(
+        base_url.rstrip("/") + "/v1/chat/completions",
+        payload,
+        timeout=timeout,
+    )
+    text, source = extract_openai_chat_text(body)
+    return {"text": text, "source": source}
+
+
+def run_probe(args: argparse.Namespace) -> Dict[str, Any]:
+    prompts = args.prompt or list(DEFAULT_PROMPTS)
+    model = args.model
+    if args.framework in {"vllm", "trtllm", "tokenspeed"} and not model:
+        model = discover_openai_model(args.url, timeout=args.timeout)
+
+    latencies: List[float] = []
+    samples: List[Dict[str, Any]] = []
+    errors: List[Dict[str, str]] = []
+
+    for request_idx in range(args.requests):
+        prompt = prompts[request_idx % len(prompts)]
+        start = time.time()
+        try:
+            if args.framework == "sglang":
+                text = sglang_request(
+                    args.url,
+                    prompt,
+                    max_tokens=args.max_tokens,
+                    timeout=args.timeout,
+                )
+                source = "generate.text"
+            else:
+                assert model is not None
+                result = openai_request(
+                    args.url,
+                    model,
+                    prompt,
+                    max_tokens=args.max_tokens,
+                    timeout=args.timeout,
+                )
+                text = result["text"]
+                source = result["source"]
+            elapsed = time.time() - start
+            latencies.append(elapsed)
+            samples.append(
+                {
+                    "prompt": prompt,
+                    "latency_s": round(elapsed, 3),
+                    "content": text[:240],
+                    "source": source,
+                    "non_empty": bool(text.strip()),
+                }
+            )
+        except Exception as exc:  # pragma: no cover - runtime probe path
+            errors.append({"prompt": prompt, "error": repr(exc)})
+
+    return {
+        "framework": args.framework,
+        "url": args.url,
+        "model": model,
+        "requests": args.requests,
+        "success": len(samples),
+        "errors": len(errors),
+        "all_non_empty": (
+            all(sample["non_empty"] for sample in samples) if samples else False
+        ),
+        "avg_latency_s": round(statistics.mean(latencies), 3) if latencies else None,
+        "p95_latency_s": round(p95(latencies), 3) if latencies else None,
+        "samples": samples[:3],
+        "error_samples": errors[:3],
+    }
+
+
+def main() -> int:
+    args = parse_args()
+    summary = run_probe(args)
+    rendered = json.dumps(summary, ensure_ascii=False, indent=2)
+    print(rendered)
+    if args.output:
+        output_path = Path(args.output).expanduser().resolve()
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        output_path.write_text(rendered + "\n", encoding="utf-8")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.claude/skills/llm-torch-profiler-analysis/scripts/render_triage_markdown_bundle.py

@@ -0,0 +1,262 @@
+"""Bundle one or more triage text reports into a single markdown document."""
+
+from __future__ import annotations
+
+import argparse
+from collections import defaultdict
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Dict, List, Optional, Sequence, Tuple
+
+FRAMEWORK_LABELS = {
+    "sglang": "SGLang",
+    "vllm": "vLLM",
+    "trtllm": "TensorRT-LLM",
+    "tokenspeed": "TokenSpeed",
+}
+
+FRAMEWORK_ORDER = {"sglang": 0, "vllm": 1, "trtllm": 2, "tokenspeed": 3}
+
+
+def parse_args(argv: Optional[Sequence[str]] = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Render multiple profiler triage text outputs into one markdown file. "
+            "Input files are expected to be the existing analysis_*.txt outputs "
+            "already emitted by analyze_llm_torch_profile.py."
+        )
+    )
+    parser.add_argument(
+        "--analysis-root",
+        type=str,
+        default=None,
+        help=(
+            "Root directory to scan recursively for analysis_*.txt files. "
+            "Parent directory names are used as model section ids."
+        ),
+    )
+    parser.add_argument(
+        "--analysis-file",
+        action="append",
+        default=[],
+        help=(
+            "Explicit analysis file entry. Use either PATH or LABEL=PATH. "
+            "When LABEL is omitted, the parent directory name is used."
+        ),
+    )
+    parser.add_argument(
+        "--title",
+        type=str,
+        default="Unified LLM Torch Profiler Triage Bundle",
+        help="Top-level markdown title.",
+    )
+    parser.add_argument(
+        "--output",
+        type=str,
+        default=None,
+        help="Write the bundled markdown to this file. Prints to stdout when omitted.",
+    )
+    parser.add_argument(
+        "--include-toc",
+        action=argparse.BooleanOptionalAction,
+        default=True,
+        help="Include a simple table of contents.",
+    )
+    args = parser.parse_args(argv)
+    if not args.analysis_root and not args.analysis_file:
+        parser.error("Provide at least one of --analysis-root or --analysis-file.")
+    return args
+
+
+def framework_key_from_path(path: Path) -> str:
+    lowered = path.name.lower()
+    if "sglang" in lowered:
+        return "sglang"
+    if "vllm" in lowered:
+        return "vllm"
+    if "trtllm" in lowered or "tensorrt" in lowered:
+        return "trtllm"
+    if "tokenspeed" in lowered or "token-speed" in lowered:
+        return "tokenspeed"
+    return "other"
+
+
+def framework_label(framework_key: str) -> str:
+    return FRAMEWORK_LABELS.get(framework_key, framework_key)
+
+
+def discover_analysis_files(root: Path) -> List[Tuple[str, Path]]:
+    entries: List[Tuple[str, Path]] = []
+    for path in sorted(root.rglob("analysis*.txt")):
+        entries.append((path.parent.name, path))
+    return entries
+
+
+def parse_explicit_entry(raw: str) -> Tuple[str, Path]:
+    if "=" in raw:
+        label, path_text = raw.split("=", 1)
+        path = Path(path_text).expanduser().resolve()
+        return label.strip(), path
+    path = Path(raw).expanduser().resolve()
+    return path.parent.name, path
+
+
+def slugify(text: str) -> str:
+    chars = []
+    last_dash = False
+    for char in text.lower():
+        if char.isalnum():
+            chars.append(char)
+            last_dash = False
+        elif not last_dash:
+            chars.append("-")
+            last_dash = True
+    return "".join(chars).strip("-")
+
+
+def extract_model_name(report_text: str) -> Optional[str]:
+    for line in report_text.splitlines():
+        if line.startswith("Model: "):
+            return line.split("Model: ", 1)[1].strip()
+    return None
+
+
+def choose_model_display_name(
+    current: Optional[str],
+    candidate: Optional[str],
+    *,
+    label: str,
+) -> str:
+    if candidate and candidate != label:
+        if not current or current == label:
+            return candidate
+        if len(candidate) > len(current):
+            return candidate
+        return current
+    if current:
+        return current
+    return label
+
+
+def normalize_report_text(report_text: str) -> str:
+    text = report_text.replace("\r\n", "\n").strip()
+    if not text:
+        return "_Empty analysis output._"
+    heading_map = {
+        "Triage View": "#### Triage View",
+        "Kernel Table": "#### Kernel Table",
+        "Overlap Opportunity Table": "#### Overlap Opportunity Table",
+        "Fuse Opportunity Table": "#### Fuse Opportunity Table",
+    }
+    normalized_lines = []
+    for line in text.splitlines():
+        normalized_lines.append(heading_map.get(line, line))
+    return "\n".join(normalized_lines)
+
+
+def build_bundle_markdown(
+    *,
+    title: str,
+    labeled_paths: Sequence[Tuple[str, Path]],
+    include_toc: bool,
+) -> str:
+    grouped: Dict[str, List[Tuple[str, Path, str]]] = defaultdict(list)
+    model_display: Dict[str, str] = {}
+
+    for label, path in labeled_paths:
+        raw_text = path.read_text(encoding="utf-8")
+        report_text = normalize_report_text(raw_text)
+        model_name = extract_model_name(report_text)
+        grouped[label].append((framework_key_from_path(path), path, report_text))
+        model_display[label] = choose_model_display_name(
+            model_display.get(label),
+            model_name,
+            label=label,
+        )
+
+    ordered_labels = sorted(
+        grouped,
+        key=lambda item: (model_display[item].lower(), item.lower()),
+    )
+
+    lines: List[str] = [f"# {title}", ""]
+    lines.append(
+        f"_Generated on {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')}_"
+    )
+    lines.append("")
+
+    if include_toc:
+        lines.append("## Contents")
+        lines.append("")
+        for label in ordered_labels:
+            lines.append(
+                f"- [{model_display[label]}](#{slugify(model_display[label])})"
+            )
+        lines.append("")
+
+    for label in ordered_labels:
+        display_name = model_display[label]
+        lines.append(f"## {display_name}")
+        lines.append("")
+        lines.append(f"Model id: `{label}`")
+        lines.append("")
+
+        records = sorted(
+            grouped[label],
+            key=lambda item: (
+                FRAMEWORK_ORDER.get(item[0], 99),
+                item[1].name.lower(),
+            ),
+        )
+
+        for framework_key, path, report_text in records:
+            lines.append(f"### {framework_label(framework_key)}")
+            lines.append("")
+            lines.append(f"Source: `{path}`")
+            lines.append("")
+            lines.append(report_text)
+            lines.append("")
+
+    return "\n".join(lines).rstrip() + "\n"
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+    args = parse_args(argv)
+
+    labeled_paths: List[Tuple[str, Path]] = []
+    if args.analysis_root:
+        labeled_paths.extend(
+            discover_analysis_files(Path(args.analysis_root).expanduser().resolve())
+        )
+    for raw_entry in args.analysis_file:
+        labeled_paths.append(parse_explicit_entry(raw_entry))
+
+    existing = []
+    missing = []
+    for label, path in labeled_paths:
+        if path.is_file():
+            existing.append((label, path))
+        else:
+            missing.append(str(path))
+    if missing:
+        raise SystemExit("Missing analysis files:\n" + "\n".join(missing))
+    if not existing:
+        raise SystemExit("No analysis files found.")
+
+    markdown = build_bundle_markdown(
+        title=args.title,
+        labeled_paths=existing,
+        include_toc=args.include_toc,
+    )
+
+    if args.output:
+        output_path = Path(args.output).expanduser().resolve()
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        output_path.write_text(markdown, encoding="utf-8")
+    else:
+        print(markdown, end="")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.claude/skills/llm-torch-profiler-analysis/scripts/run_llm_single_model_matrix_host.sh

@@ -0,0 +1,274 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_llm_single_model_matrix_host.sh \
+    --model-id gpt_oss_20b \
+    --model openai/gpt-oss-20b \
+    --root /data/bbuf/validate/unified_llm_profiler_skill/runs/20260423_h100_large_model_matrix \
+    --gpus 2,3,4,5 \
+    --sglang-port 30098 \
+    --vllm-formal-port 31098 \
+    --vllm-mapping-port 31099 \
+    --trt-formal-prefill-port 32098 \
+    --trt-formal-decode-port 32099 \
+    --trt-mapping-prefill-port 32198 \
+    --trt-mapping-decode-port 32199
+
+This script is intended to run on the H100 host. It:
+1. captures SGLang live profiling and writes `analysis_sglang.txt`
+2. captures vLLM formal + eager mapping traces and writes `analysis_vllm.txt`
+3. captures TensorRT-LLM formal + graph-off mapping traces and writes `analysis_trtllm.txt`
+4. stores one benchmark JSON per framework under the model run directory
+
+Default profiler workloads are stage-separated:
+  prefill: input 4090, output 1
+  decode:  input 1, output 2048
+
+Environment:
+  Export `HF_TOKEN` and `HUGGINGFACE_HUB_TOKEN` before running.
+EOF
+}
+
+MODEL_ID=""
+MODEL=""
+ROOT=""
+GPUS=""
+TP_SIZE=""
+SGLANG_PORT=""
+VLLM_FORMAL_PORT=""
+VLLM_MAPPING_PORT=""
+TRT_FORMAL_PREFILL_PORT=""
+TRT_FORMAL_DECODE_PORT=""
+TRT_MAPPING_PREFILL_PORT=""
+TRT_MAPPING_DECODE_PORT=""
+SGLANG_MEM_FRACTION="0.85"
+MAX_MODEL_LEN="4096"
+KV_FRACTION="0.85"
+SGLANG_SERVER_EXTRA=""
+PROFILE_WORKLOAD="both"
+PREFILL_INPUT_LEN=4090
+PREFILL_OUTPUT_LEN=1
+DECODE_INPUT_LEN=1
+DECODE_OUTPUT_LEN=2048
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TRT_IMAGE="nvcr.io/nvidia/tensorrt-llm/release:latest"
+TRT_OVERRIDE_ROOT="/data/bbuf/validate/unified_llm_profiler_skill/overrides/trtllm"
+TRT_OVERRIDE_SOURCE="$TRT_OVERRIDE_ROOT/py_executor.original.py"
+TRT_OVERRIDE_PATH="$TRT_OVERRIDE_ROOT/py_executor_with_stack.py"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model-id) MODEL_ID="$2"; shift 2 ;;
+    --model) MODEL="$2"; shift 2 ;;
+    --root) ROOT="$2"; shift 2 ;;
+    --gpus) GPUS="$2"; shift 2 ;;
+    --tp-size) TP_SIZE="$2"; shift 2 ;;
+    --sglang-port) SGLANG_PORT="$2"; shift 2 ;;
+    --vllm-formal-port) VLLM_FORMAL_PORT="$2"; shift 2 ;;
+    --vllm-mapping-port) VLLM_MAPPING_PORT="$2"; shift 2 ;;
+    --trt-formal-prefill-port) TRT_FORMAL_PREFILL_PORT="$2"; shift 2 ;;
+    --trt-formal-decode-port) TRT_FORMAL_DECODE_PORT="$2"; shift 2 ;;
+    --trt-mapping-prefill-port) TRT_MAPPING_PREFILL_PORT="$2"; shift 2 ;;
+    --trt-mapping-decode-port) TRT_MAPPING_DECODE_PORT="$2"; shift 2 ;;
+    --sglang-mem-fraction) SGLANG_MEM_FRACTION="$2"; shift 2 ;;
+    --sglang-server-extra) SGLANG_SERVER_EXTRA="$2"; shift 2 ;;
+    --max-model-len) MAX_MODEL_LEN="$2"; shift 2 ;;
+    --kv-fraction) KV_FRACTION="$2"; shift 2 ;;
+    --profile-workload) PROFILE_WORKLOAD="$2"; shift 2 ;;
+    --prefill-input-len) PREFILL_INPUT_LEN="$2"; shift 2 ;;
+    --prefill-output-len) PREFILL_OUTPUT_LEN="$2"; shift 2 ;;
+    --decode-input-len) DECODE_INPUT_LEN="$2"; shift 2 ;;
+    --decode-output-len) DECODE_OUTPUT_LEN="$2"; shift 2 ;;
+    --help|-h) usage; exit 0 ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "${HF_TOKEN:-}" && -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  echo "Set HF_TOKEN or HUGGINGFACE_HUB_TOKEN before running." >&2
+  exit 2
+fi
+if [[ -z "${HF_TOKEN:-}" ]]; then
+  HF_TOKEN="$HUGGINGFACE_HUB_TOKEN"
+fi
+if [[ -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  HUGGINGFACE_HUB_TOKEN="$HF_TOKEN"
+fi
+
+for value in \
+  MODEL_ID MODEL ROOT GPUS \
+  SGLANG_PORT VLLM_FORMAL_PORT VLLM_MAPPING_PORT \
+  TRT_FORMAL_PREFILL_PORT TRT_FORMAL_DECODE_PORT \
+  TRT_MAPPING_PREFILL_PORT TRT_MAPPING_DECODE_PORT; do
+  if [[ -z "${!value}" ]]; then
+    echo "Missing required argument: $value" >&2
+    usage >&2
+    exit 2
+  fi
+done
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tp-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+
+MODEL_ROOT="$ROOT/$MODEL_ID"
+SGLANG_ANALYSIS="$MODEL_ROOT/analysis_sglang.txt"
+VLLM_FORMAL_DIR="$MODEL_ROOT/vllm_formal"
+VLLM_MAPPING_DIR="$MODEL_ROOT/vllm_mapping"
+VLLM_ANALYSIS="$MODEL_ROOT/analysis_vllm.txt"
+TRT_FORMAL_DIR="$MODEL_ROOT/trtllm_formal"
+TRT_MAPPING_DIR="$MODEL_ROOT/trtllm_mapping"
+TRT_ANALYSIS="$MODEL_ROOT/analysis_trtllm.txt"
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$MODEL_ROOT'"
+
+if [[ ! -s "$TRT_OVERRIDE_SOURCE" ]]; then
+  echo "[bootstrap] TensorRT-LLM py_executor source snapshot"
+  docker exec sglang_bbuf bash -lc "mkdir -p '$TRT_OVERRIDE_ROOT'"
+  docker run --rm --entrypoint cat "$TRT_IMAGE" \
+    /usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py \
+    | docker exec -i sglang_bbuf bash -lc "cat > '$TRT_OVERRIDE_SOURCE'"
+fi
+echo "[bootstrap] TensorRT-LLM py_executor override with with_stack=True and rank0-only trace export"
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 make_trtllm_py_executor_override.py --source '$TRT_OVERRIDE_SOURCE' --output '$TRT_OVERRIDE_PATH'"
+
+sglang_args=(
+  --model "$MODEL"
+  --run-dir "$MODEL_ROOT"
+  --port "$SGLANG_PORT"
+  --gpus "$GPUS"
+  --tp-size "$TP_SIZE"
+  --mem-fraction "$SGLANG_MEM_FRACTION"
+  --profile-workload "$PROFILE_WORKLOAD"
+  --prefill-input-len "$PREFILL_INPUT_LEN"
+  --prefill-output-len "$PREFILL_OUTPUT_LEN"
+  --decode-input-len "$DECODE_INPUT_LEN"
+  --decode-output-len "$DECODE_OUTPUT_LEN"
+  --trust-remote-code
+)
+if [[ -n "$SGLANG_SERVER_EXTRA" ]]; then
+  sglang_args+=(--server-extra "$SGLANG_SERVER_EXTRA")
+fi
+
+echo "[1/6] SGLang server + live triage"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_sglang_torch_profile_host.sh" \
+  "${sglang_args[@]}"
+
+echo "[2/6] vLLM formal"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_vllm_torch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$VLLM_FORMAL_DIR" \
+  --port "$VLLM_FORMAL_PORT" \
+  --gpus "$GPUS" \
+  --tensor-parallel-size "$TP_SIZE" \
+  --max-model-len "$MAX_MODEL_LEN" \
+  --profile-workload "$PROFILE_WORKLOAD" \
+  --prefill-input-len "$PREFILL_INPUT_LEN" \
+  --prefill-output-len "$PREFILL_OUTPUT_LEN" \
+  --decode-input-len "$DECODE_INPUT_LEN" \
+  --decode-output-len "$DECODE_OUTPUT_LEN" \
+  --trust-remote-code
+
+echo "[3/6] vLLM mapping"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_vllm_torch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$VLLM_MAPPING_DIR" \
+  --port "$VLLM_MAPPING_PORT" \
+  --gpus "$GPUS" \
+  --tensor-parallel-size "$TP_SIZE" \
+  --profiler-active-iterations 2 \
+  --max-model-len "$MAX_MODEL_LEN" \
+  --profile-workload "$PROFILE_WORKLOAD" \
+  --prefill-input-len "$PREFILL_INPUT_LEN" \
+  --prefill-output-len "$PREFILL_OUTPUT_LEN" \
+  --decode-input-len "$DECODE_INPUT_LEN" \
+  --decode-output-len "$DECODE_OUTPUT_LEN" \
+  --trust-remote-code \
+  --enforce-eager
+
+echo "[4/6] vLLM mapping-formal analysis"
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 analyze_llm_torch_profile.py --framework vllm --mapping-input '$VLLM_MAPPING_DIR' --formal-input '$VLLM_FORMAL_DIR' > '$VLLM_ANALYSIS'"
+
+echo "[5/6] TensorRT-LLM formal + mapping captures"
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_FORMAL_DIR" \
+  --stage prefill \
+  --port "$TRT_FORMAL_PREFILL_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$PREFILL_INPUT_LEN" \
+  --output-len "$PREFILL_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --trust-remote-code
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_FORMAL_DIR" \
+  --stage decode \
+  --port "$TRT_FORMAL_DECODE_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$DECODE_INPUT_LEN" \
+  --output-len "$DECODE_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --trust-remote-code
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_MAPPING_DIR" \
+  --stage prefill \
+  --port "$TRT_MAPPING_PREFILL_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$PREFILL_INPUT_LEN" \
+  --output-len "$PREFILL_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --disable-cudagraph \
+  --trust-remote-code
+HF_TOKEN="$HF_TOKEN" HUGGINGFACE_HUB_TOKEN="$HUGGINGFACE_HUB_TOKEN" \
+  "$SCRIPT_DIR/run_trtllm_pytorch_profile_host.sh" \
+  --model "$MODEL" \
+  --run-dir "$TRT_MAPPING_DIR" \
+  --stage decode \
+  --port "$TRT_MAPPING_DECODE_PORT" \
+  --gpus "$GPUS" \
+  --tp-size "$TP_SIZE" \
+  --kv-fraction "$KV_FRACTION" \
+  --input-len "$DECODE_INPUT_LEN" \
+  --output-len "$DECODE_OUTPUT_LEN" \
+  --override-py-executor "$TRT_OVERRIDE_PATH" \
+  --disable-cudagraph \
+  --trust-remote-code
+
+echo "[6/6] TensorRT-LLM mapping-formal analysis"
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 analyze_llm_torch_profile.py --framework trtllm --mapping-input '$TRT_MAPPING_DIR' --formal-input '$TRT_FORMAL_DIR' > '$TRT_ANALYSIS'"
+
+echo "MODEL_ROOT=$MODEL_ROOT"
+echo "ANALYSIS_SGLANG=$SGLANG_ANALYSIS"
+echo "ANALYSIS_VLLM=$VLLM_ANALYSIS"
+echo "ANALYSIS_TRTLLM=$TRT_ANALYSIS"

.claude/skills/llm-torch-profiler-analysis/scripts/run_sglang_torch_profile_host.sh

@@ -0,0 +1,241 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_sglang_torch_profile_host.sh \
+    --model Qwen/Qwen3-8B \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_sglang \
+    --port 30088 \
+    --gpus 0
+
+  run_sglang_torch_profile_host.sh \
+    --model openai/gpt-oss-20b \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_sglang_4gpu \
+    --port 30088 \
+    --gpus 2,3,4,5 \
+    --tp-size 4
+
+Options:
+  --model TEXT                  Model id or local path for SGLang.
+  --run-dir PATH               Shared /data directory for logs and traces.
+  --port INT                   Server port.
+  --gpus TEXT                  CUDA_VISIBLE_DEVICES value, for example 0 or 2,3,4,5.
+  --gpu TEXT                   Alias for --gpus.
+  --tp-size INT                Tensor parallel size. Defaults to the visible GPU count.
+  --trust-remote-code          Pass --trust-remote-code.
+  --mem-fraction FLOAT         SGLang static memory fraction.
+  --request-max-tokens INT     Generation length for the probe request.
+  --prompt TEXT                Probe prompt.
+  --warmup-steps INT           Warmup steps before profiling. Defaults to 10.
+  --profile-workload TEXT      legacy|prefill|decode|both. Defaults to both.
+  --prefill-input-len INT      Synthetic prefill prompt length. Defaults to 4090.
+  --prefill-output-len INT     Synthetic prefill output length. Defaults to 1.
+  --decode-input-len INT       Synthetic decode prompt length. Defaults to 1.
+  --decode-output-len INT      Synthetic decode output length. Defaults to 2048.
+  --repo-dir PATH              SGLang repo path inside `sglang_bbuf`.
+  --server-extra TEXT          Extra args appended to launch_server.
+  --help                       Show this message.
+
+Notes:
+  - Run this on the H100 host. It uses `docker exec sglang_bbuf`.
+  - The server is launched first, then the profiler capture runs with
+    stage-separated prefill/decode workloads and `--profile-by-stage`.
+  - A small benchmark summary is written after profiling.
+EOF
+}
+
+MODEL=""
+RUN_DIR=""
+PORT=""
+GPUS=""
+TP_SIZE=""
+TRUST_REMOTE_CODE=0
+MEM_FRACTION=0.85
+REQUEST_MAX_TOKENS=12
+PROMPT="Explain the difference between CUDA graph mode and eager mode in two sentences."
+WARMUP_STEPS=10
+PROFILE_WORKLOAD="both"
+PREFILL_INPUT_LEN=4090
+PREFILL_OUTPUT_LEN=1
+DECODE_INPUT_LEN=1
+DECODE_OUTPUT_LEN=2048
+SGLANG_REPO_DIR="${SGLANG_REPO_DIR:-/data/bbuf/repos/sglang}"
+SERVER_EXTRA=""
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model)
+      MODEL="$2"
+      shift 2
+      ;;
+    --run-dir)
+      RUN_DIR="$2"
+      shift 2
+      ;;
+    --port)
+      PORT="$2"
+      shift 2
+      ;;
+    --gpu)
+      GPUS="$2"
+      shift 2
+      ;;
+    --gpus)
+      GPUS="$2"
+      shift 2
+      ;;
+    --tp-size)
+      TP_SIZE="$2"
+      shift 2
+      ;;
+    --trust-remote-code)
+      TRUST_REMOTE_CODE=1
+      shift
+      ;;
+    --mem-fraction)
+      MEM_FRACTION="$2"
+      shift 2
+      ;;
+    --request-max-tokens)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --prompt)
+      PROMPT="$2"
+      shift 2
+      ;;
+    --warmup-steps)
+      WARMUP_STEPS="$2"
+      shift 2
+      ;;
+    --profile-workload)
+      PROFILE_WORKLOAD="$2"
+      shift 2
+      ;;
+    --prefill-input-len)
+      PREFILL_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --prefill-output-len)
+      PREFILL_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-input-len)
+      DECODE_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-output-len)
+      DECODE_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --repo-dir)
+      SGLANG_REPO_DIR="$2"
+      shift 2
+      ;;
+    --server-extra)
+      SERVER_EXTRA="$2"
+      shift 2
+      ;;
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "$MODEL" || -z "$RUN_DIR" || -z "$PORT" || -z "$GPUS" ]]; then
+  usage >&2
+  exit 2
+fi
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tp-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+
+LOG_PATH="$RUN_DIR/sglang_server.log"
+ANALYSIS_PATH="$RUN_DIR/analysis_sglang.txt"
+PROFILE_ROOT="$RUN_DIR/sglang_profile_live"
+BENCHMARK_PATH="$RUN_DIR/benchmark_sglang.json"
+PID_PATH="$RUN_DIR/sglang_server.pid"
+LAUNCH_PATTERN="[s]glang.launch_server.*--port $PORT"
+SERVER_ARGS="python3 -m sglang.launch_server --model-path \"$MODEL\" --port \"$PORT\" --tp-size \"$TP_SIZE\" --mem-fraction-static \"$MEM_FRACTION\""
+
+if [[ "$TRUST_REMOTE_CODE" -eq 1 ]]; then
+  SERVER_ARGS="$SERVER_ARGS --trust-remote-code"
+fi
+if [[ -n "$SERVER_EXTRA" ]]; then
+  SERVER_ARGS="$SERVER_ARGS $SERVER_EXTRA"
+fi
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$RUN_DIR' '$PROFILE_ROOT'"
+docker exec sglang_bbuf bash -lc "pkill -f '$LAUNCH_PATTERN' >/dev/null 2>&1 || true"
+docker exec sglang_bbuf bash -lc "mkdir -p '$RUN_DIR' '$PROFILE_ROOT' && cd '$SGLANG_REPO_DIR' && rm -f '$PID_PATH' && (CUDA_VISIBLE_DEVICES=$GPUS PYTHONPATH=python nohup $SERVER_ARGS > '$LOG_PATH' 2>&1 < /dev/null & echo \$! > '$PID_PATH')"
+
+cleanup() {
+  docker exec sglang_bbuf bash -lc "pkill -f '$LAUNCH_PATTERN' >/dev/null 2>&1 || true" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+ready=0
+for _ in $(seq 1 180); do
+  if curl -sf "http://127.0.0.1:${PORT}/v1/models" >/dev/null; then
+    ready=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$ready" -ne 1 ]]; then
+  echo "SGLang server did not become ready on port ${PORT}. Recent logs:" >&2
+  ssh_log=$(docker exec sglang_bbuf bash -lc "tail -n 120 '$LOG_PATH'" 2>/dev/null || true)
+  printf '%s\n' "$ssh_log" >&2
+  exit 1
+fi
+
+python3 - <<PY
+import json
+import urllib.request
+
+payload = {
+    "text": ${PROMPT@Q},
+    "sampling_params": {
+        "temperature": 0.0,
+        "max_new_tokens": int(${REQUEST_MAX_TOKENS@Q}),
+    },
+    "stream": False,
+}
+req = urllib.request.Request(
+    "http://127.0.0.1:${PORT}/generate",
+    data=json.dumps(payload).encode(),
+    headers={"Content-Type": "application/json"},
+)
+with urllib.request.urlopen(req, timeout=600) as resp:
+    body = json.loads(resp.read().decode())
+text = body.get("text", "")
+print(text[:400])
+PY
+
+docker exec sglang_bbuf bash -lc "cd '$SCRIPT_DIR' && python3 analyze_llm_torch_profile.py --framework sglang --url http://127.0.0.1:${PORT} --output-dir '$PROFILE_ROOT' --num-steps 5 --warmup-steps '$WARMUP_STEPS' --probe-requests 1 --profile-by-stage --profile-workload '$PROFILE_WORKLOAD' --prefill-input-len '$PREFILL_INPUT_LEN' --prefill-output-len '$PREFILL_OUTPUT_LEN' --decode-input-len '$DECODE_INPUT_LEN' --decode-output-len '$DECODE_OUTPUT_LEN' > '$ANALYSIS_PATH'"
+python3 "$SCRIPT_DIR/probe_llm_server.py" \
+  --framework sglang \
+  --url "http://127.0.0.1:${PORT}" \
+  | docker exec -i sglang_bbuf bash -lc "cat > '$BENCHMARK_PATH'" >/dev/null
+docker exec sglang_bbuf bash -lc "sed -n '1,240p' '$ANALYSIS_PATH'"
+echo "BENCHMARK_PATH=$BENCHMARK_PATH"

.claude/skills/llm-torch-profiler-analysis/scripts/run_trtllm_pytorch_profile_host.sh

@@ -0,0 +1,407 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_trtllm_pytorch_profile_host.sh \
+    --model Qwen/Qwen3-8B \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example \
+    --stage prefill \
+    --port 32188 \
+    --gpus 0
+
+  run_trtllm_pytorch_profile_host.sh \
+    --model openai/gpt-oss-20b \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_4gpu \
+    --stage prefill \
+    --port 32188 \
+    --gpus 2,3,4,5 \
+    --tp-size 4
+
+Options:
+  --model TEXT                     Hugging Face model id.
+  --run-dir PATH                  Shared /data run directory for logs and traces.
+  --stage prefill|decode          Capture window. Prefill profiles 4090->1 by
+                                  default; decode profiles 1->2048 by default.
+  --port INT                      Host port for trtllm-serve.
+  --gpus TEXT                     CUDA_VISIBLE_DEVICES value, for example 0 or 2,3,4,5.
+  --gpu TEXT                      Alias for --gpus.
+  --tp-size INT                   Tensor parallel size. Defaults to the visible GPU count.
+  --image TEXT                    Container image.
+  --shared-root PATH              Shared validation root mounted into the container.
+  --hf-cache PATH                 Host Hugging Face cache path.
+  --override-py-executor PATH     Optional py_executor.py override path.
+  --disable-cudagraph             Generate/use a YAML override with cuda_graph_config: null.
+  --input-len INT                 Synthetic prompt length for this stage.
+                                  Defaults: prefill 4090, decode 1.
+  --request-max-tokens INT        Generation length for this stage.
+                                  Defaults: prefill 1, decode 2048.
+  --output-len INT                Alias for --request-max-tokens.
+  --prompt TEXT                   Probe prompt. Defaults to a synthetic prompt
+                                  sized by --input-len.
+  --warmup-steps INT              Warmup steps before the profiler window. Defaults to 10.
+  --active-steps INT              Active profiler steps to capture. Defaults to 5.
+  --max-seq-len INT               Serve max sequence length.
+  --kv-fraction FLOAT             KV cache free GPU memory fraction.
+  --container-name TEXT           Override container name.
+  --trust-remote-code             Pass --trust_remote_code to trtllm-serve.
+  --help                          Show this message.
+
+Environment:
+  HF_TOKEN or HUGGINGFACE_HUB_TOKEN must be set.
+
+Notes:
+  - Run this on the H100 host, not inside `sglang_bbuf`.
+  - It always pins TensorRT-LLM to `--backend pytorch`.
+  - The default image tag is floating; record the resolved TensorRT-LLM version
+    in the run manifest and pass --image for reproducible validation.
+  - Profiling uses `TLLM_PROFILE_START_STOP` and `TLLM_TORCH_PROFILE_TRACE`.
+  - For Python-location recovery, prefer a `py_executor.py` override with `with_stack=True`.
+  - A small benchmark summary is written after the trace is emitted.
+EOF
+}
+
+IMAGE="nvcr.io/nvidia/tensorrt-llm/release:latest"
+SHARED_ROOT="/data/bbuf/validate/unified_llm_profiler_skill"
+HF_CACHE="/data/.cache/huggingface"
+OVERRIDE_PY_EXECUTOR=""
+DISABLE_CUDAGRAPH=0
+REQUEST_MAX_TOKENS=""
+INPUT_LEN=""
+PROMPT=""
+WARMUP_STEPS=10
+ACTIVE_STEPS=5
+MAX_SEQ_LEN=4096
+KV_FRACTION=0.85
+CONTAINER_NAME=""
+TRUST_REMOTE_CODE=0
+TP_SIZE=""
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+MODEL=""
+RUN_DIR=""
+STAGE=""
+PORT=""
+GPUS=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model)
+      MODEL="$2"
+      shift 2
+      ;;
+    --run-dir)
+      RUN_DIR="$2"
+      shift 2
+      ;;
+    --stage)
+      STAGE="$2"
+      shift 2
+      ;;
+    --port)
+      PORT="$2"
+      shift 2
+      ;;
+    --gpu)
+      GPUS="$2"
+      shift 2
+      ;;
+    --gpus)
+      GPUS="$2"
+      shift 2
+      ;;
+    --tp-size)
+      TP_SIZE="$2"
+      shift 2
+      ;;
+    --image)
+      IMAGE="$2"
+      shift 2
+      ;;
+    --shared-root)
+      SHARED_ROOT="$2"
+      shift 2
+      ;;
+    --hf-cache)
+      HF_CACHE="$2"
+      shift 2
+      ;;
+    --override-py-executor)
+      OVERRIDE_PY_EXECUTOR="$2"
+      shift 2
+      ;;
+    --disable-cudagraph)
+      DISABLE_CUDAGRAPH=1
+      shift
+      ;;
+    --input-len)
+      INPUT_LEN="$2"
+      shift 2
+      ;;
+    --request-max-tokens)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --output-len)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --prompt)
+      PROMPT="$2"
+      shift 2
+      ;;
+    --warmup-steps)
+      WARMUP_STEPS="$2"
+      shift 2
+      ;;
+    --active-steps)
+      ACTIVE_STEPS="$2"
+      shift 2
+      ;;
+    --max-seq-len)
+      MAX_SEQ_LEN="$2"
+      shift 2
+      ;;
+    --kv-fraction)
+      KV_FRACTION="$2"
+      shift 2
+      ;;
+    --container-name)
+      CONTAINER_NAME="$2"
+      shift 2
+      ;;
+    --trust-remote-code)
+      TRUST_REMOTE_CODE=1
+      shift
+      ;;
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "${HF_TOKEN:-}" && -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  echo "Set HF_TOKEN or HUGGINGFACE_HUB_TOKEN before running." >&2
+  exit 2
+fi
+if [[ -z "${HF_TOKEN:-}" ]]; then
+  HF_TOKEN="$HUGGINGFACE_HUB_TOKEN"
+fi
+if [[ -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  HUGGINGFACE_HUB_TOKEN="$HF_TOKEN"
+fi
+
+if [[ -z "$MODEL" || -z "$RUN_DIR" || -z "$STAGE" || -z "$PORT" || -z "$GPUS" ]]; then
+  usage >&2
+  exit 2
+fi
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tp-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+
+case "$STAGE" in
+  prefill)
+    TRACE_PATH="$RUN_DIR/trace-prefill.json"
+    LOG_PATH="$RUN_DIR/server-prefill.log"
+    BENCHMARK_PATH="$RUN_DIR/benchmark-prefill.json"
+    if [[ -z "$INPUT_LEN" ]]; then
+      INPUT_LEN=4090
+    fi
+    if [[ -z "$REQUEST_MAX_TOKENS" ]]; then
+      REQUEST_MAX_TOKENS=1
+    fi
+    ;;
+  decode)
+    TRACE_PATH="$RUN_DIR/trace-decode.json"
+    LOG_PATH="$RUN_DIR/server-decode.log"
+    BENCHMARK_PATH="$RUN_DIR/benchmark-decode.json"
+    if [[ -z "$INPUT_LEN" ]]; then
+      INPUT_LEN=1
+    fi
+    if [[ -z "$REQUEST_MAX_TOKENS" ]]; then
+      REQUEST_MAX_TOKENS=2048
+    fi
+    ;;
+  *)
+    echo "--stage must be prefill or decode." >&2
+    exit 2
+    ;;
+esac
+
+if (( WARMUP_STEPS < 0 || ACTIVE_STEPS < 1 )); then
+  echo "--warmup-steps must be >= 0 and --active-steps must be >= 1." >&2
+  exit 2
+fi
+
+case "$STAGE" in
+  prefill)
+    profile_start=$((WARMUP_STEPS + 1))
+    ;;
+  decode)
+    profile_start=$((WARMUP_STEPS + 2))
+    ;;
+esac
+profile_stop=$((profile_start + ACTIVE_STEPS - 1))
+PROFILE_START_STOP="${profile_start}-${profile_stop}"
+
+if [[ -z "$CONTAINER_NAME" ]]; then
+  model_slug="${MODEL##*/}"
+  model_slug="${model_slug//\//-}"
+  model_slug="${model_slug//./-}"
+  model_slug="${model_slug//_/-}"
+  model_slug="${model_slug// /-}"
+  gpu_slug="${GPUS//,/-}"
+  CONTAINER_NAME="trtllm-${model_slug}-${STAGE}-g${gpu_slug}-p${PORT}"
+fi
+
+EXTRA_LLM_OPTIONS=""
+if [[ "$DISABLE_CUDAGRAPH" -eq 1 ]]; then
+  EXTRA_CFG_PATH="$SHARED_ROOT/tmp/trt_no_cudagraph.yaml"
+  docker exec sglang_bbuf bash -lc "mkdir -p '$(dirname "$EXTRA_CFG_PATH")' && printf 'cuda_graph_config: null\n' > '$EXTRA_CFG_PATH'"
+  EXTRA_LLM_OPTIONS="--extra_llm_api_options $EXTRA_CFG_PATH"
+fi
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$RUN_DIR'"
+docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+
+docker_args=(
+  run -d --rm
+  --name "$CONTAINER_NAME"
+  --gpus all
+  --ipc=host
+  --network host
+  --entrypoint bash
+  -e "CUDA_VISIBLE_DEVICES=$GPUS"
+  -e "HF_TOKEN=$HF_TOKEN"
+  -e "HUGGINGFACE_HUB_TOKEN=$HUGGINGFACE_HUB_TOKEN"
+  -e "TLLM_PROFILE_START_STOP=$PROFILE_START_STOP"
+  -e "TLLM_LLMAPI_ENABLE_NVTX=1"
+  -e "TLLM_TORCH_PROFILE_TRACE=$TRACE_PATH"
+  -e "RUN_DIR=$RUN_DIR"
+  -e "LOG_PATH=$LOG_PATH"
+  -e "MODEL_ID=$MODEL"
+  -e "SERVE_PORT=$PORT"
+  -v "$HF_CACHE:/root/.cache/huggingface"
+  -v "$SHARED_ROOT:$SHARED_ROOT"
+)
+
+if [[ -n "$OVERRIDE_PY_EXECUTOR" ]]; then
+  docker_args+=(
+    -v "$OVERRIDE_PY_EXECUTOR:/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor.py:ro"
+  )
+fi
+
+trust_remote_code_arg=""
+if [[ "$TRUST_REMOTE_CODE" -eq 1 ]]; then
+  trust_remote_code_arg="--trust_remote_code"
+fi
+
+container_cmd=$(
+  cat <<EOF
+mkdir -p "$RUN_DIR" && trtllm-serve serve "$MODEL" \
+  --backend pytorch \
+  --tp_size "$TP_SIZE" \
+  --gpus_per_node "$GPU_COUNT" \
+  --host 0.0.0.0 \
+  --port "$PORT" \
+  --max_seq_len "$MAX_SEQ_LEN" \
+  --kv_cache_free_gpu_memory_fraction "$KV_FRACTION" \
+  $trust_remote_code_arg \
+  $EXTRA_LLM_OPTIONS \
+  > "$LOG_PATH" 2>&1
+EOF
+)
+
+docker_args+=("$IMAGE" -lc "$container_cmd")
+docker "${docker_args[@]}" >/dev/null
+
+cleanup() {
+  docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+ready=0
+for _ in $(seq 1 180); do
+  if curl -sf "http://127.0.0.1:${PORT}/v1/models" >/dev/null; then
+    ready=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$ready" -ne 1 ]]; then
+  echo "Server did not become ready on port ${PORT}. Recent logs:" >&2
+  docker logs "$CONTAINER_NAME" 2>&1 | tail -n 120 >&2 || true
+  exit 1
+fi
+
+python3 - <<PY
+import json
+import sys
+import urllib.request
+
+sys.path.insert(0, ${SCRIPT_DIR@Q})
+from profile_common import extract_openai_chat_text, synthetic_prompt
+
+prompt = ${PROMPT@Q} or synthetic_prompt(int(${INPUT_LEN@Q}))
+stage = ${STAGE@Q}
+warmup_steps = int(${WARMUP_STEPS@Q})
+active_steps = int(${ACTIVE_STEPS@Q})
+request_count = warmup_steps + active_steps if stage == "prefill" else 1
+
+payload = {
+    "model": ${MODEL@Q},
+    "messages": [{"role": "user", "content": prompt}],
+    "temperature": 0,
+    "max_tokens": int(${REQUEST_MAX_TOKENS@Q}),
+}
+for request_idx in range(request_count):
+    req = urllib.request.Request(
+        "http://127.0.0.1:${PORT}/v1/chat/completions",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"},
+    )
+    with urllib.request.urlopen(req, timeout=600) as resp:
+        body = json.loads(resp.read().decode())
+text, source = extract_openai_chat_text(body)
+print(text[:400] if text else f"[empty completion; source={source}]")
+PY
+
+for _ in $(seq 1 120); do
+  if [[ -s "$TRACE_PATH" ]]; then
+    break
+  fi
+  sleep 2
+done
+
+if [[ ! -s "$TRACE_PATH" ]]; then
+  echo "Trace was not written: $TRACE_PATH" >&2
+  exit 1
+fi
+
+python3 "$SCRIPT_DIR/probe_llm_server.py" \
+  --framework trtllm \
+  --url "http://127.0.0.1:${PORT}" \
+  --model "$MODEL" \
+  | docker exec -i sglang_bbuf bash -lc "cat > '$BENCHMARK_PATH'" >/dev/null
+
+echo "TRACE_PATH=$TRACE_PATH"
+echo "LOG_PATH=$LOG_PATH"
+echo "BENCHMARK_PATH=$BENCHMARK_PATH"

.claude/skills/llm-torch-profiler-analysis/scripts/run_vllm_torch_profile_host.sh

@@ -0,0 +1,343 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+  cat <<'EOF'
+Usage:
+  run_vllm_torch_profile_host.sh \
+    --model Qwen/Qwen3-8B \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_vllm_formal \
+    --port 31088 \
+    --gpus 1
+
+  run_vllm_torch_profile_host.sh \
+    --model openai/gpt-oss-20b \
+    --run-dir /data/bbuf/validate/unified_llm_profiler_skill/runs/example_vllm_4gpu \
+    --port 31088 \
+    --gpus 2,3,4,5 \
+    --tensor-parallel-size 4
+
+Options:
+  --model TEXT                   Hugging Face model id.
+  --run-dir PATH                Shared /data directory for logs and traces.
+  --port INT                    Host port for vllm serve.
+  --gpus TEXT                   CUDA_VISIBLE_DEVICES value, for example 1 or 2,3,4,5.
+  --gpu TEXT                    Alias for --gpus.
+  --image TEXT                  Container image.
+  --hf-cache PATH               Host Hugging Face cache path.
+  --gpu-memory-util FLOAT       vLLM --gpu-memory-utilization.
+  --max-model-len INT           vLLM --max-model-len.
+    --tensor-parallel-size INT    vLLM --tensor-parallel-size. Defaults to the visible GPU count.
+    --profiler-active-iterations INT
+                                 Torch-profiler active iterations.
+    --enforce-eager               Launch vLLM with --enforce-eager for mapping traces.
+  --trust-remote-code           Pass --trust-remote-code.
+  --request-max-tokens INT      Generation length for the probe request.
+  --prompt TEXT                 Probe prompt.
+  --warmup-steps INT            Warmup steps before profiling. Defaults to 10.
+  --profile-workload TEXT       legacy|prefill|decode|both. Defaults to both.
+  --prefill-input-len INT       Synthetic prefill prompt length. Defaults to 4090.
+  --prefill-output-len INT      Synthetic prefill output length. Defaults to 1.
+  --decode-input-len INT        Synthetic decode prompt length. Defaults to 1.
+  --decode-output-len INT       Synthetic decode output length. Defaults to 2048.
+  --container-name TEXT         Override container name.
+  --help                        Show this message.
+
+Environment:
+  HF_TOKEN or HUGGINGFACE_HUB_TOKEN must be set.
+
+Notes:
+  - Run this on the H100 host, not inside `sglang_bbuf`.
+  - This uses the vLLM torch-profiler flow: `--profiler-config`, then POST
+    `/start_profile` and `/stop_profile`.
+  - Default capture is two labeled profiles: prefill 4090->1 and decode 1->2048.
+  - Current vLLM profiler config already defaults `torch_profiler_with_stack=true`.
+  - A small benchmark summary is written after profiling.
+EOF
+}
+
+IMAGE="vllm/vllm-openai:latest"
+HF_CACHE="/data/.cache/huggingface"
+GPU_MEMORY_UTIL=0.90
+MAX_MODEL_LEN=4096
+TP_SIZE=""
+ENFORCE_EAGER=0
+TRUST_REMOTE_CODE=0
+REQUEST_MAX_TOKENS=12
+PROFILER_ACTIVE_ITERATIONS=5
+PROMPT="Explain the difference between CUDA graph mode and eager mode in two sentences."
+WARMUP_STEPS=10
+PROFILE_WORKLOAD="both"
+PREFILL_INPUT_LEN=4090
+PREFILL_OUTPUT_LEN=1
+DECODE_INPUT_LEN=1
+DECODE_OUTPUT_LEN=2048
+CONTAINER_NAME=""
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+MODEL=""
+RUN_DIR=""
+PORT=""
+GPUS=""
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --model)
+      MODEL="$2"
+      shift 2
+      ;;
+    --run-dir)
+      RUN_DIR="$2"
+      shift 2
+      ;;
+    --port)
+      PORT="$2"
+      shift 2
+      ;;
+    --gpu)
+      GPUS="$2"
+      shift 2
+      ;;
+    --gpus)
+      GPUS="$2"
+      shift 2
+      ;;
+    --image)
+      IMAGE="$2"
+      shift 2
+      ;;
+    --hf-cache)
+      HF_CACHE="$2"
+      shift 2
+      ;;
+    --gpu-memory-util)
+      GPU_MEMORY_UTIL="$2"
+      shift 2
+      ;;
+    --max-model-len)
+      MAX_MODEL_LEN="$2"
+      shift 2
+      ;;
+    --tensor-parallel-size)
+      TP_SIZE="$2"
+      shift 2
+      ;;
+    --profiler-active-iterations)
+      PROFILER_ACTIVE_ITERATIONS="$2"
+      shift 2
+      ;;
+    --enforce-eager)
+      ENFORCE_EAGER=1
+      shift
+      ;;
+    --trust-remote-code)
+      TRUST_REMOTE_CODE=1
+      shift
+      ;;
+    --request-max-tokens)
+      REQUEST_MAX_TOKENS="$2"
+      shift 2
+      ;;
+    --prompt)
+      PROMPT="$2"
+      shift 2
+      ;;
+    --warmup-steps)
+      WARMUP_STEPS="$2"
+      shift 2
+      ;;
+    --profile-workload)
+      PROFILE_WORKLOAD="$2"
+      shift 2
+      ;;
+    --prefill-input-len)
+      PREFILL_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --prefill-output-len)
+      PREFILL_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-input-len)
+      DECODE_INPUT_LEN="$2"
+      shift 2
+      ;;
+    --decode-output-len)
+      DECODE_OUTPUT_LEN="$2"
+      shift 2
+      ;;
+    --container-name)
+      CONTAINER_NAME="$2"
+      shift 2
+      ;;
+    --help|-h)
+      usage
+      exit 0
+      ;;
+    *)
+      echo "Unknown argument: $1" >&2
+      usage >&2
+      exit 2
+      ;;
+  esac
+done
+
+if [[ -z "${HF_TOKEN:-}" && -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  echo "Set HF_TOKEN or HUGGINGFACE_HUB_TOKEN before running." >&2
+  exit 2
+fi
+if [[ -z "${HF_TOKEN:-}" ]]; then
+  HF_TOKEN="$HUGGINGFACE_HUB_TOKEN"
+fi
+if [[ -z "${HUGGINGFACE_HUB_TOKEN:-}" ]]; then
+  HUGGINGFACE_HUB_TOKEN="$HF_TOKEN"
+fi
+
+if [[ -z "$MODEL" || -z "$RUN_DIR" || -z "$PORT" || -z "$GPUS" ]]; then
+  usage >&2
+  exit 2
+fi
+
+IFS=',' read -r -a GPU_LIST <<< "$GPUS"
+GPU_COUNT="${#GPU_LIST[@]}"
+if [[ "$GPU_COUNT" -lt 1 ]]; then
+  echo "Could not parse --gpus: $GPUS" >&2
+  exit 2
+fi
+if [[ -z "$TP_SIZE" ]]; then
+  TP_SIZE="$GPU_COUNT"
+fi
+if (( TP_SIZE < 1 || TP_SIZE > GPU_COUNT )); then
+  echo "--tensor-parallel-size must be between 1 and the visible GPU count ($GPU_COUNT)." >&2
+  exit 2
+fi
+if (( PROFILER_ACTIVE_ITERATIONS < 1 )); then
+  echo "--profiler-active-iterations must be >= 1." >&2
+  exit 2
+fi
+
+PROFILE_DIR="$RUN_DIR/vllm_profile"
+LOG_PATH="$RUN_DIR/server.log"
+ANALYSIS_PATH="$RUN_DIR/analysis_vllm_live.txt"
+BENCHMARK_PATH="$RUN_DIR/benchmark_vllm.json"
+
+if [[ -z "$CONTAINER_NAME" ]]; then
+  model_slug="${MODEL##*/}"
+  model_slug="${model_slug//\//-}"
+  model_slug="${model_slug//./-}"
+  model_slug="${model_slug//_/-}"
+  gpu_slug="${GPUS//,/-}"
+  CONTAINER_NAME="vllm-${model_slug}-g${gpu_slug}-p${PORT}"
+  if [[ "$ENFORCE_EAGER" -eq 1 ]]; then
+    CONTAINER_NAME="${CONTAINER_NAME}-eager"
+  fi
+fi
+
+docker exec sglang_bbuf bash -lc "mkdir -p '$PROFILE_DIR'"
+docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+
+profiler_config=$(python3 - <<PY
+import json
+print(json.dumps({
+    "profiler": "torch",
+    "torch_profiler_dir": ${PROFILE_DIR@Q},
+    "active_iterations": int(${PROFILER_ACTIVE_ITERATIONS@Q}),
+}))
+PY
+)
+
+docker_args=(
+  run -d --rm
+  --name "$CONTAINER_NAME"
+  --gpus all
+  --ipc=host
+  --network host
+  -e "CUDA_VISIBLE_DEVICES=$GPUS"
+  -e "HF_TOKEN=$HF_TOKEN"
+  -e "HUGGINGFACE_HUB_TOKEN=$HUGGINGFACE_HUB_TOKEN"
+  -e "VLLM_RPC_TIMEOUT=1800000"
+  -v "$HF_CACHE:/root/.cache/huggingface"
+  -v "$RUN_DIR:$RUN_DIR"
+)
+
+docker_cmd=(
+  "$IMAGE"
+  "$MODEL"
+  --host 0.0.0.0
+  --port "$PORT"
+  --tensor-parallel-size "$TP_SIZE"
+  --max-model-len "$MAX_MODEL_LEN"
+  --gpu-memory-utilization "$GPU_MEMORY_UTIL"
+  --profiler-config "$profiler_config"
+)
+
+if [[ "$ENFORCE_EAGER" -eq 1 ]]; then
+  docker_cmd+=(--enforce-eager)
+fi
+if [[ "$TRUST_REMOTE_CODE" -eq 1 ]]; then
+  docker_cmd+=(--trust-remote-code)
+fi
+
+docker "${docker_args[@]}" "${docker_cmd[@]}" >/dev/null
+cleanup() {
+  docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+ready=0
+for _ in $(seq 1 180); do
+  if curl -sf "http://127.0.0.1:${PORT}/v1/models" >/dev/null; then
+    ready=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$ready" -ne 1 ]]; then
+  echo "Server did not become ready on port ${PORT}. Recent logs:" >&2
+  docker logs "$CONTAINER_NAME" 2>&1 | tail -n 120 >&2 || true
+  exit 1
+fi
+
+python3 "$SCRIPT_DIR/analyze_llm_torch_profile.py" \
+  --framework vllm \
+  --url "http://127.0.0.1:${PORT}" \
+  --output-dir "$PROFILE_DIR" \
+  --num-steps "$PROFILER_ACTIVE_ITERATIONS" \
+  --warmup-steps "$WARMUP_STEPS" \
+  --probe-requests 1 \
+  --no-profile-by-stage \
+  --profile-workload "$PROFILE_WORKLOAD" \
+  --probe-prompt "$PROMPT" \
+  --probe-max-new-tokens "$REQUEST_MAX_TOKENS" \
+  --prefill-input-len "$PREFILL_INPUT_LEN" \
+  --prefill-output-len "$PREFILL_OUTPUT_LEN" \
+  --decode-input-len "$DECODE_INPUT_LEN" \
+  --decode-output-len "$DECODE_OUTPUT_LEN" \
+  > "$ANALYSIS_PATH"
+
+profile_found=0
+for _ in $(seq 1 240); do
+  if find "$PROFILE_DIR" -type f \( -name '*.pt.trace.json' -o -name '*.pt.trace.json.gz' -o -name '*.trace.json' -o -name '*.trace.json.gz' \) | grep -q .; then
+    profile_found=1
+    break
+  fi
+  sleep 2
+done
+if [[ "$profile_found" -ne 1 ]]; then
+  echo "No vLLM profiler traces appeared under $PROFILE_DIR" >&2
+  docker logs "$CONTAINER_NAME" 2>&1 | tail -n 120 >&2 || true
+  exit 1
+fi
+
+python3 "$SCRIPT_DIR/probe_llm_server.py" \
+  --framework vllm \
+  --url "http://127.0.0.1:${PORT}" \
+  --model "$MODEL" \
+  | docker exec -i sglang_bbuf bash -lc "cat > '$BENCHMARK_PATH'" >/dev/null
+
+docker logs "$CONTAINER_NAME" 2>&1 | docker exec -i sglang_bbuf bash -lc "cat > '$LOG_PATH'" || true
+sed -n '1,240p' "$ANALYSIS_PATH"
+echo "PROFILE_DIR=$PROFILE_DIR"
+echo "LOG_PATH=$LOG_PATH"
+echo "ANALYSIS_PATH=$ANALYSIS_PATH"
+echo "BENCHMARK_PATH=$BENCHMARK_PATH"

.claude/skills/mechanical-refactor-verify/SKILL.md

@@ -0,0 +1,50 @@
+---
+name: mechanical-refactor-verify
+description: Make mechanical refactoring (file splits, function moves, module extractions, renames) machine-checkable instead of eyeballed. Reproduce a relocation commit byte-for-byte from faithful primitives, and split an extraction into a verifiable prepare + move + postpare. Use when doing or reviewing such changes.
+---
+
+# Mechanical Refactor — Machine-Checkable Verification
+
+## 1. Overview
+
+- The correctness of a mechanical change (file split, function move, module extraction,
+  rename) must be **machine-checkable, not eyeballed** — the proof is something anyone can
+  re-run, whoever made the change and whenever.
+- **One property**: *a commit is a pure relocation*. **One proof**: **reproduce** —
+  regenerate the move from the base commit with faithful primitives, run the formatter,
+  byte-diff against the target.
+- Empty diff = the proof. Any residual = a bundled non-move change, surfaced for review.
+- A reshape must not ride along: split into optional **prepare** + certified **move** +
+  optional **postpare** (`guide-split.md`).
+
+## 2. What do you want to do?
+
+- **Split a change into commits** (extract, move, file split) → `guide-split.md`: the
+  prepare + move + postpare rule, the case recipes, and the anti-patterns.
+- **Construct the proof for a move commit** → `guide-construct-proof.md`: run
+  `scripts/mechanical_refactor_proof_generator.py`, or hand-write a `Repro` when the
+  generator reports `UNSUPPORTED`.
+- **Verify someone's proof** → `guide-verify-proof.md`: re-run it, read the verdict, audit
+  the authored surfaces.
+- **Decide whether a change counts as a clean move** → `spec-reproduction-utils.md`: the
+  property, the whole whitelist / not-allowed list, and each primitive's contract. The
+  source of truth for the reproduction module; if any other file disagrees, it wins.
+
+## 3. Files
+
+- [`guide-split.md`](guide-split.md) — split a change into prepare + move + postpare: the
+  case recipes, what stays mechanical, and the anti-patterns.
+- [`guide-construct-proof.md`](guide-construct-proof.md) — produce the proof: the
+  generator, the hand-written `Repro`, and publishing the proof with the PR.
+- [`guide-verify-proof.md`](guide-verify-proof.md) — consume the proof: re-run, verdicts,
+  and the audit checklist for authored surfaces.
+- [`spec-reproduction-utils.md`](spec-reproduction-utils.md) — the normative spec of the
+  clean-move property and the reproduction primitives.
+- [`scripts/mechanical_refactor_proof_generator.py`](scripts/mechanical_refactor_proof_generator.py) —
+  the **generator**: infers a reproduce recipe from a commit's diff and emits/runs a
+  standalone, auditable script per commit, with a `PASS` / `RESIDUAL` / `UNSUPPORTED` verdict.
+- [`scripts/mechanical_refactor_reproduction_utils.py`](scripts/mechanical_refactor_reproduction_utils.py) — the
+  **proof engine**: the `Repro` builder's faithful relocation primitives plus the worktree +
+  pre-commit + byte-diff scaffold. Self-contained — only git and the standard library.
+- [`scripts/tests/`](scripts/tests/) — pytest suites, one folder per module:
+  `reproduction_utils/` for the proof engine, `proof_generator/` for the generator.

.claude/skills/mechanical-refactor-verify/guide-construct-proof.md

@@ -0,0 +1,157 @@
+# Construct a proof for a move commit
+
+## 1. What a proof is
+
+- A runnable script that regenerates the commit from its base with the faithful relocation
+  primitives and byte-diffs the result against it.
+- The property and primitive contracts: `spec-reproduction-utils.md`. Splitting the change
+  so the move is provable: `guide-split.md`. Consuming the proof: `guide-verify-proof.md`.
+
+## 2. Auto-generate the script (primary path)
+
+- `mechanical_refactor_proof_generator.py` infers the recipe from a commit's diff and
+  before-state AST.
+- It emits and runs a standalone, auditable script — no one hand-writes it.
+
+### 2.1 Commands
+
+```bash
+# one commit: print the inferred script and run it (non-zero exit unless PASS)
+python3 .claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_proof_generator.py <commit>
+
+# a range: write a self-contained folder
+python3 .claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_proof_generator.py \
+    <base>..<tip> --match -move: --out repro_out
+```
+
+### 2.2 The range product
+
+Self-contained, auditable without the skill installed:
+
+- `repro_scripts/<sha>.py` — one script per commit;
+- `output.log` + `output.html` — the verdicts;
+- a copy of `mechanical_refactor_reproduction_utils.py` — the scripts' only dependency.
+
+### 2.3 What the inference covers
+
+- **Method → existing class**: call sites lowered (`Owner.m(recv, …)` → `recv.m(…)`), the
+  orphaned local import removed.
+- **Method → module-level free function**: call sites requalified (`Owner.m(…)` → `m(…)`).
+- **Free function → existing module**: the call stays bare; callers repath their import
+  (`repath_import` when function-scoped; module-level repoints realised as remove-old +
+  add-new).
+- **New-module extract of scattered defs**: `extract_symbols_to_new_module` under the
+  audited header; a constant that relocated into the header is dropped from the source.
+  A contiguous-tail source still uses `extract_to_new_module`.
+- **A source file the commit deletes** once its defs relocated: `delete_file`.
+- **The module-level import diff**, realised directly from the target: gained names added
+  (a wholly new module's statement verbatim, wrapping kept), lost names removed with
+  `remove_imported_name`.
+- Non-Python files in the commit do not block inference; their diff is noted and left to
+  the residual.
+
+### 2.4 What it reports `UNSUPPORTED`
+
+- Single-commit mode prints the verdict with notes and exits non-zero; range mode
+  records it in the outputs.
+- Review such a commit as prepare, or hand-write the `Repro` (§3).
+- The cases:
+    - **no definition relocated** — a rename (even a privacy flip `_foo` → `foo`) or a
+      statement-level reorder; reshapes belong in prepare;
+    - **a new-module extract whose symbols are not all top-level in the source** — a
+      method still inside a class; prepare must de-self it out first;
+    - **an extract drawing from more than one source file**, and an **inline-block
+      extract-function** — compose `extract_function` by hand (the body must be unchanged;
+      a de-self / restructure is a separate semantic commit).
+
+## 3. Hand-write the `Repro` when inference falls short
+
+- Compose the transform from the same primitives (`spec-reproduction-utils.md` §3).
+- The same byte-diff then certifies it.
+
+```python
+import sys
+from pathlib import Path
+
+sys.path.append(".claude/skills/mechanical-refactor-verify/scripts")
+from mechanical_refactor_reproduction_utils import Repro
+
+r = Repro(base="<base_sha>", target="<commit>")
+# Adapt call sites / repath imports BEFORE moving, so a call to a moved method from inside
+# another moved method is lowered while still in the source and travels with the body.
+r.lower_call_sites("update_weights_from_ipc", "ModelRunner", paths=["a.py", "b.py"])
+r.remove_import("a.py", "from x import ModelRunner", in_function="update_weights_from_ipc")
+r.move_symbol("update_weights_from_ipc", src="a.py", dst="dst.py", into_class="WeightUpdater", dedent=0)
+r.add_import("dst.py", "import gc")
+r.run()   # PASS = byte-identical; otherwise prints the residual
+```
+
+## 4. A hand-written transform for a non-relocation mechanical change
+
+- For a whole-file split or rename — no single symbol relocates — write a `transform()`
+  and call `verify_mechanical_refactor`.
+- The scaffold (worktree, pre-commit, diff, reporting) lives in the skill's utils.
+
+```python
+import sys
+from pathlib import Path
+
+sys.path.append(".claude/skills/mechanical-refactor-verify/scripts")
+from mechanical_refactor_reproduction_utils import verify_mechanical_refactor, git_add_and_commit
+
+BASE_COMMIT = "<base_sha>"
+TARGET_COMMIT = "<final_sha>"
+
+def transform(dir_root: Path) -> None:
+    source = dir_root / "path/to/source.py"
+    lines = source.read_text().splitlines(keepends=True)
+    for target_path, start, end in [("path/to/a.py", 1, 50), ("path/to/b.py", 51, 120)]:
+        target = dir_root / target_path
+        target.parent.mkdir(parents=True, exist_ok=True)
+        target.write_text("".join(lines[start - 1 : end]))
+    source.unlink()
+    git_add_and_commit("split source.py", cwd=str(dir_root))
+    # A rename is just: for each file, write content.replace(OLD, NEW); commit.
+
+if __name__ == "__main__":
+    verify_mechanical_refactor(BASE_COMMIT, TARGET_COMMIT, transform)
+```
+
+## 5. Publish the proof with the PR
+
+### 5.1 What to share
+
+- Share the scripts **plus** the copied `mechanical_refactor_reproduction_utils.py` — the
+  scripts import it, so a lone raw file is not runnable.
+- Never a `python3 <(curl ...)` one-liner: process substitution gives the script no real
+  directory, so the import breaks.
+- Flat layouts work: Python puts the script's own directory on `sys.path`, so the utils
+  module can sit either next to the script or one level up (the `--out` layout).
+
+### 5.2 Author: create a gist
+
+```bash
+cd repro_out
+gh gist create --desc "mechanical-move proof for PR #NNNN" \
+    repro_scripts/*.py mechanical_refactor_reproduction_utils.py output.log
+# prints https://gist.github.com/<user>/<gist_id> -- put it in the PR description
+```
+
+- `gh gist create` flattens paths — fine per §5.1.
+- Alternatives: a PR attachment (zip the `--out` folder) or a branch holding it.
+
+### 5.3 Reviewer: download and re-run
+
+```bash
+gh gist clone <gist_id> /tmp/proof        # or: git clone https://gist.github.com/<gist_id>.git /tmp/proof
+cd <repo-root>                            # the run resolves the repo from the cwd
+python3 /tmp/proof/<sha>.py               # PASS = byte-identical to this commit
+```
+
+- Include exactly these commands in the PR description under a
+  "Mechanical move — reproducible" heading.
+
+### 5.4 Keep the PR mechanical
+
+- A mechanical PR contains **only** mechanical changes (moves, splits, renames, import
+  fixes, formatting). Semantic changes go in a separate PR.

.claude/skills/mechanical-refactor-verify/guide-split.md

@@ -0,0 +1,243 @@
+# Split a mechanical change: prepare, move, postpare
+
+## 1. Why split
+
+- A "move a method/function" change is really **two operations with different
+  correctness criteria**:
+
+| Operation | What it does | How you check it |
+|---|---|---|
+| **Semantic reshape** | method → free function or method; `self.X` → a parameter, or `self` retyped to the target class; signature / typing change | behavior unchanged: lint + tests pass |
+| **Physical move** | cut from the source, paste into the target, fix imports | the moved body is byte-identical, line for line; the only other changes are move artifacts |
+
+- Put both in one commit and the criteria contaminate each other:
+    - one hunk then holds the reshape **and** an indentation shift **and** a cross-file
+      relocation;
+    - neither a human nor a tool can mechanically confirm "the body that landed is the
+      body that left" — you must re-read the logic.
+
+## 2. The rule — up to three commits, in this order
+
+- **prepare (optional)** — a **minimal** in-place reshape the relocation needs (de-self a
+  method, retype `self`). Human-reviewed, so: small, **no cross-file def relocation, no
+  body relocation** — the code stays where it is.
+- **move** — the pure relocation; carries the **bulk**; certified by the reproduce proof
+  (`guide-construct-proof.md`; property: `spec-reproduction-utils.md`).
+- **postpare (optional)** — a **minimal** tail fixup the move cannot do mechanically (a
+  module path inside a string literal, a doc reference). Human-reviewed.
+
+Hard lines ("prep" below = the prepare phase):
+
+- Both ends are optional, minimal, and covered by tests; neither ever relocates a def
+  across files or moves a body.
+- The move-artifact whitelist is what a relocation *forces* — **not** a licence to fold
+  reshape work into the move. Anything outside the artifacts in the move's diff = the
+  reshape leaked; push it back into prep.
+- **A large semantic refactor is not a phase.** Consolidating bookkeeping, deduplicating
+  logic, restructuring control flow, redesigning an API → its **own commit**, reviewed for
+  **equivalence** (tests or a written argument). Never smuggled into prep as a "small
+  reshape".
+
+- The prep's shape depends on the destination: a module-level function (§3.1) or a class
+  (§3.2).
+- The move is the same idea in both: a pure relocation, body byte-identical.
+
+## 3. Cases
+
+### 3.1 Case 1: method → free function
+
+#### 3.1.1 Commit 1 — prep: de-self in place (no relocation)
+
+Reshape the method **in its original file and position** so it no longer needs `self`.
+The body stays put:
+
+- `self.X` (read) → pass `X` in as a parameter.
+- `self.X = v` (write) → `return v`; the caller assigns. (Or pass an explicit mutable
+  object.)
+- `self.other_method(...)` → prep that method in the same commit, or inject it as a
+  `Callable` argument.
+- Once `self` is gone → mark `@staticmethod`; the body **does not move**.
+- Call site: `self.foo(args)` → `TheClass.foo(args)`.
+
+- The decorator and the qualifier are the only artifacts the move will carry — exactly
+  what the whitelist (`spec-reproduction-utils.md` §2.1) forgives.
+
+**Check:** lint + tests pass; the diff is the body reshape plus the call-site qualifier;
+nothing moved.
+
+#### 3.1.2 Commit 2 — move: relocate to the module
+
+- Cut the `@staticmethod` block; paste into the target module.
+- Drop `@staticmethod`, dedent to module level — body **unchanged, line for line**.
+- Source file: import the moved symbol; drop now-unused imports.
+- Call site: `TheClass.foo(args)` → `foo(args)` (args untouched).
+
+**Check:** `mechanical_refactor_proof_generator.py <commit>` reports `PASS`. Cross-check:
+`git show <commit> --color-moved=dimmed-zebra --color-moved-ws=allow-indentation-change`
+marks the whole block as moved.
+
+### 3.2 Case 2: method → method on a class
+
+- For pulling **several methods and the fields they touch** into a new (or existing)
+  class.
+- Prep does **not** de-self — it builds the class and retypes `self`, body untouched.
+
+#### 3.2.1 Commit 1 — prep: build the class, retype `self`
+
+1. Create the target class with the fields the moved methods touch (a frozen dataclass is
+   simplest; drop `frozen` only if they mutate).
+2. Wire an instance into the call path — composition (`self.component = Target(...)` in
+   the source ctor), construction at the call site, or temporarily both.
+3. Retype each moved method as a `@staticmethod` whose parameter is still **named** `self`
+   but **typed** as the target class — body unchanged:
+
+   ```python
+   class Source:
+       component: Target
+
+       @staticmethod
+       def foo(self: Target) -> None:
+           ...  # body still reads self.field_a / self.field_b
+   ```
+
+4. Caller: `self.foo(...)` → `Source.foo(self.component, ...)`.
+
+Why keep the name `self`:
+
+- it is an ordinary parameter name, so every `self.X` resolves against the target class
+  statically and at runtime (the argument *is* a target-class instance);
+- renaming it would rewrite every `self.X` and destroy the "body unchanged across both
+  commits" invariant.
+
+Boundaries:
+
+- **Prep stays minimal.** Signature redesign, helper extraction, parameter objects,
+  mutate→return, renames, method splits, dead-branch removal → later non-mechanical
+  commits, never prep.
+- **Runtime-mutable state → inject a `Callable` getter (still prep).** State that changes
+  every step (counters, the current batch, running stats): inject `Callable[[], T]` into
+  the target ctor; rewrite `self.X` → `self.get_X()`. Do **not** thread it per call and do
+  **not** reach back into the source object — per-call kwargs make every call site noisy,
+  the API non-self-contained, and the threading a caller chore.
+
+  ```python
+  class Target:
+      def __init__(self, *, static_field, get_running_state: "Callable[[], State]"):
+          self.static_field = static_field
+          self.get_running_state = get_running_state
+
+      @staticmethod
+      def check(self: "Target") -> None:
+          running = self.get_running_state()   # was self.running_state
+          ...
+  ```
+
+  ```python
+  # source ctor
+  self.component = Target(
+      static_field=...,
+      get_running_state=lambda: self.running_state,
+  )
+  ```
+
+**Check:** lint + tests pass; body unchanged; types check (`self: Target` matches the
+instance the caller passes).
+
+#### 3.2.2 Commit 2 — move: relocate into the class
+
+- Cut `foo` into the target class; drop `@staticmethod` — body **unchanged, line for
+  line**.
+- Header: `def foo(self: Target)` → `def foo(self)` (type redundant inside the class).
+- Caller: `Source.foo(self.component, ...)` → `self.component.foo(...)` — the receiver
+  moves out of the argument list (replayed by `lower_call_sites`).
+
+**Check:** `mechanical_refactor_proof_generator.py <commit>` reports `PASS`. The split
+paid off: prep left the body untouched, so the move is a clean cut/paste.
+
+### 3.3 Case 3: extract to a new module — one move commit, no prep
+
+- The move gathers the defs **from wherever they sit** — no prep staging at the source
+  tail. Replayed by `extract_symbols_to_new_module`.
+- Each def/class is cut **verbatim** (the byte diff certifies the bodies); the new file's
+  small header (imports, a logger, constants, a `TYPE_CHECKING` block) is authored from
+  the target and audited (`spec-reproduction-utils.md` §2.1).
+- A module-level constant that moved into the header (e.g. `_is_hip = is_hip()`) is
+  dropped from the source too.
+- The only work outside the move: a non-mechanical reference the move cannot derive (a
+  string-literal module path) — a one-line **postpare**.
+- A symbol **not top-level** in the source (a method still in a class): prepare de-selfs
+  it out first (§3.1); the proof reports `UNSUPPORTED` until then.
+
+### 3.4 Case 4: extract-function — the bulk goes in the move
+
+- The relocated body belongs in a certified move, not buried in a prep: the
+  `extract_function` primitive cuts the inline block **verbatim** and authors only the
+  interface (signature, optional `return`, the replacing `call`).
+- Faithful **only when the body moves unchanged.** De-self, control-flow restructure, or a
+  bookkeeping change folded in → do that as a separate semantic commit (reviewed for
+  equivalence) **first**, then move the now-unchanged body.
+- An extraction that rewrites the body *as* it extracts is a semantic commit, not a
+  certifiable move — do not dress it up as one.
+
+## 4. Remarks
+
+### 4.1 A move never renames
+
+- The moved symbol keeps the **same name on both sides**.
+- A rename — even a privacy flip `_foo` → `foo` — is its own single-purpose commit
+  *before* the move (rename in place, update call sites).
+- A move that also renames cannot be machine-certified: split it — rename first, then
+  move.
+
+### 4.2 Anti-pattern: prep adds the body, move deletes it
+
+- Symptom: prep **adds** a large block to the target; the move **deletes** the same block
+  from the source. The order is reversed.
+- Correct order: prep leaves the body in the source (target skeleton, header retype,
+  caller qualification only); the move does the cut/paste.
+- The body appears and disappears exactly once — on the move side. Fix by pushing the
+  "add the body" work out of prep into the move.
+
+### 4.3 When NOT to split (single commit)
+
+- Moving an **already** module-level free function.
+- Pure file rename / whole-file move.
+- Trivial field deletion, or `getattr(obj, "x", ...)` → direct attribute access.
+- A class-internal helper relocated next to another helper in the same module.
+
+### 4.4 Which actions are mechanical vs not
+
+- Boundary: building the component correctly the first time is mechanical; reshaping it
+  *after* it exists is not.
+
+| Action | Bucket |
+|---|---|
+| target class skeleton + ctor + fields | mechanical (prep) |
+| `@dataclass(frozen=True, slots=True, kw_only=True)` decoration | mechanical (prep) |
+| composition wiring (`self.component = Target(...)`) | mechanical (prep) |
+| `Callable` getter injection for runtime-mutable state | mechanical (prep) |
+| platform conditionals carried along with the body | mechanical (prep / move) |
+| cross-file import path rewrites | mechanical (move) |
+| field-ownership migration into the component ctor | mechanical (a single pre-step) |
+| inlining an `init_*` method body into a ctor | mechanical (a single pre-step) |
+| privacy flip (`_x` ↔ `x`) | mechanical (a single rename) |
+| signature redesign (new kwargs, changed defaults, positional → kw-only) | **not** mechanical |
+| body simplification / dead-branch removal / logic rewrite | **not** mechanical |
+| semantic method rename | **not** mechanical |
+
+- The smaller the prep, the easier "behavior unchanged" is to confirm.
+- Many small, independently reviewable commits beat one big prep mixing ten flavors of
+  change.
+- Review order = commit order: prep → move → non-mechanical follow-ups.
+
+### 4.5 Naming
+
+- Consecutive commits with reserved suffixes; short kebab `<id>`:
+
+```
+<id>-prepare: <subject>    # optional: minimal in-place reshape (de-self, or retype-self)
+<id>-move: <subject>       # pure relocation, certified by the reproduce proof
+<id>-postpare: <subject>   # optional: minimal tail fixup (e.g. a string-literal path)
+```
+
+- The `<phase>:` form is what the range command's `--match -move:` regex keys on.

.claude/skills/mechanical-refactor-verify/guide-verify-proof.md

@@ -0,0 +1,62 @@
+# Verify a proof for a move commit
+
+- How the reviewer of a claimed-mechanical commit consumes its proof.
+- The certified property and primitive contracts: `spec-reproduction-utils.md`.
+- How the proof was produced and the folder it arrives in: `guide-construct-proof.md`.
+
+## 1. Re-run it
+
+- From the repo root:
+
+  ```bash
+  python3 <folder>/repro_scripts/<sha>.py
+  ```
+
+- The run *is* the proof — it replays the primitives from the base commit and byte-diffs
+  against the target in a throwaway worktree.
+- Do not trust a pasted verdict you did not re-run.
+
+## 2. Read the verdict
+
+- **PASS** — byte-identical: the commit is exactly the relocations listed in the script,
+  nothing else.
+- **RESIDUAL** — a non-empty diff: precisely the bundled non-move change. Review it as
+  semantic content; a legitimate tail fixup (string-literal module path, doc reference)
+  belongs in a postpare commit, not the move.
+- **UNSUPPORTED** — no recipe inferred (cases: `guide-construct-proof.md` §2.4). Not
+  thereby wrong, but not machine-certified: review by hand as a prepare-style reshape, or
+  ask the author for a hand-written `Repro`.
+
+## 3. Audit the authored surfaces
+
+- A PASS certifies the relocated bytes; the small **authored** surfaces are reproduced
+  from the target and need human eyes.
+- In the script, check:
+    - the `header=` of `extract_symbols_to_new_module` — the module audits its content
+      (imports / docstring / TYPE_CHECKING imports / logger / relocated `drop_assigns`
+      copies only); what remains for you: should those assignments move at all?
+    - a `leave_delegate=` on `move_symbol` — the forwarding stub is authored code in the
+      source file;
+    - the `signature=` / `return_text=` / `call=` of `extract_function` — the new
+      function's interface is authored; only its body is certified;
+    - the `drop_assigns=` list — each named constant leaves the source file.
+
+## 4. Know what a PASS does and does not assert
+
+- Requalification / lowering / repath in a script is tied to symbols the same script
+  relocates; a consumer-only call or import rewrite (no relocated definition) cannot
+  reproduce as a move — it surfaces as a residual.
+- Whatever the repo's pre-commit hooks auto-fix is absorbed on both sides
+  (`spec-reproduction-utils.md` §4) — the hook set is part of what you trust.
+- A PASS judges the **shape of a relocation**, not **intent**: "this commit is exactly
+  these relocations", not "this relocation was a good idea". Confirm the commit's subject
+  matches what the script actually moves before approving.
+
+## 5. Why the mechanism is trustworthy
+
+- It runs the real formatter and compares bytes — no diff-shape heuristic to fool
+  (`spec-reproduction-utils.md` §4).
+- The proof is the few primitive calls in the script; auditing them (plus §3) is the
+  whole human surface.
+- The folder is self-contained and re-runnable by anyone — a CI step or a reviewer —
+  without the skill installed.

.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/conftest.py

@@ -0,0 +1,24 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from generator_testlib import _git
+from mechanical_refactor_proof_generator import (
+    infer_recipe,
+    recipe_to_script,
+)
+
+
+@pytest.fixture
+def repo(tmp_path: Path) -> Path:
+    root = tmp_path / "repo"
+    root.mkdir()
+    _git(root, "init", "-q")
+    _git(root, "config", "user.email", "test@example.com")
+    _git(root, "config", "user.name", "test")
+    _git(root, "config", "commit.gpgsign", "false")
+    return root

.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/generator_testlib.py

@@ -0,0 +1,106 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from mechanical_refactor_proof_generator import (
+    infer_recipe,
+    recipe_to_script,
+)
+
+
+def _git(repo: Path, *args: str) -> str:
+    return subprocess.run(
+        ["git", *args], cwd=repo, check=True, capture_output=True, text=True
+    ).stdout.strip()
+
+
+def _write(repo: Path, **files: str | None) -> None:
+    for name, content in files.items():
+        path = repo / name.replace("__", "/")
+        if content is None:
+            path.unlink()
+        else:
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text(content)
+
+
+def _commit(repo: Path, message: str) -> str:
+    _git(repo, "add", "-A")
+    _git(repo, "commit", "-q", "-m", message)
+    return _git(repo, "rev-parse", "HEAD")
+
+
+def _method_onto_class(repo: Path) -> None:
+    """Stage a base + a 'move foo from M onto C, lower the caller' commit."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    @staticmethod\n"
+                "    def foo(self, x):\n"
+                "        return x + 1\n"
+                "\n"
+                "    def other(self):\n"
+                "        return 0\n"
+            ),
+            "comp.py": "class C:\n    def keep(self):\n        return 1\n",
+            "caller.py": (
+                "class K:\n"
+                "    def run(self):\n"
+                "        from model import M\n"
+                "\n"
+                "        return M.foo(self.c, 9)\n"
+            ),
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def other(self):\n        return 0\n",
+            "comp.py": (
+                "class C:\n"
+                "    def keep(self):\n"
+                "        return 1\n"
+                "\n"
+                "    def foo(self, x):\n"
+                "        return x + 1\n"
+            ),
+            "caller.py": (
+                "class K:\n    def run(self):\n        return self.c.foo(9)\n"
+            ),
+        },
+    )
+    _commit(repo, "move foo onto C")
+
+
+def _free_function_move_with_module_level_caller(repo: Path) -> None:
+    """Stage a free function moved model.py -> util.py whose caller imports it at module
+    level (so the repoint shows up in the symmetric module-level import diff)."""
+    _write(
+        repo,
+        **{
+            "model.py": "def keep():\n    return 0\n\n\ndef resolve(m):\n    return m\n",
+            "util.py": "import os\n",
+            "caller.py": (
+                "from model import resolve\n\n\ndef run(m):\n    return resolve(m)\n"
+            ),
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "def keep():\n    return 0\n",
+            "util.py": "import os\n\n\ndef resolve(m):\n    return m\n",
+            "caller.py": (
+                "from util import resolve\n\n\ndef run(m):\n    return resolve(m)\n"
+            ),
+        },
+    )
+    _commit(repo, "move resolve to util")

.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_infer_extracts.py

@@ -0,0 +1,230 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from generator_testlib import (  # noqa: F401
+    _commit,
+    _free_function_move_with_module_level_caller,
+    _git,
+    _method_onto_class,
+    _write,
+)
+from mechanical_refactor_proof_generator import (
+    infer_recipe,
+    recipe_to_script,
+)
+
+
+def test_infer_recipe_new_file_extract_from_class_method_unsupported(
+    repo: Path,
+) -> None:
+    """A method still inside the class cut straight into a new module cannot be cut as a
+    top-level symbol, so the extract is reported unsupported (prep must lift it out first).
+    """
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    @staticmethod\n"
+                "    def foo(self):\n"
+                "        return 1\n"
+                "\n"
+                "    def other(self):\n"
+                "        return 0\n"
+            )
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def other(self):\n        return 0\n",
+            "newmod.py": "def foo():\n    return 1\n",
+        },
+    )
+    _commit(repo, "extract foo to a new module")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert recipe.supported is False
+    assert any("not all top-level" in note for note in recipe.notes)
+
+
+def test_infer_recipe_new_file_extract_from_staged_tail(repo: Path) -> None:
+    """A staged trailing block (scaffolding + def at the source tail) cut into a new file
+    infers an extract_to_new_module, prepending the future import."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    def keep(self):\n"
+                "        return 1\n"
+                "\n"
+                "\n"
+                "import logging\n"
+                "\n"
+                "logger = logging.getLogger(__name__)\n"
+                "\n"
+                "\n"
+                "def foo(x):\n"
+                "    return x + 1\n"
+            )
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def keep(self):\n        return 1\n",
+            "newmod.py": (
+                "from __future__ import annotations\n"
+                "\n"
+                "import logging\n"
+                "\n"
+                "logger = logging.getLogger(__name__)\n"
+                "\n"
+                "\n"
+                "def foo(x):\n"
+                "    return x + 1\n"
+            ),
+        },
+    )
+    _commit(repo, "extract foo to a new module")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert recipe.supported
+    assert recipe.moves == []
+    assert recipe.extracts == [
+        {
+            "src": "model.py",
+            "dst": "newmod.py",
+            "symbols": ["foo"],
+            "future_import": True,
+        }
+    ]
+
+
+def test_infer_recipe_scattered_new_module_extract(repo: Path) -> None:
+    """Scattered top-level defs cut into a new module (no staged trailing block) infer a scatter
+    extract with the authored header and target order, not UNSUPPORTED."""
+    _write(
+        repo,
+        **{
+            "common.py": (
+                "import os\n"
+                "\n"
+                "\n"
+                "def keep():\n"
+                "    return 0\n"
+                "\n"
+                "\n"
+                "def beta():\n"
+                "    return 2\n"
+                "\n"
+                "\n"
+                "def stay():\n"
+                "    return 9\n"
+                "\n"
+                "\n"
+                "def alpha():\n"
+                "    return 1\n"
+            ),
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "common.py": (
+                "import os\n"
+                "\n"
+                "\n"
+                "def keep():\n"
+                "    return 0\n"
+                "\n"
+                "\n"
+                "def stay():\n"
+                "    return 9\n"
+            ),
+            "alloc.py": (
+                "from __future__ import annotations\n"
+                "\n"
+                "import logging\n"
+                "\n"
+                "logger = logging.getLogger(__name__)\n"
+                "\n"
+                "\n"
+                "def alpha():\n"
+                "    return 1\n"
+                "\n"
+                "\n"
+                "def beta():\n"
+                "    return 2\n"
+            ),
+        },
+    )
+    _commit(repo, "extract alpha, beta to alloc.py")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert recipe.supported
+    assert recipe.extracts == []
+    assert recipe.moves == []
+    assert len(recipe.scatter_extracts) == 1
+    sx = recipe.scatter_extracts[0]
+    assert sx["src"] == "common.py" and sx["dst"] == "alloc.py"
+    assert sorted(sx["symbols"]) == ["alpha", "beta"]
+    assert sx["order"] == ["alpha", "beta"]
+    assert sx["header"].startswith("from __future__ import annotations\n")
+    assert "logger = logging.getLogger(__name__)" in sx["header"]
+    assert sx["drop_assigns"] == []
+    script = recipe_to_script(recipe, "extract alpha, beta to alloc.py")
+    assert "extract_symbols_to_new_module" in script
+
+
+def test_infer_recipe_scatter_extract_drops_relocated_constant(repo: Path) -> None:
+    """A module-level constant relocated into the new module is inferred as a drop_assign so the
+    scatter extract removes it from the source too; a constant the source keeps is not.
+    """
+    _write(
+        repo,
+        **{
+            "common.py": (
+                "from u import is_hip\n"
+                "\n"
+                "_IS_HIP = is_hip()\n"
+                "logger = 1\n"
+                "\n"
+                "\n"
+                "def moved():\n"
+                "    return _IS_HIP\n"
+                "\n"
+                "\n"
+                "def keep():\n"
+                "    return logger\n"
+            ),
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "common.py": ("logger = 1\n\n\ndef keep():\n    return logger\n"),
+            "alloc.py": (
+                "from __future__ import annotations\n"
+                "\n"
+                "from u import is_hip\n"
+                "\n"
+                "_IS_HIP = is_hip()\n"
+                "\n"
+                "\n"
+                "def moved():\n"
+                "    return _IS_HIP\n"
+            ),
+        },
+    )
+    _commit(repo, "extract moved to alloc.py")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert len(recipe.scatter_extracts) == 1
+    assert recipe.scatter_extracts[0]["drop_assigns"] == ["_IS_HIP"]

.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_infer_imports.py

@@ -0,0 +1,168 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from generator_testlib import (  # noqa: F401
+    _commit,
+    _free_function_move_with_module_level_caller,
+    _git,
+    _method_onto_class,
+    _write,
+)
+from mechanical_refactor_proof_generator import (
+    infer_recipe,
+    recipe_to_script,
+)
+
+
+def test_infer_recipe_infers_added_module_imports(repo: Path) -> None:
+    """An import the destination module gains (the moved code needs it) is inferred."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "import gc\n"
+                "\n"
+                "class M:\n"
+                "    @staticmethod\n"
+                "    def foo(self):\n"
+                "        gc.collect()\n"
+                "        return 1\n"
+                "\n"
+                "    def other(self):\n"
+                "        return 0\n"
+            ),
+            "comp.py": "class C:\n    def keep(self):\n        return 1\n",
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def other(self):\n        return 0\n",
+            "comp.py": (
+                "import gc\n"
+                "\n"
+                "class C:\n"
+                "    def keep(self):\n"
+                "        return 1\n"
+                "\n"
+                "    def foo(self):\n"
+                "        gc.collect()\n"
+                "        return 1\n"
+            ),
+        },
+    )
+    _commit(repo, "move foo onto C")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert {"path": "comp.py", "text": "import gc"} in recipe.import_additions
+
+
+def test_infer_recipe_module_level_import_repoint_realised_by_diff(repo: Path) -> None:
+    """A module-level consumer whose import is repointed old -> new yields a remove of the old
+    name and an add of the new -- not a reliance on the formatter pruning a duplicate.
+    """
+    _free_function_move_with_module_level_caller(repo)
+    recipe = infer_recipe("HEAD", str(repo))
+    assert recipe.repaths == []
+    assert {
+        "path": "caller.py",
+        "module": "model",
+        "name": "resolve",
+        "asname": None,
+    } in recipe.module_import_removals
+    assert {"path": "caller.py", "text": "from util import resolve"} in (
+        recipe.import_additions
+    )
+
+
+def test_infer_recipe_removes_an_import_the_source_no_longer_uses(repo: Path) -> None:
+    """When the moved body took the source's only use of an import, the source's lost name is
+    realised as a removal (deterministic, not left to the formatter)."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "import gc\n"
+                "\n"
+                "class M:\n"
+                "    @staticmethod\n"
+                "    def foo(self):\n"
+                "        gc.collect()\n"
+                "        return 1\n"
+                "\n"
+                "    def other(self):\n"
+                "        return 0\n"
+            ),
+            "comp.py": "class C:\n    def keep(self):\n        return 1\n",
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def other(self):\n        return 0\n",
+            "comp.py": (
+                "import gc\n"
+                "\n"
+                "class C:\n"
+                "    def keep(self):\n"
+                "        return 1\n"
+                "\n"
+                "    def foo(self):\n"
+                "        gc.collect()\n"
+                "        return 1\n"
+            ),
+        },
+    )
+    _commit(repo, "move foo onto C")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert {
+        "path": "model.py",
+        "module": None,
+        "name": "gc",
+        "asname": None,
+    } in recipe.module_import_removals
+
+
+def test_infer_recipe_adds_wholly_new_module_import_verbatim(repo: Path) -> None:
+    """An import gained from a module not present in base is captured as the target's verbatim
+    statement (so an exploded/magic-comma wrapping is reproduced, not collapsed per-name).
+    """
+    _write(
+        repo,
+        **{
+            "model.py": "def keep():\n    return 0\n\n\ndef solve(x):\n    return x\n",
+            "util.py": "import os\n",
+            "caller.py": (
+                "from model import solve\n\n\ndef run():\n    return solve(1)\n"
+            ),
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "def keep():\n    return 0\n",
+            "util.py": "import os\n\n\ndef solve(x):\n    return x\n",
+            "caller.py": (
+                "from util import (\n    solve,\n)\n\n\ndef run():\n    return solve(1)\n"
+            ),
+        },
+    )
+    _commit(repo, "move solve to util")
+    recipe = infer_recipe("HEAD", str(repo))
+    caller_adds = [
+        a["text"] for a in recipe.import_additions if a["path"] == "caller.py"
+    ]
+    assert "from util import (\n    solve,\n)" in caller_adds
+    assert {
+        "path": "caller.py",
+        "module": "model",
+        "name": "solve",
+        "asname": None,
+    } in recipe.module_import_removals

.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_infer_moves.py

@@ -0,0 +1,290 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from generator_testlib import (  # noqa: F401
+    _commit,
+    _free_function_move_with_module_level_caller,
+    _git,
+    _method_onto_class,
+    _write,
+)
+from mechanical_refactor_proof_generator import (
+    infer_recipe,
+    recipe_to_script,
+)
+
+
+def test_infer_recipe_method_onto_class(repo: Path) -> None:
+    """A method move onto a class infers the move, the call-site lowering, and the orphaned
+    local import removal."""
+    _method_onto_class(repo)
+    recipe = infer_recipe("HEAD", str(repo))
+    assert recipe.supported
+    assert [
+        (m["name"], m["src"], m["dst"], m["into_class"], m["dedent"])
+        for m in recipe.moves
+    ] == [("foo", "model.py", "comp.py", "C", 0)]
+    assert recipe.lowerings == [
+        {"name": "foo", "owner": "M", "path": "caller.py", "kind": "lower"}
+    ]
+    assert recipe.import_removals == [
+        {"path": "caller.py", "text": "from model import M", "in_function": "run"}
+    ]
+    assert recipe.import_additions == []
+
+
+def test_infer_recipe_free_function_move_uses_requalify(repo: Path) -> None:
+    """A move to a module-level free function dedents and requalifies the call site
+    (drops the qualifier), rather than lowering a receiver."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    @staticmethod\n"
+                "    def foo(x):\n"
+                "        return x + 1\n"
+                "\n"
+                "    def other(self):\n"
+                "        return 0\n"
+            ),
+            "util.py": "import os\n",
+            "caller.py": (
+                "class K:\n"
+                "    def run(self):\n"
+                "        from model import M\n"
+                "\n"
+                "        return M.foo(9)\n"
+            ),
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def other(self):\n        return 0\n",
+            "util.py": "import os\n\n\ndef foo(x):\n    return x + 1\n",
+            "caller.py": ("class K:\n    def run(self):\n        return foo(9)\n"),
+        },
+    )
+    _commit(repo, "move foo to util as a free function")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert [(m["name"], m["into_class"], m["dedent"]) for m in recipe.moves] == [
+        ("foo", None, 4)
+    ]
+    assert recipe.lowerings == [
+        {"name": "foo", "owner": "M", "path": "caller.py", "kind": "requalify"}
+    ]
+
+
+def test_infer_recipe_excludes_the_moved_bodys_own_call(repo: Path) -> None:
+    """A same-named call on a different receiver inside the moved body is not a caller
+    lowering (only `M.foo(...)` is, not `worker.foo(...)`)."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    @staticmethod\n"
+                "    def foo(self, x):\n"
+                "        worker.foo(x)\n"
+                "        return x\n"
+                "\n"
+                "    def other(self):\n"
+                "        return 0\n"
+            ),
+            "comp.py": "class C:\n    def keep(self):\n        return 1\n",
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def other(self):\n        return 0\n",
+            "comp.py": (
+                "class C:\n"
+                "    def keep(self):\n"
+                "        return 1\n"
+                "\n"
+                "    def foo(self, x):\n"
+                "        worker.foo(x)\n"
+                "        return x\n"
+            ),
+        },
+    )
+    _commit(repo, "move foo onto C")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert recipe.lowerings == []
+
+
+def test_infer_recipe_skips_nested_functions(repo: Path) -> None:
+    """A def nested inside a moved method is not inferred as its own move."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    def wrap(self):\n"
+                "        def inner(z):\n"
+                "            return z\n"
+                "        return inner\n"
+                "\n"
+                "    def other(self):\n"
+                "        return 0\n"
+            ),
+            "comp.py": "class C:\n    def keep(self):\n        return 1\n",
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "class M:\n    def other(self):\n        return 0\n",
+            "comp.py": (
+                "class C:\n"
+                "    def keep(self):\n"
+                "        return 1\n"
+                "\n"
+                "    def wrap(self):\n"
+                "        def inner(z):\n"
+                "            return z\n"
+                "        return inner\n"
+            ),
+        },
+    )
+    _commit(repo, "move wrap onto C")
+    recipe = infer_recipe("HEAD", str(repo))
+    names = [m["name"] for m in recipe.moves]
+    assert names == ["wrap"]
+    assert any("inner" in n for n in recipe.notes)
+
+
+def test_infer_recipe_free_function_source_move_repaths_caller(repo: Path) -> None:
+    """A free function moved to an existing module becomes a move_symbol with the call left
+    bare; a caller's function-scoped import is repathed."""
+    _write(
+        repo,
+        **{
+            "model.py": "def keep():\n    return 0\n\n\ndef resolve(m):\n    return m\n",
+            "util.py": "import os\n",
+            "caller.py": (
+                "class K:\n"
+                "    def run(self):\n"
+                "        from model import resolve\n"
+                "\n"
+                "        return resolve(self.m)\n"
+            ),
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "def keep():\n    return 0\n",
+            "util.py": "import os\n\n\ndef resolve(m):\n    return m\n",
+            "caller.py": (
+                "class K:\n"
+                "    def run(self):\n"
+                "        from util import resolve\n"
+                "\n"
+                "        return resolve(self.m)\n"
+            ),
+        },
+    )
+    _commit(repo, "move resolve to util")
+    recipe = infer_recipe("HEAD", str(repo))
+    assert recipe.supported
+    assert [(m["name"], m["src"], m["dst"], m["into_class"]) for m in recipe.moves] == [
+        ("resolve", "model.py", "util.py", None)
+    ]
+    assert recipe.lowerings == []
+    assert recipe.repaths == [
+        {
+            "path": "caller.py",
+            "old_module": "model",
+            "new_module": "util",
+            "name": "resolve",
+        }
+    ]
+
+
+def test_infer_recipe_survives_a_non_python_file_in_the_commit(repo: Path) -> None:
+    """A commit also touching a .md file infers the move and notes the non-Python path."""
+    _write(
+        repo,
+        **{
+            "model.py": "def foo():\n    return 1\n\n\ndef keep():\n    return 0\n",
+            "util.py": "x = 1\n",
+            "README.md": "hello\n",
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": "def keep():\n    return 0\n",
+            "util.py": "x = 1\n\n\ndef foo():\n    return 1\n",
+            "README.md": "hello world, this is plain markdown text\n",
+        },
+    )
+    commit = _commit(repo, "move foo and touch docs")
+
+    recipe = infer_recipe(commit, str(repo))
+
+    assert [mv["name"] for mv in recipe.moves] == ["foo"]
+    assert any("README.md" in note for note in recipe.notes)
+
+
+def test_infer_recipe_records_the_source_class_for_disambiguation(repo: Path) -> None:
+    """A method move carries from_class so the cut cannot hit a same-named other method."""
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    def foo(self, x):\n"
+                "        return x + 1\n"
+                "\n"
+                "\n"
+                "class Other:\n"
+                "    def foo(self, x):\n"
+                "        return x + 2\n"
+            ),
+            "comp.py": "class C:\n    def keep(self):\n        return 1\n",
+        },
+    )
+    _commit(repo, "base")
+    _write(
+        repo,
+        **{
+            "model.py": (
+                "class M:\n"
+                "    pass\n"
+                "\n"
+                "\n"
+                "class Other:\n"
+                "    def foo(self, x):\n"
+                "        return x + 2\n"
+            ),
+            "comp.py": (
+                "class C:\n"
+                "    def keep(self):\n"
+                "        return 1\n"
+                "\n"
+                "    def foo(self, x):\n"
+                "        return x + 1\n"
+            ),
+        },
+    )
+    commit = _commit(repo, "move M.foo onto C")
+
+    recipe = infer_recipe(commit, str(repo))
+
+    assert [mv["from_class"] for mv in recipe.moves] == ["M"]
+    script = recipe_to_script(recipe, "move M.foo onto C")
+    assert "from_class='M'" in script

.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_script_and_diff.py

@@ -0,0 +1,54 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+from generator_testlib import (  # noqa: F401
+    _commit,
+    _free_function_move_with_module_level_caller,
+    _git,
+    _method_onto_class,
+    _write,
+)
+from mechanical_refactor_proof_generator import (
+    infer_recipe,
+    recipe_to_script,
+)
+
+
+def test_recipe_to_script_is_self_contained_and_ordered(repo: Path) -> None:
+    """The emitted script imports only the reproduce util and lowers before moving."""
+    _method_onto_class(repo)
+    script = recipe_to_script(infer_recipe("HEAD", str(repo)), "move foo onto C")
+    assert "from mechanical_refactor_reproduction_utils import Repro" in script
+    assert script.index("lower_call_sites") < script.index("move_symbol")
+    assert "r.run()" in script
+    # importing nothing else from the skill keeps the script auditable in isolation
+    assert "mechanical_refactor_verify_utils" not in script
+    assert "mechanical_refactor_proof_generator" not in script
+
+
+def test_recipe_to_script_orders_import_ops_after_moves(repo: Path) -> None:
+    """The emitted script applies module-level import add/remove AFTER the move, matching
+    build_repro's run order so the script and the in-process verdict cannot diverge."""
+    _free_function_move_with_module_level_caller(repo)
+    script = recipe_to_script(infer_recipe("HEAD", str(repo)), "move resolve to util")
+    assert script.index("move_symbol") < script.index("remove_imported_name")
+    assert script.index("move_symbol") < script.index("add_import")
+
+
+def test_per_file_diff_keeps_content_lines_starting_with_plus_signs(repo: Path) -> None:
+    """An added content line beginning with '++' is collected, not mistaken for a header."""
+    from mechanical_refactor_proof_generator import _per_file_diff
+
+    _write(repo, **{"notes.py": "a = 1\n"})
+    _commit(repo, "base")
+    _write(repo, **{"notes.py": 'a = 1\nb = "++x"\n'})
+    commit = _commit(repo, "add plus-plus line")
+
+    files = _per_file_diff(commit, str(repo))
+
+    assert files["notes.py"]["added"] == ['b = "++x"']

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/conftest.py

@@ -0,0 +1,36 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _git
+
+
+@pytest.fixture
+def repo(tmp_path: Path) -> Path:
+    root = tmp_path / "repo"
+    root.mkdir()
+    _git(root, "init", "-q")
+    _git(root, "config", "user.email", "test@example.com")
+    _git(root, "config", "user.name", "test")
+    _git(root, "config", "commit.gpgsign", "false")
+    return root
+
+
+# --- exec_command --------------------------------------------------------------

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/reproduction_testlib.py

@@ -0,0 +1,49 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+
+
+def _apply(repro: Repro, root: Path) -> None:
+    """Run a built Repro's recorded operations against a plain directory (no git)."""
+    for op in repro.ops:
+        op(root)
+
+
+def _git(repo: Path, *args: str) -> str:
+    return subprocess.run(
+        ["git", *args], cwd=repo, check=True, capture_output=True, text=True
+    ).stdout.strip()
+
+
+def _write(repo: Path, **files: str | None) -> None:
+    for name, content in files.items():
+        path = repo / name.replace("__", "/")
+        if content is None:
+            path.unlink()
+        else:
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text(content)
+
+
+def _commit(repo: Path, message: str) -> str:
+    _git(repo, "add", "-A")
+    _git(repo, "commit", "-q", "-m", message)
+    return _git(repo, "rev-parse", "HEAD")

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_add_imports.py

@@ -0,0 +1,142 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- add_import ----------------------------------------------------------------
+
+
+def test_add_import_appends_after_last_top_level_import(tmp_path: Path) -> None:
+    """A new import is inserted right after the last module-level import."""
+    (tmp_path / "m.py").write_text("import os\nimport sys\n\nx = 1\n")
+    r = Repro("b", "t").add_import("m.py", "from pkg import Thing")
+    _apply(r, tmp_path)
+    assert (
+        tmp_path / "m.py"
+    ).read_text() == "import os\nimport sys\nfrom pkg import Thing\n\nx = 1\n"
+
+
+# --- repath_import / add_typechecking_import -----------------------------------
+
+
+def test_add_typechecking_import_inserts_in_block(tmp_path: Path) -> None:
+    """The import is appended inside the existing TYPE_CHECKING block."""
+    (tmp_path / "m.py").write_text(
+        "from typing import TYPE_CHECKING\n"
+        "\n"
+        "if TYPE_CHECKING:\n"
+        "    from a import X\n"
+        "\n"
+        "\n"
+        "def f():\n"
+        "    pass\n"
+    )
+    r = Repro("b", "t").add_typechecking_import("m.py", "from b import Y")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == (
+        "from typing import TYPE_CHECKING\n"
+        "\n"
+        "if TYPE_CHECKING:\n"
+        "    from a import X\n"
+        "    from b import Y\n"
+        "\n"
+        "\n"
+        "def f():\n"
+        "    pass\n"
+    )
+
+
+def test_add_import_into_an_empty_file(tmp_path: Path) -> None:
+    """Adding an import to an empty file writes just the statement."""
+    (tmp_path / "m.py").write_text("")
+    r = Repro("b", "t").add_import("m.py", "import os")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "import os\n"
+
+
+def test_add_import_lands_below_a_module_docstring(tmp_path: Path) -> None:
+    """In a file with only a docstring, the new import must land below the docstring."""
+    (tmp_path / "m.py").write_text('"""Module doc."""\n\nx = 1\n')
+    r = Repro("b", "t").add_import("m.py", "import os")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text().startswith('"""Module doc."""')
+
+
+def test_add_typechecking_import_matches_qualified_typing_form(tmp_path: Path) -> None:
+    """A `if typing.TYPE_CHECKING:` block is recognized and receives the import."""
+    (tmp_path / "m.py").write_text(
+        "import typing\n"
+        "\n"
+        "if typing.TYPE_CHECKING:\n"
+        "    from a import X\n"
+        "\n"
+        "\n"
+        "def f():\n"
+        "    pass\n"
+    )
+    r = Repro("b", "t").add_typechecking_import("m.py", "from b import Y")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == (
+        "import typing\n"
+        "\n"
+        "if typing.TYPE_CHECKING:\n"
+        "    from a import X\n"
+        "    from b import Y\n"
+        "\n"
+        "\n"
+        "def f():\n"
+        "    pass\n"
+    )
+
+
+def test_add_typechecking_import_after_a_multiline_final_import(tmp_path: Path) -> None:
+    """The insert lands after the closing paren of a multi-line final guarded import."""
+    (tmp_path / "m.py").write_text(
+        "from typing import TYPE_CHECKING\n"
+        "\n"
+        "if TYPE_CHECKING:\n"
+        "    from a import (\n"
+        "        X,\n"
+        "    )\n"
+        "\n"
+        "x = 1\n"
+    )
+    r = Repro("b", "t").add_typechecking_import("m.py", "from b import Y")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == (
+        "from typing import TYPE_CHECKING\n"
+        "\n"
+        "if TYPE_CHECKING:\n"
+        "    from a import (\n"
+        "        X,\n"
+        "    )\n"
+        "    from b import Y\n"
+        "\n"
+        "x = 1\n"
+    )
+
+
+def test_add_typechecking_import_raises_without_a_block(tmp_path: Path) -> None:
+    """A file lacking a TYPE_CHECKING block fails loudly."""
+    (tmp_path / "m.py").write_text("import os\n\nx = 1\n")
+    r = Repro("b", "t").add_typechecking_import("m.py", "from b import Y")
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_call_sites.py

@@ -0,0 +1,158 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+
+def test_lowered_call_text_preserves_magic_trailing_comma(tmp_path: Path) -> None:
+    """A magic trailing comma in the original call survives the textual lowering."""
+    (tmp_path / "m.py").write_text("x = Old.foo(\n    self.r,\n    a,\n    b,\n)\n")
+    r = Repro("b", "t").lower_call_sites("foo", "Old", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "x = self.r.foo(\n    a,\n    b,\n)\n"
+
+
+# --- lower_call_sites ----------------------------------------------------------
+
+
+def test_lower_call_sites_moves_receiver_out_of_args(tmp_path: Path) -> None:
+    """Owner.foo(receiver, rest) becomes receiver.foo(rest)."""
+    (tmp_path / "m.py").write_text("x = ModelRunner.foo(self.r, a, b)\n")
+    r = Repro("b", "t").lower_call_sites("foo", "ModelRunner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "x = self.r.foo(a, b)\n"
+
+
+def test_lower_call_sites_handles_only_receiver_arg(tmp_path: Path) -> None:
+    """Owner.foo(receiver) becomes receiver.foo() without re-lowering the result."""
+    (tmp_path / "m.py").write_text("ModelRunner.foo(self.r)\n")
+    r = Repro("b", "t").lower_call_sites("foo", "ModelRunner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "self.r.foo()\n"
+
+
+def test_lower_call_sites_ignores_a_different_owner(tmp_path: Path) -> None:
+    """A same-named call on another receiver (e.g. the moved body's own call) is untouched."""
+    (tmp_path / "m.py").write_text("worker.foo(zmq)\n")
+    r = Repro("b", "t").lower_call_sites("foo", "ModelRunner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "worker.foo(zmq)\n"
+
+
+def test_lower_call_sites_preserves_magic_trailing_comma(tmp_path: Path) -> None:
+    """A magic trailing comma is kept so the formatter re-explodes the lowered call."""
+    (tmp_path / "m.py").write_text("ModelRunner.foo(\n    self.r,\n    a,\n)\n")
+    r = Repro("b", "t").lower_call_sites("foo", "ModelRunner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "self.r.foo(\n    a,\n)\n"
+
+
+# --- requalify_call_sites ------------------------------------------------------
+
+
+# --- requalify_call_sites ------------------------------------------------------
+
+
+def test_requalify_call_sites_drops_the_qualifier(tmp_path: Path) -> None:
+    """Owner.bar(args) becomes bar(args) when bar moves to a free function."""
+    (tmp_path / "m.py").write_text("y = ModelRunner.bar(a, b)\n")
+    r = Repro("b", "t").requalify_call_sites("bar", "ModelRunner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "y = bar(a, b)\n"
+
+
+# --- adversarial audit: call-site rewrites ---------------------------------------
+
+
+# --- adversarial audit: call-site rewrites ---------------------------------------
+
+
+def test_requalify_call_sites_matches_a_zero_argument_call(tmp_path: Path) -> None:
+    """Owner.bar() with no arguments is requalified to bar()."""
+    (tmp_path / "m.py").write_text("y = Owner.bar()\n")
+    r = Repro("b", "t").requalify_call_sites("bar", "Owner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "y = bar()\n"
+
+
+def test_lower_call_sites_preserves_comments_inside_a_multiline_call(
+    tmp_path: Path,
+) -> None:
+    """A comment between arguments of the rewritten call must survive."""
+    (tmp_path / "m.py").write_text(
+        "x = Old.foo(\n    self.r,\n    a,  # keep me\n    b,\n)\n"
+    )
+    r = Repro("b", "t").lower_call_sites("foo", "Old", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert "# keep me" in (tmp_path / "m.py").read_text()
+
+
+def test_lower_call_sites_preserves_arg_literal_spelling(tmp_path: Path) -> None:
+    """Hex literals and quote styles inside the rewritten call must not be normalized."""
+    (tmp_path / "m.py").write_text('x = Old.foo(self.r, 0x10, "s")\n')
+    r = Repro("b", "t").lower_call_sites("foo", "Old", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == 'x = self.r.foo(0x10, "s")\n'
+
+
+def test_lower_call_sites_lowers_a_nested_matching_call_too(tmp_path: Path) -> None:
+    """A matching call nested inside another matching call is lowered as well."""
+    (tmp_path / "m.py").write_text("x = Old.foo(self.r, Old.foo(self.q, 1))\n")
+    r = Repro("b", "t").lower_call_sites("foo", "Old", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "x = self.r.foo(self.q.foo(1))\n"
+
+
+def test_lower_call_sites_magic_comma_with_sole_receiver_arg_stays_valid(
+    tmp_path: Path,
+) -> None:
+    """Lowering a magic-comma call whose only argument is the receiver stays valid Python."""
+    (tmp_path / "m.py").write_text("Owner.foo(\n    self.r,\n)\n")
+    r = Repro("b", "t").lower_call_sites("foo", "Owner", paths=["m.py"])
+    _apply(r, tmp_path)
+    out = (tmp_path / "m.py").read_text()
+    compile(out, "m.py", "exec")
+
+
+def test_call_rewrite_is_column_accurate_on_non_ascii_lines(tmp_path: Path) -> None:
+    """A call after a non-ASCII string on the same line is rewritten at the right columns."""
+    (tmp_path / "m.py").write_text('x = "中文"; y = Owner.foo(self.r, 1)\n')
+    r = Repro("b", "t").lower_call_sites("foo", "Owner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == 'x = "中文"; y = self.r.foo(1)\n'
+
+
+def test_call_rewrite_survives_a_form_feed_line_start(tmp_path: Path) -> None:
+    """A form feed at a line start must not shift the rewrite onto the wrong line."""
+    (tmp_path / "m.py").write_text("a = 1\n\x0cb = 2\ny = Owner.foo(self.r, 1)\n")
+    r = Repro("b", "t").lower_call_sites("foo", "Owner", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "a = 1\n\x0cb = 2\ny = self.r.foo(1)\n"
+
+
+def test_requalify_call_sites_preserves_redundant_parens_in_kwargs(
+    tmp_path: Path,
+) -> None:
+    """Redundant parentheses around a keyword value survive the requalification."""
+    (tmp_path / "m.py").write_text("y = Old.bar(\n    a=1,\n    b=(2),\n)\n")
+    r = Repro("b", "t").requalify_call_sites("bar", "Old", paths=["m.py"])
+    _apply(r, tmp_path)
+    assert "b=(2)" in (tmp_path / "m.py").read_text()

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_delete_file.py

@@ -0,0 +1,49 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+
+def test_delete_file_removes_emptied_source(tmp_path: Path) -> None:
+    """delete_file removes a source module left empty after its defs relocated."""
+    (tmp_path / "gone.py").write_text("import os\n")
+    r = Repro("b", "t").delete_file("gone.py")
+    _apply(r, tmp_path)
+    assert not (tmp_path / "gone.py").exists()
+
+
+def test_delete_file_refuses_a_file_with_remaining_definitions(tmp_path: Path) -> None:
+    """Deleting a module that still contains defs must fail loudly."""
+    (tmp_path / "live.py").write_text("def still_used():\n    return 42\n")
+    r = Repro("b", "t").delete_file("live.py")
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+    assert (tmp_path / "live.py").exists()
+
+
+def test_delete_file_on_a_missing_path_is_a_no_op(tmp_path: Path) -> None:
+    """Deleting an already-absent file does nothing and raises nothing."""
+    r = Repro("b", "t").delete_file("nope.py")
+    _apply(r, tmp_path)
+    assert not (tmp_path / "nope.py").exists()
+
+
+# --- adversarial audit: extract_function -----------------------------------------

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_extract_function.py

@@ -0,0 +1,190 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- extract_function ----------------------------------------------------------
+
+
+def test_extract_function_relocates_body_and_replaces_with_call(tmp_path: Path) -> None:
+    """An inline block is cut verbatim, re-indented under the new signature, and the call site
+    replaced; the body lands at function-body indent."""
+    (tmp_path / "src.py").write_text(
+        "class Q:\n"
+        "    def run(self, n):\n"
+        "        total = 0\n"
+        "        for i in range(n):\n"
+        "            total += i * i\n"
+        "        return total\n"
+    )
+    (tmp_path / "dst.py").write_text("def existing():\n    return 0\n")
+    body = "        total = 0\n        for i in range(n):\n            total += i * i\n"
+    r = Repro("b", "t").extract_function(
+        "src.py",
+        "dst.py",
+        name="sum_squares",
+        signature="def sum_squares(n):",
+        body=body,
+        body_indent=8,
+        call="        total = sum_squares(n)\n",
+        return_text="    return total\n",
+    )
+    _apply(r, tmp_path)
+    src_out = (tmp_path / "src.py").read_text()
+    assert "        total = sum_squares(n)\n" in src_out
+    assert "for i in range(n)" not in src_out
+    assert (
+        "def sum_squares(n):\n"
+        "    total = 0\n"
+        "    for i in range(n):\n"
+        "        total += i * i\n"
+        "    return total\n"
+    ) in (tmp_path / "dst.py").read_text()
+
+
+def test_extract_function_inserts_before_named_sibling(tmp_path: Path) -> None:
+    """With before=, the new function lands immediately above that sibling at module level."""
+    (tmp_path / "src.py").write_text("x = compute()\n")
+    (tmp_path / "dst.py").write_text(
+        "def a():\n    return 1\n\n\ndef c():\n    return 3\n"
+    )
+    r = Repro("b", "t").extract_function(
+        "src.py",
+        "dst.py",
+        name="b",
+        signature="def b():",
+        body="x = compute()\n",
+        body_indent=0,
+        call="x = b()\n",
+        return_text="    return x\n",
+        before="c",
+    )
+    _apply(r, tmp_path)
+    dst_out = (tmp_path / "dst.py").read_text()
+    assert dst_out.index("def a") < dst_out.index("def b") < dst_out.index("def c")
+    assert "x = b()\n" == (tmp_path / "src.py").read_text()
+
+
+def test_extract_function_asserts_block_not_unique(tmp_path: Path) -> None:
+    """A block that occurs more than once in the source raises, so the cut is unambiguous."""
+    (tmp_path / "src.py").write_text("p = f()\np = f()\n")
+    (tmp_path / "dst.py").write_text("def z():\n    return 0\n")
+    r = Repro("b", "t").extract_function(
+        "src.py",
+        "dst.py",
+        name="g",
+        signature="def g():",
+        body="p = f()\n",
+        body_indent=0,
+        call="p = g()\n",
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+# --- adversarial audit: module extraction ----------------------------------------
+
+
+# --- adversarial audit: extract_function -----------------------------------------
+
+
+def test_extract_function_does_not_pad_blank_lines_in_the_body(tmp_path: Path) -> None:
+    """Interior blank lines of the extracted body stay bare newlines, unpadded."""
+    (tmp_path / "src.py").write_text("        a = 1\n\n        b = 2\n")
+    (tmp_path / "dst.py").write_text("def z():\n    return 0\n")
+    r = Repro("b", "t").extract_function(
+        "src.py",
+        "dst.py",
+        name="g",
+        signature="def g():",
+        body="        a = 1\n\n        b = 2\n",
+        body_indent=8,
+        call="        g()\n",
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == "        g()\n"
+    assert (tmp_path / "dst.py").read_text() == (
+        "def z():\n    return 0\n\ndef g():\n    a = 1\n\n    b = 2\n"
+    )
+
+
+def test_extract_function_does_not_reindent_string_literal_interiors(
+    tmp_path: Path,
+) -> None:
+    """Triple-quoted string interior lines keep their exact bytes through the extraction."""
+    (tmp_path / "src.py").write_text(
+        "TEMPLATE = '''\nliteral line\n'''\nx = TEMPLATE\n"
+    )
+    (tmp_path / "dst.py").write_text("def existing():\n    return 0\n")
+    r = Repro("b", "t").extract_function(
+        "src.py",
+        "dst.py",
+        name="make",
+        signature="def make():",
+        body="TEMPLATE = '''\nliteral line\n'''\nx = TEMPLATE\n",
+        body_indent=0,
+        call="x = make()\n",
+        return_text="    return x\n",
+    )
+    _apply(r, tmp_path)
+    assert "\nliteral line\n" in (tmp_path / "dst.py").read_text()
+
+
+def test_extract_function_rejects_a_mid_line_substring_match(tmp_path: Path) -> None:
+    """A body that only matches mid-line must fail loudly instead of splicing the call."""
+    (tmp_path / "src.py").write_text("value = prefix_total = 0\n")
+    (tmp_path / "dst.py").write_text("def z():\n    return 0\n")
+    r = Repro("b", "t").extract_function(
+        "src.py",
+        "dst.py",
+        name="g",
+        signature="def g():",
+        body="total = 0\n",
+        body_indent=0,
+        call="total = g()\n",
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_extract_function_into_class_indents_body_to_method_depth(
+    tmp_path: Path,
+) -> None:
+    """Extracting into a class must indent the relocated body to method depth."""
+    (tmp_path / "src.py").write_text("val = compute_thing()\n")
+    (tmp_path / "dst.py").write_text(
+        "class H:\n    def last(self):\n        return 0\n"
+    )
+    r = Repro("b", "t").extract_function(
+        "src.py",
+        "dst.py",
+        name="helper",
+        signature="    def helper(self):",
+        body="val = compute_thing()\n",
+        body_indent=0,
+        call="val = h.helper()\n",
+        return_text="        return val\n",
+        into_class="H",
+    )
+    _apply(r, tmp_path)
+    out = (tmp_path / "dst.py").read_text()
+    compile(out, "dst.py", "exec")
+    assert "    def helper(self):\n        val = compute_thing()\n" in out

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_extract_symbols_to_new_module.py

@@ -0,0 +1,199 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- extract_symbols_to_new_module ---------------------------------------------
+
+
+def test_extract_symbols_to_new_module_gathers_scattered_defs(tmp_path: Path) -> None:
+    """Scattered top-level defs are cut from the source and assembled under the authored header
+    in the given order; the source keeps everything else."""
+    (tmp_path / "src.py").write_text(
+        "import os\n"
+        "\n"
+        "\n"
+        "def keep_a():\n"
+        "    return 1\n"
+        "\n"
+        "\n"
+        "def moved_b():\n"
+        "    return 2\n"
+        "\n"
+        "\n"
+        "def keep_c():\n"
+        "    return 3\n"
+        "\n"
+        "\n"
+        "def moved_a():\n"
+        "    return 4\n"
+    )
+    header = (
+        "from __future__ import annotations\n"
+        "\n"
+        "import logging\n"
+        "\n"
+        "logger = logging.getLogger(__name__)\n"
+    )
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py",
+        "new.py",
+        symbols=["moved_b", "moved_a"],
+        header=header,
+        order=["moved_a", "moved_b"],
+    )
+    _apply(r, tmp_path)
+    src_out = (tmp_path / "src.py").read_text()
+    assert "def moved_a" not in src_out and "def moved_b" not in src_out
+    assert "def keep_a" in src_out and "def keep_c" in src_out
+    new_out = (tmp_path / "new.py").read_text()
+    assert new_out.startswith("from __future__ import annotations\n")
+    assert "logger = logging.getLogger(__name__)" in new_out
+    assert new_out.index("def moved_a") < new_out.index("def moved_b")
+    assert "    return 4\n" in new_out and "    return 2\n" in new_out
+
+
+def test_extract_symbols_to_new_module_asserts_order_permutes_symbols(
+    tmp_path: Path,
+) -> None:
+    """An order that is not a permutation of the symbols raises, so a wrong recipe fails."""
+    (tmp_path / "src.py").write_text(
+        "def a():\n    return 1\n\n\ndef b():\n    return 2\n"
+    )
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py", "n.py", symbols=["a", "b"], header="", order=["a"]
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_extract_symbols_to_new_module_asserts_when_symbol_absent(
+    tmp_path: Path,
+) -> None:
+    """A symbol that is not a top-level def/class in the source raises."""
+    (tmp_path / "src.py").write_text("def a():\n    return 1\n")
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py", "n.py", symbols=["a", "missing"], header="", order=["a", "missing"]
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_extract_symbols_to_new_module_drops_relocated_assigns(tmp_path: Path) -> None:
+    """A module-level constant that moved into the new module's header is deleted from the
+    source (its copy lives in the authored header); a kept assignment stays."""
+    (tmp_path / "src.py").write_text(
+        "import os\n"
+        "\n"
+        "_FLAG = os.cpu_count()\n"
+        "stay = 1\n"
+        "\n"
+        "\n"
+        "def moved():\n"
+        "    return _FLAG\n"
+    )
+    header = (
+        "from __future__ import annotations\n"
+        "\n"
+        "import os\n"
+        "\n"
+        "_FLAG = os.cpu_count()\n"
+    )
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py",
+        "new.py",
+        symbols=["moved"],
+        header=header,
+        order=["moved"],
+        drop_assigns=["_FLAG"],
+    )
+    _apply(r, tmp_path)
+    src_out = (tmp_path / "src.py").read_text()
+    assert "_FLAG = os.cpu_count()" not in src_out
+    assert "stay = 1" in src_out
+    assert "_FLAG = os.cpu_count()" in (tmp_path / "new.py").read_text()
+
+
+def test_extract_symbols_to_new_module_asserts_unknown_drop_assign(
+    tmp_path: Path,
+) -> None:
+    """A drop_assigns name that is not assigned at module level in the source raises."""
+    (tmp_path / "src.py").write_text("X = 1\n\n\ndef m():\n    return X\n")
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py", "n.py", symbols=["m"], header="", order=["m"], drop_assigns=["Y"]
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+# --- extract_function ----------------------------------------------------------
+
+
+def test_extract_symbols_to_new_module_joins_blocks_with_two_blank_lines(
+    tmp_path: Path,
+) -> None:
+    """Relocated blocks are joined with exactly two blank lines (the formatter's spacing)."""
+    (tmp_path / "src.py").write_text(
+        "def moved_a():\n    return 1\n\n\n\n\ndef moved_b():\n    return 2\n"
+    )
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py",
+        "new.py",
+        symbols=["moved_a", "moved_b"],
+        header="",
+        order=["moved_a", "moved_b"],
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "new.py").read_text() == (
+        "def moved_a():\n    return 1\n\n\ndef moved_b():\n    return 2\n"
+    )
+
+
+def test_extract_symbols_to_new_module_leaves_a_comment_above_a_moved_def(
+    tmp_path: Path,
+) -> None:
+    """A section comment directly above a moved def stays behind in the source."""
+    (tmp_path / "src.py").write_text(
+        "x = 1\n\n\n# --- movers ---\ndef moved():\n    return 2\n"
+    )
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py", "new.py", symbols=["moved"], header="", order=["moved"]
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == "x = 1\n\n\n# --- movers ---\n"
+    assert (tmp_path / "new.py").read_text() == "def moved():\n    return 2\n"
+
+
+def test_extract_symbols_drop_assigns_preserves_other_targets_of_chained_assign(
+    tmp_path: Path,
+) -> None:
+    """Dropping A from `A = B = 1` must not delete B's binding from the source."""
+    (tmp_path / "src.py").write_text("A = B = 1\n\n\ndef moved():\n    return A\n")
+    r = Repro("b", "t").extract_symbols_to_new_module(
+        "src.py",
+        "new.py",
+        symbols=["moved"],
+        header="A = 1\n",
+        order=["moved"],
+        drop_assigns=["A"],
+    )
+    _apply(r, tmp_path)
+    assert "B" in (tmp_path / "src.py").read_text()

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_extract_to_new_module.py

@@ -0,0 +1,126 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- extract_to_new_module -----------------------------------------------------
+
+
+def test_extract_to_new_module_cuts_trailing_block(tmp_path: Path) -> None:
+    """Cuts the trailing scaffolding+def block into a new file, prepending the future import."""
+    (tmp_path / "src.py").write_text(
+        "class M:\n"
+        "    def keep(self):\n"
+        "        return 1\n"
+        "\n"
+        "\n"
+        "import logging\n"
+        "\n"
+        "logger = logging.getLogger(__name__)\n"
+        "\n"
+        "\n"
+        "def foo(x):\n"
+        "    return x + 1\n"
+    )
+    r = Repro("b", "t").extract_to_new_module(
+        "src.py", "new.py", symbols=["foo"], future_import=True
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == (
+        "class M:\n    def keep(self):\n        return 1\n\n\n"
+    )
+    assert (tmp_path / "new.py").read_text() == (
+        "from __future__ import annotations\n"
+        "import logging\n"
+        "\n"
+        "logger = logging.getLogger(__name__)\n"
+        "\n"
+        "\n"
+        "def foo(x):\n"
+        "    return x + 1\n"
+    )
+
+
+def test_extract_to_new_module_carries_a_trailing_class(tmp_path: Path) -> None:
+    """A class in the staged tail (not just a def) travels with the cut block."""
+    (tmp_path / "src.py").write_text(
+        "class M:\n"
+        "    pass\n"
+        "\n"
+        "\n"
+        "from dataclasses import dataclass\n"
+        "\n"
+        "\n"
+        "@dataclass\n"
+        "class Cfg:\n"
+        "    x: int\n"
+        "\n"
+        "\n"
+        "def foo():\n"
+        "    return Cfg(1)\n"
+    )
+    r = Repro("b", "t").extract_to_new_module(
+        "src.py", "new.py", symbols=["Cfg", "foo"], future_import=False
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == "class M:\n    pass\n\n\n"
+    assert "class Cfg:" in (tmp_path / "new.py").read_text()
+    assert "def foo():" in (tmp_path / "new.py").read_text()
+
+
+# --- extract_symbols_to_new_module ---------------------------------------------
+
+
+# --- adversarial audit: module extraction ----------------------------------------
+
+
+def test_extract_to_new_module_asserts_when_symbol_not_in_the_tail(
+    tmp_path: Path,
+) -> None:
+    """A wanted symbol above a non-scaffolding statement is not in the tail and raises."""
+    (tmp_path / "src.py").write_text(
+        "def wanted():\n    return 1\n\n\nprint('side effect')\n"
+    )
+    r = Repro("b", "t").extract_to_new_module("src.py", "n.py", symbols=["wanted"])
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_extract_to_new_module_refuses_a_trailing_main_guard(tmp_path: Path) -> None:
+    """A trailing __main__ guard is executable code, not scaffolding: the tail cut raises."""
+    (tmp_path / "src.py").write_text(
+        "class Keep:\n"
+        "    pass\n"
+        "\n"
+        "\n"
+        "def foo():\n"
+        "    return 1\n"
+        "\n"
+        "\n"
+        'if __name__ == "__main__":\n'
+        "    foo()\n"
+    )
+    r = Repro("b", "t").extract_to_new_module(
+        "src.py", "new.py", symbols=["foo"], future_import=False
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+    assert "__main__" in (tmp_path / "src.py").read_text()

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_infra_helpers.py

@@ -0,0 +1,153 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- exec_command --------------------------------------------------------------
+
+
+def test_exec_command_returns_stripped_stdout_on_success() -> None:
+    """A successful command returns its stdout with surrounding whitespace stripped."""
+    assert exec_command("echo hello") == "hello"
+
+
+def test_exec_command_respects_cwd(tmp_path: Path) -> None:
+    """The command runs in the supplied working directory."""
+    sub = tmp_path / "workdir"
+    sub.mkdir()
+    assert exec_command("pwd", cwd=str(sub)) == str(sub.resolve())
+
+
+def test_exec_command_check_true_raises_on_failure() -> None:
+    """With check=True a non-zero exit status raises RuntimeError with the command."""
+    with pytest.raises(RuntimeError, match="exit 7"):
+        exec_command("exit 7", check=True)
+
+
+def test_exec_command_check_false_returns_stdout_without_exiting() -> None:
+    """With check=False a failing command returns its stdout and does not exit."""
+    assert exec_command("echo partial; exit 3", check=False) == "partial"
+
+
+# --- git_add_and_commit --------------------------------------------------------
+
+
+# --- git_add_and_commit --------------------------------------------------------
+
+
+def test_git_add_and_commit_stages_and_commits(repo: Path) -> None:
+    """It stages every change in the cwd and records a commit with the message."""
+    _write(repo, **{"file.txt": "content\n"})
+    git_add_and_commit("add file", cwd=str(repo))
+    assert _git(repo, "log", "-1", "--format=%s") == "add file"
+    assert _git(repo, "status", "--porcelain") == ""
+
+
+@pytest.mark.parametrize(
+    "message",
+    [
+        "subject with spaces",
+        "has 'single' and \"double\" quotes",
+        "shell $HOME && rm -rf / ; metacharacters",
+        "trailing parens (a, b) and pipe | semicolon ;",
+    ],
+)
+def test_git_add_and_commit_message_round_trips_with_metacharacters(
+    repo: Path, message: str
+) -> None:
+    """Messages with shell metacharacters are quoted safely and survive verbatim."""
+    _write(repo, **{"file.txt": "content\n"})
+    git_add_and_commit(message, cwd=str(repo))
+    assert _git(repo, "log", "-1", "--format=%B") == message
+
+
+# --- dedent --------------------------------------------------------------------
+
+
+# --- dedent --------------------------------------------------------------------
+
+
+def test_dedent_with_zero_leaves_text_unchanged() -> None:
+    """Dedenting by zero spaces returns the text untouched."""
+    text = "  indented\nplain\n"
+    assert dedent(text, 0) == text
+
+
+def test_dedent_removes_exactly_n_leading_spaces() -> None:
+    """Exactly n leading spaces are removed from each qualifying line."""
+    assert dedent("    four\n        eight\n", 4) == "four\n    eight\n"
+
+
+def test_dedent_leaves_lines_with_fewer_than_n_spaces_unchanged() -> None:
+    """A line with fewer than n leading spaces is not modified at all."""
+    assert dedent("    four\n  two\nzero\n", 4) == "four\n  two\nzero\n"
+
+
+def test_dedent_does_not_strip_tabs() -> None:
+    """Tab characters are never treated as the spaces dedent removes."""
+    assert dedent("\t\ttabbed\n", 2) == "\t\ttabbed\n"
+
+
+def test_dedent_preserves_blank_lines_and_trailing_newline() -> None:
+    """Blank lines and a final newline are preserved across line boundaries."""
+    assert dedent("    a\n\n    b\n", 4) == "a\n\nb\n"
+
+
+def test_dedent_preserves_absence_of_trailing_newline() -> None:
+    """A text without a trailing newline keeps it absent after dedenting."""
+    assert dedent("    a\n    b", 4) == "a\nb"
+
+
+# --- span / call helpers -------------------------------------------------------
+
+
+# --- span / call helpers -------------------------------------------------------
+
+
+def test_replace_span_single_line() -> None:
+    """A span within one line is replaced in place."""
+    assert _replace_span("ab cd ef\n", 1, 3, 1, 5, "XY") == "ab XY ef\n"
+
+
+def test_replace_span_across_lines() -> None:
+    """A span crossing lines collapses to the replacement between the kept prefix/suffix."""
+    text = "a = foo(\n    x,\n) + 1\n"
+    assert _replace_span(text, 1, 4, 3, 1, "bar()") == "a = bar() + 1\n"
+
+
+def test_slice_span_returns_the_overwritten_text() -> None:
+    """_slice_span returns exactly the region _replace_span would overwrite."""
+    text = "a = foo(\n    x,\n) + 1\n"
+    assert _slice_span(text, 1, 4, 3, 1) == "foo(\n    x,\n)"
+
+
+def test_find_def_span_includes_decorators() -> None:
+    """A def's span starts at its first decorator and ends at its last body line."""
+    src = "class C:\n    @staticmethod\n    def foo(self):\n        return 1\n"
+    node = _find_def(rr.ast.parse(src), "foo")
+    assert node is not None and _def_span(node) == (2, 4)
+
+
+def test_find_class_returns_named_class_or_none() -> None:
+    """_find_class locates a class by name and returns None when absent."""
+    tree = rr.ast.parse("class A:\n    pass\nclass B:\n    pass\n")
+    assert _find_class(tree, "B").name == "B"
+    assert _find_class(tree, "Z") is None

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_symbol.py

@@ -0,0 +1,360 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+
+def test_move_symbol_drops_self_annotation_into_class(tmp_path: Path) -> None:
+    """Moving a `def foo(self: Target)` into Target drops the now-redundant annotation."""
+    (tmp_path / "src.py").write_text(
+        "class M:\n"
+        "    @staticmethod\n"
+        "    def foo(self: Target, x):\n"
+        "        return self.y + x\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class Target:\n    def keep(self):\n        return 1\n"
+    )
+    r = Repro("b", "t").move_symbol(
+        "foo",
+        src="src.py",
+        dst="dst.py",
+        into_class="Target",
+        dedent=0,
+        drop_self_annotation=True,
+    )
+    _apply(r, tmp_path)
+    text = (tmp_path / "dst.py").read_text()
+    assert "def foo(self, x):" in text
+    assert "self: Target" not in text
+
+
+# --- adversarial audit: import primitives ----------------------------------------
+
+
+# --- move_symbol ---------------------------------------------------------------
+
+
+def test_move_symbol_into_class_drops_decorator_and_appends(tmp_path: Path) -> None:
+    """The def leaves the source, its @staticmethod is dropped, and it lands at the end of
+    the destination class with its body verbatim."""
+    (tmp_path / "src.py").write_text(
+        "class Old:\n"
+        "    @staticmethod\n"
+        "    def foo(x):\n"
+        "        return x + 1\n"
+        "\n"
+        "    def keep(self):\n"
+        "        return 0\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class New:\n    def existing(self):\n        return 1\n"
+    )
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class="New")
+    _apply(r, tmp_path)
+    src_out = (tmp_path / "src.py").read_text()
+    dst_out = (tmp_path / "dst.py").read_text()
+    assert "def foo" not in src_out and "def keep" in src_out
+    assert "@staticmethod" not in dst_out
+    assert dst_out.index("def existing") < dst_out.index("def foo")
+    assert "        return x + 1\n" in dst_out
+
+
+def test_move_symbol_to_module_level_with_dedent(tmp_path: Path) -> None:
+    """With into_class=None and a dedent, the def lands at module level, dedented."""
+    (tmp_path / "src.py").write_text(
+        "class Old:\n    @staticmethod\n    def helper(x):\n        return x * 2\n"
+    )
+    (tmp_path / "dst.py").write_text("import os\n")
+    r = Repro("b", "t").move_symbol(
+        "helper", src="src.py", dst="dst.py", into_class=None, dedent=4
+    )
+    _apply(r, tmp_path)
+    assert "def helper(x):\n    return x * 2\n" in (tmp_path / "dst.py").read_text()
+    assert "def helper" not in (tmp_path / "src.py").read_text()
+
+
+def test_move_symbol_before_inserts_above_named_sibling(tmp_path: Path) -> None:
+    """With before=, the relocated def lands immediately above that sibling, not at the end."""
+    (tmp_path / "src.py").write_text(
+        "class Old:\n    @staticmethod\n    def moved(self):\n        return 1\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class New:\n"
+        "    def first(self):\n        return 0\n"
+        "\n"
+        "    def last(self):\n        return 2\n"
+    )
+    r = Repro("b", "t").move_symbol(
+        "moved", src="src.py", dst="dst.py", into_class="New", before="last"
+    )
+    _apply(r, tmp_path)
+    dst_out = (tmp_path / "dst.py").read_text()
+    assert (
+        dst_out.index("def first")
+        < dst_out.index("def moved")
+        < dst_out.index("def last")
+    )
+
+
+# --- adversarial audit: move_symbol edge cases -----------------------------------
+
+
+def test_move_symbol_moves_an_async_def_verbatim(tmp_path: Path) -> None:
+    """An async def relocates with its `async` keyword and body byte-identical."""
+    (tmp_path / "src.py").write_text(
+        "class Old:\n"
+        "    async def foo(self):\n"
+        "        return 1\n"
+        "\n"
+        "    def keep(self):\n"
+        "        return 0\n"
+    )
+    (tmp_path / "dst.py").write_text("class New:\n    def e(self):\n        return 1\n")
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class="New")
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == (
+        "class Old:\n\n    def keep(self):\n        return 0\n"
+    )
+    assert (tmp_path / "dst.py").read_text() == (
+        "class New:\n"
+        "    def e(self):\n"
+        "        return 1\n"
+        "\n"
+        "    async def foo(self):\n"
+        "        return 1\n"
+    )
+
+
+def test_move_symbol_moves_a_def_within_the_same_file(tmp_path: Path) -> None:
+    """With src == dst the def is cut and re-inserted above its sibling in one file."""
+    (tmp_path / "m.py").write_text(
+        "def a():\n    return 1\n\n\ndef b():\n    return 2\n"
+    )
+    r = Repro("b", "t").move_symbol(
+        "b", src="m.py", dst="m.py", into_class=None, before="a"
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == (
+        "def b():\n    return 2\n\ndef a():\n    return 1\n\n\n"
+    )
+
+
+def test_move_symbol_prefers_a_module_level_def_over_an_earlier_class_method(
+    tmp_path: Path,
+) -> None:
+    """When a class method and a module-level def share a name, the module-level def moves."""
+    (tmp_path / "src.py").write_text(
+        "class C:\n"
+        "    def foo(self):\n"
+        "        return 'method'\n"
+        "\n"
+        "\n"
+        "def foo():\n"
+        "    return 'module'\n"
+    )
+    (tmp_path / "dst.py").write_text("x = 1\n")
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class=None)
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == (
+        "class C:\n    def foo(self):\n        return 'method'\n\n\n"
+    )
+    assert (tmp_path / "dst.py").read_text() == (
+        "x = 1\n\ndef foo():\n    return 'module'\n"
+    )
+
+
+def test_move_symbol_keeps_a_real_decorator_while_dropping_classmethod(
+    tmp_path: Path,
+) -> None:
+    """@classmethod is shed on the move but any other decorator travels verbatim."""
+    (tmp_path / "src.py").write_text(
+        "import functools\n"
+        "\n"
+        "\n"
+        "class Old:\n"
+        "    @classmethod\n"
+        "    @functools.lru_cache(maxsize=None)\n"
+        "    def foo(cls, x):\n"
+        "        return x + 1\n"
+    )
+    (tmp_path / "dst.py").write_text("def z():\n    return 0\n")
+    r = Repro("b", "t").move_symbol(
+        "foo", src="src.py", dst="dst.py", into_class=None, dedent=4
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "dst.py").read_text() == (
+        "def z():\n"
+        "    return 0\n"
+        "\n"
+        "@functools.lru_cache(maxsize=None)\n"
+        "def foo(cls, x):\n"
+        "    return x + 1\n"
+    )
+
+
+def test_move_symbol_leaves_a_comment_above_the_def_in_the_source(
+    tmp_path: Path,
+) -> None:
+    """A comment above the def is not part of its span, so it stays behind in the source."""
+    (tmp_path / "src.py").write_text(
+        "# explains foo\ndef foo():\n    return 1\n\n\ndef keep():\n    return 2\n"
+    )
+    (tmp_path / "dst.py").write_text("x = 1\n")
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class=None)
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == (
+        "# explains foo\n\n\ndef keep():\n    return 2\n"
+    )
+    assert (tmp_path / "dst.py").read_text() == "x = 1\n\ndef foo():\n    return 1\n"
+
+
+def test_move_symbol_without_trailing_newlines_keeps_moved_bytes(
+    tmp_path: Path,
+) -> None:
+    """Files lacking a final newline lose no bytes of the moved def or the remainder."""
+    (tmp_path / "src.py").write_text(
+        "def keep():\n    return 0\n\n\ndef foo():\n    return 1"
+    )
+    (tmp_path / "dst.py").write_text("x = 1")
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class=None)
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == "def keep():\n    return 0\n\n\n"
+    assert (tmp_path / "dst.py").read_text() == "x = 1\ndef foo():\n    return 1"
+
+
+def test_move_symbol_dedent_leaves_string_literal_interior_lines(
+    tmp_path: Path,
+) -> None:
+    """Dedent only strips lines with exactly n leading spaces, so string interiors survive."""
+    (tmp_path / "src.py").write_text(
+        "class Old:\n"
+        "    class Deep:\n"
+        "        def foo(self):\n"
+        "            s = '''raw\n"
+        "    partial\n"
+        "'''\n"
+        "            return s\n"
+    )
+    (tmp_path / "dst.py").write_text("import os\n")
+    r = Repro("b", "t").move_symbol(
+        "foo", src="src.py", dst="dst.py", into_class=None, dedent=8
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "dst.py").read_text() == (
+        "import os\n"
+        "\n"
+        "def foo(self):\n"
+        "    s = '''raw\n"
+        "    partial\n"
+        "'''\n"
+        "    return s\n"
+    )
+
+
+def test_move_symbol_asserts_when_destination_class_missing(tmp_path: Path) -> None:
+    """Naming an into_class absent from the destination fails loudly."""
+    (tmp_path / "src.py").write_text("def foo():\n    return 1\n")
+    (tmp_path / "dst.py").write_text("x = 1\n")
+    r = Repro("b", "t").move_symbol(
+        "foo", src="src.py", dst="dst.py", into_class="Nope"
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_move_symbol_preserves_staticmethod_inside_moved_body(tmp_path: Path) -> None:
+    """A @staticmethod on a nested def inside the moved body must survive the move."""
+    (tmp_path / "src.py").write_text(
+        "class Old:\n"
+        "    @staticmethod\n"
+        "    def foo(x):\n"
+        "        class Inner:\n"
+        "            @staticmethod\n"
+        "            def helper(y):\n"
+        "                return y\n"
+        "        return Inner.helper(x)\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class New:\n    def keep(self):\n        return 0\n"
+    )
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class="New")
+    _apply(r, tmp_path)
+    dst_out = (tmp_path / "dst.py").read_text()
+    assert "            @staticmethod\n            def helper(y):\n" in dst_out
+
+
+def test_move_symbol_rejects_ambiguous_duplicate_names(tmp_path: Path) -> None:
+    """Two same-named defs at equal depth must raise instead of silently picking one."""
+    (tmp_path / "src.py").write_text(
+        "class A:\n"
+        "    def foo(self):\n"
+        "        return 'A'\n"
+        "\n"
+        "class B:\n"
+        "    def foo(self):\n"
+        "        return 'B'\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class New:\n    def keep(self):\n        return 0\n"
+    )
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class="New")
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_move_symbol_asserts_when_before_sibling_missing(tmp_path: Path) -> None:
+    """A before= anchor absent from the destination must raise, not fall back to append."""
+    (tmp_path / "src.py").write_text("def moved():\n    return 1\n")
+    (tmp_path / "dst.py").write_text("def z():\n    return 0\n")
+    r = Repro("b", "t").move_symbol(
+        "moved", src="src.py", dst="dst.py", into_class=None, before="NO_SUCH_DEF"
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_move_symbol_preserves_crlf_line_endings(tmp_path: Path) -> None:
+    """Moving a def in a CRLF file must keep every line ending CRLF."""
+    (tmp_path / "src.py").write_bytes(
+        b"class Old:\r\n    def foo(self):\r\n        return 1\r\n"
+    )
+    (tmp_path / "dst.py").write_bytes(
+        b"class New:\r\n    def keep(self):\r\n        return 0\r\n"
+    )
+    r = Repro("b", "t").move_symbol("foo", src="src.py", dst="dst.py", into_class="New")
+    _apply(r, tmp_path)
+    dst_bytes = (tmp_path / "dst.py").read_bytes()
+    assert dst_bytes.count(b"\n") == dst_bytes.count(b"\r\n")
+
+
+def test_move_symbol_negative_dedent_indents_into_the_class(tmp_path: Path) -> None:
+    """Moving a module-level def into a class with dedent=-4 must indent it as a method."""
+    (tmp_path / "src.py").write_text("def helper(x):\n    return x\n")
+    (tmp_path / "dst.py").write_text("class New:\n    def e(self):\n        return 0\n")
+    r = Repro("b", "t").move_symbol(
+        "helper", src="src.py", dst="dst.py", into_class="New", dedent=-4
+    )
+    _apply(r, tmp_path)
+    assert "    def helper(x):\n        return x\n" in (tmp_path / "dst.py").read_text()
+
+
+# --- adversarial audit: leave_delegate stubs -------------------------------------

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_symbol_delegate.py

@@ -0,0 +1,195 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+
+def test_move_symbol_leave_delegate_keeps_forwarding_stub(tmp_path: Path) -> None:
+    """With leave_delegate, the source keeps a forwarding stub through the named field and the
+    destination gets the full method body."""
+    (tmp_path / "src.py").write_text(
+        "class Mixin:\n"
+        "    def compute(self, n: int) -> int:\n"
+        "        return n + self.cfg.base\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class Cfg:\n    def existing(self):\n        return 0\n"
+    )
+    r = Repro("b", "t").move_symbol(
+        "compute",
+        src="src.py",
+        dst="dst.py",
+        into_class="Cfg",
+        leave_delegate="cfg",
+    )
+    _apply(r, tmp_path)
+    src_out = (tmp_path / "src.py").read_text()
+    dst_out = (tmp_path / "dst.py").read_text()
+    assert "def compute(self, n: int) -> int:" in src_out
+    assert "return self.cfg.compute(n)" in src_out
+    assert "return n + self.cfg.base" not in src_out
+    assert "return n + self.cfg.base" in dst_out
+
+
+def test_move_symbol_leave_delegate_does_not_absorb_leading_comments(
+    tmp_path: Path,
+) -> None:
+    """A leading comment before the first statement is not an AST node, so it must not be
+    pulled into the forwarding stub -- the delegate is just the header plus the return.
+    """
+    (tmp_path / "src.py").write_text(
+        "class Mixin:\n"
+        "    def compute(self, n: int) -> int:\n"
+        "        # explain the maths\n"
+        "        # second comment line\n"
+        "        return n + self.cfg.base\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class Cfg:\n    def existing(self):\n        return 0\n"
+    )
+    r = Repro("b", "t").move_symbol(
+        "compute", src="src.py", dst="dst.py", into_class="Cfg", leave_delegate="cfg"
+    )
+    _apply(r, tmp_path)
+    src_out = (tmp_path / "src.py").read_text()
+    dst_out = (tmp_path / "dst.py").read_text()
+    assert "# explain the maths" not in src_out
+    assert (
+        src_out == "class Mixin:\n"
+        "    def compute(self, n: int) -> int:\n"
+        "        return self.cfg.compute(n)\n"
+    )
+    assert "# explain the maths" in dst_out
+
+
+# --- adversarial audit: move_symbol edge cases -----------------------------------
+
+
+# --- adversarial audit: leave_delegate stubs -------------------------------------
+
+
+def test_move_symbol_leave_delegate_keeps_a_multiline_signature_verbatim(
+    tmp_path: Path,
+) -> None:
+    """A multi-line header is carried into the stub byte-for-byte via the bracket scan."""
+    (tmp_path / "src.py").write_text(
+        "class Mixin:\n"
+        "    def compute(\n"
+        "        self,\n"
+        "        n: int,\n"
+        "        *,\n"
+        "        scale: float = 1.0,\n"
+        "    ) -> int:\n"
+        "        return int(n * scale) + self.cfg.base\n"
+    )
+    (tmp_path / "dst.py").write_text("class Cfg:\n    def e(self):\n        return 0\n")
+    r = Repro("b", "t").move_symbol(
+        "compute", src="src.py", dst="dst.py", into_class="Cfg", leave_delegate="cfg"
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == (
+        "class Mixin:\n"
+        "    def compute(\n"
+        "        self,\n"
+        "        n: int,\n"
+        "        *,\n"
+        "        scale: float = 1.0,\n"
+        "    ) -> int:\n"
+        "        return self.cfg.compute(n, scale=scale)\n"
+    )
+    assert (tmp_path / "dst.py").read_text() == (
+        "class Cfg:\n"
+        "    def e(self):\n"
+        "        return 0\n"
+        "\n"
+        "    def compute(\n"
+        "        self,\n"
+        "        n: int,\n"
+        "        *,\n"
+        "        scale: float = 1.0,\n"
+        "    ) -> int:\n"
+        "        return int(n * scale) + self.cfg.base\n"
+    )
+
+
+def test_move_symbol_leave_delegate_forwards_posonly_vararg_kwonly_kwargs(
+    tmp_path: Path,
+) -> None:
+    """Every parameter kind is forwarded correctly in the delegate's return call."""
+    (tmp_path / "src.py").write_text(
+        "class Mixin:\n"
+        "    def compute(self, a, /, b, *args, c, d=3, **kw):\n"
+        "        return a\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class Cfg:\n    def keep(self):\n        return 0\n"
+    )
+    r = Repro("b", "t").move_symbol(
+        "compute", src="src.py", dst="dst.py", into_class="Cfg", leave_delegate="cfg"
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "src.py").read_text() == (
+        "class Mixin:\n"
+        "    def compute(self, a, /, b, *args, c, d=3, **kw):\n"
+        "        return self.cfg.compute(a, b, *args, c=c, d=d, **kw)\n"
+    )
+
+
+def test_move_symbol_leave_delegate_survives_paren_in_string_default(
+    tmp_path: Path,
+) -> None:
+    """A string default containing '(' must not break the delegate's header scan."""
+    (tmp_path / "src.py").write_text(
+        "class Mixin:\n"
+        "    def compute(\n"
+        "        self,\n"
+        '        sep: str = "(",\n'
+        "        n: int = 0,\n"
+        "    ) -> int:\n"
+        "        return n + self.cfg.base\n"
+    )
+    (tmp_path / "dst.py").write_text(
+        "class Cfg:\n    def keep(self):\n        return 0\n"
+    )
+    r = Repro("b", "t").move_symbol(
+        "compute", src="src.py", dst="dst.py", into_class="Cfg", leave_delegate="cfg"
+    )
+    _apply(r, tmp_path)
+    src_out = (tmp_path / "src.py").read_text()
+    compile(src_out, "src.py", "exec")
+    assert "return self.cfg.compute(sep, n)" in src_out
+
+
+def test_move_symbol_async_leave_delegate_awaits_the_forwarded_call(
+    tmp_path: Path,
+) -> None:
+    """An async method's delegate stub must await the forwarded coroutine."""
+    (tmp_path / "src.py").write_text(
+        "class Mixin:\n"
+        "    async def compute(self, n):\n"
+        "        return n + self.cfg.base\n"
+    )
+    (tmp_path / "dst.py").write_text("class Cfg:\n    def e(self):\n        return 0\n")
+    r = Repro("b", "t").move_symbol(
+        "compute", src="src.py", dst="dst.py", into_class="Cfg", leave_delegate="cfg"
+    )
+    _apply(r, tmp_path)
+    assert "return await self.cfg.compute(n)" in (tmp_path / "src.py").read_text()

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_remove_import.py

@@ -0,0 +1,126 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- remove_import -------------------------------------------------------------
+
+
+def test_remove_import_scoped_leaves_module_level_same_text(tmp_path: Path) -> None:
+    """Scoped to a function, it removes the local import but not a same-text module-level
+    one (e.g. a TYPE_CHECKING guard), and drops the import's trailing blank line."""
+    (tmp_path / "m.py").write_text(
+        "from typing import TYPE_CHECKING\n"
+        "\n"
+        "if TYPE_CHECKING:\n"
+        "    from pkg.mod import Thing\n"
+        "\n"
+        "def caller(self):\n"
+        "    from pkg.mod import Thing\n"
+        "\n"
+        "    return Thing.go(self.x)\n"
+    )
+    r = Repro("b", "t").remove_import(
+        "m.py", "from pkg.mod import Thing", in_function="caller"
+    )
+    _apply(r, tmp_path)
+    out = (tmp_path / "m.py").read_text()
+    assert out.count("from pkg.mod import Thing") == 1
+    assert "if TYPE_CHECKING:\n    from pkg.mod import Thing" in out
+    assert "def caller(self):\n    return Thing.go(self.x)\n" in out
+
+
+def test_remove_import_removes_every_occurrence_in_scope(tmp_path: Path) -> None:
+    """All matching local imports in the function are removed, not just the first."""
+    (tmp_path / "m.py").write_text(
+        "def caller(self):\n"
+        "    from pkg import M\n"
+        "\n"
+        "    M.a(self.x)\n"
+        "    if cond:\n"
+        "        from pkg import M\n"
+        "\n"
+        "        M.b(self.y)\n"
+    )
+    r = Repro("b", "t").remove_import("m.py", "from pkg import M", in_function="caller")
+    _apply(r, tmp_path)
+    assert "from pkg import M" not in (tmp_path / "m.py").read_text()
+
+
+# --- remove_imported_name ------------------------------------------------------
+
+
+# --- adversarial audit: import primitives ----------------------------------------
+
+
+def test_remove_import_unscoped_removes_module_level_import_and_blank(
+    tmp_path: Path,
+) -> None:
+    """Without in_function the matching module-level import and its trailing blank go."""
+    (tmp_path / "m.py").write_text("import os\nfrom pkg import Thing\n\nx = Thing\n")
+    r = Repro("b", "t").remove_import("m.py", "from pkg import Thing")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "import os\nx = Thing\n"
+
+
+def test_remove_import_keeps_a_code_line_directly_after_the_import(
+    tmp_path: Path,
+) -> None:
+    """Only a blank line after the import is absorbed; a code line stays untouched."""
+    (tmp_path / "m.py").write_text("import os\nx = 1\n")
+    r = Repro("b", "t").remove_import("m.py", "import os")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "x = 1\n"
+
+
+def test_remove_import_asserts_when_text_absent(tmp_path: Path) -> None:
+    """Removing an import text that matches nothing fails loudly."""
+    (tmp_path / "m.py").write_text("import os\n")
+    r = Repro("b", "t").remove_import("m.py", "from pkg import Q")
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_remove_import_asserts_when_scope_function_missing(tmp_path: Path) -> None:
+    """Scoping to a function that does not exist fails loudly."""
+    (tmp_path / "m.py").write_text("def f():\n    import os\n")
+    r = Repro("b", "t").remove_import("m.py", "import os", in_function="nope")
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_remove_import_leaves_other_statements_on_a_semicolon_line(
+    tmp_path: Path,
+) -> None:
+    """Removing 'import os' from a semicolon-joined line must keep 'import sys'."""
+    (tmp_path / "m.py").write_text("import os; import sys\nprint(sys.path)\n")
+    r = Repro("b", "t").remove_import("m.py", "import os")
+    _apply(r, tmp_path)
+    out = (tmp_path / "m.py").read_text()
+    assert "import sys" in out and "print(sys.path)" in out
+
+
+def test_remove_import_does_not_overmatch_a_submodule_import(tmp_path: Path) -> None:
+    """Removing 'import os' must not also remove 'import os.path'."""
+    (tmp_path / "m.py").write_text("import os\nimport os.path\nprint(os.path.sep)\n")
+    r = Repro("b", "t").remove_import("m.py", "import os")
+    _apply(r, tmp_path)
+    assert "import os.path\n" in (tmp_path / "m.py").read_text()

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_remove_imported_name.py

@@ -0,0 +1,113 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- remove_imported_name ------------------------------------------------------
+
+
+def test_remove_imported_name_drops_one_name_from_a_multi_name_import(
+    tmp_path: Path,
+) -> None:
+    """One name is dropped from a `from m import a, b, c`; the others stay on the line."""
+    (tmp_path / "m.py").write_text("from pkg import a, moved, b\n\nx = a + b\n")
+    r = Repro("b", "t").remove_imported_name("m.py", module="pkg", name="moved")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "from pkg import a, b\n\nx = a + b\n"
+
+
+def test_remove_imported_name_drops_whole_statement_when_sole_name(
+    tmp_path: Path,
+) -> None:
+    """Dropping the only name removes the whole `from` statement."""
+    (tmp_path / "m.py").write_text("from pkg import moved\nimport os\n\nx = 1\n")
+    r = Repro("b", "t").remove_imported_name("m.py", module="pkg", name="moved")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "import os\n\nx = 1\n"
+
+
+def test_remove_imported_name_drops_a_plain_import_with_module_none(
+    tmp_path: Path,
+) -> None:
+    """With module=None a plain `import name` statement is removed."""
+    (tmp_path / "m.py").write_text("import gc\nimport os\n\nx = 1\n")
+    r = Repro("b", "t").remove_imported_name("m.py", module=None, name="gc")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "import os\n\nx = 1\n"
+
+
+def test_remove_imported_name_matches_an_asname(tmp_path: Path) -> None:
+    """The alias is matched on both the name and the asname, so `import numpy as np` is found."""
+    (tmp_path / "m.py").write_text("import numpy as np\nimport os\n\nx = 1\n")
+    r = Repro("b", "t").remove_imported_name(
+        "m.py", module=None, name="numpy", asname="np"
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "import os\n\nx = 1\n"
+
+
+def test_remove_imported_name_asserts_when_absent(tmp_path: Path) -> None:
+    """Removing a name that is not imported raises, so a wrong recipe fails loudly."""
+    (tmp_path / "m.py").write_text("from pkg import a, b\n")
+    r = Repro("b", "t").remove_imported_name("m.py", module="pkg", name="missing")
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+# --- add_import ----------------------------------------------------------------
+
+
+def test_remove_imported_name_collapses_a_multiline_import_to_one_line(
+    tmp_path: Path,
+) -> None:
+    """Pruning a name from a parenthesized import rebuilds it as a single sorted-later line."""
+    (tmp_path / "m.py").write_text(
+        "from pkg import (\n    a,\n    moved,\n    b,\n)\n\nx = a + b\n"
+    )
+    r = Repro("b", "t").remove_imported_name("m.py", module="pkg", name="moved")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "from pkg import a, b\n\nx = a + b\n"
+
+
+def test_remove_imported_name_matches_a_relative_module(tmp_path: Path) -> None:
+    """A relative `from .pkg import` is matched via its level dots."""
+    (tmp_path / "m.py").write_text("from .pkg import a, moved\n\nx = a\n")
+    r = Repro("b", "t").remove_imported_name("m.py", module=".pkg", name="moved")
+    _apply(r, tmp_path)
+    assert (tmp_path / "m.py").read_text() == "from .pkg import a\n\nx = a\n"
+
+
+def test_remove_imported_name_preserves_comments_in_a_multiline_import(
+    tmp_path: Path,
+) -> None:
+    """Comments on surviving lines of a pruned parenthesized import must not vanish."""
+    (tmp_path / "m.py").write_text(
+        "from pkg import (\n"
+        "    a,  # used by frobnicator\n"
+        "    moved,\n"
+        "    b,\n"
+        ")\n"
+        "\n"
+        "x = a + b\n"
+    )
+    r = Repro("b", "t").remove_imported_name("m.py", module="pkg", name="moved")
+    _apply(r, tmp_path)
+    assert "# used by frobnicator" in (tmp_path / "m.py").read_text()

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_repath_import.py

@@ -0,0 +1,94 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- repath_import / add_typechecking_import -----------------------------------
+
+
+def test_repath_import_rewrites_nested_import(tmp_path: Path) -> None:
+    """A function-scoped import is repathed in place; the bare call is untouched."""
+    (tmp_path / "c.py").write_text(
+        "class K:\n"
+        "    def run(self):\n"
+        "        from old.mod import foo\n"
+        "\n"
+        "        return foo(1)\n"
+    )
+    r = Repro("b", "t").repath_import(
+        "c.py", old_module="old.mod", new_module="new.mod", name="foo"
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "c.py").read_text() == (
+        "class K:\n"
+        "    def run(self):\n"
+        "        from new.mod import foo\n"
+        "\n"
+        "        return foo(1)\n"
+    )
+
+
+def test_repath_import_leaves_a_module_level_import(tmp_path: Path) -> None:
+    """Only nested imports are repathed; a module-level import is left to the sorter."""
+    (tmp_path / "c.py").write_text("from old.mod import foo\n\n\nx = foo(1)\n")
+    r = Repro("b", "t").repath_import(
+        "c.py", old_module="old.mod", new_module="new.mod", name="foo"
+    )
+    with pytest.raises(AssertionError):
+        _apply(r, tmp_path)
+
+
+def test_repath_import_repaths_a_multiline_aliased_nested_import(
+    tmp_path: Path,
+) -> None:
+    """A nested multi-line from-import with an alias is repathed on its first line."""
+    (tmp_path / "c.py").write_text(
+        "def run():\n"
+        "    from old.mod import (\n"
+        "        foo as f,\n"
+        "    )\n"
+        "\n"
+        "    return f(1)\n"
+    )
+    r = Repro("b", "t").repath_import(
+        "c.py", old_module="old.mod", new_module="new.mod", name="foo"
+    )
+    _apply(r, tmp_path)
+    assert (tmp_path / "c.py").read_text() == (
+        "def run():\n"
+        "    from new.mod import (\n"
+        "        foo as f,\n"
+        "    )\n"
+        "\n"
+        "    return f(1)\n"
+    )
+
+
+def test_repath_import_rewrites_a_relative_nested_import(tmp_path: Path) -> None:
+    """A nested `from .mod import` matched by module name must actually be repathed."""
+    (tmp_path / "c.py").write_text(
+        "def run():\n    from .mod import foo\n\n    return foo(1)\n"
+    )
+    r = Repro("b", "t").repath_import(
+        "c.py", old_module="mod", new_module="pkg.mod", name="foo"
+    )
+    _apply(r, tmp_path)
+    assert "from pkg.mod import foo" in (tmp_path / "c.py").read_text()

.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_verify_and_run.py

@@ -0,0 +1,180 @@
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
+
+import mechanical_refactor_reproduction_utils as rr
+from mechanical_refactor_reproduction_utils import (
+    Repro,
+    _def_span,
+    _find_class,
+    _find_def,
+    _replace_span,
+    _slice_span,
+    dedent,
+    exec_command,
+    git_add_and_commit,
+    verify_mechanical_refactor,
+)
+from reproduction_testlib import _apply, _commit, _git, _write  # noqa: F401
+
+# --- verify_mechanical_refactor ------------------------------------------------
+
+
+def _silence_precommit(monkeypatch) -> None:
+    real = rr.exec_command
+
+    def fake(cmd: str, cwd=None, check=True):
+        if cmd.startswith("pre-commit"):
+            return ""
+        return real(cmd, cwd=cwd, check=check)
+
+    monkeypatch.setattr(rr, "exec_command", fake)
+
+
+def test_reproduce_passes_when_transform_matches_target(
+    repo: Path, tmp_path: Path, monkeypatch, capsys
+) -> None:
+    """A transform that recreates the target tree reports PASS and does not exit."""
+    _write(repo, **{"src.py": "line1\nline2\nline3\n"})
+    base = _commit(repo, "base")
+    _write(repo, **{"src.py": None, "a.py": "line1\nline2\n", "b.py": "line3\n"})
+    target = _commit(repo, "split")
+
+    def transform(root: Path) -> None:
+        lines = (root / "src.py").read_text().splitlines(keepends=True)
+        (root / "a.py").write_text("".join(lines[0:2]))
+        (root / "b.py").write_text("".join(lines[2:3]))
+        (root / "src.py").unlink()
+        rr.git_add_and_commit("split", cwd=str(root))
+
+    monkeypatch.chdir(repo)
+    monkeypatch.setattr(rr.tempfile, "mkdtemp", lambda prefix="": str(tmp_path / "wt"))
+    _silence_precommit(monkeypatch)
+
+    verify_mechanical_refactor(base, target, transform)
+    assert "PASS" in capsys.readouterr().out
+
+
+def test_reproduce_exits_when_transform_diverges(
+    repo: Path, tmp_path: Path, monkeypatch
+) -> None:
+    """A transform that produces a different tree fails with a non-zero exit."""
+    _write(repo, **{"src.py": "line1\nline2\nline3\n"})
+    base = _commit(repo, "base")
+    _write(repo, **{"src.py": None, "a.py": "line1\nline2\n", "b.py": "line3\n"})
+    target = _commit(repo, "split")
+
+    def wrong_transform(root: Path) -> None:
+        (root / "a.py").write_text("WRONG\n")
+        (root / "b.py").write_text("line3\n")
+        (root / "src.py").unlink()
+        rr.git_add_and_commit("split", cwd=str(root))
+
+    monkeypatch.chdir(repo)
+    monkeypatch.setattr(rr.tempfile, "mkdtemp", lambda prefix="": str(tmp_path / "wt"))
+    _silence_precommit(monkeypatch)
+
+    with pytest.raises(SystemExit):
+        verify_mechanical_refactor(base, target, wrong_transform)
+
+
+def test_reproduce_creates_verify_branch_on_pass(
+    repo: Path, tmp_path: Path, monkeypatch, capsys
+) -> None:
+    """A PASS run leaves a verify-mechanical-<base[:8]> branch in the repo."""
+    _write(repo, **{"src.py": "line1\nline2\nline3\n"})
+    base = _commit(repo, "base")
+    _write(repo, **{"src.py": None, "a.py": "line1\nline2\n", "b.py": "line3\n"})
+    target = _commit(repo, "split")
+
+    def transform(root: Path) -> None:
+        lines = (root / "src.py").read_text().splitlines(keepends=True)
+        (root / "a.py").write_text("".join(lines[0:2]))
+        (root / "b.py").write_text("".join(lines[2:3]))
+        (root / "src.py").unlink()
+        rr.git_add_and_commit("split", cwd=str(root))
+
+    monkeypatch.chdir(repo)
+    monkeypatch.setattr(rr.tempfile, "mkdtemp", lambda prefix="": str(tmp_path / "wt"))
+    _silence_precommit(monkeypatch)
+
+    verify_mechanical_refactor(base, target, transform)
+    assert "PASS" in capsys.readouterr().out
+    branch = f"verify-mechanical-{base[:8]}"
+    assert _git(repo, "branch", "--list", branch).endswith(branch)
+
+
+def _precommit_writes_file(monkeypatch, filename: str, contents: str) -> None:
+    real = rr.exec_command
+
+    def fake(cmd: str, cwd=None, check=True):
+        if cmd.startswith("pre-commit"):
+            (Path(cwd) / filename).write_text(contents)
+            return ""
+        return real(cmd, cwd=cwd, check=check)
+
+    monkeypatch.setattr(rr, "exec_command", fake)
+
+
+def test_reproduce_commits_pre_commit_fixes_when_tree_left_dirty(
+    repo: Path, tmp_path: Path, monkeypatch, capsys
+) -> None:
+    """When pre-commit reformats and leaves the tree dirty, a 'pre-commit fixes' commit
+    is created on top of the transform commit."""
+    _write(repo, **{"src.py": "hello\n"})
+    base = _commit(repo, "base")
+    _write(repo, **{"src.py": "hello world\n", "formatted.py": "auto\n"})
+    target = _commit(repo, "edit")
+
+    def transform(root: Path) -> None:
+        (root / "src.py").write_text("hello world\n")
+        rr.git_add_and_commit("transform", cwd=str(root))
+
+    monkeypatch.chdir(repo)
+    monkeypatch.setattr(rr.tempfile, "mkdtemp", lambda prefix="": str(tmp_path / "wt"))
+    _precommit_writes_file(monkeypatch, "formatted.py", "auto\n")
+
+    verify_mechanical_refactor(base, target, transform)
+    assert "PASS" in capsys.readouterr().out
+    branch = f"verify-mechanical-{base[:8]}"
+    subjects = _git(repo, "log", "--format=%s", "-2", branch).splitlines()
+    assert subjects == ["pre-commit fixes", "transform"]
+
+
+# --- Repro.run end-to-end ------------------------------------------------------
+
+
+def test_repro_run_passes_on_a_faithful_call_site_lowering(
+    repo: Path, monkeypatch, capsys
+) -> None:
+    """End-to-end: a lowering reproduces the commit byte-for-byte (pre-commit stubbed)."""
+    _write(repo, **{"c.py": "r = Old.foo(self.n, 5)\n"})
+    base = _commit(repo, "base")
+    _write(repo, **{"c.py": "r = self.n.foo(5)\n"})
+    target = _commit(repo, "lower the call site")
+    monkeypatch.chdir(repo)
+    _silence_precommit(monkeypatch)
+
+    diff = Repro(base, target).lower_call_sites("foo", "Old", paths=["c.py"]).run()
+    assert diff == ""
+    assert "PASS" in capsys.readouterr().out
+
+
+def test_repro_run_reports_residual_when_a_change_is_bundled(
+    repo: Path, monkeypatch, capsys
+) -> None:
+    """A bundled non-relocation change surfaces as a non-empty residual diff."""
+    _write(repo, **{"c.py": "r = Old.foo(self.n, 5)\nUNRELATED = 1\n"})
+    base = _commit(repo, "base")
+    _write(repo, **{"c.py": "r = self.n.foo(5)\nUNRELATED = 2\n"})
+    target = _commit(repo, "lower the call AND change a constant")
+    monkeypatch.chdir(repo)
+    _silence_precommit(monkeypatch)
+
+    diff = Repro(base, target).lower_call_sites("foo", "Old", paths=["c.py"]).run()
+    assert "UNRELATED" in diff
+    assert "RESIDUAL" in capsys.readouterr().out

.claude/skills/mechanical-refactor-verify/spec-reproduction-utils.md

@@ -0,0 +1,164 @@
+# Reproduction utils — specification (source of truth)
+
+## 1. Scope
+
+- Source of truth for `scripts/mechanical_refactor_reproduction_utils.py`: the
+  **clean-move property** its primitives implement (§2), each primitive's contract (§3),
+  and the byte-diff arbiter's semantics (§4).
+- The module, its tests, and the guides defer to this file; on any disagreement, this
+  file wins.
+- Elsewhere: commit splitting → `guide-split.md`; producing a proof →
+  `guide-construct-proof.md`; reading one → `guide-verify-proof.md`.
+
+## 2. The property — a "clean move"
+
+> A commit is a **clean move** iff every change it makes is code **relocated in the same
+> order** — allowing one **uniform indentation shift** of the whole block — plus a small
+> fixed set of **move artifacts**, and nothing else.
+
+- Equivalently: the commit is reproducible by composing only the primitives of §3.
+- The whitelist (§2.1) is exactly what they do; the not-allowed list (§2.2) is what they
+  refuse, so it surfaces as a residual diff.
+
+### 2.1 Allowed — the whole whitelist
+
+- A line **relocated in order**, modulo one **uniform** leading-indentation shift of the
+  whole block.
+- **Defs/classes gathered from scattered positions** into a **new module**, each cut
+  verbatim, assembled under an **audited authored header**:
+    - the byte diff certifies the bodies; the header is reproduced from the target;
+    - the header audit accepts only: imports, a docstring, a TYPE_CHECKING import block,
+      a `logging.getLogger(__name__)` logger, or an unparse-equivalent copy of an
+      assignment actually deleted from the source (`drop_assigns`, e.g.
+      `_is_hip = is_hip()`);
+    - every dropped assignment must reappear in the header — anything else raises instead
+      of certifying.
+- The **body of an extracted function** — an inline block relocated verbatim into a new
+  def; the `def` signature, an optional `return`, and the replacing `call` are authored.
+  Faithful **only** when the body moves unchanged; a de-self, control-flow restructure, or
+  bookkeeping consolidation is semantic and goes in its own commit first.
+- **Import statements** — added, removed, or repathed; single-line or parenthesised.
+  Realised directly from the target (a wholly new module's statement verbatim, wrapping
+  preserved); a new-module move may add `from __future__ import annotations`.
+- A one-sided **`@staticmethod` / `@classmethod`** — method ↔ free function.
+- A **`self` type annotation dropped** from the moved definition — relocating
+  `@staticmethod def foo(self: Target)` into `Target` as `def foo(self)`.
+- A **call-site requalification** — `Owner.foo(x)` → `foo(x)`: same symbol, same argument
+  bytes, only the qualifier dropped. (An `Old.foo(x)` → `New.foo(x)` owner swap is not a
+  primitive; it surfaces as a residual.)
+- A **call-site lowering** — `Owner.method(receiver, rest)` → `receiver.method(rest)`:
+  the receiver moves out of the argument list.
+- **Deleting a source file the relocation emptied** — nothing left beyond a docstring,
+  imports, or a `TYPE_CHECKING` block (`delete_file` refuses anything else).
+- **Blank-line changes** — ignored (§2.3).
+
+### 2.2 Not allowed — the commit is **not** a clean move
+
+- A **reorder** of lines within the moved block.
+- A **statement-level reorder** that relocates no definition — it changes evaluation
+  order: a reshape a human must confirm, not a certifiable relocation.
+- A **non-uniform** indentation change — it can change Python semantics.
+- A **trailing-whitespace** change, an internal-whitespace change, or a **line
+  merge/split**.
+- A **changed argument** in an otherwise-requalified call.
+- A **call rewrite for a symbol that did not move** in this commit.
+- A **signature change** other than dropping the `self` annotation.
+- A **rename** of the moved symbol (even a privacy flip `_foo` → `foo`).
+- **Scaffolding or a constant authored into an existing module** — a logger, a module
+  constant, a `TYPE_CHECKING` guard, a re-derived `_flag = compute_flag()`. (A *new*
+  module's header is authored from the target, §2.1; an existing module's body is not a
+  place to author fresh code.)
+- A **changed body in an extracted function** — de-self, control-flow restructure, or a
+  folded-in bookkeeping change: a semantic rewrite, not a relocation.
+
+- Reshape work (rename, fresh scaffolding, statement reorder, changed extraction body)
+  belongs in the prepare/postpare phases of `guide-split.md`.
+- The proof reports it as a residual — never certifies it.
+
+### 2.3 Blank lines are ignored
+
+- A blank line never changes Python behavior; PEP 8 separator blanks legitimately collapse
+  on relocation.
+- The formatter normalises both the reproduced and target sides, so a blank-line-only
+  difference cannot reach the byte diff.
+- Assumption: the **target commit is itself pre-commit-clean** (true for any commit that
+  passed this repo's hooks); a target that skipped the formatter can show blank-line
+  residuals.
+
+## 3. The faithful relocation primitives
+
+- Each primitive does only a relocation-faithful edit — AST-located, spliced as original
+  source text, never regenerated.
+- Therefore a byte match after the formatter certifies the commit is *exactly* that
+  relocation.
+
+- `move_symbol(name, *, src, dst, into_class, from_class, dedent, drop_self_annotation,
+  before, leave_delegate, delegate_name)`:
+    - cuts a `def` (functions only; a class moves via the extract primitives) with its
+      decorators; drops its own `@staticmethod`/`@classmethod`;
+    - shifts indentation uniformly (negative `dedent` indents into a class);
+    - pastes at a class end, at module level, or above the named sibling `before`;
+    - same-named defs need `from_class`; an ambiguous name or missing anchor raises;
+    - `leave_delegate` **authors** a forwarding stub in the source (original header + one
+      `return self.<attr>.<name>(...)`, `await`ed for async) — audit it like any header.
+- `extract_to_new_module(src, dst, *, symbols, future_import)`:
+    - cuts the contiguous source tail: the moved defs/classes plus leading scaffolding
+      (imports, TYPE_CHECKING guards, name-target assignments only);
+    - an executable trailing statement stops the cut;
+    - prepends `from __future__ import annotations` when the move adds it.
+- `extract_symbols_to_new_module(src, dst, *, symbols, header, order, drop_assigns)`:
+    - cuts the named defs/classes from **scattered** positions; assembles the new module
+      under the audited `header` (§2.1);
+    - `drop_assigns` deletes a relocated module-level constant from the source; a chained
+      `A = B = 1` keeps the surviving bindings.
+- `extract_function(src, dst, *, name, signature, body, body_indent, call, return_text,
+  before, into_class)`:
+    - cuts an inline `body` verbatim (must match at a line boundary);
+    - re-indents under the authored `signature` — multi-line string interiors keep their
+      exact bytes;
+    - replaces the block with the authored `call`.
+- `lower_call_sites(name, owner, *, paths)` — `Owner.m(receiver, rest)` →
+  `receiver.m(rest)` by splicing the original argument bytes (literal spelling, comments,
+  magic trailing comma survive); nested matching calls are all rewritten.
+- `requalify_call_sites(name, owner, *, paths)` — `Owner.m(args)` → `m(args)`; only the
+  qualifier span changes.
+- `remove_import(rel, import_text, *, in_function)` — function-scoped or module-level;
+  whole-statement match with token boundaries (`import os` cannot hit `import os.path`);
+  removes exactly the matched import even on a semicolon-joined line.
+- `remove_imported_name(rel, *, module, name, asname)` — drops one name from a
+  `from m import a, b` (or a plain `import x`), realising a lost import directly (this
+  repo's ruff has no F811); an import carrying comments loses only the dropped alias's own
+  line.
+- `add_import(rel, import_stmt)` — the import sorter places it; with no existing imports
+  it lands below the module docstring.
+- `add_typechecking_import(rel, import_stmt)` — appends inside the destination's
+  `if TYPE_CHECKING:` block; the sorter orders it.
+- `repath_import(rel, *, old_module, new_module, name)` — repaths a function-scoped
+  `from old import … name …` (relative imports included) in place; module-level repaths
+  fall out of add/remove + the sorter.
+- `delete_file(path)` — deletes a source module the relocation emptied; refuses anything
+  beyond a docstring, imports, or a `TYPE_CHECKING` block.
+
+Cross-cutting guarantees:
+
+- CRLF sources round-trip byte-for-byte; synthesized lines follow the file's newline
+  style.
+- Column arithmetic is UTF-8-byte-accurate; non-ASCII text does not shift a rewrite.
+
+## 4. The arbiter — reproduce and byte-diff
+
+`Repro.run()` (and the lower-level `verify_mechanical_refactor`):
+
+- checks out the base commit in a throwaway worktree;
+- replays the recorded primitives;
+- runs the repo's pre-commit hooks on the changed files;
+- byte-diffs against the target commit — an empty diff is the proof; a non-empty diff is
+  returned as the residual, exactly what the relocation does not account for.
+
+Properties:
+
+- It runs the **real formatter**: a call split across an `= (` line, or a reflow leaving a
+  closing bracket as context, reproduces exactly — no diff-shape heuristic to fool.
+- Explicit tradeoff: whatever the **pre-commit hooks auto-fix is absorbed** on both sides
+  (e.g. ruff's F401 removing a now-unused import). A hook-introduced change rides under a
+  byte match, so the hook set is part of the trusted base.

.claude/skills/scripted-runtime-notes/SKILL.md

@@ -0,0 +1,29 @@
+---
+name: scripted-runtime-notes
+description: Requirements for the SGLang scripted runtime, chiefly when to add (vs not add) a harness API. Use for anything related to the scripted runtime.
+---
+
+# Scripted Runtime — Notes
+
+Notes for anything related to the SGLang scripted runtime.
+
+## When to Add an API
+
+Tests read `r.req.*` and `t._scheduler.*` directly — there is no encapsulation boundary. A thin wrapper buys zero isolation; it only grows the surface.
+
+Add an API only if it does real work:
+
+1. **Control primitive** — drives the engine through a real path (`start_req`, `pause_generation`, `abort`, `evict_radix`, `exhaust_kv`). Reuse the real path; never hand-mutate state.
+2. **Hook-backed** — value cannot be read from a snapshot; accumulate via `scheduler_hook.on_run_batch` or the recv proxy (`chunks_done`). Read-only; never monkey-patch; never add `*_count` to `srt/`.
+3. **Multi-structure derivation, widely reused** — scans `chunked_req` + `waiting_queue` + `running_batch` + `last_batch` (`is_idle`, `status`, `batch_composition`).
+
+Else: don't. Read `r.req.X` / `t._scheduler.X` in the test; inline single-use accessors.
+
+Never:
+
+- Weaken an assertion to fit a missing probe.
+- Probe implementation details ("field non-None", "which branch ran") — assert the consequence.
+
+## Other Tips
+
+- **Engine-self-driven behavior: drive the real loop, don't call the private.** Never synchronously call a scheduler private (e.g. `scheduler._abort_on_waiting_timeout()`) from the harness/test — it runs at the wrong loop phase, bypasses the ordered `recv_requests` → `process_input_requests` injection, and can fire in states the real loop never reaches (e.g. while paused). For sweeps the engine runs itself (timeout/idle), enable the config/env and advance the loop with `yield`.

.claude/skills/sglang-bisect-ci-regression/SKILL.md

@@ -0,0 +1,224 @@
+---
+name: sglang-bisect-ci-regression
+description: Investigate consistently failing SGLang CI tests by extracting the failure signature from scheduled or rerun workflows, bisecting the passing/failing commit window, checking runner or hardware specificity, and optionally reproducing on a remote GPU host.
+---
+
+# SGLang Bisect CI Regression
+
+Investigate a consistently failing CI test to find the root cause - whether it's a code regression from a specific PR, a hardware/runner-specific issue, or an environment change. Optionally reproduce the failure on a remote GPU server.
+
+## Slash Command
+
+`/sglang-bisect-ci-regression <test_name_or_ci_url> [ssh_target] [docker_container]`
+
+## When to Use This Skill
+
+- A CI test is failing consistently on main (scheduled runs)
+- You need to find which PR introduced a regression
+- You suspect a runner-specific or GPU-specific issue
+- You want to reproduce a CI failure on a remote server
+
+## Arguments
+
+- **First argument (required)**: Test file name (e.g. `test_lora_tp.py`) or a GitHub Actions job URL
+- **Second argument (optional)**: SSH target for remote reproduction (e.g. `user@host`)
+- **Third argument (optional)**: Docker container name on the SSH target (e.g. `sglang_dev`)
+
+If SSH target and docker container are not provided, the skill will only perform the CI log analysis and bisection, without remote reproduction. **Ask the user** for these if reproduction is needed and they weren't provided.
+
+## Background: Scheduled CI Runs
+
+SGLang uses the `pr-test.yml` workflow with **scheduled runs** (cron-triggered) to periodically test the `main` branch. These runs are the primary data source for detecting regressions:
+
+- **Workflow**: `pr-test.yml` with `event: schedule`
+- **Branch**: `main`
+- **Dashboard**: https://github.com/sgl-project/sglang/actions/workflows/pr-test.yml?query=event%3Aschedule
+- **Frequency**: Runs multiple times daily, each pinned to the HEAD of `main` at trigger time
+- **Purpose**: Catches regressions that slip through PR-level CI (e.g., interaction bugs between merged PRs, hardware-specific issues)
+
+Always use these scheduled runs (not PR-triggered runs) when bisecting regressions on `main`. The `--event schedule` filter in `gh run list` ensures you only see these periodic main-branch runs.
+
+## Workflow
+
+### Phase 1: Extract the Failure Signature
+
+1. **Get the failing test details from CI logs.** If given a URL, fetch logs directly. If given a test name, find recent scheduled runs of `pr-test.yml` on `main` that failed:
+
+```bash
+# List recent scheduled runs targeting main (the primary source of truth for regressions)
+# These are cron-triggered runs visible at:
+# https://github.com/sgl-project/sglang/actions/workflows/pr-test.yml?query=event%3Aschedule
+gh run list --repo sgl-project/sglang --workflow="pr-test.yml" --event schedule --branch main --limit 20 --json databaseId,conclusion,createdAt,headSha
+
+# Find the job containing the test
+gh run view {RUN_ID} --repo sgl-project/sglang --json jobs --jq '.jobs[] | select(.conclusion == "failure") | {name, conclusion, databaseId}'
+
+# Get the failure details
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E -B 5 -A 30 "AssertionError|FAIL|Error|{TEST_NAME}"
+```
+
+2. **Record the failure signature:**
+   - Exact error message and assertion
+   - Affected test method name
+   - Model/config involved
+   - Numeric values (e.g., tolerance diffs, scores)
+   - Whether the failure is deterministic (same values across runs)
+
+### Phase 2: Temporal Bisection
+
+3. **Find the boundary between passing and failing runs.** Walk through the scheduled run history (from the `pr-test.yml` schedule runs on `main`) to identify:
+   - Last known PASSING run (sha + date)
+   - First known FAILING run (sha + date)
+
+```bash
+# For each scheduled run, check the specific partition/job status
+gh run view {RUN_ID} --repo sgl-project/sglang --json jobs --jq '.jobs[] | select(.name == "{JOB_NAME}") | {conclusion, databaseId}'
+
+# Verify a specific test passed or failed in a run
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E "{TEST_NAME}|PASSED|FAILED|logprobs mismatch" | head -10
+```
+
+4. **List commits between the boundary:**
+
+```bash
+git log --oneline {LAST_PASS_SHA}..{FIRST_FAIL_SHA}
+```
+
+5. **Filter for relevant commits** that touch files related to the failing test (model layers, kernels, test utilities, etc.):
+
+```bash
+git log --oneline {LAST_PASS_SHA}..{FIRST_FAIL_SHA} -- {relevant_paths}
+```
+
+### Phase 3: Runner/Hardware Analysis
+
+6. **Check if the failure is runner-specific.** Extract the runner identity from each failing and passing run:
+
+```bash
+# Get runner name and machine
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E "Runner name|Machine name" | head -5
+
+# Get GPU/driver info
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -i -E "NVIDIA-SMI|Driver Version|CUDA Version" | head -5
+
+# Get package versions
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E "sgl.kernel.*==|flashinfer.*==" | head -5
+```
+
+7. **Correlate runners with pass/fail outcomes.** Build a table:
+
+| Run ID | Date | Runner | GPU Type | Driver | Result |
+|--------|------|--------|----------|--------|--------|
+
+If all failures map to a specific runner type/GPU and all passes map to another, the issue is **hardware-specific**, not a code regression.
+
+### Phase 4: Code Analysis
+
+8. **If a code regression is suspected** (failures not runner-specific), examine the candidate commits:
+   - Read the changed files
+   - Understand how the changes could affect the failing test
+   - Look for prefill-vs-decode differences, TP-specific paths, kernel changes
+
+9. **If a hardware issue is suspected**, analyze:
+   - Kernel compatibility (CUDA compute capability)
+   - Driver version differences
+   - All-reduce / NCCL behavior differences
+   - CUDA graph capture differences across GPU architectures
+
+### Phase 5: Remote Reproduction (Optional)
+
+Only if SSH target and docker container were provided.
+
+10. **Verify the remote environment:**
+
+```bash
+ssh {SSH_TARGET} "docker exec {CONTAINER} nvidia-smi --query-gpu=name,driver_version --format=csv"
+ssh {SSH_TARGET} "docker exec {CONTAINER} pip show sgl-kernel sglang flashinfer-python 2>&1 | grep -E 'Name:|Version:'"
+```
+
+11. **Ensure latest code is installed.** If the container is stale, update:
+
+```bash
+# Try fetching latest main
+ssh {SSH_TARGET} "docker exec {CONTAINER} bash -c 'cd /path/to/sglang && git fetch origin main && git checkout origin/main'"
+# Or download and install from tarball if git auth fails
+ssh {SSH_TARGET} "docker exec {CONTAINER} bash -c 'cd /tmp && curl -L https://github.com/sgl-project/sglang/archive/refs/heads/main.tar.gz | tar xz && cd sglang-main && pip install -e \"python[all]\"'"
+# Reinstall (after git fetch)
+ssh {SSH_TARGET} "docker exec {CONTAINER} bash -c 'cd /path/to/sglang && pip install -e \"python[all]\"'"
+# Install test dependencies if needed
+ssh {SSH_TARGET} "docker exec {CONTAINER} pip install peft rouge-score"
+```
+
+12. **Create a minimal reproduction script** that:
+    - Uses `if __name__ == '__main__'` with `mp.set_start_method("spawn")`
+    - Runs the specific failing test configuration
+    - Prints key metrics (diffs, scores, outputs)
+    - Exits with code 1 on failure
+
+13. **Copy and run the reproduction script:**
+
+```bash
+scp /tmp/repro_script.py {SSH_TARGET}:/tmp/
+ssh {SSH_TARGET} "docker cp /tmp/repro_script.py {CONTAINER}:/tmp/"
+ssh {SSH_TARGET} "docker exec -e CUDA_VISIBLE_DEVICES=0,1 {CONTAINER} python3 /tmp/repro_script.py"
+```
+
+14. **Run control experiments** to isolate the variable:
+    - If suspecting TP issue: run with TP=1 as control
+    - If suspecting GPU issue: compare same code on different GPU
+    - If suspecting a specific commit: test before/after that commit
+
+### Phase 6: Report
+
+15. **Produce a structured report:**
+
+```markdown
+## CI Regression Bisection Report
+
+### Failure Signature
+- **Test**: {test_file}::{test_method}
+- **Error**: {exact error message}
+- **Key metrics**: {numeric values}
+- **Deterministic**: Yes/No
+
+### Root Cause Classification
+One of:
+- **Code Regression**: PR #{number} introduced the bug
+- **Hardware-Specific**: Fails on {GPU_TYPE}, passes on others
+- **Environment Change**: New runner/driver/package version
+- **Pre-existing Flakiness**: Intermittent, not a new regression
+
+### Evidence
+| Condition | Result |
+|-----------|--------|
+| {condition1} | PASS/FAIL |
+| {condition2} | PASS/FAIL |
+
+### Timeline
+- {date}: Last known pass ({sha}, {runner})
+- {date}: First known fail ({sha}, {runner})
+- {date}: Confirmed reproduction on {server}
+
+### Recommended Fix
+- **Short-term**: {workaround}
+- **Long-term**: {proper fix}
+```
+
+## Key Patterns to Recognize
+
+| Pattern | Diagnosis |
+|---------|-----------|
+| Same SHA passes on runner A, fails on runner B | Hardware/runner-specific |
+| All runners fail after commit X | Code regression from commit X |
+| Intermittent - same runner sometimes passes/fails | Flaky test or race condition |
+| Prefill OK but decode fails | TP/all-reduce issue in decode path |
+| Works with TP=1, fails with TP>1 | Tensor parallelism bug |
+| Exact same numeric diff every time | Deterministic bug, not flakiness |
+
+## Important Notes
+
+- **Always check runner identity** before concluding it's a code regression. Many "consistent" failures are actually runner-specific.
+- **Test partition assignments change over time** as tests are added/removed. A test may move between partitions, landing on different runner types.
+- **H200 runners** use `/root/actions-runner/` path and machine names like `gpu-h200-worker-*`. Non-H200 runners use `/public_sglang_ci/runner-*` paths.
+- When running remote reproduction, use `run_in_background` for long-running tests and check output with `TaskOutput`.
+- Container environments may be stale - always verify package versions match CI before drawing conclusions.

.claude/skills/sglang-cherrypick/SKILL.md

@@ -0,0 +1,331 @@
+---
+name: sglang-cherrypick
+description: Trigger the bot-cherry-pick workflow for a batch of merged PRs onto a release branch and monitor each run to completion. Use when an SGLang release manager asks to cherry-pick a list of PRs to a release branch.
+---
+
+# SGLang Cherry-Pick
+
+Trigger `.github/workflows/bot-cherry-pick.yml` for each PR in a list, then monitor the resulting workflow runs and report per-PR success/failure with links to the created cherry-pick PRs (or the failure reason).
+
+## Slash Command
+
+`/sglang-cherrypick <target_branch> <pr1> [pr2 pr3 ...]`
+
+Examples:
+- `/sglang-cherrypick release/v0.5.7 25956 25958 25987`
+- `/sglang-cherrypick release/v0.5.7 25956,25958,25987` (comma-separated also accepted)
+
+## Arguments
+
+- **`target_branch`** (required): Release branch in the form `release/vX.Y` or `release/vX.Y.Z`. Must already exist on `origin` (i.e., `sgl-project/sglang`).
+- **`pr_numbers`** (required, one or more): Merged PR numbers to cherry-pick. Each must be a positive integer.
+
+## Repository
+
+Always targets the upstream repo `sgl-project/sglang`. The workflow's job guard (`if: github.repository == 'sgl-project/sglang'`) means triggering it on a fork is a no-op.
+
+## Workflow
+
+### Step 1 — Validate arguments
+
+Fail fast before triggering anything.
+
+```bash
+# target branch shape (matches the workflow's own validator)
+[[ "$TARGET_BRANCH" =~ ^release/v[0-9]+\.[0-9]+(\.[0-9]+)?$ ]] || die "Invalid target_branch"
+
+# branch exists on upstream
+gh api "repos/sgl-project/sglang/branches/$TARGET_BRANCH" --jq '.name' >/dev/null || die "Branch not found"
+
+# each PR is numeric, exists, MERGED, and has a recorded merge commit
+declare -A PR_TO_SHA=()
+declare -A PR_TO_TITLE=()
+for PR in "${PRS[@]}"; do
+  [[ "$PR" =~ ^[0-9]+$ ]] || die "PR '$PR' is not a positive integer"
+  PR_JSON=$(gh pr view "$PR" --repo sgl-project/sglang --json state,mergeCommit,title) \
+    || die "PR #$PR not found"
+  STATE=$(jq -r .state <<<"$PR_JSON")
+  [[ "$STATE" == "MERGED" ]] || die "PR #$PR is not MERGED (state=$STATE)"
+  SHA=$(jq -r '.mergeCommit.oid // empty' <<<"$PR_JSON")
+  [[ -n "$SHA" ]] || die "PR #$PR has no merge commit recorded"
+  PR_TO_SHA[$PR]="$SHA"
+  PR_TO_TITLE[$PR]=$(jq -r .title <<<"$PR_JSON")
+done
+```
+
+Report any failures and **stop** — do not trigger partial batches.
+
+### Step 2 — Pre-flight: list changed files and detect conflicts locally
+
+Before dispatching any workflow, simulate each cherry-pick locally with `git merge-tree` to (a) show the user which files would change and (b) catch conflicts before paying for a CI run. `git merge-tree --write-tree` is a side-effect-free 3-way merge — it touches neither the working tree nor any ref.
+
+**2a. Locate the upstream remote (`sgl-project/sglang`).** Both the dual-remote (`upstream` + `origin` fork) and single-remote setups need to work.
+
+```bash
+UPSTREAM_REMOTE=$(git remote -v \
+  | awk '$2 ~ /[:\/]sgl-project\/sglang(\.git)?$/ && $3 == "(fetch)" {print $1; exit}')
+[[ -n "$UPSTREAM_REMOTE" ]] || die "No remote points to sgl-project/sglang"
+```
+
+**2b. Fetch the target branch and each PR's merge commit.** Fetch the commits by SHA (in case they're not on a ref the user has locally) and the target branch in one call.
+
+```bash
+git fetch "$UPSTREAM_REMOTE" "$TARGET_BRANCH" "${PR_TO_SHA[@]}" --quiet \
+  || die "Failed to fetch from $UPSTREAM_REMOTE"
+
+TARGET_REF="refs/remotes/$UPSTREAM_REMOTE/$TARGET_BRANCH"
+```
+
+**2c. Index existing cherry-pick PRs on the target branch.** One `gh pr list` call gets every cherry-pick PR ever filed against this branch (any state). For each input PR, we cross-reference by the title suffix `(#<PR>)` that the bot workflow always uses.
+
+```bash
+# Fetch all cherry-pick PRs against this branch (any state), then bucket by
+# source-PR number using the title pattern "(#<source_pr>)".
+EXISTING_CP_JSON=$(gh pr list --repo sgl-project/sglang \
+  --base "$TARGET_BRANCH" \
+  --label cherry-pick \
+  --state all \
+  --limit 200 \
+  --json number,title,url,state)
+
+declare -A PR_TO_EXISTING_CP=()   # source_pr -> JSON array of existing cherry-pick PRs
+for PR in "${PRS[@]}"; do
+  PR_TO_EXISTING_CP[$PR]=$(jq -c \
+    "[.[] | select(.title | contains(\"(#${PR})\"))]" <<<"$EXISTING_CP_JSON")
+done
+```
+
+For each input PR, classify the existing cherry-picks:
+
+- **`MERGED`** present → the cherry-pick already landed. **Skip** this PR in Step 3.
+- **`OPEN`** present (and no `MERGED`) → a previous dispatch is still in flight. **Warn**, ask the user whether to skip or re-dispatch, but default to **skip** (re-dispatching creates a duplicate).
+- Only `CLOSED` (no merged, no open) → previous attempts were abandoned; safe to re-dispatch.
+- Empty → no prior attempt; proceed normally.
+
+**2d. For each PR, run `git merge-tree` and diff the result.** The semantics of `cherry-pick` are: 3-way-merge with base = parent of source commit, ours = target tip, theirs = source commit. For merge commits the workflow uses `-m 1`, which means base = **first** parent — `${SHA}^` resolves to `${SHA}^1` for both regular and merge commits, so one form covers both.
+
+```bash
+declare -A PR_TO_CONFLICTS=()
+declare -A PR_TO_FILES=()
+
+for PR in "${PRS[@]}"; do
+  SHA="${PR_TO_SHA[$PR]}"
+
+  # --write-tree: print the resulting tree SHA on success
+  # Exit 0 = clean merge; exit 1 = conflicts
+  if MERGE_OUT=$(git merge-tree --write-tree \
+                   --merge-base="${SHA}^" \
+                   "$TARGET_REF" "$SHA" 2>&1); then
+    RESULT_TREE=$(head -1 <<<"$MERGE_OUT")
+    PR_TO_CONFLICTS[$PR]=""
+    # Show files that actually differ between target tip and the merged tree.
+    # This is more accurate than `git show --name-status $SHA` because it
+    # accounts for changes already present on the release branch. As a
+    # side-effect, an already-cherry-picked commit shows up here as "0 files".
+    PR_TO_FILES[$PR]=$(git diff --name-status "$TARGET_REF" "$RESULT_TREE")
+  else
+    # Conflict output format (git ≥2.40): first line is the (partial) tree,
+    # remaining lines list conflicted paths and informational messages.
+    # We just capture and surface it; user decides what to do.
+    PR_TO_CONFLICTS[$PR]="$MERGE_OUT"
+    PR_TO_FILES[$PR]=$(git show --name-status --format= "$SHA" 2>/dev/null)
+  fi
+done
+```
+
+**2e. Print a pre-flight report.** One table summarizing each PR, followed by per-PR file lists. The "Prior cherry-pick" column uses the classification from 2c.
+
+```markdown
+## Cherry-Pick Pre-Flight — `release/vX.Y.Z`
+
+| PR     | Title                    | Merge SHA | Prior cherry-pick    | Conflicts    | # files |
+|--------|--------------------------|-----------|----------------------|--------------|---------|
+| #25733 | [Bug] Fix V4-Pro NaN ... | 79ea30d1  | ✅ merged as #26063  | clean        | 0       |
+| #25562 | [bugfix] Fix wrong ...   | b19052c9  | none                 | **CONFLICT** | —       |
+| #25585 | [Bugfix] Fix missing ... | 86c6c77f  | none                 | clean        | 2       |
+
+### Files (PR #25585 — clean)
+M  python/sglang/srt/layers/communicator.py
+M  python/sglang/srt/models/deepseek_v4.py
+
+### Conflict detail (PR #25562)
+<merge-tree output: conflicted paths and reasons>
+```
+
+**2f. Gate before dispatching.** Stop and report if any PR is in either of these states:
+
+- `git merge-tree` reports a **conflict** — the workflow would just fail; let the user fix or remove that PR.
+- Already has a **MERGED** cherry-pick PR on the target branch — re-dispatching would create a redundant PR. Skip it (or, if the user really wants a re-run, they can pass an explicit override list).
+- Has an **OPEN** cherry-pick PR with no merged one — default to skipping with a warning; surface the open PR's URL so the user can review/merge/close it before re-dispatching.
+
+Only PRs that are **clean** AND have **no merged-or-open** prior cherry-pick should proceed to Step 3.
+
+As a sanity check, a clean pre-flight that shows **0 files changed** is the structural signature of "this commit is already on the branch" — if you see it without an existing merged cherry-pick PR being detected (rare, e.g. the original PR was force-merged onto the release branch directly), surface that too and skip the dispatch.
+
+### Step 3 — Dispatch each PR's workflow run
+
+`gh workflow run` (gh ≥2.45) prints the dispatched run's URL on stdout — parse it directly. Fall back to the snapshot/diff polling only if the URL isn't returned (older gh).
+
+```bash
+# Snapshot once up front in case we need the fallback path.
+mapfile -t SEEN < <(gh run list \
+  --workflow=bot-cherry-pick.yml \
+  --repo sgl-project/sglang \
+  --limit 50 \
+  --json databaseId --jq '.[].databaseId')
+
+declare -A PR_TO_RUN=()  # pr_number -> run_id
+
+for PR in "${PRS[@]}"; do
+  DISPATCH_OUT=$(gh workflow run bot-cherry-pick.yml \
+    --repo sgl-project/sglang \
+    -f pr_number="$PR" \
+    -f target_branch="$TARGET_BRANCH" 2>&1) || { echo "$DISPATCH_OUT"; die "dispatch failed for PR #$PR"; }
+
+  # Preferred path: gh prints the run URL like
+  #   https://github.com/sgl-project/sglang/actions/runs/26275460359
+  RUN_URL=$(grep -oE 'https://github.com/[^[:space:]]+/actions/runs/[0-9]+' \
+              <<<"$DISPATCH_OUT" | head -1)
+  RUN_ID="${RUN_URL##*/}"
+
+  # Fallback for older gh that doesn't print the URL: poll the runs list,
+  # filter to workflow_dispatch events we haven't seen yet.
+  if [[ -z "$RUN_ID" ]]; then
+    for _ in $(seq 1 30); do
+      sleep 2
+      CANDIDATE=$(gh run list \
+        --workflow=bot-cherry-pick.yml \
+        --repo sgl-project/sglang \
+        --limit 10 \
+        --json databaseId,event \
+        --jq '[.[] | select(.event=="workflow_dispatch") | .databaseId] | .[0]')
+      if [[ -n "$CANDIDATE" ]] \
+         && ! printf '%s\n' "${SEEN[@]}" | grep -qx "$CANDIDATE" \
+         && ! printf '%s\n' "${PR_TO_RUN[@]}" | grep -qx "$CANDIDATE"; then
+        RUN_ID="$CANDIDATE"
+        break
+      fi
+    done
+  fi
+
+  if [[ -z "$RUN_ID" ]]; then
+    echo "::warning::No new workflow run detected for PR #$PR within 60s"
+    PR_TO_RUN[$PR]="UNKNOWN"
+  else
+    PR_TO_RUN[$PR]="$RUN_ID"
+  fi
+done
+```
+
+**Notes:**
+- The workflow has `concurrency: cherry-pick-${{ target_branch }}` with `cancel-in-progress: false`. So multiple dispatches against the same target branch **queue serially**, not in parallel. That's fine — we batch the triggers and the GitHub side serializes execution.
+- `gh workflow run` is fire-and-forget; the dispatched run shows up in `gh run list` within a few seconds.
+
+### Step 4 — Monitor each run to completion
+
+Use `gh run watch` per run id, sequentially (since they execute serially anyway).
+
+```bash
+for PR in "${PRS[@]}"; do
+  RUN_ID="${PR_TO_RUN[$PR]}"
+  [[ "$RUN_ID" == "UNKNOWN" ]] && continue
+
+  gh run watch "$RUN_ID" \
+    --repo sgl-project/sglang \
+    --exit-status \
+    --interval 15 \
+    >/dev/null 2>&1 || true   # we read conclusion below; don't abort the loop on fail
+done
+```
+
+`gh run watch` blocks until the run completes. Use `--interval 15` to be polite on rate limits.
+
+### Step 5 — Collect outcomes per PR
+
+For each PR, fetch the run conclusion and (if successful) the URL of the created cherry-pick PR.
+
+```bash
+for PR in "${PRS[@]}"; do
+  RUN_ID="${PR_TO_RUN[$PR]}"
+
+  if [[ "$RUN_ID" == "UNKNOWN" ]]; then
+    echo "PR #$PR: UNKNOWN (no run found)"
+    continue
+  fi
+
+  CONCLUSION=$(gh run view "$RUN_ID" --repo sgl-project/sglang \
+    --json conclusion,status,url \
+    --jq '"\(.status) \(.conclusion) \(.url)"')
+
+  STATUS=$(awk '{print $1}' <<<"$CONCLUSION")
+  RESULT=$(awk '{print $2}' <<<"$CONCLUSION")
+  RUN_URL=$(awk '{print $3}' <<<"$CONCLUSION")
+
+  if [[ "$RESULT" == "success" ]]; then
+    # Find the cherry-pick PR created by this run. Title format from the workflow:
+    #   "[Cherry-pick to <BRANCH>] <ORIG TITLE> (#<PR>)"
+    CP_PR=$(gh pr list --repo sgl-project/sglang \
+      --base "$TARGET_BRANCH" \
+      --label cherry-pick \
+      --state all \
+      --limit 30 \
+      --json number,title,url,createdAt \
+      --jq "[.[] | select(.title | contains(\"(#${PR})\"))][0]")
+
+    CP_URL=$(jq -r '.url // "N/A"' <<<"$CP_PR")
+    CP_NUM=$(jq -r '.number // "?"' <<<"$CP_PR")
+    echo "PR #$PR  -> SUCCESS  cherry-pick PR #$CP_NUM ($CP_URL)  [run: $RUN_URL]"
+  else
+    # Failure: pull the cherry-pick step's last error line so the user sees why.
+    REASON=$(gh run view "$RUN_ID" --repo sgl-project/sglang --log-failed 2>/dev/null \
+      | grep -m1 -E "::error::" \
+      | sed -E 's/^[^:]*::error::?//' \
+      || echo "(see run logs)")
+    echo "PR #$PR  -> $RESULT  reason: $REASON  [run: $RUN_URL]"
+  fi
+done
+```
+
+### Step 6 — Final summary
+
+Print one table sorted by input order:
+
+```markdown
+## Cherry-Pick Batch Summary — `release/v0.5.7`
+
+| PR  | Status   | Cherry-pick PR | Run | Notes |
+|-----|----------|----------------|-----|-------|
+| #25956 | success | #26031 | run/12345 | — |
+| #25958 | failure | —      | run/12346 | Cherry-pick of <SHA> onto release/v0.5.7 failed due to conflicts |
+| #25987 | success | #26032 | run/12347 | — |
+
+**Totals:** N succeeded, M failed, K unknown.
+```
+
+For any **failure**, suggest the manual fallback from the workflow's own error message:
+
+> Resolve locally: `git checkout release/v0.5.7 && git cherry-pick <SHA>`, fix conflicts, push a branch, and open the PR by hand.
+
+## Common Failure Modes
+
+| Symptom | Cause | Action |
+|---------|-------|--------|
+| `PR #X is not merged (state=OPEN)` | PR not yet merged | Wait for merge or pass `--commit-sha` (not supported by this slash command) |
+| `Target branch '...' does not exist` | Typo or branch not cut yet | Confirm branch name; release manager may not have cut it |
+| `Cherry-pick of <SHA> onto <BRANCH> failed due to conflicts` | Code drift on release branch (should already have been caught in Step 2 pre-flight) | Do it manually as instructed above |
+| Pre-flight `git merge-tree` reports a conflict | Same as above, caught locally before any CI run | Remove that PR from the batch and resolve manually |
+| `No remote points to sgl-project/sglang` | Skill invoked from a checkout that only has a fork remote | Add the upstream remote: `git remote add upstream https://github.com/sgl-project/sglang.git` |
+| Pre-flight `git merge-tree` errors with `unknown option` | git < 2.38 | Upgrade git, or run the skill on a machine with a modern git |
+| Pre-flight reports `Prior cherry-pick: merged as #N` | The PR has already been cherry-picked and merged onto this release branch | Skip this PR — re-dispatching would create a duplicate PR. Verify #N is the right one before removing from the list. |
+| Pre-flight reports `Prior cherry-pick: OPEN as #N` | A previous dispatch is still in flight (PR not yet merged or closed) | Default: skip and ask the user to land or close #N first. Re-dispatching creates a parallel duplicate that needs to be cleaned up afterwards. |
+| Pre-flight is clean but `# files = 0` and no prior cherry-pick PR was found | Commit landed on the release branch by direct merge (not via the bot), or via a rebase that rewrote the SHA | Skip the dispatch — the change is already there. Surface this anomaly so the user knows the bot wasn't the source. |
+| Multiple runs but only one detected | Two dispatches landed in the same `gh run list` poll cycle | Re-run for the missing PR, or look up its run by hand: `gh run list --workflow=bot-cherry-pick.yml --event workflow_dispatch -L 20` |
+| `403` on `gh workflow run` | Missing `actions:write` on the token | Use a token that has workflow dispatch rights on `sgl-project/sglang` |
+
+## Notes
+
+- The skill **never modifies** the workflow file. It only dispatches it.
+- The skill operates on the upstream repo only (`sgl-project/sglang`); the user's fork is irrelevant here.
+- Per-branch concurrency means picking 20 PRs to the same release branch will take ~20× the runtime of one. There is no parallelism to be gained client-side. If the user batches across **different** target branches, those run concurrently.
+- Do not skip the merged-state precheck — the workflow will reject unmerged PRs, but we want a single batched validation report up front rather than N individual workflow failures.
+- The skill should be invoked with `gh auth status` already passing; if not, surface the auth error and stop.

.claude/skills/sglang-prod-incident-triage/SKILL.md

@@ -0,0 +1,291 @@
+---
+name: sglang-prod-incident-triage
+description: Replay-first debug flow for SGLang serving problems. Use when a live or recent server shows health-check failures, latency or throughput regressions, queue growth, timeouts, distributed stalls, crash dumps, wrong outputs after deploys, or PD/EP/HiCache issues, and the job is to turn the problem into a replay plus the right next debug tool.
+---
+
+# SGLang Serving Debug
+
+## Overview
+
+Use this skill to turn a live serving problem into a debug path you can replay.
+
+Use one loop:
+
+- collect a baseline bundle
+- save the failing request or crash dump
+- replay on a clean target
+- only then switch tools
+
+Do not start with profiling.
+
+This skill should work with more focused skills instead of re-implementing them:
+
+- `debug-cuda-crash` when replay plus coredump points to a CUDA crash path
+- `debug-distributed-hang` when the problem is clearly a TP/PP/DP/EP hang
+- `llm-torch-profiler-analysis` when the issue is already narrowed to a
+  compute-side path
+
+Three examples are included:
+
+- TTFT spike with low queue time
+- replay-first CUDA crash flow
+- request-shaped distributed hang flow
+
+## Output Contract
+
+Return:
+
+- problem class
+- what was checked
+- strongest signal so far
+- current best guess
+- what was ruled out
+- next step
+- production risk
+
+## When To Use It
+
+- `/health` or `/health_generate` is unhealthy
+- latency or throughput regressed under serving load
+- queue size grows while health still looks green
+- one request class times out or hangs
+- the server crashes only after some requests
+- outputs changed after a deploy, topology change, or weight switch
+- one older commit is known-good and a newer commit is known-bad
+
+## Workflow
+
+### 1. Collect a baseline bundle
+
+If a live server is reachable, collect a read-only bundle before anything more
+intrusive:
+
+```bash
+python3 scripts/incident_artifact_tool.py collect-bundle \
+  --base-url http://127.0.0.1:30000 \
+  --outdir /tmp/incident_bundle
+
+python3 scripts/incident_artifact_tool.py summarize-bundle \
+  /tmp/incident_bundle
+```
+
+If the server is protected:
+
+```bash
+python3 scripts/incident_artifact_tool.py collect-bundle \
+  --base-url http://127.0.0.1:30000 \
+  --token "$SGLANG_BEARER_TOKEN" \
+  --outdir /tmp/incident_bundle
+```
+
+The bundle script collects:
+
+- `/health`
+- `/health_generate`
+- `/model_info`
+- `/server_info`
+- `/v1/loads?include=all`
+- `/v1/loads?include=core,queues,disagg,spec`
+- `/metrics`
+- `/hicache/storage-backend` on a best-effort basis
+
+Use the summary for a quick read on:
+
+- health vs. active health state
+- topology and runtime flags
+- point-in-time queue and token usage
+- TTFT / E2E / queue-time heuristics from Prometheus metrics
+
+If the summary says the bundle was captured while the server was idle, recollect
+it during traffic or move quickly to dump plus replay.
+
+If no live server is reachable, start from the best dump or log already available:
+
+- crash dump
+- request dump
+- logs
+- CUDA coredump
+- OTel trace
+- torch profile
+
+### 2. Save the failing request
+
+Read [references/decision-tree.md](references/decision-tree.md) only if the
+problem class is still unclear:
+
+- server down or unhealthy
+- latency or throughput regression
+- wrong output or behavior regression
+- intermittent timeout or hang
+
+Then preserve the request payload that actually triggers the problem:
+
+- crash path: use `--crash-dump-folder`
+- non-crash path: enable request dump or save the exact trigger request
+
+Do not jump straight from a live symptom to low-level debugging without first
+saving something you can replay.
+
+### 3. Replay on a clean target
+
+Read [references/endpoints-and-signals.md](references/endpoints-and-signals.md)
+when you need help reading the baseline bundle or the replay target.
+
+Read [references/replay-trace-profile.md](references/replay-trace-profile.md)
+when you need the replay, trace, profile, or bisect paths.
+
+Standard order:
+
+1. collect baseline bundle
+2. capture request dump or crash dump
+3. restart a clean debug target if needed
+4. replay the same issue
+5. collect replay-time logs and dumps
+
+### 4. Only go deeper after replay
+
+#### Replay
+
+Use replay when:
+
+- a crash dump exists
+- a request dump exists
+- the problem depends on request shape or workload mix
+
+If a crash dump exists, summarize it first:
+
+```bash
+python3 scripts/incident_artifact_tool.py summarize-dump \
+  --input-file /path/to/crash_dump.pkl
+```
+
+Then replay:
+
+```bash
+python3 /path/to/sglang/scripts/playground/replay_request_dump.py \
+  --input-file /path/to/crash_dump.pkl \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --parallel 128
+```
+
+If `safe_pickle_load` blocks a locally captured trusted dump, use:
+
+```bash
+python3 scripts/replay_trusted_request_dump.py \
+  --input-file /path/to/request_dump.pkl \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --parallel 1
+```
+
+If replay indicates a CUDA crash path, restart the same build with coredumps
+enabled before reproducing again:
+
+```bash
+SGLANG_CUDA_COREDUMP=1 \
+SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \
+python -m sglang.launch_server \
+  --model-path ... \
+  --crash-dump-folder /tmp/sglang_crash_dump \
+  ...
+```
+
+Then inspect the generated coredump:
+
+```bash
+cuda-gdb "$(which python3)" \
+  -ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"
+```
+
+For a replay-first crash example, read
+[references/case-studies.md](references/case-studies.md).
+
+#### OTel trace
+
+Use tracing when:
+
+- request-stage timing is unclear
+- router vs. worker attribution is unclear
+- PD prefill/decode transfer may be implicated
+
+If tracing was enabled at startup, you can change the level without restart:
+
+```bash
+curl "http://127.0.0.1:30000/set_trace_level?level=1"
+curl "http://127.0.0.1:30000/set_trace_level?level=2"
+```
+
+#### Torch profile
+
+Use profiling when:
+
+- the issue is already narrowed to compute-side ownership
+- replay already reproduces the problem
+- metrics and loads do not explain the regression
+
+At that point, switch to `llm-torch-profiler-analysis`. Do not duplicate
+its profiling workflow here.
+
+For a low-noise latency example, read
+[references/case-studies.md](references/case-studies.md).
+
+#### Distributed hang
+
+If this looks like a collective stall, save the failing request, replay it on a
+clean target, collect the replay-time bundle and stacks, then switch to
+`debug-distributed-hang`.
+
+For an example of that flow, read
+[references/case-studies.md](references/case-studies.md).
+
+#### Regression between two commits
+
+If one commit is known-good and another is known-bad, build a deterministic
+harness before doing deeper manual debugging:
+
+1. choose a stable reproducer: request replay, benchmark command, or correctness check
+2. make the harness return `0` on good behavior and non-zero on bad behavior
+3. run `git bisect start <bad> <good>`
+4. run `git bisect run <harness>`
+5. return here only after a candidate commit is isolated
+
+Prefer replay-backed bisect when the regression depends on request shape or
+long-running serving state.
+
+### 6. Switch tools when the boundary is clear
+
+Switch tools once the fault class is clear:
+
+- `llm-torch-profiler-analysis` for kernel and overlap attribution
+- `debug-distributed-hang` for collective or rank-divergence hangs
+- `debug-cuda-crash` for CUDA crash reproduction and kernel API logging
+
+Do not switch tools before collecting the first bundle unless the user already has
+decisive logs or dumps.
+
+## References
+
+Load only what the current step needs:
+
+- [references/decision-tree.md](references/decision-tree.md)
+  - problem classes, tool switch points, return shape
+- [references/endpoints-and-signals.md](references/endpoints-and-signals.md)
+  - endpoint behavior, auth notes, field reading
+- [references/replay-trace-profile.md](references/replay-trace-profile.md)
+  - request dump, crash dump, replay, trace, profiler step, bisect
+- [references/case-studies.md](references/case-studies.md)
+  - compact examples for replay-first CUDA crash, latency, and distributed-hang triage
+
+## Scripts
+
+- [scripts/incident_artifact_tool.py](scripts/incident_artifact_tool.py)
+  - collect a read-only live bundle
+  - summarize a collected bundle into a compact debug note
+  - summarize a trusted request dump or crash dump before replay
+- [scripts/replay_trusted_request_dump.py](scripts/replay_trusted_request_dump.py)
+  - replay a trusted request dump when `safe_pickle_load` blocks stock replay
+
+If a live bundle was collected, include its path.
+
+If replay, trace, or profiling was chosen, say why bundle plus dump were not enough.

.claude/skills/sglang-prod-incident-triage/references/case-studies.md

@@ -0,0 +1,81 @@
+# Case Studies
+
+Use these examples only after the live bundle and request dump point toward the
+same class of failure. They are patterns for how to reason from replayable
+evidence, not recipes to copy blindly.
+
+## CUDA Crash: Upstream Top-K Corruption, Downstream MoE OOB
+
+Use when a replayed CUDA crash lands in a MoE align or shared-memory kernel but
+the suspicious data was produced by an earlier routing kernel.
+
+Shape that made the original case useful:
+
+- model family: Qwen3 MoE
+- visible crash: `moe_align_block_size_kernel`
+- likely producer: `topkGatingSoftmax` / MoE top-k routing
+- evidence path: crash dump -> replay -> CUDA coredump -> walk one kernel
+  upstream from the visible fault
+
+Triage loop:
+
+```text
+summarize crash dump
+  -> replay the exact request
+  -> enable CUDA coredump on the replay target
+  -> identify the failing kernel
+  -> inspect the immediately preceding producer kernel and tensors
+```
+
+Key lesson: a consumer kernel can be the first one to fault even when the bad
+index was produced earlier. Preserve the request shape before changing prompts.
+
+## Latency: TTFT Spike With Low Queue Time
+
+Use when `/health` and `/health_generate` are green, queue depth is low, but TTFT
+is still high.
+
+Signals from the original case:
+
+- `waiting=0`
+- average queue time was tiny
+- TTFT was high
+- scheduler stage timing pointed to prefill forward time
+
+Triage loop:
+
+```text
+collect live bundle
+  -> save the slow request
+  -> replay the same request on a clean target
+  -> profile only after replay reproduces compute-side ownership
+```
+
+Key lesson: rule out queue pressure with `/v1/loads`, `/metrics`, and stage
+timing before opening a profiler trace.
+
+## Distributed Hang: Request-Shaped TP Collective Mismatch
+
+Use when one request hangs, ranks stop making progress differently, and the
+failure looks like a generic serving stall until replay isolates it.
+
+Shape that made the original case useful:
+
+- a prompt tokenized to a specific extend length
+- one TP rank skipped a logits `all_gather`
+- the peer rank still entered the real collective
+- the request never returned
+
+Triage loop:
+
+```text
+collect healthy bundle
+  -> save the trigger request
+  -> replay on a clean target
+  -> collect rank stacks and replay-time bundle
+  -> switch to debug-distributed-hang
+```
+
+Key lesson: once the symptom looks like rank divergence or a collective mismatch,
+do not keep profiling kernels. Preserve the replay and move to distributed-hang
+debugging.

.claude/skills/sglang-prod-incident-triage/references/decision-tree.md

@@ -0,0 +1,197 @@
+# SGLang First Checks
+
+Use this reference when the problem class is still unclear and you need a fast
+starting point.
+
+## Default Order
+
+1. classify the symptom
+2. collect the fastest useful signal
+3. save the failing request or dump
+4. replay before you profile
+
+Do not start with `torch.profiler` unless the issue is already clearly
+compute-side.
+
+If one commit is known-good and another is known-bad, turn the problem into a
+stable `git bisect run <harness>` first.
+
+## Problem Classes
+
+### Server down or unhealthy
+
+Check:
+
+- `/health`
+- `/health_generate`
+- `/server_info`
+- recent stderr/stdout
+- crash dump status if `--crash-dump-folder` is enabled
+
+Likely directions:
+
+- startup or weight-load failure
+- deadlock or blocked scheduler
+- CUDA crash or OOM
+- auth or routing mismatch
+
+### High latency or low throughput
+
+Check:
+
+- `/v1/loads?include=all`
+- `/metrics`
+- `/server_info`
+- the exact request shape or benchmark command
+
+Likely directions:
+
+- queueing or capacity pressure
+- cache hit rate collapse
+- PD or EP topology mismatch
+- speculative decoding disabled or ineffective
+- kernel or backend regression
+
+### Wrong output or behavior regression
+
+Check:
+
+- exact request and expected output
+- `/model_info`
+- `/server_info`
+- current weights or recent config change
+
+Likely directions:
+
+- wrong weights or wrong revision
+- chat template, parser, or tool config drift
+- multimodal preprocessing drift
+- quantization or kernel correctness bug
+
+### Timeout or hang
+
+Check:
+
+- `/health`
+- `/health_generate`
+- `/v1/loads?include=all`
+- request dumps if enabled
+- per-rank logs
+- OTel trace if already enabled
+
+Likely directions:
+
+- distributed divergence or collective hang
+- queue starvation or retraction storm
+- PD transfer stall
+- storage or HiCache backend stall
+
+## Quick Paths
+
+### TTFT spike
+
+Start with:
+
+- `/v1/loads?include=all`
+- `/metrics`
+- `/server_info`
+
+Watch for:
+
+- `num_waiting_reqs` growth
+- `token_usage` saturation
+- `cache_hit_rate` drop
+- PD queue buildup
+
+If queue pressure does not explain the slowdown, save the slow request and
+replay it.
+
+### Throughput collapse
+
+Start with:
+
+- `/v1/loads?include=all`
+- `/metrics`
+- benchmark reproduction if available
+
+Watch for:
+
+- low `gen_throughput`
+- queue growth
+- low cache hit rate
+- speculative metrics collapse
+- PD transfer or decode prealloc queues backing up
+
+### Crash after some requests
+
+Start with:
+
+- crash dump folder
+- stderr/stdout
+- request dump folder if available
+
+Then replay the crash dump or recent request dump.
+
+### Regression between two commits
+
+Start with:
+
+- known-good commit
+- known-bad commit
+- one stable pass/fail harness
+
+Best move:
+
+- `git bisect run <harness>`
+
+### One request class fails
+
+Start with:
+
+- exact request payload
+- request dump if available
+- smallest reproduction request
+
+Typical categories:
+
+- multimodal edge case
+- parser or structured output bug
+- model-specific kernel path
+- tool-call formatting issue
+
+## When To Switch Tools
+
+### Use replay when
+
+- a crash dump or request dump already exists
+- the issue depends on request shape or workload mix
+- you need one stable reproducer before going deeper
+
+### Use OTel trace when
+
+- request-stage timing is unclear
+- router vs. worker ownership is unclear
+- PD boundaries may be involved
+
+### Use torch profiler when
+
+- replay already reproduces the issue
+- queueing and routing are mostly ruled out
+- you need kernel-level attribution
+
+At that point, switch to `llm-torch-profiler-analysis`.
+
+### Use lower-level debug paths when
+
+- replay plus trace still leave ambiguity
+- the problem looks like a specific crash, hang, or correctness bug
+
+## What To Return
+
+- problem class
+- what was checked
+- strongest signal so far
+- current best guess
+- what was ruled out
+- next step
+- production risk

.claude/skills/sglang-prod-incident-triage/references/endpoints-and-signals.md

@@ -0,0 +1,218 @@
+# SGLang Endpoints and Signals
+
+Use this reference when checking a live server.
+
+## Auth
+
+Most read endpoints are public unless the server is protected by `api_key` or
+`admin_api_key`.
+
+Use:
+
+```bash
+curl -H "Authorization: Bearer <token>" ...
+```
+
+Rules:
+
+- normal protected endpoints require `api_key`
+- admin endpoints require `admin_api_key`
+- some HiCache endpoints fail if `admin_api_key` is not configured at all
+- `/health` and metrics-style health checks are usually still exposed
+
+## Core Endpoints
+
+### `/health`
+
+Cheap liveness check.
+
+- `200`: process is alive enough to answer health
+- `503`: starting, shutting down, or unhealthy
+
+`/health` alone is not enough for latency or hang diagnosis.
+
+### `/health_generate`
+
+Active health check.
+
+- exercises a real generate or embedding path
+- catches stuck schedulers or broken worker paths that `/health` can miss
+
+Use this when requests time out but `/health` is still green.
+
+### `/model_info`
+
+Use for model identity:
+
+- `model_path`
+- `tokenizer_path`
+- `is_generation`
+- `weight_version`
+- multimodal flags
+- model type or architectures
+
+This is the first check for wrong-output or wrong-weight problems.
+
+### `/server_info`
+
+Use for runtime shape:
+
+- serialized `server_args`
+- scheduler info
+- per-DP `internal_states`
+- SGLang version
+
+This is usually the single best live snapshot.
+
+## Load And Capacity
+
+### `/v1/loads?include=all`
+
+Best structured load endpoint for a first pass.
+
+Useful fields:
+
+- `num_running_reqs`
+- `num_waiting_reqs`
+- `num_total_tokens`
+- `num_used_tokens`
+- `token_usage`
+- `gen_throughput`
+- `cache_hit_rate`
+- `memory`
+- `speculative`
+- `disaggregation`
+- `queues`
+
+Useful queries:
+
+```bash
+curl -s http://127.0.0.1:30000/v1/loads
+curl -s "http://127.0.0.1:30000/v1/loads?include=all"
+curl -s "http://127.0.0.1:30000/v1/loads?include=core,queues,disagg"
+curl -s "http://127.0.0.1:30000/v1/loads?format=prometheus"
+```
+
+What to look for:
+
+- high `num_waiting_reqs` with low compute throughput usually means queueing or capacity pressure
+- `token_usage` near `1.0` usually means KV or token-capacity pressure
+- low `cache_hit_rate` after a deploy can explain TTFT regressions
+- PD queue fields often explain transfer or prealloc bottlenecks hidden by plain queue size
+
+### `/metrics`
+
+Prometheus endpoint. Use it when you need trends rather than one live snapshot.
+
+High-value metrics:
+
+- `sglang:time_to_first_token_seconds`
+- `sglang:time_per_output_token_seconds`
+- `sglang:e2e_request_latency_seconds`
+- `sglang:num_running_reqs`
+- `sglang:num_queue_reqs`
+- `sglang:num_used_tokens`
+- `sglang:cache_hit_rate`
+- `sglang:gen_throughput`
+- `sglang:token_usage`
+
+## Request Capture
+
+### `/configure_logging`
+
+Used by `python -m sglang.srt.managers.configure_logging`.
+
+Main use:
+
+- enable request logging
+- set request logging level
+- enable request dump folder
+- set request dump threshold
+
+Typical payload:
+
+```json
+{
+  "log_requests": true,
+  "log_requests_level": 3,
+  "dump_requests_folder": "/tmp/sglang_request_dump",
+  "dump_requests_threshold": 100
+}
+```
+
+Use this when the problem is ongoing and you need the next failing request
+without restarting the service.
+
+## HiCache
+
+### `GET /hicache/storage-backend`
+
+Returns tokenizer-side HiCache storage status:
+
+- `hicache_storage_backend`
+- `hicache_storage_backend_extra_config`
+- `hicache_storage_prefetch_policy`
+- `hicache_write_policy`
+
+Use this when long-context or PD problems may involve storage-backed KV reuse.
+
+### `PUT /hicache/storage-backend`
+### `DELETE /hicache/storage-backend`
+
+Runtime attach or detach. These are operational actions, not passive checks.
+
+## Profiling And Tracing Controls
+
+### `/start_profile`
+### `/stop_profile`
+
+Use only after the problem is already narrowed down.
+
+### `/set_trace_level?level=N`
+
+Changes trace verbosity when tracing was enabled at startup.
+
+Levels:
+
+- `0`: disabled
+- `1`: important slices
+- `2`: all slices except nested ones
+- `3`: all slices
+
+## Quick Reads By Problem Type
+
+### TTFT spike
+
+Read:
+
+- `/server_info`
+- `/v1/loads?include=all`
+- `/metrics`
+
+Compare:
+
+- queue size
+- token usage
+- cache hit rate
+- PD disaggregation queues
+
+### Hang or timeout
+
+Read:
+
+- `/health`
+- `/health_generate`
+- `/server_info`
+- `/v1/loads?include=all`
+
+If tracing is already enabled, look at trace data before heavier profiling.
+
+### Wrong model behavior
+
+Read:
+
+- `/model_info`
+- `/server_info`
+- exact request payload and parser or template config
+
+Do not jump to kernel profiling until config drift is ruled out.

.claude/skills/sglang-prod-incident-triage/references/replay-trace-profile.md

@@ -0,0 +1,236 @@
+# Replay, Trace, Profile, and Bisect
+
+Use this reference after the first live checks. The goal is to turn the problem
+into something repeatable.
+
+## Save Requests
+
+### Request dump
+
+```bash
+python3 -m sglang.srt.managers.configure_logging \
+  --url http://127.0.0.1:30000 \
+  --dump-requests-folder /tmp/sglang_request_dump \
+  --dump-requests-threshold 100
+```
+
+Use this when:
+
+- the problem is intermittent
+- you need the real request shape
+- you do not want to restart the server
+
+### Crash dump
+
+If the server already runs with:
+
+```bash
+--crash-dump-folder /tmp/crash_dump
+```
+
+SGLang saves recent requests before a crash. Treat that dump as the best
+starting point.
+
+Summarize it first:
+
+```bash
+python3 scripts/incident_artifact_tool.py summarize-dump \
+  --input-file /path/to/crash_dump.pkl
+```
+
+Current crash-dump tests show at least:
+
+- `server_args`
+- `requests`
+- `launch_command`
+
+## Replay
+
+Use the stock replay tool:
+
+```bash
+python3 scripts/playground/replay_request_dump.py \
+  --input-file /path/to/crash_dump.pkl \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --parallel 128
+```
+
+Or replay a folder:
+
+```bash
+python3 scripts/playground/replay_request_dump.py \
+  --input-folder /path/to/request_dump_dir \
+  --file-number 10 \
+  --parallel 128
+```
+
+If `safe_pickle_load` blocks a locally captured trusted dump, use:
+
+```bash
+python3 scripts/replay_trusted_request_dump.py \
+  --input-file /path/to/request_dump.pkl \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --parallel 1
+```
+
+If that happens, the allowlist is the problem, not the dump.
+
+Use replay before profiling when:
+
+- the issue depends on workload mix
+- it only appears after some number of requests
+- you need to compare two builds on the same traffic
+
+## CUDA Restart-And-Replay
+
+If replay points to a CUDA crash path, restart the same build with coredumps:
+
+```bash
+SGLANG_CUDA_COREDUMP=1 \
+SGLANG_CUDA_COREDUMP_DIR=/tmp/sglang_cuda_coredumps \
+python -m sglang.launch_server \
+  --model-path ... \
+  --crash-dump-folder /tmp/sglang_crash_dump \
+  ...
+```
+
+Then inspect the coredump:
+
+```bash
+cuda-gdb "$(which python3)" \
+  -ex "target cudacore /tmp/sglang_cuda_coredumps/cuda_coredump_<host>.<pid>.<ts>"
+```
+
+Good first commands:
+
+- `where`
+- `info cuda kernels`
+- `x/10i <pc>`
+
+Use the coredump to find the failing kernel, not automatically the root-cause
+kernel.
+
+See:
+
+- [case-studies.md](case-studies.md)
+
+## Trace
+
+Tracing must be enabled at startup:
+
+```bash
+python -m sglang.launch_server \
+  --enable-trace \
+  --otlp-traces-endpoint localhost:4317 \
+  ...
+```
+
+Optional router command:
+
+```bash
+python -m sglang_router.launch_router \
+  --enable-trace \
+  --otlp-traces-endpoint localhost:4317 \
+  ...
+```
+
+Useful environment variables:
+
+```bash
+export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500
+export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64
+```
+
+If tracing is already enabled, change the level without restart:
+
+```bash
+curl "http://127.0.0.1:30000/set_trace_level?level=1"
+curl "http://127.0.0.1:30000/set_trace_level?level=2"
+curl "http://127.0.0.1:30000/set_trace_level?level=3"
+```
+
+Use tracing for:
+
+- router vs. worker delay
+- tokenizer / scheduler / detokenizer timing
+- PD transfer timing
+- request timing across processes
+
+If you already have OTEL JSON or JSONL, convert it for timeline inspection:
+
+```bash
+python3 scripts/convert_otel_2_perfetto.py \
+  --input /tmp/otel_trace.json \
+  --output /tmp/sglang_trace_perfetto.json
+```
+
+## Torch Profiler
+
+Switch to `llm-torch-profiler-analysis` when:
+
+- replay already reproduces the issue
+- metrics and loads do not explain it
+- the problem now looks compute-side
+
+This skill should decide when to profile, not duplicate the profiler workflow.
+
+## Bisect
+
+If one commit is known-good and a newer commit is known-bad:
+
+1. build a deterministic harness from the problem
+2. prefer replay-based harnesses when the failure depends on request mix
+3. use `git bisect run <harness>`
+4. only then go back to trace or profile if needed
+
+Example:
+
+```bash
+git bisect start <bad> <good>
+git bisect run bash ./repro_or_check.sh
+```
+
+## Common Paths
+
+### Crash
+
+1. crash dump
+2. summarize dump
+3. replay
+4. CUDA coredump plus `cuda-gdb`
+5. `debug-cuda-crash` or narrower instrumentation
+
+### TTFT regression
+
+1. baseline metrics and loads
+2. request dump
+3. replay the slow request
+4. trace if stage ownership is unclear
+5. `llm-torch-profiler-analysis` if it still looks compute-side
+
+See:
+
+- [case-studies.md](case-studies.md)
+
+### Distributed hang
+
+1. healthy baseline bundle
+2. save the trigger request
+3. replay on a clean target
+4. collect replay-time bundle and stacks
+5. identify the NCCL or collective path
+6. switch to `debug-distributed-hang`
+
+See:
+
+- [case-studies.md](case-studies.md)
+
+### Throughput regression after deploy
+
+1. compare `server_info`
+2. compare `/metrics` and `/v1/loads`
+3. replay stable workload
+4. bisect if one older commit is known-good
+5. profile only if compute still looks suspicious

.claude/skills/sglang-prod-incident-triage/scripts/incident_artifact_tool.py

@@ -0,0 +1,735 @@
+#!/usr/bin/env python3
+"""Collect or inspect serving bundles and dumps for SGLang debug."""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import math
+import os
+import pickle
+import re
+import time
+from collections import defaultdict
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, Optional, Sequence
+from urllib import error, parse, request
+
+METRIC_RE = re.compile(
+    r"^(?P<name>[^{\s]+)(?:\{(?P<labels>[^}]*)\})?\s+(?P<value>[-+]?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)$"
+)
+LABEL_RE = re.compile(r'([a-zA-Z_:][a-zA-Z0-9_:]*)="((?:[^"\\]|\\.)*)"')
+ENDPOINT_SPECS = (
+    ("text", "health.txt", "/health"),
+    ("text", "health_generate.txt", "/health_generate"),
+    ("text", "metrics.txt", "/metrics"),
+    ("json", "model_info.json", "/model_info"),
+    ("json", "server_info.json", "/server_info"),
+    ("json", "loads_all.json", "/v1/loads?include=all"),
+    (
+        "json",
+        "loads_core_queues_disagg.json",
+        "/v1/loads?include=core,queues,disagg,spec",
+    ),
+    ("json", "hicache_storage_backend.json", "/hicache/storage-backend"),
+)
+BUNDLE_NOTES = [
+    "This bundle is read-only. It does not start profiling or change trace level.",
+    "HiCache status may fail if admin_api_key is not configured or the wrong bearer token was used.",
+    "loads_all.json is the best point-in-time load snapshot in this bundle.",
+    "metrics.txt is raw Prometheus text intended for follow-up parsing.",
+]
+
+
+def request_text(
+    base_url: str,
+    path: str,
+    token: Optional[str],
+    timeout: float = 10.0,
+) -> tuple[bool, int, str]:
+    url = parse.urljoin(base_url.rstrip("/") + "/", path.lstrip("/"))
+    req = request.Request(url)
+    if token:
+        req.add_header("Authorization", f"Bearer {token}")
+    try:
+        with request.urlopen(req, timeout=timeout) as resp:
+            body = resp.read().decode("utf-8", errors="replace")
+            return True, resp.status, body
+    except error.HTTPError as e:
+        body = e.read().decode("utf-8", errors="replace")
+        return False, e.code, body
+    except Exception as e:  # noqa: BLE001
+        return False, -1, f"{type(e).__name__}: {e}"
+
+
+def request_endpoint(
+    base_url: str,
+    path: str,
+    token: Optional[str],
+    parse_json: bool,
+    timeout: float = 10.0,
+) -> Dict[str, Any]:
+    ok, status, body = request_text(base_url, path, token, timeout=timeout)
+    result: Dict[str, Any] = {"ok": ok, "status": status, "path": path}
+    if not ok:
+        result["error"] = body
+        return result
+    if not parse_json:
+        result["text"] = body
+        return result
+    try:
+        result["json"] = json.loads(body)
+    except json.JSONDecodeError:
+        result["text"] = body
+        result["decode_error"] = "response was not valid JSON"
+    return result
+
+
+def write_json(path: Path, obj: Dict[str, Any]) -> None:
+    path.write_text(
+        json.dumps(obj, indent=2, ensure_ascii=False) + "\n", encoding="utf-8"
+    )
+
+
+def write_text(path: Path, text: str) -> None:
+    path.write_text(text, encoding="utf-8")
+
+
+def format_summary_line(filename: str, result: Dict[str, Any]) -> str:
+    if result.get("ok"):
+        return f"{filename}: ok"
+    return (
+        f"{filename}: failed status={result.get('status')} "
+        f"error={result.get('error')}"
+    )
+
+
+def collect_bundle(
+    base_url: str,
+    token: Optional[str],
+    outdir: Optional[str],
+    timeout: float,
+) -> Path:
+    timestamp = time.strftime("%Y%m%d_%H%M%S")
+    bundle_dir = Path(outdir or f"./incident_bundle_{timestamp}").resolve()
+    bundle_dir.mkdir(parents=True, exist_ok=True)
+
+    metadata = {
+        "artifact_type": "incident_bundle",
+        "base_url": base_url,
+        "collected_at": timestamp,
+        "token_provided": bool(token),
+        "timeout_seconds": timeout,
+    }
+    write_json(bundle_dir / "metadata.json", metadata)
+
+    summary_lines = []
+    for kind, filename, path in ENDPOINT_SPECS:
+        result = request_endpoint(
+            base_url, path, token, parse_json=(kind == "json"), timeout=timeout
+        )
+        output_path = bundle_dir / filename
+        if kind == "text" and result.get("ok"):
+            write_text(output_path, str(result.get("text", "")))
+        else:
+            write_json(
+                (
+                    output_path
+                    if kind == "json"
+                    else bundle_dir / f"{filename}.error.json"
+                ),
+                result,
+            )
+        summary_lines.append(format_summary_line(filename, result))
+
+    write_text(
+        bundle_dir / "SUMMARY.txt",
+        "\n".join(summary_lines + [""] + BUNDLE_NOTES) + "\n",
+    )
+    return bundle_dir
+
+
+def load_json(path: Path) -> Optional[Dict[str, Any]]:
+    if not path.exists():
+        return None
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def unwrap_result(path: Path) -> Optional[Dict[str, Any]]:
+    obj = load_json(path)
+    if obj is None:
+        return None
+    if isinstance(obj, dict) and "json" in obj:
+        return obj.get("json")
+    return obj
+
+
+def read_text(path: Path) -> Optional[str]:
+    if not path.exists():
+        return None
+    return path.read_text(encoding="utf-8")
+
+
+def endpoint_ok(bundle_dir: Path, stem: str) -> bool:
+    return (bundle_dir / f"{stem}.txt").exists() and not (
+        bundle_dir / f"{stem}.txt.error.json"
+    ).exists()
+
+
+def parse_labels(raw: Optional[str]) -> Dict[str, str]:
+    if not raw:
+        return {}
+    labels = {}
+    for key, value in LABEL_RE.findall(raw):
+        labels[key] = bytes(value, "utf-8").decode("unicode_escape")
+    return labels
+
+
+def parse_metrics(metrics_text: str) -> Dict[str, list[dict[str, Any]]]:
+    series: Dict[str, list[dict[str, Any]]] = defaultdict(list)
+    for line in metrics_text.splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        match = METRIC_RE.match(line)
+        if not match:
+            continue
+        series[match.group("name")].append(
+            {
+                "labels": parse_labels(match.group("labels")),
+                "value": float(match.group("value")),
+            }
+        )
+    return series
+
+
+def metric_sum(metrics: Dict[str, list[dict[str, Any]]], name: str) -> float:
+    return sum(item["value"] for item in metrics.get(name, []))
+
+
+def safe_div(
+    numerator: Optional[float], denominator: Optional[float]
+) -> Optional[float]:
+    if numerator is None or denominator in (None, 0):
+        return None
+    return numerator / denominator
+
+
+def coalesce(*values: Any) -> Any:
+    for value in values:
+        if value is not None:
+            return value
+    return None
+
+
+def fmt_float(value: Optional[float], digits: int = 3) -> str:
+    if value is None or (
+        isinstance(value, float) and (math.isnan(value) or math.isinf(value))
+    ):
+        return "n/a"
+    return f"{value:.{digits}f}"
+
+
+def is_positive_number(value: Any, threshold: float = 0.0) -> bool:
+    return (
+        isinstance(value, (int, float))
+        and not math.isnan(value)
+        and not math.isinf(value)
+        and value > threshold
+    )
+
+
+def compute_stage_averages(
+    metrics: Dict[str, list[dict[str, Any]]], sum_name: str, count_name: str
+) -> Dict[str, float]:
+    grouped_sum: Dict[str, float] = defaultdict(float)
+    grouped_count: Dict[str, float] = defaultdict(float)
+    for item in metrics.get(sum_name, []):
+        stage = item["labels"].get("stage", "")
+        rank = item["labels"].get("tp_rank", "")
+        grouped_sum[f"{stage}|{rank}"] += item["value"]
+    for item in metrics.get(count_name, []):
+        stage = item["labels"].get("stage", "")
+        rank = item["labels"].get("tp_rank", "")
+        grouped_count[f"{stage}|{rank}"] += item["value"]
+
+    result: Dict[str, float] = {}
+    for key, total_sum in grouped_sum.items():
+        stage, _rank = key.split("|", 1)
+        avg = safe_div(total_sum, grouped_count.get(key))
+        if avg is None:
+            continue
+        result[stage] = max(result.get(stage, 0.0), avg)
+    return result
+
+
+def add_signal(signals: list[str], text: str) -> None:
+    if text not in signals:
+        signals.append(text)
+
+
+def build_bundle_summary(bundle_dir: Path) -> Dict[str, Any]:
+    metadata = load_json(bundle_dir / "metadata.json") or {}
+    model_info = unwrap_result(bundle_dir / "model_info.json") or {}
+    server_info = unwrap_result(bundle_dir / "server_info.json") or {}
+    loads_info = unwrap_result(bundle_dir / "loads_all.json") or {}
+    metrics_text = read_text(bundle_dir / "metrics.txt") or ""
+    metrics = parse_metrics(metrics_text)
+
+    aggregate = loads_info.get("aggregate") or {}
+    loads = loads_info.get("loads") or []
+    load0 = loads[0] if loads else {}
+    internal_states = server_info.get("internal_states") or []
+    runtime_state = internal_states[0] if internal_states else {}
+    memory_usage = runtime_state.get("memory_usage") or load0.get("memory") or {}
+
+    ttft_avg = safe_div(
+        metric_sum(metrics, "sglang:time_to_first_token_seconds_sum"),
+        metric_sum(metrics, "sglang:time_to_first_token_seconds_count"),
+    )
+    e2e_avg = safe_div(
+        metric_sum(metrics, "sglang:e2e_request_latency_seconds_sum"),
+        metric_sum(metrics, "sglang:e2e_request_latency_seconds_count"),
+    )
+    queue_avg = safe_div(
+        metric_sum(metrics, "sglang:queue_time_seconds_sum"),
+        metric_sum(metrics, "sglang:queue_time_seconds_count"),
+    )
+    per_stage_avg = compute_stage_averages(
+        metrics,
+        "sglang:per_stage_req_latency_seconds_sum",
+        "sglang:per_stage_req_latency_seconds_count",
+    )
+
+    summary: Dict[str, Any] = {
+        "artifact_type": "incident_bundle",
+        "bundle_dir": str(bundle_dir),
+        "base_url": metadata.get("base_url"),
+        "collected_at": metadata.get("collected_at"),
+        "health": {
+            "health_ok": endpoint_ok(bundle_dir, "health"),
+            "health_generate_ok": endpoint_ok(bundle_dir, "health_generate"),
+        },
+        "model": {
+            "model_path": model_info.get("model_path") or server_info.get("model_path"),
+            "served_model_name": server_info.get("served_model_name"),
+            "weight_version": model_info.get("weight_version")
+            or server_info.get("weight_version"),
+            "model_type": model_info.get("model_type"),
+            "is_generation": model_info.get("is_generation"),
+        },
+        "topology": {
+            "tp_size": server_info.get("tp_size"),
+            "dp_size": server_info.get("dp_size"),
+            "pp_size": server_info.get("pp_size"),
+            "ep_size": server_info.get("ep_size"),
+            "disaggregation_mode": server_info.get("disaggregation_mode"),
+            "attention_backend": server_info.get("attention_backend"),
+            "sampling_backend": server_info.get("sampling_backend"),
+            "schedule_policy": server_info.get("schedule_policy"),
+            "enable_trace": server_info.get("enable_trace"),
+            "enable_metrics": server_info.get("enable_metrics"),
+        },
+        "capacity": {
+            "max_total_num_tokens": server_info.get("max_total_num_tokens"),
+            "max_req_input_len": server_info.get("max_req_input_len"),
+            "effective_max_running_requests_per_dp": coalesce(
+                runtime_state.get("effective_max_running_requests_per_dp"),
+                load0.get("max_running_requests"),
+            ),
+            "weight_gb": coalesce(
+                memory_usage.get("weight"), memory_usage.get("weight_gb")
+            ),
+            "kv_cache_gb": coalesce(
+                memory_usage.get("kvcache"), memory_usage.get("kv_cache_gb")
+            ),
+            "graph_gb": coalesce(
+                memory_usage.get("graph"), memory_usage.get("graph_gb")
+            ),
+            "token_capacity": memory_usage.get("token_capacity"),
+        },
+        "point_in_time_load": {
+            "running_reqs": coalesce(
+                aggregate.get("total_running_reqs"), load0.get("num_running_reqs")
+            ),
+            "waiting_reqs": coalesce(
+                aggregate.get("total_waiting_reqs"), load0.get("num_waiting_reqs")
+            ),
+            "total_reqs": coalesce(
+                aggregate.get("total_reqs"), load0.get("num_total_reqs")
+            ),
+            "token_usage": coalesce(
+                aggregate.get("avg_token_usage"), load0.get("token_usage")
+            ),
+            "avg_throughput": coalesce(
+                aggregate.get("avg_throughput"), load0.get("gen_throughput")
+            ),
+            "avg_utilization": coalesce(
+                aggregate.get("avg_utilization"), load0.get("utilization")
+            ),
+            "cache_hit_rate": load0.get("cache_hit_rate"),
+            "queues": load0.get("queues"),
+            "disaggregation": load0.get("disaggregation"),
+        },
+        "metrics": {
+            "request_count": metric_sum(metrics, "sglang:num_requests_total"),
+            "prompt_tokens_total": metric_sum(metrics, "sglang:prompt_tokens_total"),
+            "generation_tokens_total": metric_sum(
+                metrics, "sglang:generation_tokens_total"
+            ),
+            "avg_ttft_seconds": ttft_avg,
+            "avg_e2e_seconds": e2e_avg,
+            "avg_queue_time_seconds": queue_avg,
+            "stage_avg_seconds_max_tp_rank": per_stage_avg,
+        },
+        "signals": [],
+    }
+
+    signals = summary["signals"]
+    health = summary["health"]
+    point_in_time_load = summary["point_in_time_load"]
+    running_reqs = point_in_time_load.get("running_reqs")
+    waiting_reqs = point_in_time_load.get("waiting_reqs")
+
+    if health["health_ok"] and not health["health_generate_ok"]:
+        add_signal(
+            signals,
+            "/health is green but /health_generate failed. Suspect runtime or scheduler path, not just HTTP liveness.",
+        )
+    if not health["health_ok"]:
+        add_signal(
+            signals,
+            "/health failed. Start with startup, crash, or global unhealthy paths.",
+        )
+    if is_positive_number(waiting_reqs):
+        add_signal(
+            signals,
+            f"Point-in-time load shows queue buildup: waiting_reqs={waiting_reqs}.",
+        )
+    if (
+        point_in_time_load.get("token_usage") is not None
+        and point_in_time_load["token_usage"] >= 0.9
+    ):
+        add_signal(
+            signals,
+            "Token usage is near saturation. KV or token-capacity pressure may explain latency.",
+        )
+    if (
+        ttft_avg is not None
+        and queue_avg is not None
+        and ttft_avg > 2.0
+        and queue_avg < 0.2
+    ):
+        add_signal(
+            signals,
+            f"Average TTFT is high ({fmt_float(ttft_avg)}s) while average queue time is low ({fmt_float(queue_avg)}s). This looks more like prefill or request-path work than queue pressure.",
+        )
+    prefill_forward = per_stage_avg.get("prefill_forward")
+    request_process = per_stage_avg.get("request_process")
+    if (
+        prefill_forward is not None
+        and request_process is not None
+        and prefill_forward > max(0.5, request_process * 10)
+    ):
+        add_signal(
+            signals,
+            f"Prefill forward dominates quick stage timing: prefill_forward~{fmt_float(prefill_forward)}s vs request_process~{fmt_float(request_process)}s.",
+        )
+    if running_reqs == 0 and waiting_reqs == 0:
+        add_signal(
+            signals,
+            "Bundle snapshot was captured while the server was effectively idle. Reproduce under live traffic or replayed workload if the problem is intermittent.",
+        )
+
+    return summary
+
+
+def render_bundle_text(summary: Dict[str, Any]) -> str:
+    health = summary["health"]
+    model = summary["model"]
+    topology = summary["topology"]
+    capacity = summary["capacity"]
+    load = summary["point_in_time_load"]
+    metrics = summary["metrics"]
+    stage_avgs = metrics["stage_avg_seconds_max_tp_rank"]
+
+    lines = [
+        f"Bundle: {summary['bundle_dir']}",
+        f"Base URL: {summary.get('base_url') or 'n/a'}",
+        f"Collected At: {summary.get('collected_at') or 'n/a'}",
+        "",
+        f"Health: /health={'ok' if health['health_ok'] else 'failed'} /health_generate={'ok' if health['health_generate_ok'] else 'failed'}",
+        f"Model: {model.get('model_path') or 'n/a'} weight_version={model.get('weight_version') or 'n/a'} type={model.get('model_type') or 'n/a'}",
+        "Topology: "
+        f"tp={topology.get('tp_size')} dp={topology.get('dp_size')} pp={topology.get('pp_size')} ep={topology.get('ep_size')} "
+        f"disagg={topology.get('disaggregation_mode')} trace={topology.get('enable_trace')} metrics={topology.get('enable_metrics')}",
+        "Capacity: "
+        f"max_total_tokens={capacity.get('max_total_num_tokens')} "
+        f"max_running_reqs={capacity.get('effective_max_running_requests_per_dp')} "
+        f"weight_gb={fmt_float(capacity.get('weight_gb'))} "
+        f"kv_cache_gb={fmt_float(capacity.get('kv_cache_gb'))} "
+        f"graph_gb={fmt_float(capacity.get('graph_gb'))}",
+        "Point-in-time load: "
+        f"running={load.get('running_reqs')} waiting={load.get('waiting_reqs')} total={load.get('total_reqs')} "
+        f"token_usage={fmt_float(load.get('token_usage'))} throughput={fmt_float(load.get('avg_throughput'))} "
+        f"cache_hit_rate={fmt_float(load.get('cache_hit_rate'))}",
+        "Metrics: "
+        f"requests={fmt_float(metrics.get('request_count'), 0)} "
+        f"prompt_tokens={fmt_float(metrics.get('prompt_tokens_total'), 0)} "
+        f"generation_tokens={fmt_float(metrics.get('generation_tokens_total'), 0)} "
+        f"avg_ttft_s={fmt_float(metrics.get('avg_ttft_seconds'))} "
+        f"avg_e2e_s={fmt_float(metrics.get('avg_e2e_seconds'))} "
+        f"avg_queue_s={fmt_float(metrics.get('avg_queue_time_seconds'))}",
+    ]
+
+    if stage_avgs:
+        stage_parts = [
+            f"{name}={fmt_float(value)}s" for name, value in sorted(stage_avgs.items())
+        ]
+        lines.append("Stage Averages (max across TP ranks): " + ", ".join(stage_parts))
+
+    queues = load.get("queues") or {}
+    if queues:
+        lines.append(
+            "Queues: "
+            + ", ".join(f"{key}={value}" for key, value in sorted(queues.items()))
+        )
+
+    disagg = load.get("disaggregation") or {}
+    if disagg:
+        lines.append(
+            "Disaggregation: "
+            + ", ".join(f"{key}={value}" for key, value in sorted(disagg.items()))
+        )
+
+    lines.append("")
+    lines.append("What stands out:")
+    if summary["signals"]:
+        lines.extend(f"- {signal}" for signal in summary["signals"])
+    else:
+        lines.append("- No strong signal from this bundle.")
+
+    return "\n".join(lines) + "\n"
+
+
+def get_field(obj: Any, name: str, default: Any = None) -> Any:
+    if obj is None:
+        return default
+    if isinstance(obj, dict):
+        return obj.get(name, default)
+    return getattr(obj, name, default)
+
+
+def iter_dump_files(
+    input_file: Optional[str], input_folder: Optional[str]
+) -> Sequence[Path]:
+    if input_file:
+        return [Path(input_file)]
+    if input_folder:
+        return [Path(p) for p in sorted(glob.glob(f"{input_folder}/*.pkl"))]
+    raise SystemExit("Either --input-file or --input-folder must be provided.")
+
+
+def load_dump_payload(path: Path) -> dict[str, Any]:
+    with path.open("rb") as fh:
+        payload = pickle.load(fh)
+    if isinstance(payload, dict):
+        return payload
+    return {"requests": payload}
+
+
+def pick_text_preview(req: Any) -> str:
+    candidates = [
+        get_field(req, "origin_input_text"),
+        get_field(req, "text"),
+        get_field(req, "prompt"),
+    ]
+    for value in candidates:
+        if isinstance(value, str) and value:
+            return value
+        if isinstance(value, list) and value:
+            first = value[0]
+            if isinstance(first, str) and first:
+                return first
+    return ""
+
+
+def format_timestamp(ts: Any) -> str:
+    if not isinstance(ts, (int, float)):
+        return "n/a"
+    return datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
+
+
+def summarize_request(
+    record: tuple[Any, dict[str, Any], Any, Any], idx: int, preview_chars: int
+) -> list[str]:
+    req, output, start_time, end_time = record
+    preview = pick_text_preview(req).replace("\n", " ").strip()
+    if len(preview) > preview_chars:
+        preview = preview[: preview_chars - 3] + "..."
+
+    output_dict = output if isinstance(output, dict) else {}
+    meta_info = get_field(output_dict, "meta_info", {}) or {}
+    rid = get_field(req, "rid") or get_field(meta_info, "id")
+    stream = bool(get_field(req, "stream", False))
+    prompt_tokens = get_field(meta_info, "prompt_tokens")
+    completion_tokens = get_field(meta_info, "completion_tokens")
+    duration = (
+        end_time - start_time
+        if isinstance(start_time, (int, float)) and isinstance(end_time, (int, float))
+        else None
+    )
+
+    elapsed_str = f"{duration:.3f}" if duration is not None else "n/a"
+    lines = [
+        f"[{idx}] rid={rid or 'n/a'} stream={stream} "
+        f"prompt_tokens={prompt_tokens if prompt_tokens is not None else 'n/a'} "
+        f"completion_tokens={completion_tokens if completion_tokens is not None else 'n/a'} "
+        f"start={format_timestamp(start_time)} elapsed_s={elapsed_str}"
+    ]
+    if preview:
+        lines.append(f"      text={preview}")
+    return lines
+
+
+def summarize_dump_file(path: Path, max_requests: int, preview_chars: int) -> str:
+    payload = load_dump_payload(path)
+    requests = payload.get("requests") or []
+    server_args = payload.get("server_args")
+    launch_command = payload.get("launch_command")
+
+    model_path = get_field(server_args, "model_path")
+    tp_size = get_field(server_args, "tp_size")
+    dp_size = get_field(server_args, "dp_size")
+    pp_size = get_field(server_args, "pp_size")
+    host = get_field(server_args, "host")
+    port = get_field(server_args, "port")
+
+    timestamps = [
+        record[2]
+        for record in requests
+        if isinstance(record, tuple)
+        and len(record) >= 4
+        and isinstance(record[2], (int, float))
+    ]
+    time_span = (
+        max(timestamps) - min(timestamps)
+        if len(timestamps) >= 2
+        else 0.0 if len(timestamps) == 1 else None
+    )
+
+    lines = [
+        f"File: {path}",
+        "Dump Type: request_or_crash_dump",
+        f"Requests: {len(requests)}",
+        f"Model: {model_path or 'n/a'}",
+        f"Topology: tp={tp_size if tp_size is not None else 'n/a'} "
+        f"dp={dp_size if dp_size is not None else 'n/a'} "
+        f"pp={pp_size if pp_size is not None else 'n/a'}",
+        f"Endpoint: {host or 'n/a'}:{port if port is not None else 'n/a'}",
+        (
+            f"Time span seconds: {time_span:.3f}"
+            if time_span is not None
+            else "Time span seconds: n/a"
+        ),
+    ]
+    if launch_command:
+        lines.append(f"Launch command: {launch_command}")
+
+    for idx, record in enumerate(requests[:max_requests]):
+        if not isinstance(record, tuple) or len(record) < 4:
+            lines.append(f"[{idx}] Unsupported record shape: {type(record)!r}")
+            continue
+        lines.extend(summarize_request(record, idx, preview_chars))
+
+    if len(requests) > max_requests:
+        lines.append(f"... truncated {len(requests) - max_requests} more requests")
+    return "\n".join(lines)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Collect or inspect serving bundles and dumps for SGLang debug."
+    )
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    collect_parser = subparsers.add_parser(
+        "collect-bundle", help="Collect a read-only live bundle from a running server"
+    )
+    collect_parser.add_argument("--base-url", required=True)
+    collect_parser.add_argument(
+        "--token",
+        default=os.environ.get("SGLANG_BEARER_TOKEN"),
+        help="Bearer token for protected endpoints. Defaults to $SGLANG_BEARER_TOKEN.",
+    )
+    collect_parser.add_argument("--outdir", default=None)
+    collect_parser.add_argument("--timeout", type=float, default=10.0)
+
+    bundle_parser = subparsers.add_parser(
+        "summarize-bundle", help="Summarize a bundle directory"
+    )
+    bundle_parser.add_argument("bundle_dir")
+    bundle_parser.add_argument("--out", default=None)
+    bundle_parser.add_argument("--json-out", default=None)
+    bundle_parser.add_argument("--stdout-json", action="store_true")
+
+    dump_parser = subparsers.add_parser(
+        "summarize-dump", help="Summarize a trusted request dump or crash dump"
+    )
+    dump_parser.add_argument("--input-file", default=None)
+    dump_parser.add_argument("--input-folder", default=None)
+    dump_parser.add_argument("--max-requests", type=int, default=20)
+    dump_parser.add_argument("--preview-chars", type=int, default=160)
+
+    args = parser.parse_args()
+
+    if args.command == "collect-bundle":
+        bundle_dir = collect_bundle(
+            args.base_url, args.token, args.outdir, args.timeout
+        )
+        print(bundle_dir)
+        return 0
+
+    if args.command == "summarize-bundle":
+        bundle_dir = Path(args.bundle_dir).resolve()
+        if not bundle_dir.is_dir():
+            raise SystemExit(
+                f"bundle_dir does not exist or is not a directory: {bundle_dir}"
+            )
+        summary = build_bundle_summary(bundle_dir)
+        out_text = render_bundle_text(summary)
+        text_path = Path(args.out) if args.out else bundle_dir / "SUMMARY_REPORT.txt"
+        json_path = (
+            Path(args.json_out) if args.json_out else bundle_dir / "SUMMARY_REPORT.json"
+        )
+        text_path.write_text(out_text, encoding="utf-8")
+        json_path.write_text(
+            json.dumps(summary, indent=2, ensure_ascii=False) + "\n",
+            encoding="utf-8",
+        )
+        if args.stdout_json:
+            print(json.dumps(summary, indent=2, ensure_ascii=False))
+        else:
+            print(out_text, end="")
+        return 0
+
+    files = iter_dump_files(args.input_file, args.input_folder)
+    if not files:
+        raise SystemExit("No .pkl files matched the provided input.")
+    for idx, path in enumerate(files):
+        if idx:
+            print()
+        print(
+            summarize_dump_file(
+                path=path,
+                max_requests=args.max_requests,
+                preview_chars=args.preview_chars,
+            )
+        )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.claude/skills/sglang-prod-incident-triage/scripts/replay_trusted_request_dump.py

@@ -0,0 +1,219 @@
+#!/usr/bin/env python3
+"""Replay a trusted SGLang request dump directly over HTTP.
+
+Use this only for locally captured or otherwise trusted dump files.
+It uses plain pickle loading to bypass SafeUnpickler restrictions that may block
+the stock replay helper on newer SGLang builds.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import pickle
+import time
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import asdict, is_dataclass
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Sequence
+
+import requests
+
+Record = tuple[object, dict[str, Any], float, float]
+
+
+def normalize_mm_data_item(item: Any) -> Any:
+    if isinstance(item, dict) and "url" in item:
+        return item["url"]
+    return item
+
+
+def normalize_mm_data(data: Any) -> Any:
+    if data is None:
+        return None
+    if isinstance(data, list):
+        return [
+            (
+                [normalize_mm_data_item(item) for item in sublist]
+                if isinstance(sublist, list)
+                else normalize_mm_data_item(sublist)
+            )
+            for sublist in data
+        ]
+    return normalize_mm_data_item(data)
+
+
+def normalize_request_data(json_data: dict[str, Any]) -> dict[str, Any]:
+    for field in ["image_data", "video_data", "audio_data"]:
+        if field in json_data and json_data[field] is not None:
+            json_data[field] = normalize_mm_data(json_data[field])
+    return json_data
+
+
+def to_plain_dict(obj: Any) -> dict[str, Any]:
+    if obj is None:
+        return {}
+    if isinstance(obj, dict):
+        return dict(obj)
+    if is_dataclass(obj):
+        return asdict(obj)
+
+    model_dump = getattr(obj, "model_dump", None)
+    if callable(model_dump):
+        dumped = model_dump()
+        if isinstance(dumped, dict):
+            return dumped
+
+    dict_method = getattr(obj, "dict", None)
+    if callable(dict_method):
+        dumped = dict_method()
+        if isinstance(dumped, dict):
+            return dumped
+
+    obj_dict = getattr(obj, "__dict__", None)
+    if isinstance(obj_dict, dict):
+        return {
+            key: value for key, value in obj_dict.items() if not key.startswith("_")
+        }
+
+    raise TypeError(f"Unsupported request object type: {type(obj)!r}")
+
+
+def request_to_json_data(req: Any) -> dict[str, Any]:
+    json_data = normalize_request_data(to_plain_dict(req))
+    sampling_params = json_data.get("sampling_params")
+    if sampling_params is not None and not isinstance(sampling_params, dict):
+        json_data["sampling_params"] = to_plain_dict(sampling_params)
+    return json_data
+
+
+def load_records(path: Path) -> list[Record]:
+    with path.open("rb") as fh:
+        payload = pickle.load(fh)
+    if isinstance(payload, dict) and "requests" in payload:
+        return payload["requests"]
+    return payload
+
+
+def iter_files(args: argparse.Namespace) -> Sequence[Path]:
+    if args.input_file:
+        return [Path(args.input_file)]
+    if args.input_folder:
+        return [
+            Path(p)
+            for p in sorted(glob.glob(f"{args.input_folder}/*.pkl"))[: args.file_number]
+        ]
+    raise SystemExit("Either --input-file or --input-folder must be provided.")
+
+
+def run_one_request(
+    record: Record,
+    args: argparse.Namespace,
+    replay_init_time: float,
+    base_time: float,
+    idx: int,
+) -> None:
+    req, output, start_time, end_time = record
+    relative_start = start_time - base_time
+    delay = max(0.0, (relative_start - (time.time() - replay_init_time)) / args.speed)
+    if delay:
+        time.sleep(delay)
+
+    json_data = request_to_json_data(req)
+    if args.ignore_eos:
+        json_data.setdefault("sampling_params", {})["ignore_eos"] = True
+        completion_tokens = output.get("meta_info", {}).get("completion_tokens")
+        if completion_tokens:
+            json_data["sampling_params"]["max_new_tokens"] = completion_tokens
+
+    t0 = time.time()
+    response = requests.post(
+        f"http://{args.host}:{args.port}/generate",
+        json=json_data,
+        timeout=args.timeout,
+        stream=bool(json_data.get("stream")),
+    )
+    elapsed = time.time() - t0
+
+    if json_data.get("stream"):
+        last = None
+        for chunk in response.iter_lines(decode_unicode=False):
+            decoded = chunk.decode("utf-8")
+            if decoded and decoded.startswith("data:"):
+                if decoded == "data: [DONE]":
+                    break
+                last = json.loads(decoded[5:].strip())
+        result = last or {}
+    else:
+        result = response.json()
+
+    meta = result.get("meta_info", {})
+    print(
+        json.dumps(
+            {
+                "idx": idx,
+                "status_code": response.status_code,
+                "elapsed_seconds": round(elapsed, 3),
+                "prompt_tokens": meta.get("prompt_tokens"),
+                "completion_tokens": meta.get("completion_tokens"),
+                "rid": meta.get("id"),
+            },
+            ensure_ascii=False,
+        )
+    )
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Replay a trusted SGLang request dump or crash dump directly over HTTP."
+    )
+    parser.add_argument("--host", default="127.0.0.1")
+    parser.add_argument("--port", type=int, default=30000)
+    parser.add_argument("--input-folder", default=None)
+    parser.add_argument("--input-file", default=None)
+    parser.add_argument("--file-number", type=int, default=1)
+    parser.add_argument("--req-number", type=int, default=1_000_000)
+    parser.add_argument("--req-start", type=int, default=0)
+    parser.add_argument("--parallel", type=int, default=1)
+    parser.add_argument("--ignore-eos", action="store_true")
+    parser.add_argument("--speed", type=float, default=1.0)
+    parser.add_argument("--timeout", type=float, default=120.0)
+    args = parser.parse_args()
+
+    files = iter_files(args)
+    print(f"Replay files: {[str(p) for p in files]}")
+
+    records: list[Record] = []
+    for path in files:
+        records.extend(load_records(path))
+
+    if not records:
+        print("No requests found.")
+        return 0
+
+    records.sort(key=lambda x: x[-2])
+    records = records[args.req_start : args.req_start + args.req_number]
+    print(f"Replay requests: {len(records)}")
+    base_time = records[0][-2]
+    print(
+        "Base time: " + datetime.fromtimestamp(base_time).strftime("%Y-%m-%d %H:%M:%S")
+    )
+
+    replay_init_time = time.time()
+    with ThreadPoolExecutor(max_workers=args.parallel) as executor:
+        futures = []
+        for idx, record in enumerate(records):
+            futures.append(
+                executor.submit(
+                    run_one_request, record, args, replay_init_time, base_time, idx
+                )
+            )
+        for future in futures:
+            future.result()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

.claude/skills/sglang-runtime-context/SKILL.md

@@ -0,0 +1,195 @@
+---
+name: sglang-runtime-context
+description: How SGLang's runtime configuration and process-global state are organized (RuntimeContext tiers, resolve-at-end ServerArgs, override entry point, resource/stream/buffer leases, per-forward flags), the CI guardrails that enforce the design, and the idioms for developing and testing against it. Load this before touching server_args, model overrides, module-level state, or per-forward state in sglang.
+---
+
+# SGLang runtime-context architecture
+
+One container owns process-static runtime state: `sglang.srt.runtime_context.RuntimeContext`
+(a process singleton reached via `get_context()`). Everything below is a tier on it.
+
+| Tier | Accessor | Holds | Lifecycle |
+|------|----------|-------|-----------|
+| config | `get_server_args()` | the process-wide **resolved** `ServerArgs` | resolved once in `__post_init__`; audited mutations only |
+| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
+| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
+| per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults |
+| parallel | `get_parallel()` | read-through wrapper over topology getters (tp/pp/moe/attn sizes, ranks, groups) | stateless; `override()` for tests |
+
+`reset_context()` (unit-test teardown) drops the published server_args and installs fresh
+flags/resources/forward tiers.
+
+## Config: the resolve-at-end contract
+
+**After `ServerArgs.__post_init__` returns, the fields ARE the resolved configuration.**
+Model overrides and normalization passes *declare* values during resolution (into a
+provenance stash); `materialize_declarations()` applies them once at the very end of
+`__post_init__` (gate order, last writer wins). Consequences:
+
+- **Reading config**: read fields directly, in any process, at any time after construction.
+  For global access use `runtime_context.get_server_args()` (the blessed accessor —
+  `get_global_server_args()` is a legacy shim over the same slot and its call-site count is
+  ratcheted; do not add new ones).
+- **Mutating config after resolution**: the ONLY entry point is
+  `ServerArgs.override(source, **fields)`. It records provenance (`_runtime_mutations`),
+  keeps whitelisted resolvable fields consistent with the declaration stash (so a republish
+  resolves the same values), and bypasses the strict guard. Bare `server_args.x = ...`
+  after resolution **raises** under `SGLANG_STRICT_CONFIG_MUTATION=1`, which the test
+  harness (`sglang.test.test_utils`) turns on by default.
+- **Mid-resolution code** (inside `server_args.py` / `arg_groups/` only): fields are
+  read-only during resolution; handlers and hooks read the in-flight state through
+  `resolved_view(server_args)` / `self._resolved()`. This is pipeline-internal — never use
+  `resolved_view` outside the pipeline. (One sanctioned exception: helpers that the pipeline
+  itself invokes mid-resolution, e.g. `adaptive_spec_params`.)
+
+### Adding a model-specific config adjustment
+
+Never assign `server_args` fields from model code. Declare instead
+(`sglang/srt/arg_groups/overrides.py`):
+
+- Constant per-arch values → `MODEL_OVERRIDES["MyArchForCausalLM"] = {...}`.
+- Derived values → `@register_model_override("MyArchForCausalLM")` returning a dict; the
+  callable receives *pristine* `server_args` + `hf_config` and must not write.
+- Normalization that must see earlier declarations → a post-process pass invoked via
+  `run_post_process_pass` at its slot (reads a view, returns a declaration dict).
+- Values only knowable at weight-load time → `declare_load_time_override(source, {...})`
+  (validates the whitelist, routes through `override()`).
+
+Declarable fields form a whitelist: `Arg(..., resolvable=True)` in the `ServerArgs`
+dataclass. A declaration against a non-whitelisted field fails at its slot.
+
+### Load-time vs resolution-time (critical)
+
+`__post_init__` runs in the launcher process before any model/platform import. Logic that
+consults an **extensible registry** (e.g. out-of-tree platforms registering attention
+backends in `init_backend()`, which runs at `model_runner` import) must stay at load time
+(ModelRunner init), writing through `override()`. Before moving any load-time logic into
+resolution, verify everything it reads is already complete at construction time.
+
+## Runtime flags (`get_flags()`)
+
+For state that init-time code *derives* and runtime code reads — parsed enums, platform
+probes, swappable ACTIVE values. Not for config mirrors (those died with resolve-at-end:
+read the field).
+
+- Groups are typed dataclasses on `Flags` (`capture` / `moe` / `dp`): typo-safe writes,
+  transactional test-only `override(**kw)` context manager.
+- `flags.moe` is materialized by `initialize_moe_config(server_args)` at scheduler init;
+  accessors (`get_moe_a2a_backend` etc.) are thin shims with lazy defaults. The speculative
+  contexts (`speculative_moe_backend_context`) swap the ACTIVE leaves around draft forwards.
+- `flags.dp` is materialized by `initialize_dp_attention`; `is_dp_attention_enabled()` is a
+  shim over `flags.dp.enabled`.
+- Adding a leaf: declare the dataclass field with a default equal to the pre-init behavior,
+  materialize it at the owning subsystem's init, keep any public accessor as a shim.
+
+## Resources (`get_resources()`)
+
+Named slots + two keyed-lazy registries:
+
+- `get_stream(name)` — get-or-create a named CUDA side stream; `set_stream(name, stream)`
+  installs explicitly. **Name leases by subsystem ROLE**: all model alternate streams share
+  `"alt"`; the offloader's copy stream is `"offload"`; DP-TBO comm is `"dp_tbo_comm"`; LoRA
+  side stream is `"lora_side"`. Two call sites may share a name only if their work belongs
+  on one stream — sharing across roles serializes intended overlap.
+- `get_buffer(name, factory)` — get-or-create a named persistent buffer. Grow-only or
+  per-device semantics manage their `resources.buffers` entries directly (see tokenspeed /
+  SM120 split / Marlin workspace). Buffer names are per-backend today; do not silently
+  share.
+- Singletons with manager semantics (EP dispatcher buffers, EPLB recorder/metadata, graph
+  memory pool) keep their owning accessors/classes as facades; only the *state* lives in a
+  resources entry. Preserve exact semantics in the shim: lazy defaults (the EPLB recorder
+  defaults to a Noop instance, not None), publish-once asserts, event-reuse contracts.
+- Stream/buffer creation is a driver call — it must happen outside cuda-graph capture;
+  keep lease points at init/warmup time.
+
+## Per-forward flags (`get_forward()`)
+
+Contextvar-backed; a new thread sees the defaults; `scoped(**kw)` is the regular write path
+(transactional, restores on exit and on exception); `set(name, value)` exists for legacy
+sticky setters (`is_extend_in_batch` is intentionally sticky within a thread). Use this
+tier for anything set-per-forward and read-within-forward. Before adding cross-thread
+state here, prove the readers' thread affinity: contextvars do NOT propagate to already-
+running or newly spawned threads. Note TBO ("two-batch overlap") interleaves ubatches on
+ONE thread — do not design for TBO threads that don't exist.
+
+## Testing idioms
+
+- **Force a code path by overriding causes, not effects**: compose
+  `get_context().override_server_args(**fields)` (config tier: publishes a fresh
+  dummy-boundary `ServerArgs` carrying the overrides — `with`-scoped, or
+  `install()`/`restore()` for fixture-lifetime use) + `get_parallel().override(...)` +
+  `get_flags().<group>.override(...)` + `get_forward().scoped(...)` +
+  `get_resources().override(...)`. All are scoped and transactional. Tests control
+  execution through the context — do not hand-build and publish config objects.
+  Note `override_server_args` is itself transitional (to be deprecated): it exists
+  while production still branches on raw `server_args` fields at runtime; prefer the
+  finer-grained tier overrides wherever they already cover the path you need.
+- **Never monkeypatch import bindings** (`module.get_x = lambda: ...`): production code may
+  read a different accessor over the same slot and your patch silently stops intercepting.
+  Publish/inject for real: `get_context().override_server_args(...)` for config;
+  `monkeypatch.setattr(get_resources(), "slot", fake)` for resources.
+- Fixtures standing in for `ServerArgs` need an `override` method if the code under test
+  mutates config (`_fake_server_args`-style: SimpleNamespace + write-through override).
+  MagicMock swallows `override()` calls silently — prefer SimpleNamespace so misses raise.
+- `reset_context()` in teardown; `_IsolatedServerArgs`-style save/restore when a test
+  publishes.
+- `ServerArgs(model_path="dummy")` early-returns `__post_init__` (no materialization, no
+  strict guard) — fine for lightweight fixtures.
+
+## Guardrails (these fail CI; what to do when they fire)
+
+1. **Strict mutation guard** (`SGLANG_STRICT_CONFIG_MUTATION=1`, default-on in tests):
+   bare `server_args.x = ...` after resolution raises. Fix: route through
+   `override(source, ...)`, or move genuinely config-decidable logic into resolution.
+2. **Mutation ratchet** (`test_server_args_mutation_ratchet.py`, exact pin 0 over the whole
+   package minus the pipeline / multimodal_gen / the mock-fixture factory): textual scan
+   for assignment forms. Never raise the baseline.
+3. **Legacy-accessor ratchet** (`test_legacy_global_ratchet.py`): `get_global_server_args`
+   call sites must not grow — new code uses `runtime_context.get_server_args()`.
+4. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the
+   flag-owning layers are pinned by name. A new module-level runtime global belongs on a
+   flags group / resources slot instead; migrating a pinned survivor must shrink the pin.
+
+## Hard-won pitfalls (check these before/while refactoring)
+
+- **Moving code drops first-line guards**: early returns (`if self.is_draft_worker: return`)
+  are the easiest thing to lose when relocating a method body. Draft workers share the
+  target's `server_args` object — a draft-side write poisons the target.
+- **Registry-completeness timing**: a gate that consults an extensible list is only correct
+  after the registrars ran (platform `init_backend()` at module import). See "load-time vs
+  resolution-time".
+- **Late function-scope imports shadow module names** for the WHOLE function
+  (UnboundLocalError at earlier lines). Audit moves with AST, not grep.
+- **Lease names are per-role**, not per-API-shape (the offloader-vs-"alt" lesson).
+- **Storage matrix for state read inside torch.compile-traced model code**
+  (piecewise cuda graph compiles the whole model forward): contextvars are
+  untraceable (hard error); dict-slot values are guarded per value — for a
+  per-forward int that is one recompile per distinct size, straight into the
+  recompile limit; **class/instance attributes are the only compile-friendly
+  form** (attribute-source ints get automatic-dynamic after the first size
+  change). Bools (≤2 values) are tolerable in any form — see
+  `ForwardFlags._GRAPH_VISIBLE`. Before moving such state, prove its readers
+  sit outside compile coverage; a piecewise-prefill boot of a small model is
+  the fast check (recompile storms show as `torch._dynamo hit
+  config.recompile_limit` during the compile pass).
+- **Engine-booting e2e tests are the only coverage for launcher-path code**; a child crash
+  kills the process tree and pytest dies silently — run with `PYTHONUNBUFFERED=1` and read
+  child logs.
+- CI arms `SGLANG_ENABLE_ASYNC_ASSERT=1` (device-side `torch._assert_async` probes, e.g.
+  KV-cache OOB): a fired device assert kills the tree with no Python traceback, and the
+  same bug is *silent corruption* locally with the flag off. Arm it when reproducing CI
+  crashes.
+- CI startup logs print the full `server_args=ServerArgs(...)`; diffing that dump between
+  runs is the fastest config-divergence check.
+
+## Where to read the code
+
+Key source files: `python/sglang/srt/runtime_context.py` (the container and every tier),
+`python/sglang/srt/arg_groups/overrides.py` (override registry, passes,
+`declare_load_time_override`, `resolved_view`), `python/sglang/srt/server_args.py`
+(`override`, `__setattr__`, `materialize_declarations` call,
+`_handle_model_capability_adjustments`), and the guardrail tests under
+`test/registered/unit/` (`test_server_args_mutation_ratchet.py`,
+`test_legacy_global_ratchet.py`, `test_module_state_ratchet.py`,
+`test_runtime_context.py` — the last one doubles as executable documentation of
+every tier's semantics).

.claude/skills/speculative-naming/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: speculative-naming
+description: Naming conventions for SGLang speculative decoding identifiers. Use when adding, renaming, or reviewing identifiers in speculative decoding code — anything under `python/sglang/srt/speculative/`, related attention backends, scheduler accumulators, IPC fields, observability metrics, or CLI flags.
+---
+
+# Speculative Decoding — Naming Conventions
+
+Apply this skill when adding, renaming, or reviewing identifiers in speculative decoding code (anything under `python/sglang/srt/speculative/`, related attention backends, scheduler accumulators, IPC fields, observability metrics, or CLI flags).
+
+## Rule 1 — Verb form, drop `-ed`
+
+Use the verb form `accept` everywhere. Don't use the past-participle form `accepted`.
+
+| Don't | Do |
+|---|---|
+| `num_accepted_tokens` | `num_accept_tokens` |
+| `accepted_indices` | `accept_indices` |
+| `accepted_token_ids` | `accept_tokens` (also see Rule 3) |
+
+## Rule 2 — The extra/bonus token is `bonus_token` / `bonus_tokens`
+
+The "+1" token that the target model always emits in addition to verifying drafts is the **bonus token**. Use `bonus_token` / `bonus_tokens` per Rule 7.
+
+| Don't | Do |
+|---|---|
+| `verified_id` / `verified_ids` | `bonus_token` / `bonus_tokens` |
+| `output_id` / `output_ids` (when referring to the bonus) | `bonus_token` / `bonus_tokens` |
+
+`req.output_ids` (the full output history of a request) is unrelated and stays as is.
+
+## Rule 3 — `accept` includes bonus; `correct` excludes bonus
+
+The semantic distinction lives in the **verb**, not the noun. Don't enumerate noun pairs.
+
+| Verb | Meaning |
+|---|---|
+| **`accept_*`** | Includes the bonus token |
+| **`correct_*`** | Drafts only, no bonus |
+
+Pair with whatever noun fits the data (`tokens`, `drafts`, `indices`, …). No required pairing, but **preferred default nouns**: `accept_tokens` and `correct_drafts` — `correct` semantically describes drafts (what got verified), `accept` describes the resulting token sequence (incl. bonus).
+
+| Form | Meaning |
+|---|---|
+| `accept_tokens` / `accept_indices` | Include bonus |
+| `correct_drafts` | Drafts only, no bonus |
+| `num_accept_tokens` | Count incl. bonus |
+| `num_correct_drafts` | Count excl. bonus |
+
+### Exception: `accept_rate` / `accept_length` follow paper convention
+
+These two metric names are entrenched in the spec-decoding literature and in external-facing fields (`meta_info`, Prometheus). Their semantics are paper-defined, not Rule-3-defined:
+
+| Name | Paper term | Bonus? | Definition |
+|---|---|---|---|
+| `accept_rate` | $\alpha$ (Leviathan 2023) | **No** | per-draft-token acceptance probability = `correct_drafts / proposed_drafts` |
+| `accept_length` | $\tau$ (EAGLE) | **Yes** | avg tokens per verify step = `completion_tokens / verify_ct` |
+
+Internal counters still follow Rule 3 strict semantics: `num_correct_drafts` (no bonus), `num_accept_tokens` (with bonus).
+
+## Rule 4 — `num_` for counts; `_ct` for counters; `_rate` for rates; no prefix for IDs
+
+Each form has its own marker. **Never mix** (no `num_X_ct`, no `num_accept_rate`).
+
+| Form | Pattern | Meaning | Examples |
+|---|---|---|---|
+| **Count** | `num_X` | Snapshot quantity at one point in time (often a tensor or scalar) | `num_accept_tokens`, `num_correct_drafts`, `num_proposed_drafts` |
+| **Counter** | `X_ct` | Monotonically incrementing accumulator over time | `spec_verify_ct`, `forward_ct` |
+| **Rate / ratio** | `X_rate` | Fractional value in `[0, 1]` | `accept_rate` |
+| **Tokens / content array** | no prefix | The actual token data, not a count | `accept_tokens`, `correct_drafts`, `bonus_token` |
+
+## Rule 5 — Drop redundant `_token_id` / `_token_ids` suffix in spec scope
+
+`_id` / `_ids` and `_token` / `_tokens` are both fine. But don't combine — `_token_id` / `_token_ids` is redundant **inside spec decoding**, because spec code only ever deals with vocab integers.
+
+The semantic differs by scope:
+
+| Scope | Example | What `_token_id` means |
+|---|---|---|
+| **Framework / multimodal / tokenizer** | `image_token_id`, `pad_token_id`, `eos_token_id`, `mask_token_id`, `bos_token_id` | A specific named/role token's vocab ID. The prefix names the role; `_token_id` says it's the integer ID for that role. Both halves carry information. |
+| **Spec decoding** | `accepted_token_ids`, `curr_token_id`, `out_token_ids` | Redundant. Spec only deals with vocab integers; `_id` adds nothing beyond `_token`. |
+
+### Renames
+
+| Don't | Do |
+|---|---|
+| `accepted_token_ids` | `accept_tokens` (Rule 1 + 3) |
+| `curr_token_id` | `current_token` |
+| `out_token_ids` | `out_tokens` |
+| `_resolve_spec_overlap_token_ids` | `_resolve_spec_overlap_tokens` |
+
+## Rule 6 — Singular vs plural
+
+Plural for any non-scalar tensor (`[bs]`-shaped, flat, or multi-dim); singular only for scalars (kernel `tl.load` results, single-int locals). Applies to all spec-decoding tensors (tokens, indices, etc.).
+
+```python
+accept_tokens: torch.Tensor     # [total_accepted] flat - plural
+accept_indices: torch.Tensor    # [bs, num_draft_tokens] - plural
+draft_tokens: torch.Tensor      # [bs * num_draft_tokens] flat - plural
+bonus_tokens: torch.Tensor      # [bs] - plural
+accept_token = tl.load(...)     # int32 scalar in a kernel iteration - singular
+bonus_token = tl.load(...)      # int32 scalar inside a kernel - singular
+```
+
+## Out of scope (these names stay as is)
+
+These rules apply to **spec-decoding-specific** identifiers. Pre-existing or framework-level names are kept.
+
+- **PyTorch / ecosystem**: `seq_lens`, `extend_seq_lens`, `cu_seqlens_q`
+- **Framework / multimodal vocab**: `image_token_id`, `pad_token_id`, `eos_token_id`, `mask_token_id`, `hot_token_id`, `bos_token_id`, `topk_id`
+- **Request-level state**: `req.input_ids`, `req.output_ids`, `req.origin_input_ids`, `next_token_ids` (`model_runner.sample` output)
+- **Frozen C++ kwargs**: `accept_token_num` (sgl-kernel)
+- **Non-token IDs**: `req_id`, `gpu_id`, `layer_id`, `program_id`
+- **`_len` / `_lens` names**: `num_X` is preferred for counts (Rule 4), but `_len` / `_lens` names are acceptable. Triton kernel params in particular often use `_lens` / `_len` to align with the PyTorch ecosystem (`seq_lens`, `cu_seqlens_q`). Rule 1 still requires the `-ed`-less form (`accept_length` OK, `accepted_length` not).

.claude/skills/write-sglang-test/SKILL.md

@@ -0,0 +1,452 @@
+---
+name: write-sglang-test
+description: Guide for writing SGLang CI/UT tests. Covers CustomTestCase, CI registration, server fixtures, model selection, mock testing, and test placement. Always read test/README.md for the full CI layout, how to run tests, and extra tips. Use when creating new tests, adding CI test cases, writing unit tests, or when the user asks to add tests for SGLang features.
+---
+
+# Writing SGLang CI / UT Tests
+
+This skill covers **how to write and register tests**. For CI pipeline internals (stage ordering, fast-fail, gating, partitioning, debugging CI failures), see the [CI workflow guide](../ci-workflow-guide/SKILL.md).
+
+## Core Rules
+
+1. **Always use `CustomTestCase`** — never raw `unittest.TestCase`. It ensures `tearDownClass` runs even when `setUpClass` fails, preventing resource leaks in CI.
+2. **`tearDownClass` must be defensive** — use `hasattr`/null checks before accessing resources (e.g. `cls.process`) that `setUpClass` may not have finished allocating.
+3. **Place tests in `test/registered/<category>/`** — including JIT kernel tests and benchmarks, which live in `test/registered/jit/` and `test/registered/jit/benchmark/` (nested subfolders are allowed)
+4. **Reuse server fixtures** — inherit from `DefaultServerBase` or write `setUpClass`/`tearDownClass` with `popen_launch_server`
+5. **Prefer mock over real server** — when testing logic that doesn't need a server / engine launch (middleware, request routing, config validation, argument parsing), use `unittest.mock.patch` / `MagicMock` and place tests in `test/registered/unit/`. Only launch a real server when the test genuinely needs inference results or server lifecycle behavior.
+
+JIT kernel notes:
+- If the task is adding or updating code under `python/sglang/jit_kernel/`, prefer the `add-jit-kernel` skill first.
+- JIT kernel correctness tests use `test/registered/jit/**/test_*.py`.
+- JIT kernel benchmarks use `test/registered/jit/benchmark/**/bench_*.py`.
+- Those files are executed by `test/run_suite.py` through dedicated kernel suites (`base-b-kernel-*`); a `register_*_ci(...)` call placed under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook.
+
+---
+
+## Model & Backend Selection
+
+| Scenario | Model | CI Registration | Suite |
+|----------|-------|-----------------|-------|
+| **Unit tests** (no server / engine launch) | None | `register_cpu_ci` (prefer) or `register_cuda_ci` | `base-a-test-cpu` or `base-b-test-1-gpu-small` |
+| **Common / backend-independent** (middleware, abort, routing, config, arg parsing) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` only | `base-b-test-1-gpu-small` |
+| **Model-agnostic functionality** (sampling, session, OpenAI API features) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` (+ AMD if relevant) | `base-b-test-1-gpu-small` |
+| **General performance** (single node, no spec/DP/parallelism) | `DEFAULT_MODEL_NAME_FOR_TEST` (8B) | `register_cuda_ci` | `base-b-test-1-gpu-large` |
+| **Bigger features** (spec, DP, TP, disaggregation) | Case by case | Case by case | See suite table below |
+
+**Key principle for E2E tests**: Do NOT add `register_amd_ci` unless the test specifically exercises AMD/ROCm code paths. Common E2E tests just need any GPU to run — duplicating across backends wastes CI time with no extra coverage.
+
+### All model constants
+
+Defined in `python/sglang/test/test_utils.py`:
+
+| Constant | Model | When to use |
+|----------|-------|-------------|
+| `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` | Llama-3.2-1B-Instruct | Common features, model-agnostic tests |
+| `DEFAULT_SMALL_MODEL_NAME_FOR_TEST_BASE` | Llama-3.2-1B | Base (non-instruct) model tests |
+| `DEFAULT_MODEL_NAME_FOR_TEST` | Llama-3.1-8B-Instruct | General performance (single node) |
+| `DEFAULT_MOE_MODEL_NAME_FOR_TEST` | Mixtral-8x7B-Instruct | MoE-specific tests |
+| `DEFAULT_SMALL_EMBEDDING_MODEL_NAME_FOR_TEST` | — | Embedding tests |
+| `DEFAULT_SMALL_VLM_MODEL_NAME_FOR_TEST` | — | Vision-language tests |
+
+### Naming Conventions
+
+A per-commit suite name is **generated** from registration metadata as `{stage}-test-{runner_config}` — you don't hand-write it:
+
+- **`stage`** — the CI stage (e.g. `base-b`, `base-b-kernel-unit`, `base-c`).
+- **`runner_config`** — a runner-pool key from `scripts/ci/runner_configs.yml`, which maps it to the physical runner label (so `1-gpu-large` runs on `1-gpu-h100`). AMD/NPU use their own keys (e.g. `amd`).
+- **Suite** — `register_cuda_ci(stage="base-b", runner_config="1-gpu-small")` → `base-b-test-1-gpu-small`, the name you pass to `run_suite.py --suite`. The `-test-` is just the connector; never put it in `register_*_ci`.
+
+> Legacy single-string `suite=` is only for suites that don't fit that shape — nightly/stress/weekly and some AMD/CPU/NPU pools (e.g. `suite="nightly-kernel-1-gpu", nightly=True`). Per-commit tests always use `stage=` + `runner_config=`.
+
+### All CI Suites
+
+#### Per-commit (CUDA)
+
+| Suite | Runner (label) | Description |
+|-------|----------------|-------------|
+| `base-a-test-1-gpu-small` | `1-gpu-5090` | Quick checks on a small NVIDIA GPU before heavier stages |
+| `base-a-test-cpu` | `ubuntu-latest` | CPU-only unit tests |
+| `base-b-test-1-gpu-small` | `1-gpu-5090` | Core engine tests that fit a 5090-class card |
+| `base-b-test-1-gpu-large` | `1-gpu-h100` | Tests that need H100-class memory or kernels (e.g. FA3) |
+| `base-b-test-2-gpu-large` | `2-gpu-h100` | Two-GPU correctness and parallelism (TP/PP) on H100 |
+| `base-b-test-4-gpu-b200` | `4-gpu-b200` | Early Blackwell coverage (SM100+ paths) on four GPUs |
+| `base-b-kernel-unit-test-1-gpu-large` | `1-gpu-h100` | JIT kernel correctness tests under `test/registered/jit/` |
+| `base-b-kernel-unit-test-4-gpu-b200` | `4-gpu-b200` | JIT kernel correctness tests for Blackwell / SM100-specific paths |
+| `base-b-kernel-unit-test-8-gpu-h200` | `8-gpu-h200` | Multi-GPU JIT kernel correctness tests under `test/registered/jit/` |
+| `base-b-kernel-benchmark-test-1-gpu-large` | `1-gpu-h100` | JIT kernel benchmark files under `test/registered/jit/benchmark/` |
+| `base-c-test-4-gpu-h100` | `4-gpu-h100` | Large 4-GPU H100 integration and scaling tests |
+| `base-c-test-8-gpu-h200` | `8-gpu-h200` | Large 8-GPU H200 runs for big models and parallelism |
+| `base-c-test-8-gpu-h20` | `8-gpu-h20` | Large 8-GPU H20 runs for big models |
+| `base-c-test-deepep-4-gpu-h100` | `4-gpu-h100` | DeepEP expert-parallel and networking on four H100s |
+| `base-c-test-8-gpu-b200` | `8-gpu-b200` | 8-GPU B200 suite (registered but not yet wired to a workflow) |
+| `base-c-test-4-gpu-b200` | `4-gpu-b200` | 4-GPU B200 suite for large models on Blackwell |
+| `base-c-test-4-gpu-b200-small` | `4-gpu-b200` | Smaller 4-GPU B200 suite split onto low-disk B200 runners |
+| `base-c-test-4-gpu-gb200` | `4-gpu-gb200` | 4-GPU GB200 suite for Grace Blackwell; registered in `run_suite.py`, but the PR workflow is currently disabled until a runner is provisioned |
+
+#### Per-commit (AMD)
+
+| Suite | Runner (label) | Description |
+|-------|----------------|-------------|
+| `stage-a-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Quick checks on one MI325-class GPU |
+| `stage-b-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Core 1-GPU AMD tests (14 partitions) |
+| `stage-b-test-1-gpu-small-amd-nondeterministic` | `linux-mi325-1gpu-sglang` | Non-deterministic 1-GPU AMD tests |
+| `stage-b-test-1-gpu-small-amd-mi35x` | `linux-mi35x-gpu-1` | 1-GPU tests on MI35x hardware |
+| `stage-b-test-1-gpu-large-amd` | `linux-mi325-1gpu-sglang` | Large 1-GPU AMD tests (2 partitions) |
+| `stage-b-test-2-gpu-large-amd` | `linux-mi325-2gpu-sglang` | 2-GPU ROCm correctness and parallel setups |
+| `stage-b-test-large-8-gpu-mi35x-disaggregation-amd` | `linux-mi35x-gpu-8.fabric` | PD disaggregation and RDMA on 8×MI35x fabric |
+| `stage-c-test-4-gpu-amd` | `linux-mi325-4gpu-sglang` | 4-GPU AMD integration (2 partitions) |
+| `stage-c-test-large-8-gpu-amd` | `linux-mi325-8gpu-sglang` | 8-GPU MI325 scaling and integration |
+| `stage-c-test-large-8-gpu-amd-mi35x` | `linux-mi35x-gpu-8` | 8-GPU MI35x scaling (2 partitions) |
+
+
+### Per-commit (Ascend NPU)
+
+| Suite | Runner (label) | Description |
+| --- | --- | --- |
+| `per-commit-1-npu-a2` | `linux-aarch64-a2-1` | 1-NPU LLM CI machine |
+| `per-commit-2-npu-a2` | `linux-aarch64-a2-2` | 2-NPU LLM CI machine |
+| `per-commit-4-npu-a3` | `linux-aarch64-a3-4` | 4-NPU LLM CI machine |
+| `per-commit-16-npu-a3` | `linux-aarch64-a3-16` | 16-NPU LLM CI machine  |
+| `multimodal-gen-test-1-npu-a3` | `linux-aarch64-a3-2` | 1-NPU multimodal CI machine |
+| `multimodal-gen-test-2-npu-a3` | `linux-aarch64-a3-16` | 2-NPU multimodal CI machine |
+| `multimodal-gen-test-8-npu-a3` | `linux-aarch64-a3-16` | 8-NPU multimodal CI machine |
+
+#### Nightly
+
+Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. Examples:
+
+- `nightly-1-gpu` (CUDA)
+- `nightly-kernel-1-gpu` (CUDA, JIT kernel full grids)
+- `nightly-kernel-8-gpu-h200` (CUDA, multi-GPU JIT kernel nightly)
+- `nightly-8-gpu-h200` (CUDA)
+- `nightly-eval-vlm-2-gpu` (CUDA)
+- `nightly-amd` (AMD)
+- `nightly-amd-8-gpu-mi35x` (AMD)
+- `nightly-1-npu-a3` (NPU)
+- `nightly-2-npu-a3` (NPU)
+- `nightly-4-npu-a3` (NPU)
+- `nightly-8-npu-a3` (NPU)
+- `nightly-16-npu-a3` (NPU)
+
+> **Note**: Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
+
+### Choosing a Suite
+
+Use the lightest suite that meets your test's needs:
+
+- **No GPU required** → `base-a-test-cpu`
+- **Most small GPU tests** → `base-b-test-1-gpu-small` (default choice)
+- **Need H100 memory or Hopper features** → `base-b-test-1-gpu-large`
+- **JIT kernel correctness** → `base-b-kernel-unit-test-1-gpu-large`
+- **JIT kernel correctness for B200 / SM100 paths** → `base-b-kernel-unit-test-4-gpu-b200`
+- **JIT kernel benchmarks** → `base-b-kernel-benchmark-test-1-gpu-large`
+- **Multi-GPU** → only when the test actually needs multiple GPUs
+
+---
+
+## Test File Templates
+
+### Unit Tests (no server / engine launch)
+
+See `test/registered/unit/README.md` for quick-start and rules. Unit tests live in `test/registered/unit/`, mirroring `python/sglang/srt/`:
+
+```python
+"""Unit tests for srt/<module>"""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from sglang.srt.<module> import TargetClass
+from sglang.test.ci.ci_register import register_cpu_ci
+from sglang.test.test_utils import CustomTestCase
+
+register_cpu_ci(est_time=5, suite="base-a-test-cpu")
+# Prefer CPU. Only use register_cuda_ci when the test truly needs a GPU.
+
+class TestTargetClass(CustomTestCase):
+    def test_basic_behavior(self):
+        obj = TargetClass(...)
+        self.assertEqual(obj.method(), expected)
+
+    @patch("sglang.srt.<module>.some_dependency")
+    def test_with_mock(self, mock_dep):
+        mock_dep.return_value = MagicMock()
+        # test logic with dependency mocked
+        ...
+
+
+if __name__ == "__main__":
+    unittest.main()
+```
+
+Use `unittest.mock.patch` / `MagicMock` to mock dependencies and isolate the logic under test. If the module transitively imports GPU-only packages (e.g. `sgl_kernel`), they can be stubbed so the test runs on CPU CI. Do not modify `sys.modules` at module level — use `patch.dict` (as a class decorator or with `start`/`stop`) to ensure cleanup and avoid cross-test pollution. See `test/registered/unit/README.md` for details and examples.
+
+**Quality bar** — test real logic (validation boundaries, state transitions, error paths, branching, etc.). Skip tests that just verify Python itself works (e.g., "does calling an abstract method raise `NotImplementedError`?", "does a dataclass store the field I assigned?"). Consolidate repetitive patterns into parameterized tests. No production code changes in test PRs.
+
+### E2E test (small model, server needed)
+
+```python
+import unittest
+
+import requests
+
+from sglang.srt.utils import kill_process_tree
+from sglang.test.ci.ci_register import register_cuda_ci
+from sglang.test.test_utils import (
+    DEFAULT_SMALL_MODEL_NAME_FOR_TEST,
+    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+    DEFAULT_URL_FOR_TEST,
+    CustomTestCase,
+    popen_launch_server,
+)
+
+register_cuda_ci(est_time=60, suite="base-b-test-1-gpu-small")
+
+
+class TestMyFeature(CustomTestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
+        cls.base_url = DEFAULT_URL_FOR_TEST
+        cls.process = popen_launch_server(
+            cls.model,
+            cls.base_url,
+            timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+            other_args=["--arg1", "value1"],  # feature-specific args
+        )
+
+    @classmethod
+    def tearDownClass(cls):
+        if hasattr(cls, "process") and cls.process:
+            kill_process_tree(cls.process.pid)
+
+    def test_basic_functionality(self):
+        response = requests.post(
+            self.base_url + "/generate",
+            json={"text": "Hello", "sampling_params": {"max_new_tokens": 32}},
+        )
+        self.assertEqual(response.status_code, 200)
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=3)
+```
+
+### E2E test (8B model, server needed, performance)
+
+```python
+import time
+import unittest
+
+import requests
+
+from sglang.srt.utils import kill_process_tree
+from sglang.test.ci.ci_register import register_cuda_ci
+from sglang.test.test_utils import (
+    DEFAULT_MODEL_NAME_FOR_TEST,
+    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+    DEFAULT_URL_FOR_TEST,
+    CustomTestCase,
+    popen_launch_server,
+)
+
+register_cuda_ci(est_time=300, suite="base-b-test-1-gpu-large")
+
+
+class TestMyFeaturePerf(CustomTestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.model = DEFAULT_MODEL_NAME_FOR_TEST
+        cls.base_url = DEFAULT_URL_FOR_TEST
+        cls.process = popen_launch_server(
+            cls.model,
+            cls.base_url,
+            timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+        )
+
+    @classmethod
+    def tearDownClass(cls):
+        if hasattr(cls, "process") and cls.process:
+            kill_process_tree(cls.process.pid)
+
+    def test_latency(self):
+        start = time.perf_counter()
+        response = requests.post(
+            self.base_url + "/generate",
+            json={"text": "Hello", "sampling_params": {"max_new_tokens": 128}},
+        )
+        elapsed = time.perf_counter() - start
+        self.assertEqual(response.status_code, 200)
+        self.assertLess(elapsed, 5.0, "Latency exceeded threshold")
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=3)
+```
+
+---
+
+## Server Fixture Reuse
+
+For tests that only need a standard server, inherit from `DefaultServerBase` and override class attributes:
+
+```python
+from sglang.test.server_fixtures.default_fixture import DefaultServerBase
+
+class TestMyFeature(DefaultServerBase):
+    model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
+    other_args = ["--enable-my-feature"]
+
+    def test_something(self):
+        ...
+```
+
+Available fixtures in `python/sglang/test/server_fixtures/`:
+
+| Fixture | Use case |
+|---------|----------|
+| `DefaultServerBase` | Standard single-server tests |
+| `EagleServerBase` | EAGLE speculative decoding |
+| `PDDisaggregationServerBase` | Disaggregated prefill/decode |
+| `MMMUServerBase` | Multimodal VLM tests |
+
+---
+
+## CI Registration
+
+Every CI-discovered test file must call a registration function at module level:
+
+```python
+from sglang.test.ci.ci_register import (
+    register_cuda_ci,
+    register_amd_ci,
+    register_cpu_ci,
+    register_npu_ci,
+)
+
+# Per-commit test (small 1-gpu, runs on 5090)
+register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small")
+
+# Per-commit test (large 1-gpu, runs on H100)
+register_cuda_ci(est_time=120, suite="base-b-test-1-gpu-large")
+
+# Nightly-only test
+register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True)
+
+# Multi-backend test (only when testing backend-specific code paths)
+register_cuda_ci(est_time=80, suite="base-a-test-1-gpu-small")
+register_amd_ci(est_time=120, suite="stage-a-test-1-gpu-small-amd")
+register_npu_ci(est_time=400, suite="nightly-8-npu-a3", nightly=True)
+
+# Temporarily disabled test
+register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small", disabled="flaky - see #12345")
+```
+
+Parameters:
+- `est_time`: estimated runtime in seconds (used for CI partitioning)
+- `suite`: which CI suite to run in (see suite tables above)
+- `nightly=True`: for nightly-only tests (default `False` = per-commit)
+- `disabled="reason"`: temporarily disable with explanation
+
+**Key principle**: Only add `register_amd_ci` / `register_npu_ci` when the test exercises backend-specific code paths. Common E2E tests just need `register_cuda_ci` — duplicating across backends wastes CI time.
+
+### JIT Kernel Registration
+
+JIT kernel files live outside `test/registered/` but still use registration:
+
+```python
+from sglang.test.ci.ci_register import register_cuda_ci
+
+# Correctness tests in test/registered/jit/
+register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
+register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
+register_cuda_ci(est_time=120, stage="base-b-kernel-unit", runner_config="8-gpu-h200")
+
+# Benchmarks in test/registered/jit/benchmark/
+register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")
+
+# Optional nightly registration — nightly suites use the legacy single-string suite=
+register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True)
+register_cuda_ci(est_time=120, suite="nightly-kernel-8-gpu-h200", nightly=True)
+```
+
+The `stage` + `runner_config` calls generate suites like `base-b-kernel-unit-test-1-gpu-large`; nightly keeps the legacy `suite=` string. Keep `est_time`, `stage`, `runner_config`, and `suite` as **literal values** — `run_suite.py` collects them by AST parsing.
+
+---
+
+## Test Placement
+
+```
+test/
+├── registered/          # CI tests (auto-discovered by run_suite.py)
+│   ├── unit/            # No server / engine launch (see test/registered/unit/README.md)
+│   ├── kernels/         # CUDA kernel correctness (no server, GPU required)
+│   ├── sampling/        # test_penalty.py, test_sampling_params.py ...
+│   ├── sessions/        # test_session_control.py ...
+│   ├── openai_server/   # basic/, features/, validation/ ...
+│   ├── spec/            # eagle/, utils/ ...
+│   ├── models/          # model-specific accuracy tests
+│   ├── perf/            # performance benchmarks
+│   └── <category>/      # create new category if needed
+├── manual/              # Non-CI: debugging, one-off, manual verification
+└── run_suite.py         # CI runner (scans registered/ plus jit_kernel test/benchmark files)
+
+python/sglang/jit_kernel/
+├── tests/               # JIT kernel correctness tests (CI-discovered by test/run_suite.py)
+└── benchmark/           # JIT kernel benchmarks (CI-discovered by test/run_suite.py)
+```
+
+**Decision rule** (see also `test/registered/README.md`):
+- Component logic, no server → `registered/unit/`
+- JIT kernel correctness / benchmarks → `test/registered/jit/` or `test/registered/jit/benchmark/`
+- Other kernel correctness → `registered/kernels/`
+- Server needed → `registered/<category>/`
+- Local debugging → `manual/`
+
+---
+
+## Eval Accuracy Mixins
+
+**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (run_eval, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length).
+
+Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults.
+
+```python
+class TestMyFeature(CustomTestCase, MMLUMixin):
+    mmlu_score_threshold = 0.65
+    mmlu_num_examples = 64
+    mmlu_num_threads = 32
+    # test_mmlu is inherited — no code needed
+```
+
+---
+
+## Key Utilities
+
+```python
+from sglang.test.test_utils import (
+    CustomTestCase,              # base class with retry logic
+    popen_launch_server,         # launch server subprocess
+    DEFAULT_URL_FOR_TEST,        # auto-configured base URL
+    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,  # 600s default
+    run_bench_serving,           # benchmark helper (launch + bench)
+)
+from sglang.srt.utils import kill_process_tree  # cleanup server
+```
+
+---
+
+## Checklist
+
+Before submitting a test:
+
+- [ ] Inherits from `CustomTestCase` (not `unittest.TestCase`)
+- [ ] Has `register_*_ci(...)` call at module level
+- [ ] Placed in `test/registered/<category>/` (JIT kernel test/benchmark → `test/registered/jit/` or `test/registered/jit/benchmark/`)
+- [ ] JIT kernel work: test files live in `test/registered/jit/`; only test-only helpers stay under `python/sglang/jit_kernel/`
+- [ ] Backend-independent tests: `register_cuda_ci` only + smallest model
+- [ ] Logic that doesn't need a server / engine launch → unit test in `registered/unit/` (see Unit Tests section)
+- [ ] `setUpClass` launches server, `tearDownClass` kills it (if server-based)
+- [ ] `tearDownClass` is defensive — uses `hasattr`/null checks before accessing resources that may not have been allocated
+- [ ] Has `if __name__ == "__main__": unittest.main()`
+- [ ] `est_time` is reasonable (measure locally)

.codespellrc

@@ -0,0 +1,3 @@
+[codespell]
+ignore-words-list = ans, als, hel, boostrap, childs, te, vas, hsa, ment, cann, thi, makro, wil, rouge, PRIS, ather, MIS, medias, allready, inout, nd, fo, visibles, nothink, renderD, ond, tbe, CopyIn, notin, subtile, subtiles, dout, IST, kInf
+skip = *.json, *.jsonl, *.patch, *.txt, *.lock

.coveragerc

@@ -0,0 +1,16 @@
+[run]
+source = python/sglang/srt
+omit =
+    */test/*
+    */__pycache__/*
+
+[report]
+show_missing = true
+exclude_lines =
+    pragma: no cover
+    if __name__ == .__main__.:
+    raise NotImplementedError
+    if TYPE_CHECKING
+
+[html]
+directory = htmlcov

.devcontainer/Dockerfile

@@ -0,0 +1,33 @@
+FROM lmsysorg/sglang:dev
+
+# Create non-root user with specified UID and GID
+# NOTE: Replace with your own UID and GID. This is a workaround from https://github.com/microsoft/vscode-remote-release/issues/49#issuecomment-489060908.
+ARG HOST_UID=1003
+ARG HOST_GID=1003
+RUN groupadd -g $HOST_GID devuser && \
+    useradd -m -u $HOST_UID -g $HOST_GID -s /bin/zsh devuser
+
+# Give devuser sudo access
+RUN apt-get update && apt-get install -y sudo && \
+    echo "devuser ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/devuser && \
+    rm -rf /var/lib/apt/lists/* && \
+    apt-get clean
+
+# Set up oh-my-zsh for devuser
+RUN cp -r /root/.oh-my-zsh /home/devuser/.oh-my-zsh && \
+    cp /root/.zshrc /home/devuser/.zshrc && \
+    sed -i 's|/root/.oh-my-zsh|/home/devuser/.oh-my-zsh|g' /home/devuser/.zshrc && \
+    chown -R devuser:devuser /home/devuser/
+
+# Set workspace directory and ownership
+WORKDIR /sgl-workspace/sglang
+RUN chown -R devuser:devuser /sgl-workspace
+
+# Switch to devuser
+USER devuser
+
+# Install uv
+RUN curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Install rust
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

.devcontainer/devcontainer.json

@@ -0,0 +1,30 @@
+{
+    "name": "sglang",
+    "build": {
+        "dockerfile": "Dockerfile"
+    },
+    "remoteUser": "devuser",
+    "customizations": {
+        "vscode": {
+            "extensions": [
+                // Python development
+                "ms-python.python",
+                "charliermarsh.ruff",
+                // Rust development
+                "rust-lang.rust-analyzer",
+                "tamasfe.even-better-toml"
+            ]
+        }
+    },
+    "forwardPorts": [],
+    "runArgs": [
+        "--gpus",
+        "all"
+    ],
+    // The two lines below ensures that your local changes in the sglang
+    // repo is automatically synced to the sglang pip package installed
+    // in the dev docker container. You can remove / comment out these
+    // two lines if you prefer to sync code changes manually.
+    "workspaceMount": "source=${localWorkspaceFolder},target=/sgl-workspace/sglang,type=bind",
+    "workspaceFolder": "/sgl-workspace/sglang"
+}

.dockerignore

@@ -0,0 +1 @@
+.gitignore

.github/CODEOWNERS

@@ -0,0 +1,105 @@
+.github @merrymercy @Fridge003 @ispobock @Kangyan-Zhou @HaiShaw @bingxche
+/docker @Fridge003 @ispobock @HaiShaw @ishandhanani @yctseng0211 @sogalin
+/docker/npu.Dockerfile @ping1jing2 @iforgetmyname @whybeyoung
+/docs @wisclmy0611 @zijiexia @sogalin
+/docs_new @wisclmy0611 @zijiexia @Richardczl98 @JustinTong0323 @sogalin
+/python/pyproject.toml @merrymercy @Fridge003 @ispobock
+/python/sglang/jit_kernel @DarkSharpness @BBuf @celve @HydraQYH @yuan-luo
+/python/sglang/jit_kernel/diffusion @yingluosanqian @BBuf @mickqian
+/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf
+/python/sglang/multimodal_gen/runtime/cache @DefTruth
+/python/sglang/multimodal_gen/test/server/ascend  @ping1jing2 @ssshinigami @Makcum888e @e-martirosian
+/python/sglang/srt/batch_invariant_ops @Fridge003 @hebiao064
+/python/sglang/srt/compilation @hebiao064 @Oasis-Git
+/python/sglang/srt/constrained @hnyls2002 @DarkSharpness @JustinTong0323
+/python/sglang/srt/disaggregation @ByronHsu @hnyls2002 @ShangmingCai @HaiShaw @sogalin
+/python/sglang/srt/disaggregation/ascend @ping1jing2 @iforgetmyname
+/python/sglang/srt/disaggregation/encode_receiver.py @ShangmingCai @liusy58 @ZhengWG @gty111
+/python/sglang/srt/disaggregation/encode_server.py @ShangmingCai @liusy58 @ZhengWG @gty111
+/python/sglang/srt/disaggregation/mori @Duyi-Wang @kkHuang-amd @billishyahao
+/python/sglang/srt/distributed @yizhang2077 @merrymercy @ch-wan
+/python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py @ShangmingCai @stmatengss
+/python/sglang/srt/dllm @ClawSeven @btw616
+/python/sglang/srt/entrypoints @ispobock @CatherineSue @slin1237 @merrymercy @JustinTong0323
+/python/sglang/srt/entrypoints/anthropic @JustinTong0323
+/python/sglang/srt/entrypoints/openai @JustinTong0323
+/python/sglang/srt/entrypoints/engine_score_mixin.py @sundar24295s @chanh @fortunecookiee
+/python/sglang/srt/entrypoints/grpc_server.py @CatherineSue @slin1237
+/python/sglang/srt/entrypoints/openai/serving_score.py @sundar24295s @chanh @fortunecookiee
+/python/sglang/srt/eplb @fzyzcjy @ch-wan @xutizhou
+/python/sglang/srt/function_call @JustinTong0323
+/python/sglang/srt/grpc @CatherineSue @slin1237
+/python/sglang/srt/hardware_backend/mlx @yeahdongcn
+/python/sglang/srt/hardware_backend/musa @yeahdongcn
+/python/sglang/srt/hardware_backend/npu @ping1jing2 @iforgetmyname @whybeyoung
+/python/sglang/srt/hardware_backend/gpu/quantization @Alisehen
+/python/sglang/srt/hardware_backend/npu/quantization @OrangeRedeng @TamirBaydasov @Alisehen
+/python/sglang/srt/layers @merrymercy @Ying1123 @Fridge003 @ispobock @HaiShaw @ch-wan @BBuf @Edwardf0t1
+/python/sglang/srt/layers/attention @merrymercy @Fridge003 @ispobock @Qiaolin-Yu @hebiao064 @HaiShaw
+/python/sglang/srt/layers/attention/fla @yizhang2077 @hebiao064 @yuan-luo
+/python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py @yizhang2077 @hebiao064 @hanming-lu @yuan-luo
+/python/sglang/srt/layers/attention/mamba @yizhang2077 @hebiao064
+/python/sglang/srt/layers/attention/dsa @1am9trash @hubertlu-tw @kkHuang-amd @HaiShaw @Fridge003 @YAMY1234 @rainj-me
+/python/sglang/srt/layers/attention/vision.py @mickqian @yuan-luo @yhyang201
+/python/sglang/srt/layers/quantization @ch-wan @BBuf @Edwardf0t1 @FlamingoPg @AniZpZ @HaiShaw @b8zhong @OrangeRedeng @TamirBaydasov @Alisehen
+/python/sglang/srt/layers/quantization/quark @kkHuang-amd @yichiche @hubertlu-tw @1am9trash @BowenBao
+/python/sglang/srt/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
+/python/sglang/srt/managers @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann
+/python/sglang/srt/managers/scheduler_pp_mixin.py @ShangmingCai @XucSh
+/python/sglang/srt/managers/tokenizer_manager_score_mixin.py @sundar24295s @chanh @fortunecookiee
+/python/sglang/srt/mem_cache @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann @hanming-lu @yizhang2077 @hzh0425 @ispobock @alphabetc1
+/python/sglang/srt/mem_cache/allocator @hnyls2002 @ispobock @alphabetc1
+/python/sglang/srt/mem_cache/storage/mooncake_store @huangtingwei9988
+/python/sglang/srt/mem_cache/storage/mooncake_store/embedding_cache_controller.py @liusy58
+/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_embedding_store.py @liusy58
+/python/sglang/srt/model_executor @merrymercy @Ying1123 @hnyls2002 @Fridge003 @ispobock
+/python/sglang/srt/model_executor/piecewise_cuda_graph_runner.py @hebiao064
+/python/sglang/srt/model_loader @b8zhong
+/python/sglang/srt/models/deepseek_common @Fridge003 @ispobock @fzyzcjy @ch-wan
+/python/sglang/srt/models/deepseek_v2.py @fzyzcjy @zhyncs @ispobock @ch-wan @merrymercy @Fridge003
+/python/sglang/srt/models/transformers.py @adarshxs
+/python/sglang/srt/multimodal @mickqian @JustinTong0323 @yhyang201 @yuan-luo
+/python/sglang/srt/observability @merrymercy @fzyzcjy @sufeng-buaa
+/python/sglang/srt/parser @JustinTong0323
+/python/sglang/srt/platforms @merrymercy @whybeyoung @alexnails
+/python/sglang/srt/ray @Qiaolin-Yu @xyuzh
+/python/sglang/srt/speculative @Ying1123 @merrymercy @hnyls2002 @Qiaolin-Yu
+/python/sglang/srt/utils/hf_transformers @JustinTong0323
+/sgl-kernel @ispobock @BBuf @yizhang2077 @merrymercy @FlamingoPg @HaiShaw
+/sgl-kernel/csrc/musa @yeahdongcn
+/sgl-model-gateway @slin1237 @CatherineSue
+/sgl-model-gateway/benches @slin1237
+/sgl-model-gateway/bindings/python @CatherineSue @key4ng @slin1237
+/sgl-model-gateway/e2e_test @CatherineSue @key4ng
+/sgl-model-gateway/examples/wasm @slin1237
+/sgl-model-gateway/src/config @slin1237
+/sgl-model-gateway/src/core @slin1237
+/sgl-model-gateway/src/data_connector @key4ng
+/sgl-model-gateway/src/grpc_client @CatherineSue @slin1237
+/sgl-model-gateway/src/mcp @key4ng @slin1237
+/sgl-model-gateway/src/policies @slin1237 @ByronHsu
+/sgl-model-gateway/src/proto @CatherineSue @slin1237
+/sgl-model-gateway/src/protocols @CatherineSue @key4ng
+/sgl-model-gateway/src/reasoning_parser @CatherineSue
+/sgl-model-gateway/src/routers @CatherineSue @key4ng @slin1237
+/sgl-model-gateway/src/tokenizer @slin1237 @CatherineSue
+/sgl-model-gateway/src/tool_parser @slin1237 @CatherineSue
+/sgl-model-gateway/src/wasm @slin1237
+/sgl-model-gateway/examples/wasm @slin1237
+/test/registered/prefill_only @sundar24295s @chanh @fortunecookiee
+/benchmark/prefill_only/bench_score.py @sundar24295s @chanh @fortunecookiee
+/test/registered/ascend @ping1jing2 @ssshinigami  @e-martirosian
+/test/srt/test_modelopt* @Edwardf0t1
+/python/sglang/srt/layers/gemma4_fused_ops.py @merrymercy @Ying1123 @Fridge003 @ispobock @HaiShaw @ch-wan @BBuf @Edwardf0t1 @kpham-sgl @pyc96
+/python/sglang/srt/function_call/gemma4_detector.py @CatherineSue @JustinTong0323 @kpham-sgl @pyc96
+/python/sglang/srt/models/gemma4_*.py @kpham-sgl @pyc96
+/python/sglang/srt/multimodal/processors/gemma4.py @kpham-sgl @pyc96
+/python/sglang/test/ascend @ping1jing2 @ssshinigami @e-martirosian
+/docs_new/cookbook/autoregressive/Google/Gemma4.mdx @wisclmy0611 @zijiexia @Richardczl98 @kpham-sgl @pyc96
+/docs_new/src/snippets/autoregressive/gemma4-deployment.jsx @wisclmy0611 @zijiexia @Richardczl98 @kpham-sgl @pyc96
+/python/sglang/srt/speculative/ngram_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
+/python/sglang/srt/speculative/adaptive_*.py @Qiaolin-Yu @alphabetc1
+/python/sglang/srt/speculative/cpp_ngram @hnyls2002 @Qiaolin-Yu @kpham-sgl
+/python/sglang/srt/speculative/frozen_kv_mtp_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl @pyc96
+/python/sglang/jit_kernel/ngram_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
+/python/sglang/jit_kernel/csrc/ngram_corpus @hnyls2002 @Qiaolin-Yu @kpham-sgl

.github/FOLDER_README.md

@@ -0,0 +1,12 @@
+# Maintenance Tools
+
+This folder contains tools and workflows for automating maintenance tasks.
+
+## CI Permissions
+
+`CI_PERMISSIONS.json` defines the CI permissions granted to each user.
+Maintainers can directly edit the file to add entries with `"reason": "custom override"`.
+Maintainers can also run `update_ci_permission.py` to update it with some auto rules (e.g., top contributors in the last 90 days get full permissions).
+
+## Others
+- `MAINTAINER.md` defines the code maintenance model.

.github/ISSUE_TEMPLATE/1-bug-report.yml

@@ -0,0 +1,35 @@
+name: 🐞 Bug report
+description: Report a bug to help us reproduce and fix it.
+title: "[Bug] "
+labels: ['Bug']
+
+body:
+- type: checkboxes
+  attributes:
+    label: Checklist
+    options:
+      - label: I searched related issues but found no solution.
+      - label: The bug persists in the latest version.
+      - label: Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
+      - label: If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
+      - label: Please use English. Otherwise, it will be closed.
+- type: textarea
+  attributes:
+    label: Describe the bug
+    description: A clear, concise description of the bug.
+  validations:
+    required: true
+- type: textarea
+  attributes:
+    label: Reproduction
+    description: Command/script run and model used.
+    placeholder: Paste the command here.
+  validations:
+    required: true
+- type: textarea
+  attributes:
+    label: Environment
+    description: Run `python3 -m sglang.check_env` and paste output here. Issues without this will be closed.
+    placeholder: Paste environment output here.
+  validations:
+    required: true

.github/ISSUE_TEMPLATE/2-feature-request.yml

@@ -0,0 +1,23 @@
+name: 🚀 Feature request
+description: Suggest an idea for this project
+title: "[Feature] "
+
+body:
+- type: checkboxes
+  attributes:
+    label: Checklist
+    options:
+      - label: If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
+      - label: Please use English. Otherwise, it will be closed.
+- type: textarea
+  attributes:
+    label: Motivation
+    description: |
+      Clearly and concisely describe the feature's motivation.
+  validations:
+    required: true
+- type: textarea
+  attributes:
+    label: Related resources
+    description: |
+      Provide official releases or third-party implementations if available.

.github/ISSUE_TEMPLATE/3-playground-verified-cell.yml

@@ -0,0 +1,108 @@
+name: 🧪 Playground - Verified Cell Submission
+description: Submit a deployment recipe you verified in the cookbook Playground. A maintainer will review and promote it into the cell catalog via PR.
+title: "[Playground] Verified cell: <fill from playground>"
+labels: ["cookbook", "playground-submission"]
+
+body:
+- type: markdown
+  attributes:
+    value: |
+      Thanks for verifying a deployment recipe!
+
+      The cookbook **Playground** section of each model page pre-fills this
+      form. When your overrides diverge from the verified cell, click
+      **Submit ↗**, confirm the SGLang version / benchmark / notes in the
+      **Submit verified cell** dialog and tick the attestations, then
+      **Open submission on GitHub →**. The model, combination, proposed cell
+      snippet, and existing cell are filled automatically — please double-check
+      the snippet matches what you ran.
+
+      Once a maintainer with access to the listed hardware reproduces the
+      recipe, they'll convert this issue into a PR against the cookbook cell
+      catalog and close this issue.
+
+- type: input
+  id: model
+  attributes:
+    label: Cookbook model
+    description: Which model's cookbook does this cell belong to? Auto-filled by the playground — use the model id as it appears in the cookbook config.
+    placeholder: "deepseek-ai/deepseek-v4"
+  validations:
+    required: true
+
+- type: input
+  id: combination
+  attributes:
+    label: Combination
+    description: "Format: hw / variant / quant / strategy / nodes (auto-filled by the playground)."
+    placeholder: "b200 / flash / fp4 / low-latency / single"
+  validations:
+    required: true
+
+- type: textarea
+  id: cell-snippet
+  attributes:
+    label: Proposed cell snippet
+    description: |
+      The cell object exactly as it should land in the `cells: [...]` array
+      of the cookbook config. Auto-generated by the playground — please do not
+      hand-edit unless you're sure.
+    render: javascript
+  validations:
+    required: true
+
+- type: textarea
+  id: existing-cell
+  attributes:
+    label: Existing cell at this match (for diff)
+    description: |
+      The current verified cell at the same `match` tuple, if any. The
+      playground fills this so the maintainer can see exactly what changed.
+      Leave empty if no cell yet exists at this match.
+    render: javascript
+  validations:
+    required: false
+
+- type: input
+  id: sglang-version
+  attributes:
+    label: SGLang version
+    description: Version, tag, or git SHA you tested against.
+    placeholder: "sglang==0.5.4  (or git SHA abc1234)"
+  validations:
+    required: true
+
+- type: textarea
+  id: bench-result
+  attributes:
+    label: Benchmark result (optional but encouraged)
+    description: |
+      One-line perf numbers (TTFT, TPOT, tokens/sec, accepted-rate, etc.).
+      Helps maintainers decide whether this recipe should replace the
+      existing one or be added as an alternative.
+    placeholder: "TTFT 95 ms / TPOT 18 ms / 1820 tok/s @ bs=64"
+  validations:
+    required: false
+
+- type: textarea
+  id: notes
+  attributes:
+    label: Notes / caveats
+    description: |
+      Anything unusual: cluster config, env-var quirks, NIC mappings,
+      multi-node bootstrap details, etc.
+  validations:
+    required: false
+
+- type: checkboxes
+  id: attestation
+  attributes:
+    label: Attestation
+    description: All three must be ticked. We trust contributors but maintainers will still re-verify before merging.
+    options:
+      - label: I ran this exact command on the listed hardware.
+        required: true
+      - label: The server reached READY and answered a cURL request successfully.
+        required: true
+      - label: Output looked correct on at least one prompt.
+        required: true

.github/MAINTAINER.md

@@ -0,0 +1,167 @@
+# SGLang Code Maintenance Model
+This document describes the code maintenance model for the SGLang project.
+Since SGLang is a large project involving multiple organizations and hardware platforms, we designed this model with the following goals:
+- Ensure a responsive and smooth review process.
+- Allow for fast iteration, so maintainers can sometimes bypass flaky CI tests for important PRs.
+
+## Role Descriptions
+There are four roles in this maintenance model. Some are custom roles, while others are predefined by GitHub.
+
+- **Merge Oncall**: The person who drives the PR merge process. They have strong area-specific expertise and uphold a high bar for code quality.
+  - Permission: Merge PRs. Bypass branch protection rules if needed.
+  - Responsibility: Shepherd the merge of PRs assigned to their area. Revert or hotfix any issues related to their merge (especially if they bypass).
+- **Codeowner**: The person who protects critical code. Without a bypass, each PR needs at least one Codeowner approval for each modified file protected by [CODEOWNERS](./CODEOWNERS). Please note that this role is not an honor but a significant responsibility because PRs cannot be merged without your approval (except when bypassed by a Merge Oncall).
+  - Permission: Approve PRs, allowing them to be merged without a bypass.
+  - Responsibility: Review PRs in a timely manner.
+- **Write**: A person with write permission to the SGLang repo.
+  - Permission: Merge PRs if they have passed required tests and been approved by Codeowners. This role cannot bypass branch protection rules.
+  - Responsibility: Review and merge PRs in a timely manner.
+- **CI Oncall**: A person who manages CI runners for specific hardware platforms.
+  - Permission: Add CI runners.
+  - Responsibility: Keep the CI runners up and running.
+
+__Note__: Difference between Merge Oncall and Codeowner
+- The Merge Oncall is an active role held by someone who actively tries to help merge PRs and can bypass CI if needed.
+- The Codeowner is a passive protection role provided by GitHub; it prevents accidental changes to critical code.
+- The list of Merge Oncalls is attached below. The list of Codeowners is in the [CODEOWNERS](./CODEOWNERS) file.
+
+__Note__: The permissions to trigger CI tests are defined separately according to these [rules](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests).
+
+
+## Pull Request Merge Process
+1. The author submits a pull request (PR) and fills out the PR checklist.
+2. A bot assigns this PR to a Merge Oncall and @-mentions them. At the same time, GitHub will automatically request reviews from Codeowners.
+3. Someone tags the PR with a `run-ci` label ([help](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests)). Then the author can trigger CI by pushing new commits.
+4. The Merge Oncall coordinates the review (e.g., asking people to review) and approves the PR; the Codeowners also approve the PR. If the assigned Merge Oncall is not responsive, the author can ping other related Merge Oncalls and Reviewers in the list below.
+5. The code can now be merged:
+   - **Ideal case:** For each modified file, one Codeowner has approved the PR. The PR has also passed the required CI tests. Then, anyone with write permission can merge the PR.
+   - **Exception:** In cases where it is difficult to meet all requirements (due to flaky CI or slow responses), a Merge Oncall can bypass branch protection to merge the PR.
+
+If you meet any issues during the merge, you can discuss in [slack channels](https://slack.sglang.io/): #pull-request, #ci-cd-build-release, #dev.
+
+## The List of Merge Oncalls and Reviewers
+This section lists the oncalls for each module or feature.
+The format is @github-username (Slack username).
+
+### Scheduler
+[@merrymercy](https://github.com/merrymercy) (Lianmin Zheng), [@hnyls2002](https://github.com/hnyls2002) (Liangsheng Yin), [@cctry](https://github.com/cctry) (Shiyang Chen)
+
+related files
+- python/sglang/srt/managers
+- python/sglang/srt/model_executor
+
+### Diffusion
+[@mickqian](https://github.com/mickqian) (Mick), [@BBuf](https://github.com/BBuf) (BBuf)
+
+related files
+- python/sglang/multimodal_gen
+
+### PD disaggregation
+[@ByronHsu](https://github.com/ByronHsu) (Byron Hsu), [@cctry](https://github.com/cctry) (Shiyang Chen), [@ShangmingCai](https://github.com/ShangmingCai) (Shangming Cai)
+
+related files
+- python/sglang/srt/disaggregation
+
+### KV Cache
+[@ispobock](https://github.com/ispobock) (Ke Bao), [@xiezhq-hermann](https://github.com/xiezhq-hermann) (Zhiqiang Xie)
+
+related files
+- python/sglang/srt/mem_cache
+
+### Parallelism
+[@ch-wan](https://github.com/ch-wan) (Cheng Wan), [@fzyzcjy](https://github.com/fzyzcjy) (Tom)
+
+related files
+- python/sglang/srt/eplb
+- python/sglang/srt/distributed
+- python/sglang/srt/layers/dp_attention.py
+
+### Kernel
+[@BBuf](https://github.com/BBuf) (BBuf)
+
+related files
+- python/sglang/jit_kernel
+- sgl-kernel
+
+### Speculative decoding
+[@hnyls2002](https://github.com/hnyls2002) (Liangsheng Yin), [@Qiaolin-Yu](https://github.com/Qiaolin-Yu) (Qiaolin Yu)
+
+related files
+- python/sglang/srt/speculative
+
+### NV and model-specific optimizations
+[@Fridge003](https://github.com/Fridge003) (Baizhou Zhang), [@ishandhanani](https://github.com/ishandhanani) (Ishan Dhanani), [@Qiaolin-Yu](https://github.com/Qiaolin-Yu) (Qiaolin Yu)
+
+related files
+- python/sglang/srt/models
+- python/sglang/srt/layers/attention
+
+### AMD optimizations
+[@HaiShaw](https://github.com/HaiShaw) (Henry HAI)
+
+### NPU optimizations
+[@iforgetmyname](https://github.com/iforgetmyname) (Even Zhou)
+
+related files
+- python/sglang/srt/hardware_backend/npu
+
+### CI, Release, Package
+[@Kangyan-Zhou](https://github.com/Kangyan-Zhou) (Kangyan Zhou), [@Fridge003](https://github.com/Fridge003) (Baizhou Zhang)
+
+related files
+- .github/workflows
+
+### Router, API
+[@slin1237](https://github.com/slin1237) (Simo Lin)
+
+related files
+- sgl-model-gateway
+- python/sglang/srt/grpc
+- python/sglang/srt/entrypoints
+
+### Other Notes
+
+Now we have many Merge Oncalls mainly because the CI is flaky and the CODEOWNERS is too coarse-grained.
+In the future, we hope the CI can be improved and we only need bypass rarely. After that, most Merge Oncalls can be converted back to Write and CODEOWNERS.
+
+This list is based on the current situation. If you or someone you know would like to take on more responsibility and are qualified, please ping [Lianmin Zheng](https://github.com/merrymercy) and [Ying Sheng](https://github.com/Ying1123) in the Slack channel. They will start a nomination and internal review process.
+
+## The List of CI Oncalls
+This section lists the oncalls for each hardware platform. The format is @github-username (Slack username).
+
+### NVIDIA GPUs
+[@Kangyan-Zhou](https://github.com/Kangyan-Zhou) (Kangyan Zhou), [@ch-wan](https://github.com/ch-wan) (Cheng Wan), [@HanHan009527](https://github.com/HanHan009527) (hanhan), [@ishandhanani](https://github.com/ishandhanani) (Ishan Dhanani), [@ShangmingCai](https://github.com/ShangmingCai) (Shangming Cai), [@alisonshao](https://github.com/alisonshao) (Alison Shao).
+
+### AMD GPUs
+[@saienduri](https://github.com/saienduri) (Sai Enduri), [@HaiShaw](https://github.com/HaiShaw) (Henry HAI)
+
+### Intel CPU and XPU
+[@mingfeima](https://github.com/mingfeima) (Mingfei Ma), [@DiweiSun](https://github.com/DiweiSun) (Diwei Sun)
+
+### Ascend NPUs
+[@iforgetmyname](https://github.com/iforgetmyname) (Even Zhou)
+
+This list is based on the current situation. If you or someone you know would like to donate machines for CI, they can serve as the CI oncalls for their machines. Please ping [Lianmin Zheng](https://github.com/merrymercy) and [Ying Sheng](https://github.com/Ying1123) in the Slack channel. They will start a nomination and internal review process.
+
+## CI Maintenance Mode
+When the CI is unhealthy (e.g., the scheduled pr-test on `main` is broken for consecutive runs), the project enters **CI Maintenance Mode** by opening [issue #21065](https://github.com/sgl-project/sglang/issues/21065). While active:
+- All PR CI runs are paused. Resources are allocated to PRs that fix the CI.
+- **Merging non-CI-fix PRs is prohibited.** Only PRs that fix the CI may be merged. In severe cases, merge permissions may be revoked.
+
+Maintenance mode ends when `pr-test.yml` is all green on `main` and the issue is closed.
+
+### Rebase-Required Mode
+When a major update lands on `main` and all open PRs must rebase before CI can run (without fully pausing CI), add a line of the form `MIN_BASE_SHA: <sha>` to the body of issue #21065. **The rebase check is enforced regardless of whether the issue is open or closed** — you do not need to enter full maintenance mode (open the issue) to use this gate; just editing the body to include the directive is enough. While the directive is present:
+- CI is allowed to run only for PRs whose branch already contains `<sha>` (GitHub compare API status `ahead` or `identical` — i.e., the PR has `<sha>` in its history).
+- PRs that are `behind` or `diverged` from `<sha>` are blocked with a "rebase required" error until they rebase onto the latest `main`.
+- The `bypass-maintenance` label still bypasses this check for CI-fix PRs.
+
+Notes:
+- Only the **first** `MIN_BASE_SHA:` line in the issue body is read.
+- The SHA must be 7-40 hex characters; malformed values are ignored (with a warning in the job summary).
+- Avoid pasting the directive inside a fenced code block in the issue body — the parser does not skip code fences and may match example snippets.
+
+Remove the directive from the issue body to lift the rebase requirement (closing the issue does NOT lift it on its own).
+
+## Suspending Permissions
+If a Merge Oncall bypasses checks to merge a PR that breaks the `main` branch, merges a non-CI-fix PR during CI Maintenance Mode, or repeatedly breaks the CI due to various reasons, their privileges will be suspended for at least two days, depending on the severity of the incident.

.github/actions/check-maintenance/action.yml

@@ -0,0 +1,168 @@
+name: Check Maintenance Mode
+description: Blocks CI in two independent modes driven by issue #21065. (1) Full-pause: when the issue is open. (2) Rebase-required: whenever the issue body contains a `MIN_BASE_SHA: <sha>` directive — enforced regardless of whether the issue is open or closed, so maintainers can require all PRs to rebase past a specific commit without having to open the maintenance issue. Both modes are bypassed by the `bypass-maintenance` label on the PR, or by env PR_TEST_BYPASS_MAINTENANCE_ON_MAIN=true (PR Test workflow on main only). Merging non-CI-fix PRs is prohibited during full-pause; in severe cases, merge permissions may be revoked.
+
+inputs:
+  github-token:
+    description: GitHub token for API access
+    required: false
+    default: ${{ github.token }}
+
+runs:
+  using: composite
+  steps:
+    - name: Check maintenance mode
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.github-token }}
+      run: |
+        MAINTENANCE_ISSUE=21065
+        REPO="${{ github.repository }}"
+        PR_NUMBER="${{ github.event.pull_request.number }}"
+        PR_HEAD_SHA="${{ github.event.pull_request.head.sha }}"
+
+        # PR Test workflow only: scheduled runs and runs on main (dispatch / workflow_call) set this env
+        if [[ "${PR_TEST_BYPASS_MAINTENANCE_ON_MAIN:-}" == "true" ]]; then
+          echo "✅ PR Test on main branch; bypassing maintenance gate."
+          exit 0
+        fi
+
+        # Use curl + jq instead of `gh` because self-hosted GPU runners
+        # don't have the gh CLI installed. Without this, the action
+        # silently fell through to "Proceeding with CI" on every GPU
+        # job — the rebase gate only fired on ubuntu-latest jobs that
+        # had gh pre-installed. curl+jq are reliably available on every
+        # Linux runner.
+        gh_api() {
+          local path="$1"
+          local err_file="$2"
+          curl --silent --show-error --fail \
+            --max-time 30 \
+            -H "Authorization: Bearer $GH_TOKEN" \
+            -H "Accept: application/vnd.github+json" \
+            -H "X-GitHub-Api-Version: 2022-11-28" \
+            "https://api.github.com/$path" 2>"$err_file"
+        }
+
+        # Fetch issue state and body. Fail-open: if we can't read the issue
+        # (network blip, missing token scope), CI proceeds.
+        ERR_FILE=$(mktemp)
+        ISSUE_JSON=$(gh_api "repos/$REPO/issues/$MAINTENANCE_ISSUE" "$ERR_FILE" || true)
+        if [[ -z "$ISSUE_JSON" ]]; then
+          echo "⚠️  Issue fetch returned empty. curl stderr was:"
+          cat "$ERR_FILE" || true
+        fi
+        rm -f "$ERR_FILE"
+        ISSUE_STATE=$(printf '%s' "$ISSUE_JSON" | jq -r '.state // "UNKNOWN"' 2>/dev/null || echo "UNKNOWN")
+        # Issues API returns state in lowercase ("open"/"closed"); normalize
+        # to uppercase so existing comparisons against "OPEN" still work.
+        ISSUE_STATE=$(printf '%s' "$ISSUE_STATE" | tr '[:lower:]' '[:upper:]')
+        ISSUE_BODY=$(printf '%s' "$ISSUE_JSON" | jq -r '.body // ""' 2>/dev/null || echo "")
+        echo "DEBUG: ISSUE_STATE=$ISSUE_STATE  body_length=${#ISSUE_BODY}"
+
+        # Parse optional `MIN_BASE_SHA: <sha>` directive from the issue body
+        # (first occurrence wins). Whenever this directive is present, the
+        # rebase check is enforced regardless of whether the issue is open
+        # or closed — so maintainers can require all PRs to rebase past a
+        # specific commit without having to open the maintenance issue.
+        # `grep` exits 1 on no-match, which under `set -eo pipefail` (the
+        # default for `shell: bash` composite steps) would abort the whole
+        # script — silently failing every PR whose issue body has no
+        # MIN_BASE_SHA line. Wrap grep in a brace group with `|| true` so
+        # an empty match falls through cleanly to "no directive set".
+        MIN_BASE_SHA=$(
+          { printf '%s' "$ISSUE_BODY" | tr -d '\r' | grep -iE '^[[:space:]]*`?MIN_BASE_SHA`?[[:space:]]*[:=]' || true; } \
+            | head -n1 | sed -E 's/.*[:=][[:space:]]*//; s/`//g' | awk '{print $1}'
+        )
+        if [[ -n "$MIN_BASE_SHA" ]] && ! [[ "$MIN_BASE_SHA" =~ ^[a-fA-F0-9]{7,40}$ ]]; then
+          WARN="⚠️  Ignoring malformed MIN_BASE_SHA directive in issue #$MAINTENANCE_ISSUE: '$MIN_BASE_SHA' (must be 7-40 hex chars)"
+          echo "$WARN"
+          echo "$WARN" >> "$GITHUB_STEP_SUMMARY"
+          MIN_BASE_SHA=""
+        fi
+
+        # If neither gate is active (no MIN_BASE_SHA, issue not open), nothing to do.
+        if [[ -z "$MIN_BASE_SHA" && "$ISSUE_STATE" != "OPEN" ]]; then
+          echo "✅ Maintenance mode is OFF and no MIN_BASE_SHA directive. Proceeding with CI."
+          exit 0
+        fi
+
+        # bypass-maintenance label bypasses both gates. PR labels live on
+        # the issue resource (PRs are issues with extra metadata in the
+        # GH API), so we hit the same /issues/{n} endpoint.
+        if [[ -n "$PR_NUMBER" ]]; then
+          ERR_FILE=$(mktemp)
+          PR_JSON=$(gh_api "repos/$REPO/issues/$PR_NUMBER" "$ERR_FILE" || true)
+          rm -f "$ERR_FILE"
+          HAS_BYPASS=$(printf '%s' "$PR_JSON" | jq -r '[.labels[]?.name] | map(select(. == "bypass-maintenance")) | length' 2>/dev/null || echo "0")
+          if [[ "${HAS_BYPASS:-0}" -gt 0 ]]; then
+            echo "✅ PR #$PR_NUMBER has 'bypass-maintenance' label. Bypassing maintenance + rebase checks."
+            exit 0
+          fi
+        fi
+
+        # Rebase-required gate (independent of issue open/closed state).
+        if [[ -n "$MIN_BASE_SHA" ]]; then
+          if [[ -z "$PR_NUMBER" || -z "$PR_HEAD_SHA" ]]; then
+            echo "✅ Not a PR context; skipping rebase check."
+          else
+            # Use GitHub compare API: status is "ahead"/"identical" when MIN_BASE_SHA is reachable from PR head.
+            ERR_FILE=$(mktemp)
+            COMPARE_JSON=$(gh_api "repos/$REPO/compare/$MIN_BASE_SHA...$PR_HEAD_SHA" "$ERR_FILE" || true)
+            if [[ -z "$COMPARE_JSON" ]]; then
+              echo "⚠️  Compare API failed. curl stderr was:"
+              cat "$ERR_FILE" || true
+            fi
+            rm -f "$ERR_FILE"
+            COMPARE_STATUS=$(printf '%s' "$COMPARE_JSON" | jq -r '.status // "UNKNOWN"' 2>/dev/null || echo "UNKNOWN")
+            COMPARE_STATUS="${COMPARE_STATUS:-UNKNOWN}"
+
+            case "$COMPARE_STATUS" in
+              ahead|identical)
+                echo "✅ PR #$PR_NUMBER contains required base ${MIN_BASE_SHA:0:12} ($COMPARE_STATUS)."
+                ;;
+              UNKNOWN)
+                echo "⚠️  Could not determine rebase status via GitHub API; fail-open, allowing rebase check to pass."
+                ;;
+              *)
+                MSG=$(printf "%s\n" \
+                  "## ⚠️ Rebase Required Before CI Can Run" \
+                  "A major update has landed on \`main\`. All PRs must rebase onto the latest \`main\` before CI will run." \
+                  "Required base commit: \`${MIN_BASE_SHA:0:12}\` (your PR is \`$COMPARE_STATUS\` relative to this commit)." \
+                  "" \
+                  "What should you do?" \
+                  "- Rebase your branch onto the latest \`main\` and push again" \
+                  "- Follow CI Maintenance Mode issue: https://github.com/$REPO/issues/$MAINTENANCE_ISSUE for context" \
+                  "- CI-fix PRs may request the \`bypass-maintenance\` label to skip this check")
+                echo "$MSG" >> "$GITHUB_STEP_SUMMARY"
+                while IFS= read -r line; do
+                  echo "::error::$line"
+                done <<< "$MSG"
+                exit 1
+                ;;
+            esac
+          fi
+        fi
+
+        # Full-pause maintenance gate (only when issue is open).
+        if [[ "$ISSUE_STATE" == "OPEN" ]]; then
+          MSG=$(printf "%s\n" \
+            "## ⚠️ CI Maintenance Mode is Active" \
+            "The CI infrastructure is currently under maintenance." \
+            "All PR CI runs are paused until maintenance is complete." \
+            "**Merging non-CI-fix PRs is prohibited during maintenance mode.** In severe cases, merge permissions may be revoked." \
+            "You might also experience unexpected failures during this period." \
+            "The team is working on the issue and will update the status as soon as possible." \
+            "" \
+            "What should you do?" \
+            "- **Do NOT merge non-CI-fix PRs** until maintenance mode is lifted" \
+            "- Check back later (~12 hours)" \
+            "- Follow CI Maintenance Mode issue: https://github.com/$REPO/issues/$MAINTENANCE_ISSUE for status updates")
+
+          echo "$MSG" >> "$GITHUB_STEP_SUMMARY"
+          while IFS= read -r line; do
+            echo "::error::$line"
+          done <<< "$MSG"
+          exit 1
+        fi
+
+        echo "✅ Rebase check passed; full-pause not active. Proceeding with CI."

.github/actions/check-pr-test-health/action.yml

@@ -0,0 +1,98 @@
+name: Check PR Test Health
+description: Fail fast if any job in the current workflow run has already failed, or if the lint check (from lint.yml) has failed. Auto-skips for scheduled runs. The jobs-failed check (but not the lint check) is bypassed when the PR carries the `bypass-fastfail` label.
+
+inputs:
+  github-token:
+    description: 'GitHub token for API calls'
+    required: false
+    default: ${{ github.token }}
+
+runs:
+  using: composite
+  steps:
+    - name: Check PR test health
+      uses: actions/github-script@v8
+      env:
+        SKIP_PR_TEST_HEALTH_CHECK: ${{ env.SKIP_PR_TEST_HEALTH_CHECK }}
+      with:
+        github-token: ${{ inputs.github-token }}
+        script: |
+          // Skip when explicitly requested via env var (e.g. release branch cut)
+          if (process.env.SKIP_PR_TEST_HEALTH_CHECK === 'true') {
+            core.info('Skipping health check (SKIP_PR_TEST_HEALTH_CHECK=true)');
+            return;
+          }
+
+          // Skip for scheduled runs — they should collect all failures, not fast-fail
+          if (context.eventName === 'schedule') {
+            core.info('Skipping health check for scheduled run');
+            return;
+          }
+
+          // Check lint status from the separate Lint workflow (lint.yml).
+          // listJobsForWorkflowRun only sees jobs within the SAME run, so we use
+          // checks.listForRef which queries by commit SHA across ALL workflows.
+          const ref = context.payload.pull_request?.head?.sha || context.sha;
+          const { data } = await github.rest.checks.listForRef({
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            ref: ref,
+            check_name: 'lint',
+          });
+          const lintRun = data.check_runs.find(
+            cr => cr.app?.slug === 'github-actions'
+          );
+          if (lintRun?.status === 'completed' && lintRun?.conclusion === 'failure') {
+            core.setFailed('Fast-fail: lint check failed');
+            return;
+          }
+
+          // Skip the jobs-failed check when the PR carries the bypass-fastfail label.
+          // Lint check above still runs.
+          let labels = [];
+          if (context.payload.pull_request?.labels) {
+            labels = context.payload.pull_request.labels.map(l => l.name);
+          } else {
+            const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              commit_sha: ref,
+            });
+            if (prs.length > 0) {
+              labels = prs[0].labels.map(l => l.name);
+            }
+          }
+          if (labels.includes('bypass-fastfail')) {
+            core.info('Skipping jobs-failed check (bypass-fastfail label present)');
+            return;
+          }
+
+          const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            run_id: context.runId,
+            per_page: 100,
+          });
+          // Find jobs that failed from a real error, not from fast-fail cascade
+          const rootCauseFailures = jobs.filter(j => {
+            if (j.status !== 'completed' || j.conclusion !== 'failure') return false;
+            // h20 runners are flaky (dirty GPU state from prior runs); their failures
+            // should not cascade fast-fail to other stages. j.name shape from
+            // listJobsForWorkflowRun: "<job-key>" + optional " / <reusable-job>"
+            // + optional " (<matrix>)". Split off the base job key before exact
+            // match so we cover both inline + reusable forms without confusing
+            // 'h20' with the 'h200' prefix.
+            const baseName = j.name.split(/[ /]/)[0];
+            if (baseName === 'base-c-test-8-gpu-h20') {
+              return false;
+            }
+            // If the failing step is the health check, it's a cascade — skip it
+            const failedStep = (j.steps || []).find(s => s.conclusion === 'failure');
+            if (failedStep && (failedStep.name.includes('check-pr-test-health') || failedStep.name.includes('Check PR test health'))) {
+              return false;
+            }
+            return true;
+          });
+          if (rootCauseFailures.length > 0) {
+            core.setFailed(`Fast-fail: skipping — root cause job(s): ${rootCauseFailures.map(j => j.name).join(', ')}`);
+          }

.github/actions/upload-cuda-coredumps/action.yml

@@ -0,0 +1,101 @@
+name: Upload CUDA Coredumps
+description: Upload CUDA coredump files as artifacts, optionally signal to a tracker issue, and clean up.
+
+inputs:
+  artifact-suffix:
+    description: Suffix appended to the artifact name (e.g. matrix partition id)
+    required: false
+    default: ""
+  retention-days:
+    description: Number of days to retain the artifact
+    required: false
+    default: "7"
+  tracker-issue:
+    description: |
+      If set, post a one-line comment to sgl-project/sglang issue
+      #<tracker-issue> when at least one coredump is detected. Requires
+      `bot-token` with issues:write on sgl-project/sglang.
+    required: false
+    default: ""
+  bot-token:
+    description: PAT with issues:write on sgl-project/sglang. Required when tracker-issue is set.
+    required: false
+    default: ""
+
+runs:
+  using: composite
+  steps:
+    - name: Check for coredumps
+      id: check
+      shell: bash
+      run: |
+        # Mirror get_dump_dir() in python/sglang/srt/debug_utils/cuda_coredump.py:
+        # explicit override > per-job RUNNER_TEMP > /tmp default, then a
+        # per-(run, attempt) subdir so dumps are never mis-attributed across CI
+        # jobs that share a self-hosted runner's filesystem.
+        if [ -n "$SGLANG_CUDA_COREDUMP_DIR" ]; then
+          base="$SGLANG_CUDA_COREDUMP_DIR"
+        elif [ -n "$RUNNER_TEMP" ]; then
+          base="$RUNNER_TEMP/sglang_cuda_coredumps"
+        else
+          base="/tmp/sglang_cuda_coredumps"
+        fi
+        if [ -n "$GITHUB_RUN_ID" ]; then
+          dir="$base/${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT:-1}"
+        else
+          dir="$base"
+        fi
+        echo "dump_dir=$dir" >> "$GITHUB_OUTPUT"
+        if [ -d "$dir" ] && [ -n "$(ls -A "$dir" 2>/dev/null)" ]; then
+          echo "has_dumps=true" >> "$GITHUB_OUTPUT"
+        else
+          echo "has_dumps=false" >> "$GITHUB_OUTPUT"
+        fi
+
+    - name: Upload CUDA coredumps
+      if: steps.check.outputs.has_dumps == 'true'
+      uses: actions/upload-artifact@v4
+      with:
+        name: cuda-coredumps-${{ github.job }}${{ inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) }}
+        path: ${{ steps.check.outputs.dump_dir }}/
+        retention-days: ${{ inputs.retention-days }}
+
+    - name: Signal coredump to tracker issue
+      if: steps.check.outputs.has_dumps == 'true' && inputs.tracker-issue != '' && inputs.bot-token != ''
+      shell: bash
+      env:
+        BOT_TOKEN: ${{ inputs.bot-token }}
+        PR_NUM: ${{ github.event.pull_request.number }}
+        EVENT_NAME: ${{ github.event_name }}
+        TRACKER_ISSUE: ${{ inputs.tracker-issue }}
+      run: |
+        if [ -n "$PR_NUM" ]; then
+          ref_label="PR #${PR_NUM}"
+        else
+          ref_label="$EVENT_NAME"
+        fi
+        # Resolve own job_id via REST API: match by runner_name + status
+        # in_progress (the current job is the only one in_progress on this
+        # runner). Robust across matrix expansions and artifact-suffix shapes.
+        job_id=$(curl -sS \
+          -H "Authorization: Bearer ${BOT_TOKEN}" \
+          -H "Accept: application/vnd.github+json" \
+          "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}/jobs?per_page=100" \
+          | python3 -c 'import json,sys,os; print(next((j["id"] for j in json.load(sys.stdin)["jobs"] if j.get("runner_name")==os.environ["RUNNER_NAME"] and j.get("status")=="in_progress"), ""))')
+        if [ -n "$job_id" ]; then
+          run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/job/${job_id}"
+        else
+          # Fallback to run-attempt URL if job_id lookup failed
+          run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}"
+        fi
+        body_json=$(printf '{"body":"@hnyls2002 [Coredump Tracker] %s - %s"}' "${ref_label}" "${run_url}")
+        curl -sS -X POST \
+          -H "Authorization: Bearer ${BOT_TOKEN}" \
+          -H "Accept: application/vnd.github+json" \
+          -H "X-GitHub-Api-Version: 2022-11-28" \
+          "https://api.github.com/repos/sgl-project/sglang/issues/${TRACKER_ISSUE}/comments" \
+          -d "${body_json}"
+
+    - name: Cleanup CUDA coredumps
+      shell: bash
+      run: rm -rf "${{ steps.check.outputs.dump_dir }}"

.github/actions/wait-for-jobs/action.yml

@@ -0,0 +1,229 @@
+name: Wait for Jobs
+description: Poll and wait for specified jobs in the current workflow run to complete. Returns success immediately when the PR carries the `bypass-fastfail` label, letting downstream stages dispatch in parallel (same effect as scheduled runs).
+
+inputs:
+  stage-name:
+    description: 'Human-readable stage name for log messages (e.g. "base-a")'
+    required: true
+  jobs:
+    description: |
+      JSON array of job specs to wait for. Each element is either:
+        - a string: exact job name (e.g. "base-a-test-1-gpu-small")
+        - an object { "prefix": "...", "expected_count": N }: for matrix jobs
+    required: true
+  max-wait-minutes:
+    description: 'Maximum time to wait before timing out'
+    required: false
+    default: '240'
+  poll-interval-seconds:
+    description: 'Seconds between polling attempts'
+    required: false
+    default: '60'
+  github-token:
+    description: 'GitHub token for API calls'
+    required: false
+    default: ${{ github.token }}
+
+outputs:
+  result:
+    description: 'Overall result: success, failure, or timeout'
+    value: ${{ steps.wait.outputs.result }}
+
+runs:
+  using: composite
+  steps:
+    - name: Wait for jobs to complete
+      id: wait
+      uses: actions/github-script@v8
+      env:
+        INPUT_STAGE_NAME: ${{ inputs.stage-name }}
+        INPUT_JOBS: ${{ inputs.jobs }}
+        INPUT_MAX_WAIT_MINUTES: ${{ inputs.max-wait-minutes }}
+        INPUT_POLL_INTERVAL_SECONDS: ${{ inputs.poll-interval-seconds }}
+      with:
+        github-token: ${{ inputs.github-token }}
+        script: |
+          const stageName = process.env.INPUT_STAGE_NAME;
+          const jobSpecs = JSON.parse(process.env.INPUT_JOBS);
+          const maxWaitMinutes = parseInt(process.env.INPUT_MAX_WAIT_MINUTES);
+          const pollIntervalSeconds = parseInt(process.env.INPUT_POLL_INTERVAL_SECONDS);
+          const maxAttempts = (maxWaitMinutes * 60) / pollIntervalSeconds;
+
+          // bypass-fastfail label opts the PR out of stage-to-stage waiting,
+          // letting all stages dispatch in parallel like scheduled runs do.
+          let labels = [];
+          if (context.payload.pull_request?.labels) {
+            labels = context.payload.pull_request.labels.map(l => l.name);
+          } else {
+            const ref = context.payload.pull_request?.head?.sha || context.sha;
+            try {
+              const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                commit_sha: ref,
+              });
+              if (prs.length > 0) {
+                labels = prs[0].labels.map(l => l.name);
+              }
+            } catch (e) {
+              console.log(`Could not fetch PR labels for ${ref}: ${e.message}`);
+            }
+          }
+          if (labels.includes('bypass-fastfail')) {
+            console.log(`Skipping ${stageName} wait (bypass-fastfail label present)`);
+            core.setOutput('result', 'success');
+            return;
+          }
+
+          // Normalize job specs into a uniform format
+          const normalizedSpecs = jobSpecs.map(spec => {
+            if (typeof spec === 'string') {
+              return { prefix: spec, expected_count: 1, exact: true };
+            }
+            return { ...spec, exact: false };
+          });
+
+          const totalExpectedJobs = normalizedSpecs.reduce((sum, s) => sum + s.expected_count, 0);
+
+          const matchesSpec = (jobName, spec) => {
+            if (spec.exact) {
+              return jobName === spec.prefix;
+            }
+            // Match bare prefix or any GHA-rendered shard suffix. Inline matrix
+            // produces `<prefix> (<shard>)`; reusable-workflow callers produce
+            // `<prefix> / <called-job> (<shard>)`. Both have a space after the
+            // prefix, so a single ' '-delimited check covers both.
+            return jobName === spec.prefix || jobName.startsWith(spec.prefix + ' ');
+          };
+
+          // Use ETag conditional requests to avoid consuming rate limit when nothing changed.
+          // GitHub returns 304 Not Modified for unchanged data, which is FREE (no rate limit cost).
+          let lastEtag = '';
+          let lastJobs = null;
+          let apiCalls = 0;
+          let cachedCalls = 0;
+
+          async function fetchJobs() {
+            const url = `GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs`;
+            const params = {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              run_id: context.runId,
+              per_page: 100,
+              headers: {},
+            };
+            if (lastEtag) {
+              params.headers['if-none-match'] = lastEtag;
+            }
+
+            try {
+              const response = await github.request(url, params);
+              apiCalls++;
+              const rateRemaining = response.headers['x-ratelimit-remaining'] || '?';
+              const rateLimit = response.headers['x-ratelimit-limit'] || '?';
+              console.log(`[rate-limit] ${rateRemaining}/${rateLimit} remaining (ETag: ${lastEtag ? 'sent' : 'none'}) | this session: ${apiCalls} paid, ${cachedCalls} free`);
+              lastEtag = response.headers.etag || '';
+              const jobs = response.data.jobs;
+
+              // Handle pagination if >100 jobs
+              // ETag only covers page 1, so invalidate it to avoid stale cache
+              // when later pages change but page 1 doesn't.
+              if (response.data.total_count > 100) {
+                lastEtag = '';
+                for (let page = 2; page <= Math.ceil(response.data.total_count / 100); page++) {
+                  const { data: pageData } = await github.request(url, {
+                    ...params,
+                    page,
+                    headers: {},
+                  });
+                  jobs.push(...pageData.jobs);
+                }
+              }
+
+              lastJobs = jobs;
+              return { jobs, cached: false };
+            } catch (err) {
+              if (err.status === 304 && lastJobs) {
+                cachedCalls++;
+                console.log(`[rate-limit] 304 Not Modified | this session: ${apiCalls} paid, ${cachedCalls} free`);
+                return { jobs: lastJobs, cached: true };
+              }
+              throw err;
+            }
+          }
+
+          for (let attempt = 0; attempt < maxAttempts; attempt++) {
+            const { jobs, cached } = await fetchJobs();
+
+            let allCompleted = true;
+            let failedJobs = [];
+            let completedCount = 0;
+            let totalCount = 0;
+
+            for (const spec of normalizedSpecs) {
+              const matchingJobs = jobs.filter(job => matchesSpec(job.name, spec));
+
+              for (const job of matchingJobs) {
+                totalCount++;
+                if (!cached) {
+                  console.log(`${job.name}: status=${job.status}, conclusion=${job.conclusion}`);
+                }
+
+                if (job.status === 'completed') {
+                  completedCount++;
+                  if (job.conclusion !== 'success' && job.conclusion !== 'skipped') {
+                    failedJobs.push(job.name);
+                  }
+                } else {
+                  allCompleted = false;
+                }
+              }
+
+              if (matchingJobs.length < spec.expected_count) {
+                // Job-level `if:` is evaluated before matrix expansion. When it
+                // evaluates false, GitHub emits exactly one "skipped" entry
+                // without a "(shard)" suffix instead of N matrix entries. Two
+                // shapes are possible:
+                //   - inline matrix:     `<prefix>`
+                //   - reusable workflow: `<prefix> / <called-job>`
+                // Both lack the ` (` shard marker. Detect that precise shape so
+                // we don't poll forever — and so we don't mistake a partially
+                // materialized dynamic/reusable matrix for a skipped one.
+                const unexpandedSkip = matchingJobs.length === 1 &&
+                  !matchingJobs[0].name.includes(' (') &&
+                  matchingJobs[0].status === 'completed' &&
+                  matchingJobs[0].conclusion === 'skipped';
+                if (unexpandedSkip) {
+                  const missing = spec.expected_count - 1;
+                  totalCount += missing;
+                  completedCount += missing;
+                  if (!cached) {
+                    console.log(`${spec.prefix}: job-level skip (bare entry, conclusion=skipped); treating as all ${spec.expected_count} skipped`);
+                  }
+                } else {
+                  console.log(`${spec.prefix}: found ${matchingJobs.length}/${spec.expected_count} jobs (waiting for more)`);
+                  allCompleted = false;
+                }
+              }
+            }
+
+            console.log(`[${stageName}] Progress: ${completedCount}/${totalCount} jobs completed (expected ${totalExpectedJobs})${cached ? ' (cached, no rate limit cost)' : ''}`);
+
+            // Fail fast if any jobs failed
+            if (failedJobs.length > 0) {
+              core.setOutput('result', 'failure');
+              core.setFailed(`${stageName} jobs failed: ${failedJobs.join(', ')}`);
+              return;
+            }
+
+            if (allCompleted && totalCount >= totalExpectedJobs) {
+              core.setOutput('result', 'success');
+              return;
+            }
+
+            console.log(`Waiting ${pollIntervalSeconds}s... (attempt ${attempt + 1}/${maxAttempts})`);
+            await new Promise(resolve => setTimeout(resolve, pollIntervalSeconds * 1000));
+          }
+
+          core.setFailed(`Timeout waiting for ${stageName} jobs`);
+          core.setOutput('result', 'timeout');