chore: import upstream snapshot with attribution

docs_new/docs/advanced_features/adaptive_speculative_decoding.mdx

@@ -0,0 +1,269 @@
+---
+title: "Adaptive Speculative Decoding"
+metatags:
+    description: "Configure adaptive speculative decoding so SGLang can adjust speculative steps and draft tokens at runtime based on acceptance behavior."
+---
+
+Adaptive speculative decoding lets SGLang adjust `speculative_num_steps/speculative_num_draft_tokens` at runtime instead of keeping a single fixed value for the whole server lifetime.
+It is designed for workloads whose accept length changes over time, where one static step count is rarely optimal.
+
+## Current support
+
+- Only `--speculative-algorithm EAGLE` or `EAGLE3`
+- Only `--speculative-eagle-topk 1`
+- If either condition is not met, SGLang falls back to static speculative settings
+
+## Why adaptive steps help
+
+`speculative_num_steps` controls how many draft-model autoregressive steps run in each speculative round. In practice, the best value depends on the current workload.
+
+- If `num_steps` is too small, the draft model could have produced more accepted tokens, but the round stops too early.
+- If `num_steps` is too large, the draft model produces many candidate tokens that the target model rejects, so extra draft work is wasted.
+- At **high batch sizes**, the cost of each wasted draft step is multiplied across all sequences in the batch, so the optimal step count is often lower than at low batch sizes.
+
+Real traffic often moves between high-acceptance and low-acceptance phases, and batch sizes vary continuously. Adaptive mode follows both signals at runtime instead of hard-coding a single global `num_steps`.
+
+## Design overview
+
+The adaptive mechanism has three pieces:
+
+- `AdaptiveSpeculativeParams`: the EMA-based policy
+- `SpecRuntimeState`: the per-tier runtime state bundle
+- `AdaptiveController`: the coordinator that queries the policy for the current batch size and activates the matching runtime state
+
+### Per-batch-size independent tracking
+
+The controller maintains **independent EMA trackers for each batch size range**, so observations at small BS don't pollute the large BS signal. Each BS range can have its own candidate steps, hysteresis thresholds, and ceiling coefficient.
+
+BS ranges are defined as lower bounds in the config file (e.g., keys `"1"` and `"8"` mean BS 1–7 uses one slot, BS 8+ uses another). `SpecRuntimeState` objects are shared across BS ranges with the same step count — each state owns CUDA graphs captured for the reachable padded batch sizes of that step.
+
+```mermaid
+---
+title: "SpecRuntimeState — speculative_num_steps / speculative_num_draft_tokens"
+---
+graph LR
+  subgraph SR[" "]
+    direction LR
+    subgraph D["Draft stage"]
+      direction TB
+      d1[attn_backend]
+      d2[cuda_graph]
+    end
+    subgraph V["Verify stage"]
+      direction TB
+      v1[attn_backend]
+      v2[cuda_graph]
+    end
+    subgraph E["Extend stage"]
+      direction TB
+      e1[attn_backend]
+      e2[cuda_graph]
+    end
+  end
+```
+
+This matters because `CudaGraphRunner` is shape-dependent. Each candidate tier owns its own graph and backend state, so runtime switching is a reference swap, not an online graph recapture.
+
+## Runtime flow
+
+The adaptive update happens in two places:
+
+1. **Pre-draft**: query the optimal step for the current batch size and activate if different
+2. **Post-verify**: update the matching BS slot's EMA with observed accept lengths
+
+```mermaid
+---
+title: "EAGLEWorker.forward_batch_generation() — decode path"
+---
+flowchart TD
+  Z["⓪ activate_step_by_batch(batch_size)<br/>query optimal step for current BS range, activate if different"]
+  A["① draft(batch)<br/>draft model multi-step generation with current tier"]
+  B["② verify(batch, spec_info)<br/>target model tree verification → produces num_correct_drafts_per_req"]
+  C["③ forward_draft_extend_after_decode(batch)<br/>draft model KV-cache catch-up"]
+  D["④ adaptive_controller.on_verify_complete(num_correct_drafts_per_req, batch_size)<br/>update EMA for matching BS slot, apply warmup / interval / hysteresis gates<br/>if tier changed, select a pre-built state from pool"]
+  E["worker.apply_runtime_state(state)"]
+  Z --> A --> B --> C --> D --> E
+```
+
+> Tier switch happens after the current round completes. Backends and CUDA graphs are never swapped mid-round.
+
+## How the policy decides
+
+After each verify pass, SGLang reads the accepted draft length per request, computes the batch average, smooths it with an exponential moving average (EMA), and switches among the candidate tiers for the matching BS slot.
+
+The decision logic is intentionally conservative:
+
+- `warmup_batches` skips the first few batches
+- `update_interval` avoids switching every batch
+- `down_hysteresis` and `up_hysteresis` reduce oscillation
+- `ceiling_coeff` — an optional EMA ceiling rule can cap `num_steps` proportionally to observed draft quality, preventing over-speculation at high BS
+
+Conceptually, the policy probes one step beyond the observed acceptance:
+
+```text
+target_steps ≈ clamp(round(ema_accept_len) + 1, min(candidate_steps), max(candidate_steps))
+```
+
+So if recent requests consistently accept more drafted tokens, the policy tends to move up. If they start rejecting earlier, it tends to move down.
+
+## Usage
+
+`--speculative-adaptive-config` is optional, but the speculative setup still needs to be valid for adaptive mode.
+
+```bash
+python3 -m sglang.launch_server \
+    --model meta-llama/Llama-2-7b-chat-hf \
+    --speculative-algorithm EAGLE \
+    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
+    --speculative-eagle-topk 1 \
+    --speculative-num-steps 3 \
+    --speculative-num-draft-tokens 4 \
+    --speculative-adaptive
+```
+
+If you want to override the defaults, add `--speculative-adaptive-config /path/to/adaptive_spec.json`.
+
+Example config:
+
+```json
+{
+  "ema_alpha": 0.2,
+  "warmup_batches": 10,
+  "update_interval": 5,
+  "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0},
+  "8": {"candidate_steps": [1],       "up_hysteresis": 0.0, "down_hysteresis": 0.0,   "ceiling_coeff": 0}
+}
+```
+
+Non-integer keys (`ema_alpha`, `warmup_batches`, `update_interval`) are global overrides applied to every BS slot. Integer keys (`"1"`, `"8"`) define per-BS slots.
+
+## Config file reference
+
+The config file is optional. When provided, each integer BS-slot key must specify `candidate_steps`; all other keys fall back to defaults.
+
+### Per-BS slot parameters
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Key</th>
+      <th>Default</th>
+      <th>Meaning</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>candidate_steps</code></td>
+      <td><em>required</em></td>
+      <td>Candidate <code>speculative_num_steps</code> tiers for this BS range. Must be a non-empty list of positive ints; a slot that omits it raises a config error</td>
+    </tr>
+    <tr>
+      <td><code>down_hysteresis</code></td>
+      <td><code>-0.25</code></td>
+      <td>Extra margin before moving to a smaller step</td>
+    </tr>
+    <tr>
+      <td><code>up_hysteresis</code></td>
+      <td><code>0.0</code></td>
+      <td>Extra margin before moving to a larger step</td>
+    </tr>
+    <tr>
+      <td><code>ceiling_coeff</code></td>
+      <td><code>0</code> (disabled)</td>
+      <td>EMA ceiling coefficient; set &gt; 0 to cap steps proportionally to draft quality</td>
+    </tr>
+  </tbody>
+</table>
+
+### Global parameters
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Key</th>
+      <th>Default</th>
+      <th>Meaning</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>ema_alpha</code></td>
+      <td><code>0.2</code></td>
+      <td>EMA smoothing factor for accepted draft length</td>
+    </tr>
+    <tr>
+      <td><code>update_interval</code></td>
+      <td><code>5</code></td>
+      <td>Recompute interval, in verify batches, after warmup</td>
+    </tr>
+    <tr>
+      <td><code>warmup_batches</code></td>
+      <td><code>10</code></td>
+      <td>Number of verify batches to observe before switching</td>
+    </tr>
+  </tbody>
+</table>
+
+## Monitoring
+
+You can inspect the active tier and acceptance metric via `/server_info`:
+
+```bash
+curl -s http://127.0.0.1:30000/server_info | jq '.internal_states[0] | {speculative_num_steps, avg_spec_accept_length}'
+```
+
+- `speculative_num_steps` is the current active tier
+- `avg_spec_accept_length` helps explain whether the server is likely to move up or down
+
+## Tuning tips
+
+- Start with the built-in default (conservative) — it is safe for all draft model qualities
+- For strong draft models, use the aggressive config with ceiling rule
+- Use fewer candidate steps if you want lower startup GPU memory overhead
+- Increase `ema_alpha` to react faster, or lower it for more stability
+- Increase `warmup_batches` or `update_interval` if tier switching is too noisy
+- At high batch sizes, narrower ladders (e.g., `[1, 2]` or `[1]`) often outperform wide ones
+- If your workload is already stable and one static setting is well tuned, adaptive mode may not help much
+
+## Recommended configs
+
+The built-in default is conservative — safe for all draft models but may under-speculate for strong ones. Save one of these as a JSON file and pass via `--speculative-adaptive-config`.
+
+### Conservative (default) — for weak draft models
+
+This is the built-in default: BS 8–31 allows `[1, 3]`, and BS≥32 locks to `step=1` to avoid wasted compute. Best for models like MiniMax-M2.5, DSV4.
+
+```json
+{
+  "1":  {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0},
+  "8":  {"candidate_steps": [1, 3],    "up_hysteresis": 0.0, "down_hysteresis": 0.0,   "ceiling_coeff": 0},
+  "32": {"candidate_steps": [1],       "up_hysteresis": 0.0, "down_hysteresis": 0.0,   "ceiling_coeff": 0}
+}
+```
+
+### Aggressive — for strong or high-variance draft models
+
+Uses wider ladders with ceiling rule to cap speculation at high BS. Best for models like GLM-4.7-FP8.
+
+```json
+{
+  "1":   {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0},
+  "8":   {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 3.0},
+  "64":  {"candidate_steps": [1, 3],    "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.67},
+  "128": {"candidate_steps": [1, 3],    "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.2}
+}
+```
+
+### Custom per-model config
+
+For the best performance, benchmark your specific model across batch sizes with different static `num_steps` values, then build a per-BS config that matches each range's optimal step. A well-tuned per-model config might outperform the generic presets above.

docs_new/docs/advanced_features/attention_backend.mdx

@@ -0,0 +1,709 @@
+---
+title: "Attention Backend"
+metatags:
+    description: "SGLang attention backend guide: FlashInfer, FA3, FA4, Triton, FlashMLA, TRTLLM MLA, hybrid attention. Support matrix for MHA and MLA models."
+---
+SGLang supports a large variety of attention backends. Each of them has different pros and cons.
+You can test them according to your needs.
+
+<Warning>
+Selecting an optimal attention backend is crucial for maximizing your performance. Different backends excel in various scenarios, so choose based on your model, hardware, and use case. Not all backends are supported on all platforms and model architectures.
+
+If you don't specify `--attention-backend`, SGLang makes a best effort to automatically select the most performant backend based on your hardware and model architecture.
+</Warning>
+
+## Support Matrix
+
+The support matrix is split into two parts: MHA (standard attention) and MLA (multi-head latent attention). For an explanation of the key differences between MHA and MLA, please see the [SGLang documentation on DeepSeek MLA](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-4-mla-optimizations) and the original [DeepSeek MLA paper](https://arxiv.org/pdf/2405.04434).
+
+### MHA Backends
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Backend**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Page Size > 1 (native)**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**FP8 KV Cache**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**FP4 KV Cache**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Spec topk=1**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Spec topk>1**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Sliding Window**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**MultiModal**</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashInfer**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA3 (FlashAttention 3)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA4 (FlashAttention 4)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>128</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Triton**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Torch Native (SDPA)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlexAttention (PyTorch)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**TRTLLM MHA**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16, 32 or 64</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Dual Chunk FlashAttention**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**AITER (ROCm)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Wave (ROCm)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Ascend (NPU)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Intel XPU**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Intel AMX (CPU)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+  </tbody>
+</table>
+
+### MLA Backends
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "15%"}} />
+    <col style={{width: "15%"}} />
+    <col style={{width: "14%"}} />
+    <col style={{width: "14%"}} />
+    <col style={{width: "14%"}} />
+    <col style={{width: "14%"}} />
+    <col style={{width: "14%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Backend**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Native Page Sizes**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**FP8 KV Cache**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**FP4 KV Cache**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Chunked Prefix Cache**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Spec topk=1**</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Spec topk>1**</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashInfer MLA**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashMLA**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Cutlass MLA**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>128</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**TRTLLM MLA (Blackwell)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32 or 64</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**CuteDSL MLA (Blackwell)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32 or 64</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**TokenSpeed MLA (Blackwell)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32 or 64</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ (required)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA3 (FlashAttention 3)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>n/a</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>⚠️ (page_size=1 only)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Triton**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>n/a</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>⚠️ (page_size=1 only)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FA4**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Ascend MLA (NPU)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>128</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+    </tr>
+  </tbody>
+</table>
+
+<Note>
+Multimodal attention is selected by `--mm-attention-backend`. The "MultiModal" column indicates whether a corresponding multimodal implementation exists for that backend family.
+</Note>
+
+<Note>
+- DSA is specifically designed for [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). See the [DSA Attention Backend](#dsa-attention-backend) section and [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) for details.
+</Note>
+
+<Note>
+For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-backend to run. Except for trtllm_mha being incompatible with FA4, all other decode backends behave as shown in the table.
+</Note>
+
+<Tip>
+Speculative decoding topk: `topk` is the number of draft tokens sampled per step from the draft model. `topk = 1` follows classic EAGLE; `topk > 1` explores multiple branches and requires backend support in both draft and verification paths.
+</Tip>
+
+<Tip>
+Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching). Note that higher page sizes generally improve attention kernel performance, so prefer `page_size > 1` when prefix cache reuse is not critical.
+</Tip>
+
+Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), CuteDSL MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128).
+
+MLA page-size constraints:
+- FlashInfer MLA: page_size = 1.
+- FlashMLA: page_size = 64.
+- Cutlass MLA: page_size = 128.
+- TRTLLM MLA: page_size ∈ &#123;32, 64&#125;.
+- CuteDSL MLA: page_size ∈ &#123;32, 64&#125; (decode-only; prefill falls back to `trtllm_mla` when unset).
+- TokenSpeed MLA: page_size ∈ &#123;32, 64&#125; (Blackwell SM100/SM12x only; requires `--kv-cache-dtype fp8_e4m3`).
+
+### GDN Attention Backends
+
+GDN (Gated Delta Network) is a linear attention mechanism with O(n) complexity, used in hybrid models that alternate GDN linear attention layers with standard full attention layers. GDN is **not** selected via `--attention-backend`; it is automatically activated when the model architecture requires it (e.g., Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM).
+
+The GDN linear attention layers have their own kernel backends, selected via `--linear-attn-backend` (default: `triton`). You can override the kernel per phase with `--linear-attn-decode-backend` and `--linear-attn-prefill-backend`.
+
+On SM100/SM103 with CUDA 13+, SGLang automatically selects FlashInfer for GDN prefill when the per-phase override is unset, the base linear-attention backend is Triton, recurrent state is BF16, key/value head dimensions are 128, dynamic chunking and page-major KV layout are disabled, and `--chunked-prefill-size` is between 1 and 8192. Radix caching may be disabled or use the `no_buffer` strategy; extra-buffer strategies require state checkpoint support.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "28%"}} />
+    <col style={{width: "16%"}} />
+    <col style={{width: "24%"}} />
+    <col style={{width: "32%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Decode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Prefill / Extend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Spec Decoding (Target Verify)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (CUDA)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (AMD/ROCm)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (NPU)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (CPU)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>CuTe DSL (CUDA only)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FlashInfer (CUDA, SM90/SM100/SM103)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ linear chain; tree falls back to Triton</td>
+    </tr>
+  </tbody>
+</table>
+
+<Warning>
+GDN models are hybrid: the full-attention layers still require a standard `--attention-backend`. Platform constraints for the full-attention backend on hybrid GDN models:
+- **Blackwell (e.g., B200)**: `triton`, `trtllm_mha`, or `fa4` only.
+- **NPU (Ascend)**: `ascend` only.
+- **AMD (ROCm)**: `triton` recommended.
+- **Other CUDA (Hopper, Ampere, etc.)**: auto-selection works; no special constraints.
+</Warning>
+
+### DSA Attention Backend
+
+DSA (DeepSeek Sparse Attention) is a native sparse attention mechanism used by [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). It is activated automatically when the model architecture requires it and is selected via `--attention-backend dsa` (deprecated alias: `nsa`).
+
+Internally, the DSA backend dispatches to different sub-backends for prefill and decode phases. You can override these with `--dsa-prefill-backend` and `--dsa-decode-backend`:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "26%"}} />
+    <col style={{width: "16%"}} />
+    <col style={{width: "16%"}} />
+    <col style={{width: "42%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Sub-backend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Prefill</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Decode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>flashmla_sparse</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default prefill on Hopper and Blackwell (BF16)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>flashmla_kv</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default for FP8 on Hopper (prefill + decode)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>flashmla_auto</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Picks flashmla_sparse or flashmla_kv by KV cache dtype</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>fa3</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default decode on Hopper (BF16)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>trtllm</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default decode on Blackwell (BF16); default for FP8 on Blackwell (prefill + decode)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>tilelang</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default on AMD (ROCm)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>aiter</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>AMD-specific kernel library (requires aiter package)</td>
+    </tr>
+  </tbody>
+</table>
+
+For deployment examples, see the [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2).
+
+### Hybrid attention (different backends for prefill vs decode) (Experimental)
+
+<Warning>
+Hybrid attention is an experimental feature.
+</Warning>
+
+You can mix-and-match attention backends for prefill and decode. This is useful when one backend excels at prefill and another excels at decode. For the implementation details, please see `python/sglang/srt/layers/attention/hybrid_attn_backend.py`.
+
+```bash Command
+# Example: Prefill with FA4, Decode with TRTLLM MLA (Blackwell)
+python3 -m sglang.launch_server \
+  --model-path nvidia/DeepSeek-R1-FP4 \
+  --tp 8 \
+  --attention-backend trtllm_mla \
+  --moe-runner-backend flashinfer_trtllm \
+  --quantization modelopt_fp4 \
+  --prefill-attention-backend fa4
+```
+
+#### Speculative decoding with hybrid attention
+
+Hybrid attention also works with speculative decoding. The backend used for draft decoding and target verification depends on `--speculative-attention-mode`:
+
+- `--speculative-attention-mode decode` (recommended): draft/verify use the decode backend.
+- `--speculative-attention-mode prefill` (default): draft/verify use the prefill backend.
+
+Constraints when combining hybrid attention with speculative decoding:
+
+- If any attention backend is `trtllm_mha`, speculative decoding supports only `--speculative-eagle-topk 1`.
+- For paged MHA backends with `--page-size > 1` and `--speculative-eagle-topk > 1`, only `flashinfer` is supported.
+- CUDA Graph: the decode backend is always captured; the prefill backend is captured only when `--speculative-attention-mode prefill`.
+
+
+<Tip>
+If you set only one of `--prefill-attention-backend` or `--decode-attention-backend`, the unspecified phase inherits `--attention-backend`.
+If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase.
+</Tip>
+
+## Attention Backend Selection Guide (CUDA)
+
+If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture.
+
+### Automatic Selection Logic
+
+**1. MHA Models (e.g., Llama, Qwen)**
+- **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported.
+- **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`.
+- **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`.
+
+**2. MLA Models (e.g., DeepSeek V3)**
+- **Hopper**: Defaults to `fa3` (requires CUDA 12.3+).
+- **Blackwell**: Defaults to `flashinfer`; `trtllm_mla` is auto-selected for DeepSeek V3 models specifically.
+- **Other Architectures**: Defaults to `triton`.
+
+
+## User Guide
+
+### Launch Command for Different Attention Backends
+
+- FlashInfer (Default for Non-Hopper Machines, e.g., A100, A40)
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend flashinfer
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-V3 \
+  --attention-backend flashinfer \
+  --trust-remote-code
+```
+
+- FlashAttention 3 (Default for Hopper Machines, e.g., H100, H200, H20)
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend fa3
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-V3 \
+  --trust-remote-code \
+  --attention-backend fa3
+```
+
+- Triton
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend triton
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-V3 \
+  --attention-backend triton \
+  --trust-remote-code
+```
+
+- FlashMLA
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-R1 \
+  --attention-backend flashmla \
+  --trust-remote-code
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-R1 \
+  --attention-backend flashmla \
+  --kv-cache-dtype fp8_e4m3 \
+  --trust-remote-code
+```
+
+- TRTLLM MLA (Optimized for Blackwell Architecture, e.g., B200)
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-R1 \
+  --attention-backend trtllm_mla \
+  --trust-remote-code
+```
+
+- TRTLLM MLA with FP8 KV Cache (Higher concurrency, lower memory footprint)
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-R1 \
+  --attention-backend trtllm_mla \
+  --kv-cache-dtype fp8_e4m3 \
+  --trust-remote-code
+```
+
+- TRTLLM MHA (Optimized for Blackwell Architecture, e.g., B200)
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 4 \
+  --model Qwen/Qwen3.5-35B-A3B-FP8 \
+  --attention-backend trtllm_mha \
+  --trust-remote-code
+```
+
+- TRTLLM MHA (XQA backend) (Optimized for SM90 and SM120, e.g., H20, H200, 5090)
+  Note that TRTLLM XQA backend only works well for pagesize 64.
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 4 \
+  --model Qwen/Qwen3.5-35B-A3B-FP8 \
+  --decode-attention-backend trtllm_mha \
+  --trust-remote-code
+```
+
+- FlashAttention 4 (MHA & MLA)
+```bash Command
+# FA4 for both prefill and decode on SM90/SM100
+python3 -m sglang.launch_server \
+  --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
+  --attention-backend fa4 \
+  --page-size 128 \
+  --trust-remote-code
+
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-R1 \
+  --prefill-attention-backend fa4 \
+  --trust-remote-code
+```
+
+- Cutlass MLA
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 8 \
+  --model deepseek-ai/DeepSeek-R1 \
+  --attention-backend cutlass_mla \
+  --trust-remote-code
+```
+
+- Ascend
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend ascend
+```
+
+- Intel XPU
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend intel_xpu
+```
+
+- Wave
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend wave
+```
+
+- FlexAttention
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend flex_attention
+```
+
+- Dual Chunk FlashAttention
+```bash Command
+python3 -m sglang.launch_server \
+  --model Qwen/Qwen2.5-14B-Instruct-1M \
+  --attention-backend dual_chunk_flash_attn
+```
+
+- Torch Native
+```bash Command
+python3 -m sglang.launch_server \
+  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
+  --attention-backend torch_native
+```
+
+## Steps to add a new attention backend
+To add a new attention backend, you can learn from the existing backends
+(`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`)
+and follow the steps below.
+
+<Note>
+Linear attention kernel backends (GDN, KDA) follow a different pattern. They implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/` and are dispatched by `GDNKernelDispatcher` / `KDAKernelDispatcher` rather than registered via `@register_attention_backend`.
+</Note>
+
+1. Run without cuda graph. Support the two forward functions
+    - forward_extend
+        - Will be used for prefill, prefill with KV cache, and target verification
+        - It will be called once per layer
+    - forward_decode
+        - Will be used for normal decode, and draft decode
+        - It will be called once per layer
+    - init_forward_metadata
+        - Initialize the class and common metadata shared by all layers
+        - Call the plan function for optimizations like split_kv
+        - It will be called once per forward
+2. Run with cuda graph. It has two phases (capture and replay) and you need to implement three functions
+    - init_cuda_graph_state
+        - It will be called once during life time
+        - Create all common shared buffers
+    - init_forward_metadata_capture_cuda_graph
+        - It will be called before capturing a cuda graph
+        - It is similar to init_forward_metadata but write the medatada to some pre-defined buffers
+    - init_forward_metadata_replay_cuda_graph
+        - It will be called before replaying a cuda graph
+        - This function is in the critical path and needs to be fast

docs_new/docs/advanced_features/breakable_cuda_graph.mdx

@@ -0,0 +1,192 @@
+---
+title: "Breakable CUDA Graph"
+metatags:
+    description: "Use Breakable CUDA Graph to insert targeted eager graph breaks for debugging and CUDA graph compatibility."
+---
+
+## Motivation
+
+Standard CUDA graphs capture an entire forward pass as a single, opaque graph. This is great for performance, but creates two problems:
+
+1. **Debugging is hard.** When something goes wrong inside a captured graph (wrong outputs, numerical mismatches, crashes), there is no way to step through the operations or insert print statements because the graph replays as a monolithic unit.
+
+2. **Some ops are incompatible.** Certain operations — dynamic control flow, host-device synchronization, JIT compilation, or ops that change behavior across iterations — cannot be captured into a CUDA graph at all. Today, the only workaround is to disable CUDA graphs entirely, which sacrifices the kernel launch overhead savings for the rest of the model.
+
+**Breakable CUDA Graph** solves both problems by allowing graph breaks to be inserted at specific points. The computation is split into multiple captured graph segments with eager (non-graph) execution in between. This preserves most of the CUDA graph performance benefit while allowing targeted operations to run outside the graph.
+
+## Usage
+
+### Debug Mode: Run Everything Eagerly
+
+The simplest use case is debugging. The `--debug-cuda-graph` flag wraps the entire decode forward pass in a graph break, so every operation runs eagerly while still going through the full CUDA graph capture/replay code path. This lets you debug CUDA graph issues without changing model code.
+
+```bash
+python -m sglang.launch_server \
+    --model meta-llama/Llama-3.1-8B-Instruct \
+    --debug-cuda-graph
+```
+
+This mode is intended for debugging only — it eliminates the performance benefit of CUDA graphs since every op runs eagerly.
+
+### Selective Graph Breaks in Model Code
+
+For production use, you can mark specific functions as "non-graphable" using the `@eager_on_graph` decorator. During CUDA graph capture, these functions run eagerly between captured graph segments. Outside of capture, they behave normally.
+
+```python
+from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import eager_on_graph
+
+@eager_on_graph(enable=True)
+def my_dynamic_op(x):
+    # This op is incompatible with CUDA graph capture
+    return some_dynamic_operation(x)
+```
+
+You can also insert a bare graph break (no computation) using the `break_graph()` helper:
+
+```python
+from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import break_graph
+
+def forward(self, x):
+    x = self.layer1(x)
+    break_graph()  # force a segment split here
+    x = self.layer2(x)
+    return x
+```
+
+To enable breakable CUDA graph at the environment level (without debug mode), set the environment variable:
+
+```bash
+export SGLANG_USE_BREAKABLE_CUDA_GRAPH=1
+python -m sglang.launch_server \
+    --model meta-llama/Llama-3.1-8B-Instruct
+```
+
+### Server Args
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "18%"}} />
+    <col style={{width: "48%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--debug-cuda-graph</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable debug/eager mode. Wraps the entire forward pass in a graph break so every op runs eagerly through the capture/replay path.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_USE_BREAKABLE_CUDA_GRAPH</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>0</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Environment variable. Enables breakable CUDA graph without debug mode. Required for <code>@eager_on_graph</code> decorators to take effect.</td>
+    </tr>
+  </tbody>
+</table>
+
+## How It Works
+
+### Capture
+
+Breakable CUDA graph extends PyTorch's `torch.cuda.CUDAGraph` by splitting a single capture into multiple segments separated by graph breaks.
+
+During capture, the flow is:
+
+```
+Begin capture (segment 1)
+  ... graphable ops ...
+  @eager_on_graph function encountered:
+    1. End current capture segment
+    2. Run the function eagerly (allocates output tensors)
+    3. Record the function for later replay
+    4. Begin new capture segment
+  ... more graphable ops ...
+End capture (segment N)
+```
+
+Each segment is independently instantiated as a CUDA graph executable. The non-graph functions and their argument references are stored for replay.
+
+### Replay
+
+During replay:
+
+```
+For each segment i:
+  1. Launch CUDA graph segment i
+  2. Run the recorded non-graph function i eagerly
+Launch final CUDA graph segment
+```
+
+The non-graph functions are re-invoked with the same tensor references as capture time. Since these references point to the CUDA graph's static input/output buffers, they see updated values on each replay.
+
+### Output Writeback
+
+When a non-graph function produces output during replay, the result must be written back into the same tensor buffers that downstream graph segments reference. The mechanism handles:
+
+- **Plain tensors**: In-place `copy_()` into the original buffer.
+- **Structured outputs** (dataclasses, objects with tensor attributes): Tensor fields are copied in-place; non-tensor fields are replaced.
+- **Dicts of tensors**: Tensor values are copied in-place; non-tensor values are replaced.
+
+### Stream Fork/Join Tracking
+
+Some models fork work onto secondary CUDA streams (e.g., for overlapped computation). Breakable CUDA graph hooks `torch.cuda.Stream.wait_stream` to track which streams are forked from the capture stream. When a graph break occurs, all forked streams are automatically joined back before ending the capture segment, and re-forked after beginning the next segment.
+
+## Compatibility
+
+- **CUDA and ROCm/HIP.** Breakable CUDA graph runs on both NVIDIA and AMD GPUs. Other platforms (NPU, CPU, MPS, XPU) are unsupported; there `--debug-cuda-graph` is automatically disabled with a warning.
+- **Requires `cuda-python` on NVIDIA.** Stream-capture-status queries use the CUDA runtime via `cuda.bindings` (`pip install cuda-python`); the portable `torch.cuda.is_current_stream_capturing()` has proven unreliable on CUDA. On ROCm/HIP — where `cuda-python` is unavailable — the portable `torch.cuda` API (which maps to the HIP runtime) is used instead.
+- **Not compatible with memory saver mode.** Cannot be used together with `SGLANG_MEMORY_SAVER_CUDA_GRAPH`.
+
+## Performance
+
+When no graph breaks are inserted, breakable CUDA graph has minimal overhead compared to standard CUDA graph — the capture/replay path is nearly identical.
+
+Each graph break adds:
+- One `cudaGraphLaunch` call (to replay the segment before the break)
+- One eager Python function call
+- One `cudaStreamBeginCapture` / `cudaStreamEndCapture` pair during capture
+
+For typical use cases with a small number of graph breaks, the overhead is negligible compared to the saved kernel launch overhead from the captured segments.
+
+## Code Reference
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "52%"}} />
+    <col style={{width: "48%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>File</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Core implementation: <code>eager_on_graph</code>, <code>BreakableCUDAGraph</code>, <code>BreakableCUDAGraphCapture</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/cuda_utils.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA runtime binding utilities (NVIDIA stream-capture queries)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/runner_backend/breakable_cuda_graph_backend.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Integration with CUDA graph runner backends</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/server_args.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--debug-cuda-graph</code> flag and environment variable handling</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/environ.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>SGLANG_USE_BREAKABLE_CUDA_GRAPH</code> environment variable definition</td>
+    </tr>
+  </tbody>
+</table>

docs_new/docs/advanced_features/checkpoint_engine.mdx

@@ -0,0 +1,257 @@
+---
+title: "Checkpoint Engine Integration"
+metatags:
+    description: "SGLang checkpoint engine: distributed model weight loading, parallel multi-node setup, broadcast and P2P modes. Reduces loading time for large models."
+---
+The SGLang checkpoint engine integration provides an efficient way to load model weights using a distributed checkpoint loading system. This feature significantly reduces model loading time, especially for large models and multi-node setups, by parallelizing the weight loading process across multiple processes and nodes.
+
+## Overview
+
+The checkpoint engine integration allows SGLang to:
+- Load model weights in parallel using multiple processes
+- Distribute weight loading across multiple nodes to increase effective disk bandwidth
+- Overlap weight loading with other initialization tasks like CUDA graph capture
+- Support both single-node and multi-node deployments
+
+## Installation
+
+First, install the checkpoint engine package:
+
+```bash Command
+pip install 'checkpoint-engine[p2p]'
+```
+
+## Architecture
+
+The system consists of two main components:
+
+1. **SGLang Server**: Runs with `--wait-for-initial-weights` flag to wait for weights before becoming ready
+2. **Checkpoint Engine Workers**: Separate processes (managed by torchrun) that load and distribute model weights
+
+The checkpoint engine uses a parameter server architecture with support for:
+- **Broadcast mode**: Weights are broadcast from loading processes to inference processes
+- **P2P mode**: Direct peer-to-peer weight transfer between processes
+- **All mode**: Combination of both broadcast and P2P methods
+
+## Usage Examples
+
+### Single Node Setup
+
+**Terminal 1 - Launch SGLang Server:**
+```bash Command
+python -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --tp 8 \
+    --load-format dummy \
+    --wait-for-initial-weights
+```
+
+**Terminal 2 - Run Checkpoint Engine:**
+
+Using sglang entrypoint:
+```bash Command
+python -m sglang.srt.checkpoint_engine.update \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 8
+```
+
+Using torchrun directly:
+```bash Command
+torchrun --nproc-per-node 8 \
+    examples/checkpoint_engine/update.py \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 8
+```
+
+### Multi-Node Setup (2 Nodes)
+
+**Node 0:**
+
+Launch SGLang server:
+```bash Command
+python -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --tp 8 \
+    --load-format dummy \
+    --wait-for-initial-weights \
+    --host [IP]
+```
+
+Run checkpoint engine:
+
+Using sglang entrypoint (recommended):
+```bash Command
+python -m sglang.srt.checkpoint_engine.update \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 8
+```
+
+Using torchrun directly:
+```bash Command
+torchrun --nproc-per-node 8 \
+    --nnodes 2 \
+    --node-rank 0 \
+    --master-addr [IP] \
+    --master-port 29500 \
+    examples/checkpoint_engine/update.py \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 8
+```
+
+**Node 1:**
+
+Launch SGLang server:
+```bash Command
+python -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --tp 8 \
+    --load-format dummy \
+    --wait-for-initial-weights \
+    --host [IP]
+```
+
+Run checkpoint engine:
+
+Using sglang entrypoint (recommended):
+```bash Command
+python -m sglang.srt.checkpoint_engine.update \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 8
+```
+
+Using torchrun directly:
+```bash Command
+torchrun --nproc-per-node 8 \
+    --nnodes 2 \
+    --node-rank 1 \
+    --master-addr [IP] \
+    --master-port 29500 \
+    examples/checkpoint_engine/update.py \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 8
+```
+
+### Multi-Node Setup with Tensor Parallelism (TP=16)
+
+**Node 0:**
+
+Launch SGLang server:
+```bash Command
+python -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --tp 8 \
+    --load-format dummy \
+    --wait-for-initial-weights \
+    --host [IP] \
+    --dist-init-addr [IP]:9120 \
+    --nnodes 2 \
+    --node-rank 0
+```
+
+Run checkpoint engine:
+
+Using sglang entrypoint (recommended):
+```bash Command
+python -m sglang.srt.checkpoint_engine.update \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 16
+```
+
+Using torchrun directly:
+```bash Command
+torchrun --nproc-per-node 8 \
+    --nnodes 2 \
+    --node-rank 0 \
+    --master-addr [IP] \
+    --master-port 29500 \
+    examples/checkpoint_engine/update.py \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 16
+```
+
+**Node 1:**
+
+Launch SGLang server:
+```bash Command
+python -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --tp 8 \
+    --load-format dummy \
+    --wait-for-initial-weights \
+    --host [IP] \
+    --dist-init-addr [IP]:9120 \
+    --nnodes 2 \
+    --node-rank 1
+```
+
+Run checkpoint engine:
+
+Using sglang entrypoint (recommended):
+```bash Command
+python -m sglang.srt.checkpoint_engine.update \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 16
+```
+
+Using torchrun directly:
+```bash Command
+torchrun --nproc-per-node 8 \
+    --nnodes 2 \
+    --node-rank 1 \
+    --master-addr [IP] \
+    --master-port 29500 \
+    examples/checkpoint_engine/update.py \
+    --update-method broadcast \
+    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
+    --inference-parallel-size 16
+```
+
+## Configuration Options
+
+### SGLang Server Options
+
+- `--load-format dummy`: Use dummy format for initial loading (allows overlapping with other tasks)
+- `--wait-for-initial-weights`: Wait for checkpoint engine to provide weights before becoming ready
+- `--host`: Host address for multi-node setups
+- `--dist-init-addr`: Distributed initialization address for tensor parallelism
+
+### Checkpoint Engine Options
+
+- `--update-method`: Weight update method (`broadcast`, `p2p`, or `all`)
+- `--checkpoint-path`: Path to model checkpoint directory
+- `--inference-parallel-size`: Number of inference parallel processes
+- `--endpoint`: SGLang server endpoint (default: `http://localhost:19730`)
+- `--checkpoint-name`: Name for the checkpoint (default: `my-checkpoint-iter-0`)
+- `--save-metas-file`: File to save checkpoint metadata
+- `--load-metas-file`: File to load checkpoint metadata from
+- `--uds`: Unix domain socket path for communication
+- `--weight-version`: Version identifier for weights
+
+## Performance Benefits
+
+The checkpoint engine provides significant time savings in two main aspects:
+
+1. **Multi-node Loading**: Each node only loads a portion of weights from disk, effectively increasing disk bandwidth. More participating nodes provide greater acceleration. Preliminary tests show 20-second acceleration when loading DeepSeek-R1 on H20-3e with two nodes.
+
+2. **Single Process Optimization**: Using dummy format allows overlapping disk-to-CPU transfer with CUDA graph capture and other initialization tasks, providing additional time savings.
+
+## Troubleshooting
+
+- Ensure checkpoint engine package is installed: `pip install 'checkpoint-engine[p2p]'`
+- Verify network connectivity between nodes in multi-node setups
+- Check that the checkpoint path contains valid model files
+- Monitor logs for connection errors between SGLang server and checkpoint engine
+- Use `--sleep-time` parameter to add delays if needed for debugging
+
+## References
+
+- [Checkpoint Engine Repository](https://github.com/MoonshotAI/checkpoint-engine)

docs_new/docs/advanced_features/cuda_graph_for_multi_modal_encoder.mdx

@@ -0,0 +1,76 @@
+---
+title: "Cuda Graph for Multi-Modal Encoder in SGLang"
+metatags:
+    description: "CUDA Graph optimization for ViT in SGLang: reduce kernel launch overhead, dynamic input handling, support for Qwen2.5-VL and Qwen3-VL models."
+---
+## Motivation
+
+In multimodal reasoning services, the visual encoder (ViT / Vision Transformer) typically has a few characteristic traits:
+
+Many layers, fragmented operators: Each layer includes LN, QKV projections, attention, MLP, residual connections, etc., resulting in extremely frequent kernel launches.
+
+Server-side “small batch / low latency” is common: The batch size is very small (sometimes it looks like 1 after “flattening” the batch), so kernel launch overhead accounts for a large portion of end-to-end latency.
+
+Input token count (number of patches) varies frequently: Different image/video resolutions and different batch composition lead to different sequence lengths
+S — and this is precisely the biggest obstacle for CUDA Graph (unstable shapes).
+
+The value of CUDA Graph: It captures a long sequence of GPU kernels with fixed shapes and fixed memory addresses into a graph; later, for the same shapes, it can replay the graph directly, dramatically reducing launch overhead and making GPU scheduling more compact.
+
+This led us to seek a CUDA Graph enabled feature for ViT in order to improve ViT performance.
+
+## Design and Restrictions
+
+The new CUDA Graph enabled ViT logic is built on ViTCudaGraphRunner. This runner captures the "blocks + merger + deepstack merger (optional)" part of a vision transformer into a CUDA graph and replays it for identical shapes. See the following design consideration and restrictions for more details.
+
+### Dynamic inputs to fit static constraints of CUDA Graph
+
+Variable sequence length S is very common in ViT. While CUDA Graph requires fixed shapes. The solution is to build a graph cache by S(e.g., graph_key = S). The first time create a new S, and then capture a graph; afterwards, replay it.
+
+If there are many distinct S values, we need to increase VRAM usage which is graph-private memory pools for many graphs.
+
+### Stable addresses
+
+Everything "parameter-like" becomes a static buffer:
+
+- block_input / block_ws / block_output
+- cu_full_len / cu_window_len and their kk variants
+- sin_cos_ws
+
+In this way to solve the underlying requirement: during replay, not allowed to swap tensors, can only modify tensor contents.
+
+### Attention backend arguments
+Attention backend arguments are fixed inside the graph:
+
+TritonAttn expects [cu_seqlens, cu_seqlens_kk, max_len]
+FA3 expects [cu_seqlens, max_len]
+
+max_len is frozen as an int constant.
+cu_seqlens is cached into a dict during create_graph(), and its contents are not updated during subsequent replays.
+
+For the same graph_key = S, you not only require the input shape to match, but also require the segmentation pattern in cu_seqlens (and window seqlens) to be identical. Otherwise, attention will segment the sequence incorrectly.
+
+### Rotary buffer management
+The feature reallocates a larger sin_cos_ws when seq_len increases.
+The max_content_len is used to make sure the maximum size of the allocated rotary buffer.
+
+
+## Command Example
+You can enable CUDA Graph for ViT by setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1`, for example:
+```shell Command
+SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
+python3 -m sglang.launch_server \
+  --model Qwen/Qwen3-VL-8B-Instruct
+```
+Or you can run CUDA Graph for ViT together with Piecewise CUDA Graph feature by both setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` and setting `--enable-piecewise-cuda-graph`, for example:
+```shell Command
+SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
+python3 -m sglang.launch_server \
+  --model Qwen/Qwen3-VL-8B-Instruct \
+  --piecewise-cuda-graph-max-tokens 4096 \
+  --enable-piecewise-cuda-graph \
+  --piecewise-cuda-graph-compiler eager
+```
+
+## Known supported models
+- Qwen2.5-VL (https://github.com/sgl-project/sglang/pull/14422)
+- Qwen3-VL (https://github.com/sgl-project/sglang/pull/15320)

docs_new/docs/advanced_features/deterministic_inference.mdx

@@ -0,0 +1,215 @@
+---
+title: "Deterministic Inference"
+metatags:
+    description: "SGLang deterministic inference: consistent outputs for RL training, testing, and production. Supports FlashInfer, FA3, Triton backends with CUDA Graph."
+---
+## Why Deterministic Inference Matters
+
+Deterministic inference ensures consistent LLM outputs across runs, which is critical for:
+- **Reinforcement Learning**: Ensures consistent logprobs across runs, reducing stochastic noise and making RL training more stable, reproducible, and debuggable.
+- **Testing & Debugging**: Enables reproducible validation
+- **Production**: Improves reliability and user experience
+
+Even with `temperature=0`, standard LLM inference can produce different outputs due to dynamic batching and varying reduction orders in GPU kernels.
+
+## The Root Cause of Non-Determinism
+
+The main source is **varying batch sizes**. Different batch sizes cause GPU kernels to split reduction operations differently, leading to different addition orders. Due to floating-point non-associativity (`(a + b) + c ≠ a + (b + c)`), this produces different results even for identical inputs.
+
+
+## SGLang's Solution
+
+Building on [Thinking Machines Lab's batch-invariant operators](https://github.com/thinking-machines-lab/batch_invariant_ops), SGLang achieves fully deterministic inference while maintaining compatibility with chunked prefill, CUDA graphs, radix cache, and non-greedy sampling. The development roadmap for deterministic inference features can be found in this [issue](https://github.com/sgl-project/sglang/issues/10278).
+
+### Supported Backends
+
+Deterministic inference is only supported with the following three attention backends: **FlashInfer**, **FlashAttention 3 (FA3)**, and **Triton**.
+
+The following table shows feature compatibility for deterministic inference across different attention backends:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Attention Backend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA Graph</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Chunked Prefill</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Radix Cache</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Non-greedy Sampling (Temp > 0)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashInfer**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌ No</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**FlashAttention 3 (FA3)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Triton**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅ Yes</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
+    </tr>
+  </tbody>
+</table>
+
+## Usage
+
+### Basic Usage
+
+Enable deterministic inference by adding the `--enable-deterministic-inference` flag:
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --attention-backend fa3 \
+    --enable-deterministic-inference
+```
+
+### Server Arguments
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-deterministic-inference`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>flag; default: disabled</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable deterministic inference with batch-invariant operations</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>string; default: fa3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Choose attention backend (flashinfer, fa3, or triton)</td>
+    </tr>
+  </tbody>
+</table>
+
+### Example Configurations
+
+#### Qwen3-8B
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --attention-backend flashinfer \
+    --enable-deterministic-inference
+```
+
+#### Llama Models
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --attention-backend fa3 \
+    --enable-deterministic-inference
+```
+
+#### Qwen3-30B-A3B (MoE Model)
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen3-30B-A3B \
+    --attention-backend fa3 \
+    --enable-deterministic-inference
+```
+
+### Deterministic Inference with Non-Greedy Sampling (Temperature > 0)
+
+SGLang supports deterministic inference even with non-greedy sampling by using sampling seeds. This is particularly useful for reinforcement learning scenarios like GRPO (Group Relative Policy Optimization) where you need multiple diverse but reproducible responses.
+
+#### Default Behavior
+
+By default, SGLang uses a sampling seed of `42` for reproducible sampling:
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "Tell me a joke",
+        "sampling_params": {
+            "temperature": 0.8,  # Non-greedy sampling
+            "max_new_tokens": 128,
+        },
+    },
+)
+print(response.json())
+# This will always produce the same response across runs
+```
+
+#### Generating Multiple Reproducible Responses
+
+To sample different responses from the same prompt while maintaining reproducibility (e.g., for GRPO training), provide different sampling seeds in your requests:
+
+```python Example
+import requests
+
+# Prepare a list of sampling seeds for different responses
+sampling_seeds = [42, 43, 44, 45, 46]
+
+responses = []
+for seed in sampling_seeds:
+    response = requests.post(
+        "http://localhost:30000/generate",
+        json={
+            "text": "Tell me a joke",
+            "sampling_params": {
+                "temperature": 0.8,
+                "max_new_tokens": 128,
+                "sampling_seed": seed,  # Specify sampling seed
+            },
+        },
+    )
+    responses.append(response.json())
+
+# Each seed will produce a different but reproducible response
+# Using the same seed will always produce the same response
+```
+
+This approach ensures that:
+- Different seeds produce diverse responses
+- The same seed always produces the same response across different runs
+- Results are reproducible for debugging and evaluation
+
+
+## Verification
+
+Run deterministic tests to verify consistent outputs:
+
+```bash Command
+# Single test: same prompt, varying batch sizes
+python3 -m sglang.test.test_deterministic --test-mode single --n-trials 50
+
+# Prefix test: prompts with different prefix lengths
+python3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50
+
+# Radix Cache Consistency mode: test radix cache determinism (cached vs uncached prefill)
+python3 -m sglang.test.test_deterministic --test-mode radix_cache
+```
+
+Expected result: All tests should show `Unique samples: 1` (perfectly deterministic).

docs_new/docs/advanced_features/dp_dpa_smg_guide.mdx

@@ -0,0 +1,509 @@
+---
+title: "DP, DPA and SGLang DP Router"
+metatags:
+    description: "Learn the differences between Data Parallelism, Data Parallelism Attention, and SGLang Model Gateway routing for production DP deployments."
+---
+
+This guide explains the difference between Data Parallelism (DP) and Data Parallelism Attention (DPA), how to enable each mode correctly, and how to use the SGLang Model Gateway (SMG) for production-grade DP deployments.
+
+## Data Parallelism (DP)
+
+**Data Parallelism (DP)** is the most common parallelism strategy that replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. Each GPU set handles independent requests. With dedicated routing strategies, as we will introduce later, with those proper routing algorithms in SGLang Model Gateway, the throughput of your serving system could be multiplied nearly linearly.
+
+### Key characteristics
+
+- Each replica has a full copy of the model
+- Requests are distributed/scattered across replicas
+- No inter-replica communication during one request's inference (for simple DP)
+
+## Data Parallelism Attention (DPA)
+
+**Data Parallelism Attention (DPA)**, also known as DP Attention, is an advanced parallelism strategy. While DPA provides the most significant benefits for **Multi-Head Latent Attention (MLA)** models (such as DeepSeek, MiniMax, Kimi-K2), it also supports **standard attention models** like Qwen.
+
+### The Problem with Tensor Parallelism for MLA Models
+
+The most common parallelism strategy for inference is **Tensor Parallelism (TP)**. However, TP might not be the most efficient strategy for certain models. For example, DeepSeek models use MLA and only have **one KV head**. If we use tensor parallelism on 8 GPUs, it will lead to:
+
+- **Duplicated KV cache** across all GPUs
+- **Unwanted memory usage** that limits batch size
+- **Reduced throughput** due to memory constraints
+
+### How DPA Works
+
+DPA addresses these limitations by applying **data parallelism specifically to the attention component**.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", verticalAlign: "top", backgroundColor: "rgba(255,255,255,0.02)"}}>
+        <img src="/images/dpa.png" alt="DPA + EP Architecture" style={{width: "100%", height: "auto"}} />
+      </td>
+      <td style={{padding: "9px 12px", verticalAlign: "top", backgroundColor: "rgba(255,255,255,0.05)"}}>
+        <p><strong>Each DP replica:</strong></p>
+        <ul>
+          <li>Processes different batches independently (can be in different forward modes: prefill, decode, or idle)</li>
+          <li>Maintains its own KV cache (no duplication)</li>
+          <li>Enables significantly larger batch sizes due to memory savings</li>
+        </ul>
+        <p><strong>Communication patterns in DPA + EP:</strong></p>
+        <ul>
+          <li><strong>All2All (Dispatch)</strong>: Routes tokens to expert sub-groups based on gating decisions</li>
+          <li><strong>All2All (Combine)</strong>: Gathers computed results from experts back to original token positions</li>
+        </ul>
+      </td>
+    </tr>
+  </tbody>
+</table>
+
+### Key benefits of DPA
+
+1. **Significantly reduced KV cache memory**: Each DP replica only stores KV cache for its own batches
+2. **Larger batch sizes**: Memory savings enable larger batch sizes
+3. **Improved decoding throughput**: Significant throughput gains for MLA-based models
+4. **Independent forward modes**: Each DP replica can be in different forward modes (prefill, decode, or idle) and handles its assigned batches independently during attention computation
+
+### DPA with Expert Parallelism for MoE
+
+For MoE models like DeepSeek, DPA is **often** paired with Expert Parallelism (EP) for best throughput at scale. However, **DPA does not require EP**: you can enable DPA without EP if your deployment does not need expert sharding.
+
+- Distribute 256+ expert weights across GPUs (cannot fit on a single GPU)
+- Enable efficient all-to-all token routing via DeepEP
+- Scale to large clusters (up to 5x throughput improvement over vanilla TP)
+
+### Recommended setup for DeepSeek
+
+```bash
+python -m sglang.launch_server \
+    --model-path deepseek-ai/DeepSeek-V3 \
+    --tp 8 \
+    --dp-size 8 \
+    --ep 8 \
+    --enable-dp-attention \
+    --moe-a2a-backend deepep \
+    --moe-runner-backend deep_gemm
+```
+
+> **Note**: `--dp-size` must be explicitly set when using `--enable-dp-attention`. If `dp_size` is 1 (default), DPA will be disabled.
+
+For detailed EP configuration (DeepEP, Two-Batch Overlap, EPLB), see [Expert Parallelism](/docs/advanced_features/expert_parallelism).
+
+### Target Models
+
+DPA supports the following model architectures:
+
+- **MLA (Multi-Head Latent Attention) models** - where DPA provides the most significant benefits:
+  - DeepSeek family (DeepSeek-V2, DeepSeek-V3, DeepSeek-R1)
+  - MiniMax models
+  - Kimi-K2
+  - Other models using MLA architecture
+
+- **Standard attention models** - also supported:
+  - Qwen models (see [PR #6121](https://github.com/sgl-project/sglang/pull/6121))
+
+For models like Llama, with standard GQA, standard DP, or TP is typically recommended.
+
+To enable DPA, add `--enable-dp-attention` to your server launch command.
+
+### Activation Logic
+
+DPA is enabled explicitly via server arguments (CLI or config). You must set both `--dp-size` and `--enable-dp-attention`:
+
+```bash
+python -m sglang.launch_server \
+    --model-path deepseek-ai/DeepSeek-V3 \
+    --tp 8 \
+    --dp-size 8 \
+    --enable-dp-attention
+```
+
+**Important**: `--dp-size` must be greater than 1 for DPA to work. When `dp_size == 1` (default), `--enable-dp-attention` is automatically disabled. The constraint `tp_size % dp_size == 0` must also be satisfied.
+
+### Standard DP for MLA models
+
+Note that MLA models, of course, also support DP. Suppose you want to enable standard DP for MLA models. First, launch each MLA model's replica independently. You may launch these replicas one by one with DPA enabled. After launching each MLA model's replica, launch an SMG and connect all the replicas to the SMG. A detailed explanation of SMG is as follows.
+
+## Modern Data Parallelism SGLang Model Gateway (SMG)
+
+### Native DP Mode
+
+Native DP (built-in Data Parallelism) in SGLang creates multiple worker processes within a single SGLang instance, under the control of `DataParallelController` with the launching parameter of `dp-size`.
+
+```bash
+# Native DP mode
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --dp-size 4
+```
+
+**Limitations:**
+
+- Built-in in-process load balancing only (e.g., `round_robin`, `total_requests`, `total_tokens`)
+- No cache-aware routing
+- Limited observability and metrics
+- No fault tolerance or circuit breakers
+- Not suitable for production workloads
+
+⚠️ Native DP is **highly not recommended for use right now**. It is only used in some ancient/outdated RL frameworks. You can use SGLang Model Gateway (SMG) to power up your data parallelism in any use case.
+
+### SMG-Based DP (Recommended)
+
+Starting from September 2024, SGLang Model Gateway, i.e., SMG, formerly named as SGLang DP Router, was built especially as a production-ready DP routing system with Rust. It starts from DP routing, but later we further expanded its scope to coordinate RL, PD Disaggregation, and other scenarios. This doc only discusses SMG's usage in DP routing. For other usage, please refer to [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).
+
+> To achieve the best production-level routing performance and reduce the overhead to an extreme extent, we use Rust to build SMG, but not Python, since Python is never FAST enough.
+
+**We strongly recommend using the SGLang Model Gateway (SMG) for production-grade Data Parallelism.** SMG provides significant advantages over native DP mode.
+
+```bash
+# SMG-based DP mode (Recommended)
+python -m sglang_router.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --dp-size 4
+```
+
+⚠️ Note that **SMG and Naive DP share the same launching parameter, `--dp-size`**. But the entrypoint of Naive DP is `python -m sglang.launch_server`, and SMG's entrypoint is `python -m sglang_router.launch_server`.
+
+**Advantages of SMG-Based DP:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "28%"}} />
+    <col style={{width: "34%"}} />
+    <col style={{width: "38%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Feature</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Native DP</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>SMG-Based DP</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Load Balancing</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Built-in in-process methods</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Advanced policies (cache-aware, power-of-two, etc.)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Cache Awareness</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌ No</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes - significantly higher cache hit rate</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Throughput</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Baseline</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Significant improvement</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Multi-Node Support</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Limited</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Full support</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Worker Health Monitoring</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Basic</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Circuit breakers, health checks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Reliability</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Basic</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Retries, rate limiting, queuing</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Observability</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Basic metrics</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ 40+ Prometheus metrics, OpenTelemetry</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Hot Worker Add/Remove</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌ No</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅ Yes</td>
+    </tr>
+  </tbody>
+</table>
+
+### SMG's Performance
+
+The cache-aware routing policy in SMG significantly improves performance for workloads with shared prefixes:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Without Cache-Aware</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>With Cache-Aware SMG</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Throughput (token/s)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>82,665</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>158,596 (+92%)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Cache Hit Rate</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>20%</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>75% (+275%)</td>
+    </tr>
+  </tbody>
+</table>
+
+*Benchmark from [SGLang v0.4 blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), workload with multiple long prefix groups, 8x A100 80GB GPUs, dp-size=8*
+
+### When to Use Each
+
+**Use Native DP when:**
+
+- ~Never use Native/Naive DP~
+- Learning material of DP routing
+
+**Use SMG-Based DP when:**
+
+- In any case, when you think DP is needed
+- Production deployments
+- Multi-node distributed setups
+- Workloads with shared prefixes (high cache reuse potential)
+- You need high availability and reliability features
+- You require detailed observability and metrics
+- You want to have highly efficient RL rollout systems
+
+Note that for RL rollout systems, **there are four crucial reasons that SMG-Based DP is far better than naive DP routing**. Details can be found at [Load Balancing Router in RL](/docs/advanced_features/sglang_for_rl#load-balancing-router).
+
+### Quick Start For SMG
+
+**Installation**
+
+```bash
+pip install sglang-router
+# or
+pip install "sglang[all]"
+```
+
+**Option A: Co-launch Workers and SMG (Simplest)**
+
+This is the easiest way to get started - SMG and workers are launched together:
+
+```bash
+python -m sglang_router.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --dp-size 4 \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+**Option B: Separate Launch (Multi-Node)**
+
+For distributed deployments across multiple machines:
+
+1. Launch workers on each node
+
+```bash
+# Node 1
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --port 8000
+
+# Node 2
+python -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --port 8000
+```
+
+2. Launch SMG pointing to workers
+
+```bash
+python -m sglang_router.launch_router \
+    --worker-urls http://node1:8000 http://node2:8000 \
+    --policy cache_aware \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+**Option C: Dynamic Worker Registration**
+
+For elastic deployments where workers can be added/removed dynamically:
+
+```bash
+# Launch SMG first
+python -m sglang_router.launch_router \
+    --policy cache_aware \
+    --host 0.0.0.0 \
+    --port 30000
+
+# Register workers dynamically
+curl -X POST http://localhost:30000/workers \
+    -H "Content-Type: application/json" \
+    -d '{"url": "http://worker1:8000"}'
+
+curl -X POST http://localhost:30000/workers \
+    -H "Content-Type: application/json" \
+    -d '{"url": "http://worker2:8000"}'
+```
+
+### Load Balancing Policies
+
+SMG supports multiple load balancing policies:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "24%"}} />
+    <col style={{width: "42%"}} />
+    <col style={{width: "34%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Policy</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Best For</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>cache_aware</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Combines cache locality with load balancing</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Recommended for most workloads</strong></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>round_robin</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Cycles through workers in order</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Simple, predictable distribution</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>random</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Random worker selection</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Baseline, testing</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>power_of_two</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Samples two workers, picks lighter one</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Low latency requirements</td>
+    </tr>
+  </tbody>
+</table>
+
+**Cache-Aware Policy (Default, Recommended)**
+
+The cache-aware policy provides the best performance for most workloads:
+
+```bash
+python -m sglang_router.launch_router \
+    --worker-urls http://worker1:8000 http://worker2:8000 \
+    --policy cache_aware \
+    --cache-threshold 0.5 \
+    --balance-abs-threshold 32 \
+    --balance-rel-threshold 1.5 \
+    --eviction-interval-secs 120 \
+    --max-tree-size 67108864
+```
+
+**How it works:**
+
+1. Maintains an approximate radix tree for each worker based on request history
+2. Routes requests to workers with the highest prefix match (cache hit)
+3. Falls back to shortest-queue routing when load is imbalanced
+4. Automatically evicts old entries to prevent memory overflow
+
+### Best Practices
+
+1. **Start with `cache_aware` policy** - It provides the best balance between cache locality and load distribution for most workloads
+2. **Use SMG for production** - Prefer `sglang_router.launch_server` over `sglang.launch_server` for better reliability and observability
+3. **Enable health checks** - Configure `--router-health-check-interval-secs` to detect and remove unhealthy workers automatically
+
+**Recommended command with best practices applied:**
+
+```bash
+python -m sglang_router.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --dp-size 4 \
+    --router-policy cache_aware \
+    --router-health-check-interval-secs 30 \
+    --router-prometheus-port 10001 \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+For advanced configuration (circuit breakers, retries, Prometheus metrics, K8s integration), see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).
+
+### Verifying Traffic Distribution
+
+After launching SMG, verify that traffic is being distributed correctly:
+
+**1. Check worker status:**
+
+```bash
+curl http://localhost:30000/workers
+```
+
+**2. Check load distribution:**
+
+```bash
+curl http://localhost:30000/get_loads
+```
+
+**3. Monitor metrics (if Prometheus enabled):**
+
+```bash
+# Key metrics to check
+smg_router_requests_total{model="..."}
+smg_worker_requests_active{worker="..."}
+sglang_cache_hit_rate{source="..."}
+```
+
+For detailed metrics and monitoring setup, see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).
+
+## Reference
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Strategy</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Key Benefit</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Native DP</strong> (<code>--dp-size</code>)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Never</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Easy to understand, not rust based</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>SMG-Based DP</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Production (recommended)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Cache-aware routing, high availability</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>DPA</strong> (<code>--dp-size N --enable-dp-attention</code>)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek/MLA models</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Eliminates KV cache duplication, improved throughput</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>DPA + EP</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek MoE models</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Significant throughput improvement vs vanilla TP</td>
+    </tr>
+  </tbody>
+</table>
+
+**Recommended production setup for DeepSeek:**
+1. Enable **DPA** for attention layers (`--dp-size 8 --enable-dp-attention`)
+2. Enable **EP** for MoE layers (`--ep 8 --moe-a2a-backend deepep`)
+3. Use **SMG** with **cache_aware** policy
+
+**Related documentation:**
+- [Expert Parallelism](./expert_parallelism) - DeepEP, Two-Batch Overlap, EPLB
+- [SGLang Model Gateway Documentation](./sgl_model_gateway) - SMG configuration & troubleshooting
+- [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/) - 96 GPU deployment guide

docs_new/docs/advanced_features/dp_for_multi_modal_encoder.mdx

@@ -0,0 +1,33 @@
+---
+title: "DP for Multi-Modal Encoder in SGLang"
+metatags:
+    description: "Data parallelism for VLM vision encoder in SGLang: reduce TTFT, boost throughput. Supports Qwen2.5-VL, Qwen3-VL, InternVL, GLM-4.5V/4.6V."
+---
+A typical VLM architecture involves two main components: an multi-modal encoder and a text decoder.
+
+Most VLMs utilize a Vision Transformer (ViT) as their multi-modal encoder, it is responsible for processing visual data, extracting features (objects, colors, textures, etc.), and transforming them into a format that can be understood by the model.
+
+The text deocoder is based on LLM. It processes textual data and generates output based on the encoded visual features.
+
+However, since the size of ViT is very small compared to language decoders,
+there is relatively little gain from TP. On the other hand, TP incurs significant communication
+overhead because of all-reduce being performed after every layer.
+
+Placing the ViT in data parallel while keeping the LLM in tensor parallel consistently lowers TTFT and boosts end-to-end throughput. In this hybrid layout, the vision front-end becomes parallel and lightweight, while scarce interconnect bandwidth and collective ops are reserved for the LLM.
+
+Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel.
+
+## Command Example
+You can enable batch-level DP by setting `mm-enable-dp-encoder`, for example:
+```shell Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen2.5-VL-7B-Instruct \
+    --tp 2 \
+    --mm-enable-dp-encoder
+```
+
+## Known supported models
+- Qwen2.5-VL (<https://github.com/sgl-project/sglang/pull/13126>)
+- Qwen3-VL (<https://github.com/sgl-project/sglang/pull/13724>)
+- InternVL (<https://github.com/sgl-project/sglang/pull/13925>)
+- GLM-4.5V & GLM-4.6V (<https://github.com/sgl-project/sglang/pull/14097>)

docs_new/docs/advanced_features/epd_disaggregation.mdx

@@ -0,0 +1,197 @@
+---
+title: "EPD Disaggregation"
+metatags:
+    description: "SGLang EPD disaggregation: separate encoder, prefill, decode stages for VLM inference. Independent scaling, load balancing, three-tier architecture."
+---
+## Why and What is EPD Disaggregation?
+
+In modern Vision-Language Model (VLM) inference, request execution naturally decomposes into three distinct stages: Encoder, Prefill, and Decode.
+The Encoder stage performs vision preprocessing and ViT-based image encoding, which is highly compute-intensive but only required during request initialization. The Prefill stage processes the full multimodal input sequence to initialize the language model’s Key-Value (KV) cache, while the Decode stage is dominated by memory bandwidth and KV cache access for autoregressive token generation.
+
+Existing deployments typically colocate these stages within a unified execution engine, or at best apply Prefill–Decode (PD) disaggregation. However, such designs still tightly couple vision encoding with language prefill, leading to inefficient resource utilization, limited scalability for image-heavy workloads, and suboptimal scheduling under load.
+
+To address these challenges, we introduce Encoder–Prefill–Decode (EPD) Disaggregation in SGLang. EPD further separates vision encoding from language processing, enabling independent horizontal scaling of encoder servers, improved load balancing for multimodal requests, and seamless integration with existing PD disaggregation to form a fully decoupled three-tier inference architecture.
+
+### Usage
+
+You can launch a language-only model using `--language-only`, or an encoder-only model using `--encoder-only`.
+When launching a language-only model, you must additionally specify the encoder service endpoints via `--encoder-urls`.
+
+We support multiple encoder transfer backends, including zmq_to_scheduler, zmq_to_tokenizer, and mooncake (the default is zmq_to_scheduler). The backend can be selected using `--encoder-transfer-backend`.
+
+### Encoder transfer with Mooncake
+
+`--encoder-transfer-backend mooncake` controls **how encoder outputs are transferred** between encoder and language/prefill services. It is an encoder transfer option and can be used independently of the global multimodal embedding cache.
+
+Example:
+
+```bash Command
+# encoder
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --encoder-transfer-backend mooncake \
+  --port 30000
+
+# language-only server
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --language-only \
+  --encoder-urls http://127.0.0.1:30000 \
+  --encoder-transfer-backend mooncake \
+  --port 30002
+```
+
+### Global multimodal embedding cache with Mooncake
+
+SGLang also supports a Mooncake-backed **global multimodal embedding cache** for EPD workloads. When enabled on encoder servers, repeated image inputs can reuse previously computed ViT embeddings across instances instead of running the vision encoder again.
+
+This feature is useful when:
+
+- the deployment serves repeated or overlapping image inputs,
+- encoder compute is the bottleneck, and
+- Mooncake is already available in the cluster.
+
+At a high level, the encoder checks whether the image embedding already exists in Mooncake. Cache hits are prefetched from the global store, while misses are encoded normally and inserted into the cache in the background.
+
+To enable it:
+
+- install and configure Mooncake in the same way as other SGLang Mooncake integrations,
+- add `--enable-mm-global-cache` on the encoder server.
+
+`--enable-mm-global-cache` controls **whether multimodal embeddings are looked up and stored in the global Mooncake cache**. It is separate from `--encoder-transfer-backend`, which only controls encoder output transport.
+
+For Mooncake deployment and configuration details, see [HiCache best practices](./hicache_best_practices#deployment-with-mooncake) and the [Mooncake backend README](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md).
+
+Example:
+
+```bash Command
+# Shared Mooncake configuration
+export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata"
+export MOONCAKE_MASTER="127.0.0.1:50051"
+export MOONCAKE_PROTOCOL="rdma"
+export MOONCAKE_GLOBAL_SEGMENT_SIZE="4gb"
+
+# encoder with global multimodal cache enabled
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --enable-mm-global-cache \
+  --port 30000
+
+# language-only server
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --language-only \
+  --encoder-urls http://127.0.0.1:30000 \
+  --port 30002
+```
+
+Notes:
+
+- This cache is for **multimodal encoder embeddings**, not the language model KV cache.
+- The feature currently uses Mooncake as the shared backing store.
+- It can be enabled regardless of which `--encoder-transfer-backend` you use.
+- It is most relevant for EPD or encoder-disaggregated VLM deployments where the same images are likely to appear across requests or instances.
+
+#### Qwen VL
+
+- EP Disaggregation
+
+```bash Command
+# encoder 0
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30000
+# encoder 1
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30001
+# language-only server
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --language-only \
+  --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30002
+```
+
+- EPD Disaggregation
+
+```bash Command
+# encoder 0
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30000
+# encoder 1
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30001
+# prefill 0
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --disaggregation-mode prefill \
+  --language-only \
+  --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30002
+# decode 0
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --disaggregation-mode decode \
+  --port 30003
+# router
+python -m sglang_router.launch_router \
+  --pd-disaggregation \
+  --prefill http://$PREFILL_HOST:30002 \
+  --decode http://$DECODE_HOST:30003 \
+  --port 8000
+
+```
+
+#### gRPC Encoder (EPD)
+
+You can run the encoder as a gRPC server while keeping prefill/decode as HTTP.
+When using gRPC encoders, set `SGLANG_ENCODER_MM_RECEIVER_MODE=grpc` for the
+prefill process so it uses the gRPC receiver.
+
+```bash Command
+# gRPC encoder
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --grpc-mode \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30000
+
+# prefill (HTTP) - tell it to use gRPC receiver
+SGLANG_ENCODER_MM_RECEIVER_MODE=grpc \
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --disaggregation-mode prefill \
+  --language-only \
+  --encoder-urls grpc://127.0.0.1:30000 \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30002
+
+# decode (HTTP)
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --disaggregation-mode decode \
+  --port 30003
+
+# router
+python -m sglang_router.launch_router \
+  --pd-disaggregation \
+  --prefill http://$PREFILL_HOST:30002 \
+  --decode http://$DECODE_HOST:30003 \
+  --port 8000
+```

docs_new/docs/advanced_features/expert_parallelism.mdx

@@ -0,0 +1,290 @@
+---
+title: "Expert Parallelism"
+metatags:
+    description: "SGLang Expert Parallelism: distribute MoE experts across GPUs, DeepEP all-to-all, grouped GEMMs, TBO/SBO overlap, EPLB load balancing."
+---
+Expert Parallelism (EP) in SGLang distributes expert weights across multiple devices in Mixture-of-Experts (MoE) models, addressing memory bottlenecks and enabling efficient scaling for high-performance inference. It is particularly vital for serving large-scale MoE models where tokens are dynamically routed to specialized experts across GPUs. By leveraging optimized all-to-all communication and grouped matrix multiplications (GEMMs), EP reduces latency, boosts throughput, and minimizes idle GPU time. SGLang's EP offers strong extensibility through its modular framework, allowing seamless integration of custom kernels, backends, and optimizations without refactoring core logic, supporting diverse hardware and quantization schemes.
+
+## Supported Backends and Selection Guidance
+
+SGLang's EP integrates diverse, highly efficient backends for different use cases, allowing fine-grained control over performance trade-offs. Users specify backends via command-line flags:
+- `--moe-a2a-backend`: Selects the backend for all-to-all communication.
+- `--moe-runner-backend`: Selects the backend for MoE computation.
+
+### Backends for All-to-All Communication
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Use Cases</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`none` (default)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disables all-to-all for EP. Uses All-Reduce or All-Gather for token dispatch.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Hybrid EP and TP setups.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepep`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepEP, a communication library for efficient token shuffling in MoE models.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Large-scale EP deployments.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`mooncake`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>An extension of DeepEP for elastic inference, leveraging RDMA for high-performance data transfers.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Elastic EP serving.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>nixl</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://github.com/ai-dynamo/nixl/tree/main/examples/device/ep">NIXL-EP</a>, an elastic EP communication library built on NVIDIA's <a href="https://github.com/ai-dynamo/nixl">NIXL</a> framework with native RDMA and NVLink support.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Elastic EP serving with fault tolerance and dynamic scaling.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>mori</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MORI-EP, AMD's native all-to-all communication implementation optimized for ROCm.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>AMD GPU deployments.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flashinfer implementation of all-to-all.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Large-scale EP deployments.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`ascend_fuseep`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Ascend NPU native fused all-to-all communication.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Ascend NPU deployments.</td>
+    </tr>
+  </tbody>
+</table>
+
+DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). MORI backend only supports `normal` mode now. NIXL-EP currently operates in low-latency mode with CUDA Graph support. Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes.
+
+Currently, DeepEP, Mooncake, NIXL-EP, `ascend_fuseep` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported.
+
+### Backends for MoE Computation
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Use Cases</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`auto` (default)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Automatically selects the optimal backend based on model architecture, hardware (e.g., NVIDIA architecture like Ampere, Hopper, Blackwell), quantization scheme (e.g., FP8, FP4), and runtime conditions.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>General-purpose deployments; ensures compatibility and performance without user intervention.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Triton-based implementation for grouped GEMMs. To achieve higher performance, it's highly recommended to create <a href="https://github.com/sgl-project/sglang/blob/main/benchmark/kernels/fused_moe_triton/README.md">tuned configurations</a>.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom kernel development or scenarios requiring high extensibility with Torch compilation support.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deep_gemm`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepGEMM backend optimized for MoE matrix multiplications, supporting contiguous layouts for prefill and masked layouts for decode; often JIT-compiled for performance.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Large-scale EP deployments with FP8 block-wise quantization.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`cutlass`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUTLASS-based backend for efficient GEMMs.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA architectures with CUTLASS support.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_trtllm`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer integrated with TensorRT-LLM for accelerated MoE computations, supporting FP4 communication operators and high-performance GEMMs.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Blackwell with TRT-LLM.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>flashinfer_trtllm_routed</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer integrated with TensorRT-LLM for accelerated routed MoE computations, consuming SGLang-computed top-k expert assignments and weights. Compatible with flashinfer all-to-all.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Blackwell with TRT-LLM.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_cutlass`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer combined with CUTLASS for high-performance grouped GEMMs in MoE layers, handling FP4/FP8 quantization efficiently.  Compatible with flashinfer all-to-all.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Blackwell with FP4/FP8 models.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_mxfp4`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer variant optimized for MXFP4 (mixed FP4) quantization in MoE runners, focusing on memory-efficient low-precision inference.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Low-precision models with MXFP4.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_cutedsl`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer with a custom DSL for flexible and efficient MoE kernel generation, integrated with ModelOpt FP4 quantization. Compatible with flashinfer all-to-all.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Low-precision models with NVFP4.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Examples
+
+Launch with DeepEP and DeepGEMM for DeepSeek-V3:
+
+```bash Command
+python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --moe-a2a-backend deepep --moe-runner-backend deep_gemm --tp 8 --ep 8
+```
+
+## Extensible EP Framework
+
+SGLang's EP framework provides modular abstractions for easy integration of custom kernels, backends, and optimizations. It decouples the MoE forward pass into stages (dispatch → pre-permute → core runner → post-permute → combine), enabling seamless extensions without refactoring core logic.
+
+### Framework Overview
+
+The framework centers on `FusedMoE` as the unified entry point for a single, extensible structure. Key components include:
+- **Dispatcher**: Manages dispatch/combine for backends like DeepEP (implements `BaseDispatcher` subclasses).
+- **MoeRunner**: Orchestrates grouped-GEMM execution via `MoeRunnerCore` implementations (e.g., `TritonRunnerCore`).
+- **PermuteMethodPool**: Auto-registers layout conversions (e.g., pre/post-permute via `register_pre_permute` and `register_post_permute` for dynamic modes, or `register_fused_func` for static, torch.compile-compatible fused operations).
+- **TopK Router**: Backend-agnostic expert selection.
+
+This design supports multiple backends via `--moe-a2a-backend` and `--moe-runner-backend`, with quantization integrated through a standardized `apply()` method. The computation flow ensures modularity:
+
+```text Output
+[input_hidden_states]
+          |
+          v
+     TopK.forward -> select_experts / triton_kernels.routing / bypass
+          |
+          v
+     [TopKOutput]
+          |
+          v
+   FusedMoE.forward -> Dispatcher.dispatch -> DeepEP / bypass
+          |                     |
+          |                     v
+          |              [DispatchOutput]
+          |                     |
+          |                     v
+          |             quant_method.apply -> MoeRunner.forward
+          |                     |              |
+          |                     |              v
+          |                     | pre-permute + grouped_gemm + post-permute
+          |                     |              |
+          |                     |--------------
+          |                     v
+          |               [CombineInput]
+          |                     |
+          |                     v
+          |            Dispatcher.combine -> DeepEP / bypass
+          |                     |
+          |---------------------
+          v
+[final_hidden_states]
+```
+
+For details, see the [MoE Refactor Roadmap](https://github.com/sgl-project/sglang/issues/8715).
+
+### Implementing New Backends
+
+To add a new backend:
+1. For a new all-to-all dispatcher, implement a `BaseDispatcher` subclass with `dispatch` and `combine` methods.
+2. For a new MoE runner backend, define a `MoeRunnerCore` subclass for core operations (e.g., grouped GEMMs).
+3. Define new input/output formats for the dispatcher or model runner (e.g., `RunnerInput`, `RunnerOutput`).
+4. Register permute/unpermute methods to ensure compatibility:
+   - **Fused Mode** (static, torch.compile-compatible): Use `register_fused_func` for end-to-end operations.
+   - **Permute Mode** (dynamic): Register `register_pre_permute` and `register_post_permute` for flexible layouts.
+
+See the [MoE Refactor Implementation PR](https://github.com/sgl-project/sglang/pull/9269) for full changes, including type hints and config expansions.
+
+### Examples
+
+For an example implementation, see [moe_runner/triton.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/moe_runner/triton.py), which demonstrates Triton-based grouped GEMMs with registered fused and permutation functions.
+
+## Computation and Communication Overlap
+
+SGLang's EP employs advanced overlap techniques to hide communication latency behind computation, maximizing GPU utilization in MoE layers.
+
+### Two-Batch Overlap (TBO)
+
+TBO splits requests into micro-batches, interleaving attention computation with dispatch/combine operations. Yield points in the execution graph allow pausing for overlaps, increasing overall throughput without peak memory spikes:
+
+```python Example
+operations = [
+    self._forward_attn,
+    YieldOperation(),  # Overlap with dispatch of prior micro-batch
+    self._forward_dispatch,
+    self._forward_mlp,
+    YieldOperation(),  # Overlap with combine
+    self._forward_combine,
+]
+```
+
+Users need to specify `--enable-two-batch-overlap` to unlock up to 2x throughput. For details, see the [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#two-batch-overlap).
+
+### Single-Batch Overlap (SBO)
+
+SGLang introduces a dispatcher-hook system for Single-Batch Overlap (SBO), enabling the overlap of operations within a single batch—such as shared experts computation with communication—while decentralizing logic to enhance modularity. These hooks execute before and after the `dispatch` and `combine` operations without modifying core MoE modules. This design simplifies interfaces, reduces coupling, and improves extensibility. For implementation details and an example of overlapping shared experts with DeepEP's combine operation, refer to [PR #13327](https://github.com/sgl-project/sglang/pull/13327). Users can set `--enable-single-batch-overlap` to enable this feature.
+
+
+## Workload Balancer
+
+SGLang integrates the [Expert Parallelism Load Balancer (EPLB)](https://github.com/deepseek-ai/EPLB) from DeepSeek to address routing imbalances in MoE models. By analyzing expert activation statistics, EPLB computes an optimal expert arrangement, strategically placing or replicating experts to minimize GPU utilization variance, reduce idle cycles, and enhance scalability.
+
+To enable EPLB, use the flags `--enable-eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains.
+
+For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb).
+
+
+## Ascend NPU Guidance
+### Guidance on SGLang configuration in Ascend NPU
+- `--moe-a2a-backend` only supports `deepep` and `ascend_fuseep` backends,
+
+    - `deepep`: The mechanism is consistent with the above description.
+
+    - `ascend_fuseep`: Offer a large fused operator which integrates all operations between dispatch and combine to boost MoE computation. Only used for decode stage in PD Disaggregation Mode.
+
+- `--moe-runner-backend` parameter does not need to be configured.
+
+- `--deepep-mode`:
+
+    - In PD mixed mode, please set `--deepep-mode auto`.
+
+    - In PD Disaggregation Mode, prefill instance sets `--deepep-mode normal`, and decode instance sets `--deepep-mode low_latency`.
+
+### DeepEP Ascend Introduction
+DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP).
+It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences.
+
+Ant-moving Function can be enabled for both the dispatch and combine phases via the following environment variables:
+
+- `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`: Enable ant-moving function in dispatch stage. Indicates the number of tokens transmitted per round on each rank, default 8192.
+
+- `DEEPEP_NORMAL_LONG_SEQ_ROUND`: Enable ant-moving function in dispatch stage. Indicates the number of rounds transmitted on each rank, default 1.
+
+- `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`: Enable ant-moving function in combine stage, default 0 (means disabled).
+
+`DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * DEEPEP_NORMAL_LONG_SEQ_ROUND` means input sequence length. When the input sequence length exceeds 8192, it is recommended to enable the ant-moving function in both dispatch and combine phase.
+
+The environment variable `HCCL_BUFFSIZE` is used to configure the buffer size (MB) actually allocated. Its calculation formula is as follows:
+```text Output
+# Enable Ant-moving Function
+HCCL_BUFFSIZE >= 2 * (102MB + 4MB + DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * (hidden_size + hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE
+
+# Disable Ant-moving Function
+HCCL_BUFFSIZE >= 2 * (102MB + 4MB + TOTAL_SEQ_LEN * (hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE
+```
+Wherein the parameters are described as follows:
+
+- `hidden_size`: hidden size in model config.
+
+- `topk`: The number of selected routing experts.
+
+- `TOTAL_SEQ_LEN`: input sequence length.
+
+- `PADDING_BUFFSIZE`: A value of 20 or greater is recommended.

docs_new/docs/advanced_features/forward_hooks.mdx

@@ -0,0 +1,298 @@
+---
+title: "Model Forward Hooks"
+metatags:
+    description: "SGLang forward hooks: attach PyTorch hooks to model submodules via JSON config. Log activations, debug internals, export hidden states."
+---
+
+## Model Hooks
+
+SGLang supports attaching PyTorch forward hooks to specific submodules in the loaded model, configured entirely via `server_args` JSON.
+
+This is useful for:
+
+* Logging intermediate activations
+* Debugging model internals
+* Exporting hidden states to external tooling
+
+Hooks are attached once during `ModelRunner.initialize` and run on every forward pass.
+
+***
+### Configuration overview
+
+Hooks are configured via a `ServerArgs` field:
+
+```python Example
+class ServerArgs:
+    ...
+    # For forward hooks
+    forward_hooks: Optional[List[dict[str, Any]]] = None
+````
+
+In JSON form, a minimal configuration looks like:
+
+```jsonc Example
+{
+  "forward_hooks": [
+    {
+      "name": "outer_linear_hooks",
+      "target_modules": ["outer.0", "outer.1"],
+      "hook_factory": "my_project.hooks:dummy_hook_factory",
+      "config": {
+        "tag": "outer-layer"
+      }
+    }
+  ]
+}
+```
+
+#### Top-level fields
+
+* `forward_hooks` (optional list of objects)
+  Each element is a hook spec describing:
+
+  * Which modules to target
+  * Which Python factory to call
+  * What configuration to pass into that factory
+
+***
+### Hook spec schema
+
+Each entry in `forward_hooks` is a JSON object with the following shape:
+
+```jsonc Example
+{
+  "name": "optional-descriptive-name",
+  "target_modules": ["pattern1", "pattern2", "..."],
+  "hook_factory": "module.submodule:factory_name",
+  "config": {
+    "...": "arbitrary JSON"
+  }
+}
+```
+
+#### `name` (optional)
+
+* Human-readable name for logging.
+* Used only in log messages such as:
+
+  ```text Output
+  Registered forward hook 'outer_linear_hooks' on outer.0
+  ```
+
+#### `target_modules` (required)
+
+* List of **module name patterns** used to match entries in `model.named_modules()`.
+* Patterns are matched using `fnmatch.fnmatch`, so:
+
+  * `"outer.0"` matches exactly `"outer.0"`.
+  * `"outer.*"` matches `"outer.0"`, `"outer.1"`, `"outer.inner"`, etc.
+  * `"outer.inner.*"` matches children under `outer.inner`.
+
+> If no modules match the given patterns, hook registration does **not** fail.
+> Instead, SGLang logs a warning and continues:
+>
+> ```text
+> No modules matched hook spec 'name' patterns=['...']
+> ```
+
+#### `hook_factory` (required)
+
+* String path to the Python factory function that creates the hook.
+* Supported formats:
+
+  * `"package.module:factory_name"`
+  * `"package.module.submodule.factory_name"`
+
+The path is resolved via:
+
+```python Example
+def resolve_callable(path: Optional[str]) -> Optional[Callable]:
+    if path is None:
+        return None
+
+    if ":" in path:
+        module_name, fn_name = path.split(":", 1)
+    else:
+        parts = path.split(".")
+        if len(parts) < 2:
+            raise ValueError(
+                f"Invalid hook callable path '{path}'. "
+                "Expected 'module.submodule:factory' or 'module.submodule.factory'."
+            )
+        *mod_parts, fn_name = parts
+        module_name = ".".join(mod_parts)
+
+    module = importlib.import_module(module_name)
+    try:
+        return getattr(module, fn_name)
+    except AttributeError as e:
+        raise AttributeError(
+            f"Module '{module_name}' has no attribute '{fn_name}' "
+            f"(from hook path '{path}')"
+        ) from e
+```
+
+**Failure modes**:
+
+* If the path is malformed (not enough dots and no `:`), a `ValueError` is raised at startup.
+* If the module imports but the attribute is missing, an `AttributeError` is raised with a clear error message.
+* If the hook factory returns `None`, a warning is logged and no hook is registered for that spec (initialization continues).
+
+The first two cause initialization to fail fast with a descriptive error; the last one is non-fatal.
+
+#### `config` (optional)
+
+* Arbitrary JSON object.
+* Passed directly to the hook factory as a Python `dict`.
+* This lets you parameterize hook behavior from config (e.g. tags, log levels, sampling rates, etc.).
+
+***
+### Hook lifecycle and behavior
+
+Hooks are registered in `ModelRunner.initialize()`:
+
+```python Example
+if server_args.forward_hooks:
+    register_forward_hooks(self.model, server_args.forward_hooks)
+```
+
+The actual registration logic is implemented by `register_forward_hooks`:
+
+```python Example
+def register_forward_hooks(model: nn.Module, hook_specs: List[dict[str, Any]]) -> None:
+    """
+    hook_specs is a list of dicts from server_args.forward_hooks.
+    Attaches forward hooks to the matching modules.
+    """
+    name_to_module = dict(model.named_modules())
+
+    for spec in hook_specs:
+        spec_name = spec.get("name", "")
+        target_patterns = spec.get("target_modules", [])
+        if not target_patterns:
+            logger.warning(
+                f"Hook spec '{spec_name}' has no 'target_modules', skipping"
+            )
+            continue
+
+        hook_factory_path = spec.get("hook_factory")
+        if not hook_factory_path:
+            logger.warning(
+                f"Hook spec '{spec_name}' has no 'hook_factory', skipping"
+            )
+            continue
+
+        config = spec.get("config") or {}
+        hook_factory = resolve_callable(hook_factory_path)
+
+        hook = hook_factory(config) if hook_factory else None
+        if hook is None:
+            logger.warning(
+                f"Hook factory '{hook_factory_path}' for spec '{spec_name}' "
+                "returned None, not registering any hook"
+            )
+            continue
+
+        # Resolve patterns like "model.layers.*.mlp"
+        matched = []
+        for name, module in name_to_module.items():
+            if any(fnmatch.fnmatch(name, pattern) for pattern in target_patterns):
+                matched.append((name, module))
+
+        if not matched:
+            logger.warning(
+                f"No modules matched hook spec '{spec_name}' "
+                f"patterns={target_patterns}"
+            )
+            continue
+
+        for module_name, module in matched:
+            if hook:
+                _ = module.register_forward_hook(hook)
+                logger.info(
+                    f"Registered forward hook '{spec_name}' "
+                    f"on {module_name}"
+                )
+```
+
+Key points:
+
+* Hooks are **forward hooks only** (via `module.register_forward_hook`).
+* They are attached once at initialization.
+* Hook handles are currently not stored on `ModelRunner` (they cannot be removed later via this API).
+* Failure to match any modules is non-fatal; a warning is logged instead.
+* If a hook factory returns `None`, a warning is logged and that spec is skipped.
+
+***
+### Writing a hook factory
+
+A hook factory is a regular Python function:
+
+* Takes a `config: dict` (from JSON)
+* Returns a forward hook function with signature `(module, inputs, output)`
+
+Example:
+
+```python Example
+HOOK_CALLS = []
+
+def dummy_hook_factory(config):
+    """Factory that returns a forward hook capturing a tag from config."""
+    tag = config.get("tag", "default")
+
+    def hook(module, inputs, output):
+        HOOK_CALLS.append(
+            {
+                "module_type": type(module).__name__,
+                "tag": tag,
+                "shape": tuple(output.shape),
+            }
+        )
+        return output  # must return output if you don’t want to modify the tensor
+
+    return hook
+```
+
+In JSON:
+
+```jsonc Example
+{
+  "forward_hooks": [
+    {
+      "name": "capture_outer",
+      "target_modules": ["outer.0", "outer.1"],
+      "hook_factory": "my_project.hooks:dummy_hook_factory",
+      "config": {
+        "tag": "outer"
+      }
+    }
+  ]
+}
+```
+
+This will:
+
+* Resolve `my_project.hooks:dummy_hook_factory` to a Python callable.
+* Call it with `config = {"tag": "outer"}`.
+* Use the returned hook for all modules matching `outer.0` and `outer.1`.
+* Append metadata about each call to `HOOK_CALLS`.
+
+***
+### Summary
+
+* Define `forward_hooks` as a list of specs in `ServerArgs` to turn on the feature.
+
+* Each spec:
+
+  * selects modules via `target_modules` (glob patterns over `model.named_modules()`),
+  * points to a hook factory via `hook_factory`,
+  * passes arbitrary `config` into that factory.
+
+* Hook factories are resolved via `resolve_callable`, which supports `module:factory` and `module.submodule.factory`.
+
+* Hooks are standard PyTorch forward hooks, attached once at startup and invoked on every forward pass.
+
+* Misconfiguration is either:
+
+  * **fatal and explicit** (bad path / missing attribute), or
+  * **non-fatal with clear warnings** (no targets matched, or factory returned `None`).

docs_new/docs/advanced_features/hicache.mdx

@@ -0,0 +1,8 @@
+---
+title: "Hierarchical KV Caching (HiCache)"
+metatags:
+    description: "SGLang HiCache: three-tier KV caching (GPU, CPU, storage) for long-context and multi-turn inference. Supports Mooncake, 3FS, NIXL backends."
+---
+- [Hicache Best Practices](./hicache_best_practices)
+- [Hicache Design](./hicache_design)
+- [Hicache Storage Runtime Attach Detach](./hicache_storage_runtime_attach_detach)

docs_new/docs/advanced_features/hicache.rst

@@ -0,0 +1,9 @@
+Hierarchical KV Caching (HiCache)
+=================================
+
+.. toctree::
+   :maxdepth: 1
+
+   hicache_best_practices.md
+   hicache_design.md
+   hicache_storage_runtime_attach_detach.md

docs_new/docs/advanced_features/hicache_best_practices.mdx

@@ -0,0 +1,219 @@
+---
+title: "SGLang HiCache Best Practices"
+metatags:
+    description: "HiCache configuration guide: memory layout, prefetch policies, PD disaggregation, HF3FS and Mooncake deployment, custom storage backends."
+---
+## Why HiCache Matters
+
+SGLang HiCache extends the traditional RadixAttention with a three-tier hierarchical KV caching system that dramatically improves performance for long-context and multi-turn conversation scenarios. By intelligently managing KV caches across GPU memory, host memory, and external storage backends, HiCache addresses the fundamental capacity bottleneck that limits cache hit rates in conventional systems.
+
+## Configuration Guidelines
+
+## Core HiCache Parameters
+
+```bash Command
+# Essential HiCache flags
+--page-size 64                        # Page size for cache management
+--enable-hierarchical-cache           # Enable HiCache
+--hicache-ratio 2                     # Host memory ratio (2x GPU memory)
+--hicache-size 100                    # Host memory size in GBs, will override the above ratio
+--hicache-io-backend kernel           # The I/O backend of moving data between CPU and GPU
+--hicache-write-policy write_through  # Cache write policy from GPU to CPU
+--hicache-storage-backend             # Optional storage backend (e.g., hf3fs, mooncake, etc.)
+```
+
+Notes:
+
+- Besides configuring `--hicache-storage-backend` at startup, SGLang also supports **runtime attach/detach** of the HiCache storage backend (no restart required) via HTTP admin endpoints. See [Runtime Attach/Detach HiCache Storage Backend](./hicache_storage_runtime_attach_detach).
+
+## Key Configurations with Storage Backends Enabled
+
+### Memory Layout Optimization
+
+```bash Command
+# Page-first: Optimized for I/O efficiency with zero-copy (recommended with kernel backend)
+--hicache-mem-layout page_first
+# Page-first-direct: Optimized for direct I/O operations (Compatible with fa3 and same zero-copy performance as page_first)
+--hicache-mem-layout page_first_direct
+# Layer-first
+--hicache-mem-layout layer_first
+```
+**Layout Compatibility:**
+- `page_first`: Only compatible with `kernel` I/O backend, automatically switches to `layer_first` with `direct` backend
+- `page_first_direct`: Specifically designed for `direct` I/O backend with optimized memory organization
+
+### Heterogeneous TP Support (GQA/MHA models)
+
+HiCache storage supports cross-cluster KV reuse when different deployments use different TP sizes (for example, `tp=4` and `tp=8`) and share the same storage backend namespace.
+
+Use `tp_lcm_size` in `--hicache-storage-backend-extra-config`:
+
+```bash Command
+# Example: heterogeneous TP = {4, 8}, so lcm = 8
+--hicache-storage-backend-extra-config '{"tp_lcm_size": 8}'
+```
+
+Guidelines:
+
+- Set `tp_lcm_size` to the least common multiple (LCM) of all TP sizes that will share the same HiCache storage.
+- For MHA models with Mooncake and `page_head` layout, HiCache will split head shards based on `tp_lcm_size` to make keys reusable across heterogeneous TP deployments.
+- If all clusters use the same TP size, this option is not needed.
+
+### Prefetch Policies
+
+```bash Command
+# Best-effort: Terminate prefetch when needed
+--hicache-storage-prefetch-policy best_effort
+# Wait-complete: Ensure complete prefetch, higher cache reuse
+--hicache-storage-prefetch-policy wait_complete
+# Timeout: Balance between completion and best-effort
+--hicache-storage-prefetch-policy timeout
+```
+
+### Integration with PD Disaggregation
+
+HiCache works seamlessly with PD Disaggregation. You can choose between two configurations:
+
+1. **Prefill-only HiCache**: Enable HiCache only on Prefill nodes, allowing KV cache sharing among Prefill instances
+2. **Full HiCache with async offloading**: Enable HiCache on Prefill nodes and async KV cache offloading on Decode nodes, allowing Prefill nodes to reuse KV caches from Decode nodes in multi-turn dialogue scenarios
+
+```bash Command
+# Prefill node with HiCache enabled for cross-prefill sharing (ideal for SystemPrompt scenarios)
+python3 -m sglang.launch_server \
+  --model-path /xxx/DeepSeek-R1/ \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 10000 \
+  --enable-metrics \
+  --enable-cache-report \
+  --mem-fraction-static 0.85 \
+  --page-size 64 \
+  --enable-hierarchical-cache \
+  --hicache-ratio 2 \
+  --hicache-size 0 \
+  --hicache-mem-layout page_first_direct \
+  --hicache-io-backend direct \
+  --hicache-write-policy write_through \
+  --hicache-storage-backend hf3fs \
+  --hicache-storage-prefetch-policy wait_complete \
+  --disaggregation-ib-device mlx5_0 \
+  --disaggregation-mode prefill \
+  --disaggregation-transfer-backend mooncake
+
+# Decode node with async offloading enabled for KV cache reuse by Prefill (ideal for multi-turn conversations)
+python3 -m sglang.launch_server \
+  --model-path /xxx/DeepSeek-R1/ \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 10000 \
+  --enable-metrics \
+  --enable-cache-report \
+  --page-size 64 \
+  --hicache-ratio 2 \
+  --hicache-size 0 \
+  --hicache-mem-layout page_first_direct \
+  --hicache-io-backend direct \
+  --hicache-write-policy write_through \
+  --hicache-storage-backend hf3fs \
+  --hicache-storage-prefetch-policy wait_complete \
+  --disaggregation-decode-enable-offload-kvcache \  # Enable async KV cache offloading in decode node
+  --disaggregation-ib-device mlx5_0 \
+  --disaggregation-mode decode \
+  --disaggregation-transfer-backend mooncake
+```
+
+
+### Deployment with HF3FS
+
+Here is an example of deploying DeepSeek-R1 with HiCache-HF3FS. For more details, see the [HF3FS Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/hf3fs/docs/README.md).
+
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path /xxx/DeepSeek-R1/ \
+  --log-level info \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 10000 \
+  --enable-metrics \
+  --enable-cache-report \
+  --page-size 64 \
+  --mem-fraction-static 0.85 \
+  --enable-hierarchical-cache \
+  --hicache-ratio 2 \
+  --hicache-size 0 \
+  --hicache-mem-layout page_first_direct \
+  --hicache-io-backend direct \
+  --hicache-write-policy write_through \
+  --hicache-storage-backend hf3fs \
+  --hicache-storage-prefetch-policy wait_complete \
+```
+
+### Deployment with Mooncake
+
+Here is an example of deploying Qwen3-235B-A22B-Instruct-2507 with Mooncake. For more details, see the [Mooncake Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md).
+
+```bash Command
+# Set Mooncake environment variables
+export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata"
+export MOONCAKE_GLOBAL_SEGMENT_SIZE=816043786240
+export MOONCAKE_PROTOCOL="rdma"
+export MOONCAKE_DEVICE="$DEVICE_LIST"
+export MOONCAKE_MASTER=127.0.0.1:50051
+
+# Launch SGLang server with Mooncake backend
+python3 -m sglang.launch_server \
+  --model-path $MODEL_PATH \
+  --tp 8 \
+  --page-size 64 \
+  --enable-hierarchical-cache \
+  --hicache-ratio 2 \
+  --hicache-mem-layout page_first_direct \
+  --hicache-io-backend direct \
+  --hicache-storage-backend mooncake \
+  --hicache-write-policy write_through \
+  --hicache-storage-prefetch-policy timeout
+```
+
+
+## Custom Storage Backend Integration
+
+To integrate a new storage backend:
+
+1. **Implement three core methods:**
+   - `get(key)`: Retrieve value by key
+   - `exists(key)`: Check key existence
+   - `set(key, value)`: Store key-value pair
+
+2. **Register your backend:** Add your storage backend to the HiCache [BackendFactory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/backend_factory.py#L188)
+
+The HiCache controller handles all scheduling and synchronization automatically.
+
+### Dynamic Backend Loading
+
+Alternatively, you can use dynamic loading to avoid hard-coding your backend in the repository:
+
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path your-model \
+  --enable-hierarchical-cache \
+  --hicache-storage-backend dynamic \
+  --hicache-storage-backend-extra-config '{"backend_name":"custom_backend_name", "module_path": "your_module_path", "class_name": "YourHiCacheClassName"}'
+```
+
+**Configuration Parameters:**
+- `--hicache-storage-backend`: Set to `dynamic`
+- `--hicache-storage-backend-extra-config`: JSON configuration with:
+  - `backend_name`: Custom backend identifier
+  - `module_path`: Python module path to your implementation
+  - `class_name`: Your HiCache implementation class name
+  - `interface_v1`: 0 (disable) or 1 (enable) to control usage of batch_get_v1 and batch_set_v1 methods
+
+
+## Community and Support
+
+- **GitHub Issues**: Report bugs and feature requests
+- **Slack Channel**: Join community discussions in #sgl-kv-cache-store
+- **Documentation**: Refer to storage backend-specific guides
+
+***
+*This document will be continuously updated based on community feedback and new features. Contributions and suggestions are welcome!*

docs_new/docs/advanced_features/hicache_design.mdx

@@ -0,0 +1,170 @@
+---
+title: "HiCache System Design and Optimization"
+metatags:
+    description: "HiCache architecture: HiRadixTree metadata, L1/L2/L3 workflow, prefetch strategies, write-back policies, zero-copy transfers, multi-rank sync."
+---
+This document provides a comprehensive overview of SGLang HiCache, covering its system architecture, workflow and key components. It also details configuration parameters, optimization techniques, and integration with various L3 storage backends, serving as a complete reference for users and developers to understand and tune HiCache for efficient LLM inference.
+
+## Why and What is HiCache?
+
+In large language model inference, the prefill phase is often time-consuming: input sequences need to be first converted into Key-Value cache (KV cache) for subsequent decoding. When multiple requests share the same prefix, the KV cache for that prefix is identical. By caching and reusing these shared KV caches, redundant computation can be avoided. To address this, SGLang introduced RadixAttention, which leverages idle GPU memory to cache and reuse prefix KV caches, and **HiCache**, which extends this idea to host memory and distributed storage.
+
+Inspired by the classic three-level cache design of modern CPUs, HiCache organizes GPU memory as L1, host memory as L2, and distributed storage as L3. This hierarchy enables HiCache to fully exploit the "idle" storage space of GPUs and CPUs, while integrating distributed cache systems such as Mooncake, 3FS, NIXL, and AIBrix KVCache for global KV cache storage and scheduling. As a result, HiCache significantly expands KV cache capacity while maintaining strong read performance—especially in workloads such as multi-QA and long-context inference, where KV cache reuse is frequent. For detailed benchmark results, see [this blog](https://lmsys.org/blog/2025-09-10-sglang-hicache/).
+
+
+## System Design
+
+### Overall Architecture
+
+In many modern CPU architectures, the small but fast L1 and L2 caches are private to each core, enabling rapid access to the hottest data, while the larger L3 cache is shared across all cores to significantly reduce redundancy within the cache. Similarly, in HiCache, the L1 and L2 KV caches are private to each inference instance, whereas the L3 KV cache is shared among all inference instances within the cluster.
+
+### HiRadixTree: Metadata Organization in HiCache
+
+For KV cache data organization, HiCache builds upon the RadixTree structure introduced in RadixAttention and proposes HiRadixTree. In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory. A path from the root to a leaf node represents the prefix of a request, and shared prefixes across multiple requests can reuse the same nodes, thereby avoiding redundant storage.
+
+HiRadixTree extends this idea: each node corresponds to the KV cache of a span of consecutive tokens and records where that KV cache is stored—whether in local GPU memory, CPU memory, L3 storage, or multiple of these tiers. If stored locally, HiRadixTree maintains precise metadata, including the exact storage address. However, to reduce overhead, HiRadixTree does not store or continuously synchronize metadata for L3 KV cache. Instead, when accessing L3 data, it queries the backend in real time to retrieve the necessary metadata, such as whether the data exists and on which server and location it resides.
+
+### Overall Workflow
+
+The workflow of HiCache mainly involves three key operations: **local match**, **prefetch** and **write-back**. When the system receives a new request, it first searches the local L1 and L2 caches for matching KV caches. For parts not found locally, it attempts to prefetch from L3. After prefetching, all required KV caches are loaded into the GPU for computation. Once the prefill computation is complete, the system considers storing the newly generated data into L2 or L3.
+
+<Frame>
+  <img src="https://lmsys.org/images/blog/hicache/hicache_overview.png" alt="HiCache Workflow"/>
+</Frame>
+
+### Local Match
+
+Local matching is the first step in HiCache's workflow, where incoming request tokens are matched against the HiRadixTree to locate cached KV data in local memory tiers (L1 GPU memory and L2 host memory).
+
+The matching algorithm traverses the HiRadixTree from the root node, following child nodes that match the token sequence prefix. At each node, the incoming token sequence is compared with the node’s stored token sequence. When `page_size > 1`, matching is performed at the page granularity to optimize memory access patterns. If a match terminates within a node’s stored sequence, the node is automatically split to create an exact boundary, improving the efficiency of future matches.
+
+The algorithm returns a continuous prefix of the request, with the first part residing in L1 and the latter part in L2.
+
+Since the process only requires traversing the local HiRadixTree and does not involve any actual data copying, local matching is extremely fast.
+
+### Prefetch from L3
+
+Data prefetching is one of HiCache’s core optimization techniques, designed to proactively load KV caches from L3 storage into local L2 memory, thereby reducing access latency during subsequent operations.
+
+**Prefetch Trigger Conditions**:
+After local matching, for the parts not found in L1 or L2, the system queries L3 to retrieve metadata for the next continuous matching KV caches. If the length of hit cache in L3 exceeds a threshold (default: 256 tokens, configurable), a prefetch operation is triggered.
+
+**Prefetch Strategies**: HiCache provides three different prefetch termination strategies to address different scenario needs:
+- **best_effort**: Terminates immediately when GPU can execute prefill computation, with no waiting time, suitable for scenarios extremely sensitive to latency.
+- **wait_complete**: Must wait for all prefetch operations to complete, suitable for scenarios requiring high cache hit rates.
+- **timeout**: Terminates after specified time or when complete, balancing latency and cache hit rate needs.
+
+After prefetching stops, the data already fetched is used together with the local data for the prefill computation.
+
+For **timeout** strategy, HiCache introduces three configuration parameters to support fine-grained control over prefetch timeout conditions:
+
+* `prefetch_timeout_base`: the base timeout, representing overhead unrelated to the number of tokens (e.g., scheduling and synchronization). Default: `2` seconds.
+* `prefetch_timeout_per_ki_token`: the incremental timeout per thousand tokens. Default: `0.1` seconds per 1024 tokens.
+* `prefetch_timeout_max`: the upper bound applied to the linear timeout, preventing very long prompts from waiting unboundedly. Default: `30` seconds.
+
+The timeout is computed as:
+
+```python Example
+timeout = min(
+    prefetch_timeout_max,
+    prefetch_timeout_base + prefetch_timeout_per_ki_token * num_token_to_fetch / 1024,
+)
+```
+
+### Data Write-back
+
+The write-back mechanism is responsible for moving frequently accessed KV caches from L1 to L2 and L3, enabling larger and longer-term storage as well as cache sharing across instances.
+
+**Configurable Write-back Policies**: HiCache supports three write-back strategies:
+
+* **write_through**: Every access is immediately written back to the next level. When bandwidth is sufficient, this strategy provides the strongest caching benefit.
+* **write_through_selective**: Data is written back only after the access frequency exceeds a threshold. This strategy backs up only hot data, reducing I/O overhead.
+* **write_back**: Data is written back to the next level only when it is evicted from the upper level. This strategy alleviates storage pressure and is suitable for scenarios where storage capacity is limited but memory utilization must be maximized.
+
+**Cross-instance Sharing**: When data is written back from L2 to L3, only data not already present in L3 is transferred. KV caches stored in L3 can then be shared across all SGLang instances in the cluster (depending on the L3 backend implementation), significantly improving cache hit rates within the same memory budget.
+
+### Multi-Rank Synchronization
+
+During multi-GPU parallel computation, such as tensor parallelism (TP), HiCache must ensure consistent states across different ranks. Therefore, critical computation steps require the use of `all_reduce` for state synchronization.
+
+For example, during prefetching, `all_reduce(op=min)` is used to ensure that all ranks obtain the same number of L3 hits, preventing inconsistent judgments about whether the prefetch threshold has been reached. Similarly, after prefetching completes or terminates, `all_reduce(op=min)` is again required to guarantee consensus among ranks on the prefix length of the successfully retrieved KV cache.
+
+### Data Transfer Optimization
+
+**Zero-Copy Data Transfers**: Both prefetching and write-back involve substantial data movement. Minimizing the number of data copies can significantly improve system performance. HiCache supports passing memory addresses and sizes directly when transferring data from L2 memory to an L3 backend.
+
+**“Batch-Oriented” Data Organization**: The granularity of data reads and writes has a major impact on performance. To address this, HiCache L3 stores and transfers KV cache data at the granularity of **pages** and supports different data layouts beyond the existing `layer first` scheme, including `page first` and `page first direct`. Under the `page first` and `page first direct` layouts, all KV cache data belonging to the same page is placed in contiguous memory, allowing it to be passed as a single object to L3 using zero-copy transfers.
+
+<Frame>
+  <img src="https://lmsys.org/images/blog/hicache/hicache_layout.png" alt="HiCache L2 MEM layout"/>
+</Frame>
+
+However, because GPU KV computation is naturally performed layer by layer, the GPU inherently operates in a `layer first` layout. When transferring `page first` data from L2 to the GPU, data must be transferred at the granularity of one token per layer. The `page first direct` layout mitigates this issue by grouping together all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level.
+
+**CPU-to-GPU Transfer Optimizations**: In HiCache, moving data from CPU memory to GPU is as performance-critical as prefetching data from L3 to L2. HiCache employs several optimizations for this process:
+
+* **Compute-Transfer Overlap**: During the prefill phase, when transferring data from CPU to GPU, HiCache overlaps layers by concurrently loading the KV cache of layer N+1 while computing layer N. This effectively hides data transfer latency.
+* **GPU-assisted I/O Kernels**: On top of `cudaMemcpyAsync`, HiCache implements a set of GPU-assisted I/O kernels specifically optimized for KV cache transfers between CPU and GPU. Compared to the baseline approach, these kernels achieve up to 3x higher transfer speed.
+
+**Write-back Optimization for MLA**: For MHA (Multi-Head Attention) models under multi-TP, each rank holds `1/tp_size` of a token’s KV data. In contrast, for MLA (Multi-Layer Attention) models, all ranks hold the complete and identical KV data for each token. HiCache includes a dedicated optimization for MLA: only one rank initiates the write-back operation, ensuring that data is not redundantly stored across ranks.
+
+### Integration with PD-Disaggregation Deployment Mode
+
+SGLang supports a PD (Prefill-Decode) disaggregation deployment mode through the Mooncake TransferEngine (for details, see [this doc](./pd_disaggregation)). In the PD-disaggregation deployment mode, HiCache can be enabled on both the prefill nodes and decode nodes to optimize prefill performance. If enabled on decode nodes, the decode output will also be written back to L3.
+
+### Unified Interfaces and Rich L3 Storage Backends
+
+HiCache encapsulates all read, write, and query operations on L3 backends within the `class HiCacheStorage(ABC)`, exposing a set of simple and consistent interfaces. This design supports a wide range of L3 storage backends and allows users to select the one that best fits their specific use cases.
+
+- **Mooncake**: Mooncake is a high-performance caching system for LLM inference that leverages RDMA and multi-NIC resources to enable zero-copy, ultra-fast data transfers. Try Mooncake [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/mooncake_store).
+
+- **DeepSeek 3FS (HF3FS)**: HF3FS is a Kubernetes-native distributed storage solution with operator-based deployment. Try HF3FS [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs).
+
+- **NIXL**: NIXL provides a unified API for accessing various storage plugins, including but not limited to DeepSeek's 3FS, GPU Direct Storage (GDS) and Amazon S3-compatible object storage. Try NIXL [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/nixl).
+
+- **AIBrix KVCache**: AIBrix KVCache is a production-ready KVCache Offloading Framework, which enables efficient memory tiering and low-overhead cross-engine reuse. Try AIBrix KVCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/aibrix_kvcache).
+
+- **HiCacheFile**: A simple file-based storage backend for demonstration purposes.
+
+Specifically, **LMCache**, an efficient KV cache layer for enterprise-scale LLM inference, provides an alternative solution to HiCache. Try LMCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/lmcache).
+
+## Related Parameters
+
+- **`--enable-hierarchical-cache`**: Enable hierarchical cache functionality. This is required to use HiCache.
+
+- **`--hicache-ratio HICACHE_RATIO`**: The ratio of the size of host KV cache memory pool to the size of device pool. For example, a value of 2 means the host memory pool is twice as large as the device memory pool. The value of this parameter must be greater than 1, as the current implementation requires the host memory allocated for the KV cache to be larger than the device memory allocated for the KV cache.
+
+- **`--hicache-size HICACHE_SIZE`**: The size of host KV cache memory pool in gigabytes. This parameter overrides `hicache-ratio` if set. For example, `--hicache-size 30` allocates 30GB (1GB = 1e9 bytes) for the host memory pool **for each rank**. If there are 8 ranks, then the total memory size is 240GB. Just like `hicache-ratio`, the value of this parameter must be larger than the size of device memory allocated for KV cache.
+
+**Note**: `--hicache-ratio` and `--hicache-size` are two critical parameters. In general, a larger HiCache size leads to a higher cache hit rate, which improves prefill performance. However, the relationship between cache size and hit rate is not linear. Once most reusable KV data—especially hot tokens—are already cached, further increasing the size may yield only marginal performance gains. Users can set these parameters based on their workload characteristics and performance requirements.
+
+- **`--page-size PAGE_SIZE`**: The number of tokens per page. This parameter determines the granularity of KV cache storage and retrieval. Larger page sizes reduce metadata overhead and improve I/O efficiency for storage backends, but may lower the cache hit rate when only part of a page matches the stored KV cache. For workloads with long common prefixes, larger pages can improve performance, while workloads with more diverse prefixes may benefit from smaller pages. See [Data Transfer Optimization](#data-transfer-optimization) for how page granularity affects I/O performance.
+
+- **`--hicache-storage-prefetch-policy {best_effort,wait_complete,timeout}`**: Controls when prefetching from storage should stop. See [Prefetch from L3](#prefetch-from-l3) for details.
+  - `best_effort`: Prefetch as much as possible without blocking
+  - `wait_complete`: Wait for prefetch to complete before proceeding
+  - `timeout`: Terminates after specified time or when complete (Recommended for production environments, as setting an appropriate timeout helps the system meet required SLOs)
+
+- **`--hicache-write-policy {write_back,write_through,write_through_selective}`**: Controls how data is written from faster to slower memory tiers. See [Data Write-back](#data-write-back) for details.
+  - `write_through`: Immediately writes data to all tiers (strongest caching benefits)
+  - `write_through_selective`: Uses hit-count tracking to back up only frequently accessed data
+  - `write_back`: Writes data back to slower tiers only when eviction is needed (reduces I/O load)
+
+- **`--hicache-io-backend {direct,kernel}`**: Choose the I/O backend for KV cache transfer between CPU and GPU. See [Data Transfer Optimization](#data-transfer-optimization) for details.
+  - `direct`: Standard CUDA memory copy operations
+  - `kernel`: GPU-assisted I/O kernels (recommended for better performance)
+
+- **`--hicache-mem-layout {layer_first,page_first,page_first_direct}`**: Memory layout for the host memory pool. See [Data Transfer Optimization](#data-transfer-optimization) for details.
+  - `layer_first`: Compatible with GPU computation kernels (default for GPU memory)
+  - `page_first`: Optimized for I/O efficiency
+  - `page_first_direct`: Groups all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level
+
+- **`--hicache-storage-backend {file,mooncake,hf3fs,nixl,aibrix,dynamic}`**: Choose the storage backend for the L3 tier. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: `backend_name` (custom name), `module_path` (Python module path), `class_name` (backend class name). See [Unified Interfaces and Rich L3 Storage Backends](#unified-interfaces-and-rich-l3-storage-backends) for available backends.
+
+- **`--enable-lmcache`**: Using LMCache as an alternative hierarchical cache solution.
+
+- **`--lmcache-config-file`**: Path to the LMCache YAML configuration file.
+
+- **`--hicache-storage-backend-extra-config HICACHE_STORAGE_BACKEND_EXTRA_CONFIG`**: the extra config can be either
+  - a JSON string containing extra configuration for the storage backend, e.g., `--hicache-storage-backend-extra-config '{"prefetch_threshold":512, "prefetch_timeout_base": 0.5, "prefetch_timeout_per_ki_token": 0.25}' `, or
+  - a TOML or JSON or YAML file specifying the extra configuration for the storage backend (to differentiate from the JSON string input, prepend a `@` in front of the file name), e.g., `--hicache-storage-backend-extra-config "@config.toml"` where `config.toml` is the config file containing the complex configurations. This can be useful when the configuration consists of many or complex key-value pairs (for instance, it is preferred to use a config file for NIXL backend as its configurations can be complex).

docs_new/docs/advanced_features/hicache_storage_runtime_attach_detach.mdx

@@ -0,0 +1,133 @@
+---
+title: "Runtime Attach/Detach HiCache Storage Backend (No Restart)"
+metatags:
+    description: "Dynamically attach/detach HiCache L3 storage backends at runtime via HTTP API. No restart required, idle-state safety checks."
+---
+This document explains how to **dynamically attach/detach the HiCache L3 storage backend at runtime** (e.g., `mooncake` / `hf3fs` / `nixl` / `file` / `aibrix` / `eic`) while **SGLang is already running and serving traffic**, without restarting the process.
+
+For safety and consistency, the current implementation **strictly requires** these operations to happen only when the service is **idle**:
+
+- **No running requests**
+- **No waiting/queued requests**
+
+If the idle condition is not met, the API will fail fast (HTTP 400) and **will not modify** the current service state.
+
+***
+## 1. Background and implementation overview
+
+### 1.1 Architecture / control path
+
+The control path is:
+
+1. **HTTP Server** (`python/sglang/srt/entrypoints/http_server.py`)
+   - Exposes `PUT /hicache/storage-backend`, `DELETE /hicache/storage-backend`, `GET /hicache/storage-backend`
+2. **TokenizerManager** (`python/sglang/srt/managers/tokenizer_control_mixin.py`)
+   - Sends the request to the Scheduler via `FanOutCommunicator`
+3. **Scheduler** (`python/sglang/srt/managers/scheduler.py`)
+   - Performs a **strict idle check**
+   - Calls `tree_cache.attach_storage_backend(...)` / `detach_storage_backend(...)`
+4. **HiRadixCache** (`python/sglang/srt/mem_cache/hiradix_cache.py`)
+   - Parses `hicache_storage_backend_extra_config_json` (supports both backend config and prefetch knobs)
+   - Calls `cache_controller.attach_storage_backend(...)` / `detach_storage_backend(...)`
+5. **HiCacheController** (`python/sglang/srt/managers/cache_controller.py`)
+   - Creates/destroys the storage backend instance (via `StorageBackendFactory`)
+   - Starts/stops backend background threads at runtime (prefetch/backup)
+
+***
+## 2. Idle-state requirement (strict)
+
+The Scheduler uses `is_fully_idle()` which checks:
+
+- No running batches (including chunked prefill, overlap, pipeline-parallel, and disaggregation paths)
+- No waiting requests in any queue (waiting, grammar, disagg bootstrap/prealloc/transfer/inflight)
+- No DLLM staging requests
+
+If the condition is not met, attach/detach returns an error like:
+
+- `Reject attach: scheduler is not idle. #queue-req=... #running-req=...`
+
+<Tip>
+before switching, drain upstream traffic and wait for the server to become idle, then call attach/detach.
+</Tip>
+
+### 2.1 DP (data parallel) semantics
+
+When `dp_size > 1`, the tokenizer dispatches the request to **all DP scheduler instances** and aggregates their responses:
+
+- The final `success` is **true only if all DP ranks return success**
+- The final `message` concatenates messages from all DP ranks
+
+This is intended to prevent “silent partial success”, but it also means you may see:
+
+- Overall **failure** even though **some ranks already succeeded**
+
+Currently there is **no automatic partial rollback** across DP ranks (see TODO in code). Operationally:
+
+- Prefer to keep backend config identical across ranks
+- If attach fails, immediately call detach (best-effort/idempotent), fix config, then retry attach
+
+***
+## 3. How to use (HTTP Admin API)
+
+The examples below assume your SGLang HTTP server is at `http://127.0.0.1:30000`.
+
+### 3.1 Query current storage backend status
+
+```bash Command
+curl -s http://127.0.0.1:30000/hicache/storage-backend
+```
+
+Example response:
+
+```json Config
+{
+  "hicache_storage_backend": "mooncake",
+  "hicache_storage_backend_extra_config": "{\"master_server_address\":\"127.0.0.1:50051\", ...}"
+}
+```
+
+### 3.2 Attach (enable) a storage backend
+```bash Command
+curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "hicache_storage_backend": "mooncake"
+  }'
+```
+
+```bash Command
+curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "hicache_storage_backend": "mooncake",
+    "hicache_storage_backend_extra_config_json": "{\"master_server_address\":\"127.0.0.1:50051\",\"protocol\":\"tcp\",\"global_segment_size\":\"4gb\",\"prefetch_threshold\":256}",
+    "hicache_storage_prefetch_policy": "timeout"
+  }'
+```
+
+Notes:
+
+- `hicache_storage_backend_extra_config_json` can include both:
+  - **Backend configuration** (e.g., Mooncake master/metadata/protocol, etc.)
+  - **Prefetch configuration** (`prefetch_threshold`, `prefetch_timeout_base`, `prefetch_timeout_per_ki_token`, `prefetch_timeout_max`, `hicache_storage_pass_prefix_keys`)
+
+### 3.3 Detach (disable) the storage backend
+
+```bash Command
+curl -s -X DELETE http://127.0.0.1:30000/hicache/storage-backend
+```
+
+Notes:
+
+- Detach only makes SGLang **stop using** the L3 storage backend and stops prefetch/backup threads
+- It **does not automatically delete** data stored in Mooncake/HF3FS (or other remote backends)
+
+***
+## 4. Behavior and caveats
+
+- **No restart required**: attach/detach switches in-process at runtime
+- **Must be idle**: otherwise the request is rejected to avoid consistency issues
+- **Host KV layout constraints still apply**: for example, Mooncake still requires layouts like `page_first/page_first_direct/page_head`; if the server's HiCache host-memory layout does not satisfy the backend requirements, attach will fail with an error
+- **Observability**:
+  - After attach, `server_args.hicache_storage_backend*` is updated on both the tokenizer and scheduler sides
+  - If metrics are enabled, attach will create a storage metrics collector in `HiRadixCache` on demand

docs_new/docs/advanced_features/hisparse_guide.mdx

@@ -0,0 +1,193 @@
+---
+title: "HiSparse: Hierarchical Sparse Attention"
+metatags:
+    description: "Use HiSparse hierarchical sparse attention to reduce decode GPU KV memory with CPU pinned host storage and PD disaggregation."
+---
+
+HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency.
+
+> **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1) and **DeepSeek V4**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only.
+
+## Why HiSparse?
+
+In long-context LLM inference, each decoding request holds a full-length KV cache on GPU, limiting the number of concurrent requests a decode instance can serve. HiSparse addresses this by:
+
+- **Reducing GPU memory per request**: Each request occupies only a fixed-size device buffer (e.g., 4KB tokens) instead of the full sequence length.
+- **On-demand swap-in**: A CUDA kernel dynamically loads the top-k most relevant KV entries from host memory based on attention scores.
+- **Transparent to prefill**: HiSparse is entirely a decode-side optimization; the prefill instance requires no changes.
+
+## Design Overview
+
+### Decode Workflow
+
+Each decode step follows this flow:
+
+1. **Forward decode** — generate the next token
+2. **Top-k selection** — select the most relevant token positions via attention scores
+3. **Swap-in** — the CUDA kernel loads top-k KV entries from host to device buffer:
+   - *Short sequences* (`seq_len ≤ device_buffer_size`): fast path, all KV already in buffer
+   - *Long sequences*: hit detection → LRU reordering → miss handling (host → device copy)
+4. **Decode attention** — compute attention using the top-k device locations
+5. **Eager backup** — asynchronously copy the previous token's KV from device to host
+
+### PD Disaggregation Integration (Direct-to-Host)
+
+In PD disaggregation mode, the prefill instance transfers KV cache directly into the decode instance's host pool via RDMA, bypassing the GPU entirely on the decode side. This eliminates the transient GPU memory spike during KV transfer and removes the staging DMA step.
+
+```
+Prefill GPU  ──RDMA──▶  Decode Host Pool (CPU pinned memory)
+                              │
+                              ▼
+                     alloc device buffer (4KB)
+                              │
+                              ▼
+                     swap-in kernel (on-demand top-k)
+```
+
+For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4_indexer and C128 KV remain device-to-device transfers.
+
+## Server Arguments
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Argument</th>
+      <th>Type / Default</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>--enable-hisparse</code></td>
+      <td>flag; default: disabled</td>
+      <td>Enable HiSparse on the decode instance</td>
+    </tr>
+    <tr>
+      <td><code>--hisparse-config</code></td>
+      <td>JSON string</td>
+      <td>Configuration for HiSparse (see below)</td>
+    </tr>
+  </tbody>
+</table>
+
+### HiSparse Config Parameters
+
+Pass as a JSON string via `--hisparse-config`:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Parameter</th>
+      <th>Type / Default</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>top_k</code></td>
+      <td>int</td>
+      <td>Number of topk entries</td>
+    </tr>
+    <tr>
+      <td><code>device_buffer_size</code></td>
+      <td>int</td>
+      <td>Number of token slots in the per-request GPU device buffer</td>
+    </tr>
+    <tr>
+      <td><code>host_to_device_ratio</code></td>
+      <td>int</td>
+      <td>Ratio of logical pool size to device pool size, determining host memory capacity</td>
+    </tr>
+    <tr>
+      <td><code>swap_in_block_size</code></td>
+      <td>int / 960</td>
+      <td>CUDA thread-block size for the HiSparse swap-in kernel</td>
+    </tr>
+  </tbody>
+</table>
+
+Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'`
+
+## Deployment
+
+HiSparse currently requires **PD disaggregation mode** and is enabled only on the **decode instance**.
+
+### Prefill Instance
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path /path/to/model \
+    --trust-remote-code \
+    --port 8000 --host 0.0.0.0 \
+    --context-length 81920 \
+    --chunked-prefill-size 65536 \
+    --tp-size 8 --dp-size 8 --enable-dp-attention \
+    --mem-fraction-static 0.85 \
+    --disaggregation-mode prefill \
+    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
+    --nnodes 1 --node-rank 0
+```
+
+### Decode Instance (with HiSparse)
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path /path/to/model \
+    --trust-remote-code \
+    --port 8000 --host 0.0.0.0 \
+    --context-length 81920 \
+    --tp-size 8 --dp-size 8 --enable-dp-attention \
+    --mem-fraction-static 0.85 \
+    --disable-radix-cache \
+    --disaggregation-mode decode \
+    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
+    --dist-init-addr 127.0.0.1:5757 \
+    --nnodes 1 --node-rank 0 \
+    --enable-hisparse \
+    --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'
+```
+
+> **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`). DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend.
+
+### Benchmark
+
+```bash Command
+python3 -m sglang.bench_serving \
+    --backend sglang \
+    --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
+    --dataset-name random \
+    --random-input 40000 \
+    --random-output 20000 \
+    --num-prompts 200 \
+    --max-concurrency 200 \
+    --request-rate 40 \
+    --random-range-ratio 1.0 \
+    --host 127.0.0.1 \
+    --port 20000 \
+    --model /path/to/model \
+    --flush-cache \
+```
+
+### Key Notes
+
+- The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse.
+- On the decode instance, `--enable-hisparse` and `--hisparse-config` are required for HiSparse.
+- For DSA models, `--kv-cache-dtype bfloat16` uses `flashmla_sparse`, and `--kv-cache-dtype fp8_e4m3` uses `flashmla_kv`.
+- For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the `dsv4` attention backend and `fp8_e4m3` KV cache by default.
+- `host_to_device_ratio` should be configured based on the host machine's available memory. For example:
+  - **~1 TB** host memory → `host_to_device_ratio: 5`
+  - **~2 TB** host memory → `host_to_device_ratio: 10`
+
+## Acknowledgments
+
+We would like to thank the SGLang team and community for the implementation and generous support, especially Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Shangming Cai, Teng Ma, and many others. We also thank the Alibaba Cloud TairKVCache team and the AntGroup SCT Inference team for their valuable contributions.

docs_new/docs/advanced_features/hyperparameter_tuning.mdx

@@ -0,0 +1,82 @@
+---
+title: "Hyperparameter Tuning"
+metatags:
+    description: "SGLang performance tuning: batch size, token usage, mem-fraction-static, chunked-prefill-size, CUDA graph, DP/TP optimization."
+---
+## Achieving high throughput for offline batch inference
+
+Achieving a large batch size is the most important thing for attaining high throughput in offline batch inference.
+When the server is running at full load in a steady state, look for the following in the log:
+
+```text Output
+Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317
+```
+
+### Adjust the request submission speed to control `#queue-req`
+
+`#queue-req` indicates the number of requests in the queue.
+If you frequently see `#queue-req: 0`, it suggests that your client code is submitting requests too slowly.
+A healthy range for `#queue-req` is `100 - 2000`.
+However, avoid making `#queue-req` too large, as this will increase the scheduling overhead on the server.
+
+### Achieve a high `token usage`
+
+`token usage` indicates the KV cache memory utilization of the server. `token usage > 0.9` means good utilization.
+
+If you frequently see `token usage < 0.9` and `#queue-req > 0`, it means the server is too conservative about taking in new requests. You can decrease `--schedule-conservativeness` to a value like 0.3.
+The case of a server being too conservative can happen when users send many requests with a large `max_new_tokens` but the requests stop very early due to EOS or stop strings.
+
+On the other hand, if you see `token usage` very high and you frequently see warnings like
+`KV cache pool is full. Retract requests. #retracted_reqs: 1, #new_token_ratio: 0.9998 -> 1.0000`, you can increase `--schedule-conservativeness` to a value like 1.3.
+If you see `KV cache pool is full. Retract requests.` occasionally but not frequently (~1 time per minute), it is okay.
+
+### Tune `--mem-fraction-static` to increase KV cache pool capacity
+SGLang allocates memory as follows:
+
+Total memory usage = model weights + KV cache pool + CUDA graph buffers + activations
+
+The `--mem-fraction-static` parameter determines how much memory is allocated to the first two components:
+
+mem_fraction_static = (model weights + KV cache pool) / GPU memory capacity
+
+To support higher concurrency, you should maximize the KV cache pool capacity by setting `--mem-fraction-static` as high as possible while still reserving enough memory for activations and CUDA graph buffers.
+
+SGLang uses simple heuristics to set the default value of `--mem-fraction-static`, but you can optimize it for your use cases.
+As a rule of thumb, reserving 5–8 GB of memory for activations is typically sufficient. You can check this by inspecting the logs just before the server is ready.
+Look for log entries like this:
+
+```text Output
+[2025-08-11 17:17:03] max_total_num_tokens=665690, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4096, context_len=65536, available_gpu_mem=13.50 GB
+```
+
+Check the `available_gpu_mem` value.
+- If it is between 5–8 GB, the setting is good.
+- If it is too high (e.g., 10 - 20 GB), increase `--mem-fraction-static` to allocate more memory to the KV cache.
+- If it is too low, you risk out-of-memory (OOM) errors later, so decrease `--mem-fraction-static`.
+
+Another straightforward approach is to increase `--mem-fraction-static` in increments of 0.01 until you encounter OOM errors for your workloads.
+
+### Avoid out-of-memory errors by tuning `--chunked-prefill-size`, `--mem-fraction-static`, and `--max-running-requests`
+
+If you encounter out-of-memory (OOM) errors, you can adjust the following parameters:
+
+- If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts.
+- If OOM occurs during decoding, try lowering `--max-running-requests`.
+- You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.
+
+### Tune `--cuda-graph-max-bs-decode`
+By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256).
+However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768.
+Therefore, it may be beneficial to increase `--cuda-graph-max-bs-decode` to a larger value.
+Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time.
+
+### Tune `--dp-size` and `--tp-size`
+
+Data parallelism is better for throughput. When there is enough GPU memory, always favor data parallelism for throughput. Refer to [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for a better data parallelism rather than using `dp_size` parameter.
+
+### Try other options
+
+- `torch.compile` accelerates small models on small batch sizes. You can enable it with `--enable-torch-compile`.
+- Try other quantization (e.g. FP8 quantization with `--quantization fp8`)
+- Try other parallelism strategies (e.g. [expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) or DP attention for deepseek models (with `--enable-dp-attention --dp-size 8`).
+- If the workload has many shared prefixes, try `--schedule-policy lpm`. Here, `lpm` stands for longest prefix match. It reorders requests to encourage more cache hits but introduces more scheduling overhead.

docs_new/docs/advanced_features/llm-d.mdx

@@ -0,0 +1,41 @@
+---
+title: "llm-d"
+metatags:
+    description: "Run SGLang at fleet scale with llm-d: prefix-aware routing, distributed KV-cache management, prefill/decode disaggregation, SLO-aware autoscaling on Kubernetes."
+---
+
+[llm-d](https://llm-d.ai/) is a Kubernetes-native distributed inference framework for serving large language models at scale across a fleet of inference servers. SGLang is a supported inference engine in llm-d: llm-d coordinates a fleet of SGLang instances across a cluster so that performance holds up under real production traffic, achieving the fastest "time to state-of-the-art (SOTA) performance" for key OSS models across most hardware accelerators.
+
+llm-d is a [CNCF Sandbox project](https://www.cncf.io/blog/2026/03/24/welcome-llm-d-to-the-cncf-evolving-kubernetes-into-sota-ai-infrastructure/) founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.
+
+## What llm-d adds to an SGLang deployment
+
+A single SGLang server is fast, and RadixAttention already maximizes cache reuse within each replica. But at scale the picture changes: across many replicas, cache locality breaks under round-robin load balancing as related requests scatter and radix-cache hit rates collapse, long prompts inflate time-to-first-token, and accelerators sit underused. llm-d adds the cluster-level layer that the engine does not aim to provide on its own:
+
+- **[Prefix-aware routing](https://llm-d.ai/docs/guides/precise-prefix-cache-aware).** Instead of round-robin, the llm-d Router scores each replica on prefix-cache locality and current load, routing each request to the replica most likely to already hold its prefix — raising RadixAttention hit rates on multi-turn and shared-prefix workloads while avoiding saturated servers.
+- **[Distributed KV-cache management](https://llm-d.ai/docs/guides#advanced-kv-cache-management).** A global index tracks which token blocks live on which replica, and tiered offloading spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM.
+- **[Prefill/decode disaggregation](https://llm-d.ai/docs/guides/pd-disaggregation).** Prompt processing and token generation run on separate workers, with KV-cache moved over high-speed interconnects, lowering TTFT and steadying per-token latency on long prompts.
+- **[SLO-aware autoscaling](https://llm-d.ai/docs/guides/workload-autoscaling) and [flow control](https://llm-d.ai/docs/guides/flow-control).** Scale SGLang pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch.
+- **One control plane for mixed fleets.** llm-d schedules across engines, so platform teams can serve SGLang and vLLM pools behind the same gateway, policies, and observability instead of running parallel stacks.
+
+These capabilities are composable. Most teams start by adding prefix-aware routing over an existing SGLang pool, then layer in the rest as specific bottlenecks appear.
+
+## Kubernetes-native gateway
+
+llm-d builds on the Gateway API Inference Extension, so SGLang pools are managed through standard Kubernetes resources (Gateway, HTTPRoute, InferencePool) and work with supported gateway providers such as Istio, GKE Inference Gateway, and agentgateway, rather than a bespoke routing tier.
+
+## Performance
+
+llm-d publishes reproducible benchmarks from production-scale deployments on [Prism](https://prism.llm-d.ai/). One representative result: prefix-aware routing delivered **3x higher output throughput and 2x faster TTFT** than round-robin load balancing (Llama 3.1 70B). The mechanism carries over directly to SGLang, where RadixAttention makes the cluster-level cache hit rate a function of routing.
+
+## Get started
+
+- **Deploy the optimized baseline with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart), selecting SGLang as the inference server.** It stands up an intelligent router (the llm-d Router) over an SGLang pool on Kubernetes in a tested configuration.
+- Browse the [well-lit path guides](https://llm-d.ai/docs/guides), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload.
+- Read the [Introduction](https://llm-d.ai/docs/getting-started) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the scheduler, gateway, and model servers wrap your SGLang deployment.
+
+Questions and contributions are welcome on [GitHub](https://github.com/llm-d/llm-d) and [Slack](https://llm-d.ai/slack).
+
+## Current scope
+
+SGLang is supported across the well-lit paths — including intelligent inference scheduling, precise prefix-cache routing (SGLang publishes KV-cache events that the llm-d Router subscribes to), tiered KV-cache management, prefill/decode disaggregation, flow control, and autoscaling. The one current exception is Multi-Node Wide Expert Parallelism, which is vLLM-specific today. See the llm-d documentation for the latest per-engine support status.

docs_new/docs/advanced_features/lora.ipynb

@@ -0,0 +1,714 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# LoRA Serving"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Arguments for LoRA Serving"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The following server arguments are relevant for multi-LoRA serving:\n",
+    "\n",
+    "* `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.\n",
+    "\n",
+    "* `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.\n",
+    "\n",
+    "* `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> | <NAME>=<PATH> | JSON with schema {\"lora_name\":str,\"lora_path\":str,\"pinned\":bool}.\n",
+    "\n",
+    "* `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.\n",
+    "\n",
+    "* `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`.\n",
+    "\n",
+    "* `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out.\n",
+    "\n",
+    "* `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added.\n",
+    "\n",
+    "* `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.\n",
+    "\n",
+    "* `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters.\n",
+    "\n",
+    "* `--max-lora-chunk-size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16.\n",
+    "\n",
+    "* `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper.\n",
+    "\n",
+    "From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Usage\n",
+    "\n",
+    "### Serving Single Adaptor"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Note:** SGLang supports LoRA adapters through two APIs:\n",
+    "\n",
+    "1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions.ipynb#Using-LoRA-Adapters) for examples.\n",
+    "\n",
+    "2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import requests\n",
+    "\n",
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, terminate_process"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\n",
+    "    # Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model\n",
+    "    \"\"\"\n",
+    "python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
+    "    --max-loras-per-batch 2 \\\n",
+    "    --log-level warning \\\n",
+    "\"\"\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "json_data = {\n",
+    "    \"text\": [\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "    ],\n",
+    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
+    "    # The first input uses lora0, and the second input uses the base model\n",
+    "    \"lora_path\": [\"lora0\", None],\n",
+    "}\n",
+    "response = requests.post(\n",
+    "    url + \"/generate\",\n",
+    "    json=json_data,\n",
+    ")\n",
+    "print(f\"Output 0: {response.json()[0]['text']}\")\n",
+    "print(f\"Output 1: {response.json()[1]['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Serving Multiple Adaptors"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
+    "    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \\\n",
+    "    --max-loras-per-batch 2 \\\n",
+    "    --log-level warning \\\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "json_data = {\n",
+    "    \"text\": [\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "    ],\n",
+    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
+    "    # The first input uses lora0, and the second input uses lora1\n",
+    "    \"lora_path\": [\"lora0\", \"lora1\"],\n",
+    "}\n",
+    "response = requests.post(\n",
+    "    url + \"/generate\",\n",
+    "    json=json_data,\n",
+    ")\n",
+    "print(f\"Output 0: {response.json()[0]['text']}\")\n",
+    "print(f\"Output 1: {response.json()[1]['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Dynamic LoRA loading"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API.\n",
+    "\n",
+    "When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly \"smaller\"."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj\n",
+    "lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"  # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
+    "lora0_new = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\"  # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
+    "\n",
+    "\n",
+    "# The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified.\n",
+    "# We are adding it here just to demonstrate usage.\n",
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --cuda-graph-max-bs-decode 2 \\\n",
+    "    --max-loras-per-batch 2 \\\n",
+    "    --max-lora-rank 256\n",
+    "    --lora-target-modules all\n",
+    "    --log-level warning\n",
+    "    \"\"\")\n",
+    "\n",
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "wait_for_server(url, process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Load adapter lora0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(\n",
+    "    url + \"/load_lora_adapter\",\n",
+    "    json={\n",
+    "        \"lora_name\": \"lora0\",\n",
+    "        \"lora_path\": lora0,\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "if response.status_code == 200:\n",
+    "    print(\"LoRA adapter loaded successfully.\", response.json())\n",
+    "else:\n",
+    "    print(\"Failed to load LoRA adapter.\", response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Load adapter lora1:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(\n",
+    "    url + \"/load_lora_adapter\",\n",
+    "    json={\n",
+    "        \"lora_name\": \"lora1\",\n",
+    "        \"lora_path\": lora1,\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "if response.status_code == 200:\n",
+    "    print(\"LoRA adapter loaded successfully.\", response.json())\n",
+    "else:\n",
+    "    print(\"Failed to load LoRA adapter.\", response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Check inference output:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "json_data = {\n",
+    "    \"text\": [\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "    ],\n",
+    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
+    "    # The first input uses lora0, and the second input uses lora1\n",
+    "    \"lora_path\": [\"lora0\", \"lora1\"],\n",
+    "}\n",
+    "response = requests.post(\n",
+    "    url + \"/generate\",\n",
+    "    json=json_data,\n",
+    ")\n",
+    "print(f\"Output from lora0: \\n{response.json()[0]['text']}\\n\")\n",
+    "print(f\"Output from lora1 (updated): \\n{response.json()[1]['text']}\\n\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Unload lora0 and replace it with a different adapter:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(\n",
+    "    url + \"/unload_lora_adapter\",\n",
+    "    json={\n",
+    "        \"lora_name\": \"lora0\",\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "response = requests.post(\n",
+    "    url + \"/load_lora_adapter\",\n",
+    "    json={\n",
+    "        \"lora_name\": \"lora0\",\n",
+    "        \"lora_path\": lora0_new,\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "if response.status_code == 200:\n",
+    "    print(\"LoRA adapter loaded successfully.\", response.json())\n",
+    "else:\n",
+    "    print(\"Failed to load LoRA adapter.\", response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Check output again:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "json_data = {\n",
+    "    \"text\": [\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "    ],\n",
+    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
+    "    # The first input uses lora0, and the second input uses lora1\n",
+    "    \"lora_path\": [\"lora0\", \"lora1\"],\n",
+    "}\n",
+    "response = requests.post(\n",
+    "    url + \"/generate\",\n",
+    "    json=json_data,\n",
+    ")\n",
+    "print(f\"Output from lora0: \\n{response.json()[0]['text']}\\n\")\n",
+    "print(f\"Output from lora1 (updated): \\n{response.json()[1]['text']}\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### OpenAI-compatible API usage\n",
+    "\n",
+    "You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai_api_completions.ipynb](../basic_usage/openai_api_completions.ipynb).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### LoRA GPU Pinning"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded.\n",
+    "\n",
+    "This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations. \n",
+    "\n",
+    "In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --cuda-graph-max-bs-decode 8 \\\n",
+    "    --max-loras-per-batch 3 \\\n",
+    "    --max-lora-rank 256 \\\n",
+    "    --lora-target-modules all \\\n",
+    "    --lora-paths \\\n",
+    "        {\"lora_name\":\"lora0\",\"lora_path\":\"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\",\"pinned\":true} \\\n",
+    "        {\"lora_name\":\"lora1\",\"lora_path\":\"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"} \\\n",
+    "        lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora\n",
+    "    --log-level warning\n",
+    "    \"\"\")\n",
+    "\n",
+    "\n",
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "wait_for_server(url, process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(\n",
+    "    url + \"/unload_lora_adapter\",\n",
+    "    json={\n",
+    "        \"lora_name\": \"lora1\",\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "response = requests.post(\n",
+    "    url + \"/load_lora_adapter\",\n",
+    "    json={\n",
+    "        \"lora_name\": \"lora1\",\n",
+    "        \"lora_path\": \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\",\n",
+    "        \"pinned\": True,  # Pin the adapter to GPU\n",
+    "    },\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Verify that the results are expected:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "json_data = {\n",
+    "    \"text\": [\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "    ],\n",
+    "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
+    "    # The first input uses lora0, and the second input uses lora1\n",
+    "    \"lora_path\": [\"lora0\", \"lora1\", \"lora2\"],\n",
+    "}\n",
+    "response = requests.post(\n",
+    "    url + \"/generate\",\n",
+    "    json=json_data,\n",
+    ")\n",
+    "print(f\"Output from lora0 (pinned): \\n{response.json()[0]['text']}\\n\")\n",
+    "print(f\"Output from lora1 (pinned): \\n{response.json()[1]['text']}\\n\")\n",
+    "print(f\"Output from lora2 (not pinned): \\n{response.json()[2]['text']}\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Choosing LoRA Backend\n",
+    "\n",
+    "SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument:\n",
+    "\n",
+    "- `triton`: Basic Triton-based backend.\n",
+    "- `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios.\n",
+    "\n",
+    "The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "    python3 -m sglang.launch_server \\\n",
+    "    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --lora-backend csgmv \\\n",
+    "    --max-loras-per-batch 16 \\\n",
+    "    --lora-paths lora1=path/to/lora1 lora2=path/to/lora2\n",
+    "    \"\"\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## LoRA Overlap Loading"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a ~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\"\n",
+    "lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"\n",
+    "lora2 = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\"\n",
+    "\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "    python3 -m sglang.launch_server \\\n",
+    "    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --enable-lora-overlap-loading \\\n",
+    "    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \\\n",
+    "    lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
+    "    lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \\\n",
+    "    --max-lora-rank 256 \\\n",
+    "    --max-loras-per-batch 2 \\\n",
+    "    --max-loaded-loras 4\n",
+    "    \"\"\")\n",
+    "\n",
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "wait_for_server(url, process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "json_data = {\n",
+    "    \"text\": [\n",
+    "        \"Write a very long fairy-tale.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
+    "    ],\n",
+    "    \"sampling_params\": [\n",
+    "        {\"max_new_tokens\": 1024, \"temperature\": 0},\n",
+    "        {\"max_new_tokens\": 64, \"temperature\": 0},\n",
+    "        {\"max_new_tokens\": 64, \"temperature\": 0},\n",
+    "    ],\n",
+    "    \"lora_path\": [\"lora0\", \"lora1\", \"lora2\"],\n",
+    "}\n",
+    "\n",
+    "# lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue.\n",
+    "# lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will\n",
+    "# occur asynchronously and thus decoding for lora0's request won't be blocked.\n",
+    "response = requests.post(\n",
+    "    url + \"/generate\",\n",
+    "    json=json_data,\n",
+    ")\n",
+    "\n",
+    "for i in range(3):\n",
+    "    print(f\"Output from lora{i}: \\n{response.json()[i]['text']}\\n\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Limitations of LoRA Overlap Loading"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "However, LoRA overlap loading is not free and comes with two important caveats:\n",
+    "\n",
+    "1. **Pinned CPU memory requirement**:\n",
+    "   Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled.\n",
+    "\n",
+    "2. **Reduced multi-adapter prefill batching**:\n",
+    "   With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Example When Overlap Loading Results in Higher Latency"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms.\n",
+    "\n",
+    "1. **Baseline**:\n",
+    "  The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms`\n",
+    "\n",
+    "2. **With LoRA overlap loading enabled**:\n",
+    "  The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms`\n",
+    "\n",
+    "In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Future Works\n",
+    "\n",
+    "The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development."
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/advanced_features/lora.mdx

@@ -0,0 +1,509 @@
+---
+title: "LoRA Serving"
+metatags:
+    description: "SGLang multi-LoRA serving: S-LoRA and Punica techniques, dynamic adapter loading, GPU pinning, overlap loading, Triton and CSGMV backends."
+---
+SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs.
+
+
+## Arguments for LoRA Serving
+
+
+The following server arguments are relevant for multi-LoRA serving:
+
+* `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.
+
+* `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.
+
+* `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> | <NAME>=<PATH> | JSON with schema {"lora_name":str,"lora_path":str,"pinned":bool}.
+
+* `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.
+
+* `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`.
+
+* `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out.
+
+* `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added.
+
+* `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.
+
+* `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters.
+
+* `max_lora_chunk_size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16.
+
+* `lora_drain_wait_threshold`: When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default).
+
+* `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper.
+
+From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to.
+
+
+## Usage
+
+### Serving Single Adaptor
+
+
+**Note:** SGLang supports LoRA adapters through two APIs:
+
+1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions#using-lora-adapters) for examples.
+
+2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below).
+
+
+
+```python Example
+import json
+import requests
+
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, terminate_process
+```
+
+
+```python Example
+server_process, port = launch_server_cmd(
+    # Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model
+    """
+python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --enable-lora \
+    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
+    --max-loras-per-batch 2 \
+    --log-level warning \
+"""
+)
+
+wait_for_server(f"http://localhost:{port}")
+```
+
+
+```python Example
+url = f"http://127.0.0.1:{port}"
+json_data = {
+    "text": [
+        "List 3 countries and their capitals.",
+        "List 3 countries and their capitals.",
+    ],
+    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
+    # The first input uses lora0, and the second input uses the base model
+    "lora_path": ["lora0", None],
+}
+response = requests.post(
+    url + "/generate",
+    json=json_data,
+)
+print(f"Output 0: {response.json()[0]['text']}")
+print(f"Output 1: {response.json()[1]['text']}")
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+### Serving Multiple Adaptors
+
+
+
+```python Example
+server_process, port = launch_server_cmd(
+    """
+python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --enable-lora \
+    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
+    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \
+    --max-loras-per-batch 2 \
+    --log-level warning \
+"""
+)
+
+wait_for_server(f"http://localhost:{port}")
+```
+
+
+```python Example
+url = f"http://127.0.0.1:{port}"
+json_data = {
+    "text": [
+        "List 3 countries and their capitals.",
+        "List 3 countries and their capitals.",
+    ],
+    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
+    # The first input uses lora0, and the second input uses lora1
+    "lora_path": ["lora0", "lora1"],
+}
+response = requests.post(
+    url + "/generate",
+    json=json_data,
+)
+print(f"Output 0: {response.json()[0]['text']}")
+print(f"Output 1: {response.json()[1]['text']}")
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+### Dynamic LoRA loading
+
+
+Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API.
+
+When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly "smaller".
+
+
+
+```python Example
+lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj
+lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"  # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+lora0_new = "philschmid/code-llama-3-1-8b-text-to-sql-lora"  # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
+
+
+# The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified.
+# We are adding it here just to demonstrate usage.
+server_process, port = launch_server_cmd(
+    """
+    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --enable-lora \
+    --cuda-graph-max-bs-decode 2 \
+    --max-loras-per-batch 2 \
+    --max-lora-rank 256
+    --lora-target-modules all
+    --log-level warning
+    """
+)
+
+url = f"http://127.0.0.1:{port}"
+wait_for_server(url)
+```
+
+Load adapter lora0
+
+
+
+```python Example
+response = requests.post(
+    url + "/load_lora_adapter",
+    json={
+        "lora_name": "lora0",
+        "lora_path": lora0,
+    },
+)
+
+if response.status_code == 200:
+    print("LoRA adapter loaded successfully.", response.json())
+else:
+    print("Failed to load LoRA adapter.", response.json())
+```
+
+Load adapter lora1:
+
+
+
+```python Example
+response = requests.post(
+    url + "/load_lora_adapter",
+    json={
+        "lora_name": "lora1",
+        "lora_path": lora1,
+    },
+)
+
+if response.status_code == 200:
+    print("LoRA adapter loaded successfully.", response.json())
+else:
+    print("Failed to load LoRA adapter.", response.json())
+```
+
+Check inference output:
+
+
+
+```python Example
+url = f"http://127.0.0.1:{port}"
+json_data = {
+    "text": [
+        "List 3 countries and their capitals.",
+        "List 3 countries and their capitals.",
+    ],
+    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
+    # The first input uses lora0, and the second input uses lora1
+    "lora_path": ["lora0", "lora1"],
+}
+response = requests.post(
+    url + "/generate",
+    json=json_data,
+)
+print(f"Output from lora0: \n{response.json()[0]['text']}\n")
+print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n")
+```
+
+Unload lora0 and replace it with a different adapter:
+
+
+
+```python Example
+response = requests.post(
+    url + "/unload_lora_adapter",
+    json={
+        "lora_name": "lora0",
+    },
+)
+
+response = requests.post(
+    url + "/load_lora_adapter",
+    json={
+        "lora_name": "lora0",
+        "lora_path": lora0_new,
+    },
+)
+
+if response.status_code == 200:
+    print("LoRA adapter loaded successfully.", response.json())
+else:
+    print("Failed to load LoRA adapter.", response.json())
+```
+
+Check output again:
+
+
+
+```python Example
+url = f"http://127.0.0.1:{port}"
+json_data = {
+    "text": [
+        "List 3 countries and their capitals.",
+        "List 3 countries and their capitals.",
+    ],
+    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
+    # The first input uses lora0, and the second input uses lora1
+    "lora_path": ["lora0", "lora1"],
+}
+response = requests.post(
+    url + "/generate",
+    json=json_data,
+)
+print(f"Output from lora0: \n{response.json()[0]['text']}\n")
+print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n")
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+### OpenAI-compatible API usage
+
+You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai_api_completions](../basic_usage/openai_api_completions).
+
+
+
+### LoRA GPU Pinning
+
+
+Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded.
+
+This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations.
+
+In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported.
+
+
+
+```python Example
+server_process, port = launch_server_cmd(
+    """
+    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --enable-lora \
+    --cuda-graph-max-bs-decode 8 \
+    --max-loras-per-batch 3 \
+    --max-lora-rank 256 \
+    --lora-target-modules all \
+    --lora-paths \
+        {"lora_name":"lora0","lora_path":"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json","pinned":true} \
+        {"lora_name":"lora1","lora_path":"algoprog/fact-generation-llama-3.1-8b-instruct-lora"} \
+        lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora
+    --log-level warning
+    """
+)
+
+
+url = f"http://127.0.0.1:{port}"
+wait_for_server(url)
+```
+
+You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter:
+
+
+
+```python Example
+response = requests.post(
+    url + "/unload_lora_adapter",
+    json={
+        "lora_name": "lora1",
+    },
+)
+
+response = requests.post(
+    url + "/load_lora_adapter",
+    json={
+        "lora_name": "lora1",
+        "lora_path": "algoprog/fact-generation-llama-3.1-8b-instruct-lora",
+        "pinned": True,  # Pin the adapter to GPU
+    },
+)
+```
+
+Verify that the results are expected:
+
+
+
+```python Example
+url = f"http://127.0.0.1:{port}"
+json_data = {
+    "text": [
+        "List 3 countries and their capitals.",
+        "List 3 countries and their capitals.",
+        "List 3 countries and their capitals.",
+    ],
+    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
+    # The first input uses lora0, and the second input uses lora1
+    "lora_path": ["lora0", "lora1", "lora2"],
+}
+response = requests.post(
+    url + "/generate",
+    json=json_data,
+)
+print(f"Output from lora0 (pinned): \n{response.json()[0]['text']}\n")
+print(f"Output from lora1 (pinned): \n{response.json()[1]['text']}\n")
+print(f"Output from lora2 (not pinned): \n{response.json()[2]['text']}\n")
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+## Choosing LoRA Backend
+
+SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument:
+
+- `triton`: Basic Triton-based backend.
+- `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios.
+
+The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend.
+
+
+
+```python Example
+server_process, port = launch_server_cmd(
+    """
+    python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --enable-lora \
+    --lora-backend csgmv \
+    --max-loras-per-batch 16 \
+    --lora-paths lora1=path/to/lora1 lora2=path/to/lora2
+    """
+)
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+## LoRA Overlap Loading
+
+
+By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a ~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks).
+
+
+
+```python Example
+lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json"
+lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"
+lora2 = "philschmid/code-llama-3-1-8b-text-to-sql-lora"
+
+
+server_process, port = launch_server_cmd(
+    """
+    python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --enable-lora \
+    --enable-lora-overlap-loading \
+    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \
+    lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
+    lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \
+    --max-lora-rank 256 \
+    --max-loras-per-batch 2 \
+    --max-loaded-loras 4
+    """
+)
+
+url = f"http://127.0.0.1:{port}"
+wait_for_server(url)
+```
+
+
+```python Example
+json_data = {
+    "text": [
+        "Write a very long fairy-tale.",
+        "List 3 countries and their capitals.",
+        "List 3 countries and their capitals.",
+    ],
+    "sampling_params": [
+        {"max_new_tokens": 1024, "temperature": 0},
+        {"max_new_tokens": 64, "temperature": 0},
+        {"max_new_tokens": 64, "temperature": 0},
+    ],
+    "lora_path": ["lora0", "lora1", "lora2"],
+}
+
+# lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue.
+# lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will
+# occur asynchronously and thus decoding for lora0's request won't be blocked.
+response = requests.post(
+    url + "/generate",
+    json=json_data,
+)
+
+for i in range(3):
+    print(f"Output from lora{i}: \n{response.json()[i]['text']}\n")
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+#### Limitations of LoRA Overlap Loading
+
+
+However, LoRA overlap loading is not free and comes with two important caveats:
+
+1. **Pinned CPU memory requirement**:
+   Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled.
+
+2. **Reduced multi-adapter prefill batching**:
+   With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads).
+
+
+
+#### Example When Overlap Loading Results in Higher Latency
+
+
+For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms.
+
+1. **Baseline**:
+  The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms`
+
+2. **With LoRA overlap loading enabled**:
+  The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms`
+
+In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT.
+
+
+## Future Works
+
+The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development.

docs_new/docs/advanced_features/model_loading.mdx

@@ -0,0 +1,218 @@
+---
+title: "Model Loading"
+description: "Control how SGLang loads model weights: load formats, model loader extra config, multithreaded loading, prefetching, and remote/streaming loaders."
+---
+
+`--model-path` selects the checkpoint to serve; `--load-format` and the weight-loading flags below control how those weights are read into memory. To stream weights from cloud object storage (S3/GCS/Azure), see [Loading Models from Object Storage](./object_storage).
+
+## How loading works
+
+SGLang picks a loader from `--load-format`, falling back to auto-detection from the checkpoint or model path. The default `auto` loader reads `safetensors` and falls back to PyTorch `.bin`.
+
+```bash
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3.6-35B-A3B \
+  --load-format auto
+```
+
+Some formats are auto-detected and override `auto`:
+
+- A Mistral native checkpoint is detected and loaded with `mistral`.
+- A `.gguf` model path is detected and loaded with `gguf`.
+- An object storage URI (`s3://`, `gs://`, `az://`) is loaded with `runai_streamer`.
+- A remote URI is loaded with `remote`.
+
+## Load formats
+
+Set with `--load-format`:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "20%"}} />
+    <col style={{width: "80%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Format</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>auto</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default. Load <code>safetensors</code> if available, otherwise fall back to the PyTorch <code>.bin</code> format.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>safetensors</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load weights in the safetensors format.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>pt</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load weights in the PyTorch <code>.bin</code> format.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>npcache</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load PyTorch-format weights and store a numpy cache to speed up subsequent loads. Only supports <code>.bin</code> checkpoints.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>dummy</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Initialize weights with random values, for profiling.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>sharded_state</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Each tensor-parallel worker reads only its own pre-sharded shard rather than the full checkpoint, giving a fast load path for large TP models. See <code>examples/runtime/engine/save_sharded_state.py</code> for creating a sharded checkpoint.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>fastsafetensors</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load safetensors using the <code>fastsafetensors</code> iterator.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>layered</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load weights layer by layer, so a layer can be quantized before the next is loaded, lowering the peak memory envelope.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>gguf</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load weights in the GGUF format. Auto-detected from a <code>.gguf</code> model path.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>bitsandbytes</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load weights using bitsandbytes quantization.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>mistral</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load a Mistral native-format checkpoint. Auto-detected for such checkpoints.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>flash_rl</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load a BF16/FP16 checkpoint with native SGLang FP8 quantization for RL training. Requires <code>--rl-quant-profile</code>.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>runai_streamer</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Stream weights from SSDs, shared filesystems, or object storage. See <a href="./object_storage">Loading Models from Object Storage</a>.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>remote</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Load tensors from a remote KV/filesystem connector. Auto-detected for remote URIs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>remote_instance</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Pull weights over the network from another running SGLang instance (the "seed") rather than from disk. Configured with the <code>--remote-instance-weight-loader-*</code> flags.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Model loader extra config
+
+`--model-loader-extra-config` takes a JSON string passed to the loader selected by `--load-format`.
+
+```bash
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3.6-35B-A3B \
+  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 16}'
+```
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "20%"}} />
+    <col style={{width: "24%"}} />
+    <col style={{width: "40%"}} />
+    <col style={{width: "16%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Load format</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Key</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>auto</code> / <code>safetensors</code> / <code>pt</code> / <code>npcache</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>enable_multithread_load</code> (bool)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Read weight shards with a thread pool instead of sequentially. Disabled by default when <code>--weight-loader-prefetch-checkpoints</code> is set (to avoid I/O oversubscription with the prefetch threads); set this to <code>true</code> to opt back in.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>true</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>auto</code> / <code>safetensors</code> / <code>pt</code> / <code>npcache</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>num_threads</code> (int)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of worker threads when multithreaded loading is enabled.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>sharded_state</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>pattern</code> (str)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Filename pattern for per-rank shards.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>model-rank-&#123;rank&#125;-part-&#123;part&#125;.safetensors</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>bitsandbytes</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>qlora_adapter_name_or_path</code> (str)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>QLoRA adapter to apply on top of the bitsandbytes-quantized base weights.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>runai_streamer</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>distributed</code>, <code>concurrency</code>, <code>memory_limit</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Streaming controls. See <a href="./object_storage">Loading Models from Object Storage</a>.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>See linked page</td>
+    </tr>
+  </tbody>
+</table>
+
+## Weight-loading performance flags
+
+Top-level arguments that tune how safetensors weights are read, independent of `--load-format`.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "52%"}} />
+    <col style={{width: "14%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Flag</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--download-dir</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Directory used to download and cache Hugging Face model files.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>HF default</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}><code>--weight-loader-disable-mmap</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Disable mmap while loading safetensors. Can help on filesystems where mmap is slow.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>off</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--weight-loader-prefetch-checkpoints</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Prefetch checkpoint files into the OS page cache before loading. Each rank prefetches a fraction of the shards, cutting total network I/O on shared filesystems (NFS/Lustre) from N×checkpoint to 1×checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set <code>enable_multithread_load=true</code> in <code>--model-loader-extra-config</code> to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op).</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>off</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}><code>--weight-loader-prefetch-num-threads</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Threads per rank for checkpoint prefetching.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--weight-loader-drop-cache-after-load</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Call <code>posix_fadvise(DONTNEED)</code> on each safetensors shard after loading it, freeing page cache.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>off</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}><code>--custom-weight-loader</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Import path(s) of a custom weight-loading function, e.g. <code>my_package.weight_load_func</code>.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
+    </tr>
+  </tbody>
+</table>
+
+## See also
+
+- [Loading Models from Object Storage](./object_storage)
+- [Quantization](./quantization)
+- [Server Arguments](./server_arguments)

docs_new/docs/advanced_features/object_storage.mdx

@@ -0,0 +1,142 @@
+---
+title: "Loading Models from Object Storage"
+metatags:
+    description: "Load SGLang models directly from S3, Google Cloud Storage, Azure Blob, and S3-compatible object storage with runai_streamer."
+---
+
+SGLang can load models directly from object storage without a full local download. It uses the `runai_streamer` load format to stream model weights from cloud storage, reducing startup time and local storage requirements.
+
+## Overview
+
+When loading models from object storage, SGLang uses a two-phase approach:
+
+1. **Metadata Download** (once, before process launch): Configuration files and tokenizer files are downloaded to a local cache
+2. **Weight Streaming** (lazy, during model loading): Model weights are streamed directly from object storage as needed
+
+## Supported Storage Backends
+
+1. **Amazon S3**: `s3://bucket-name/path/to/model/`
+2. **Google Cloud Storage**: `gs://bucket-name/path/to/model/`
+3. **Azure Blob**: `az://some-azure-container/path/`
+4. **S3 compatible**: `s3://bucket-name/path/to/model/`
+
+## Quick Start
+
+### Basic Usage
+
+Simply provide an object storage URI as the model path:
+
+```bash
+# S3
+python -m sglang.launch_server \
+  --model-path s3://my-bucket/models/llama-3-8b/ \
+  --load-format runai_streamer
+
+# Google Cloud Storage
+python -m sglang.launch_server \
+  --model-path gs://my-bucket/models/llama-3-8b/ \
+  --load-format runai_streamer
+```
+
+**Note**: The `--load-format runai_streamer` is automatically detected when using object storage URIs, so you can omit it:
+
+```bash
+python -m sglang.launch_server \
+  --model-path s3://my-bucket/models/llama-3-8b/
+```
+
+### With Tensor Parallelism
+
+```bash
+python -m sglang.launch_server \
+  --model-path gs://my-bucket/models/llama-70b/ \
+  --tp 4 \
+  --model-loader-extra-config '{"distributed": true}'
+```
+
+## Configuration
+
+### Load Format
+
+The `runai_streamer` load format is designed for object storage, SSDs, and shared filesystems.
+
+```bash
+python -m sglang.launch_server \
+  --model-path s3://bucket/model/ \
+  --load-format runai_streamer
+```
+
+### Extended Configuration Parameters
+
+Use `--model-loader-extra-config` to pass additional configuration as a JSON string:
+
+```bash
+python -m sglang.launch_server \
+  --model-path s3://bucket/model/ \
+  --model-loader-extra-config '{
+    "distributed": true,
+    "concurrency": 8,
+    "memory_limit": 2147483648
+  }'
+```
+
+#### Available Parameters
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "22%"}} />
+    <col style={{width: "16%"}} />
+    <col style={{width: "44%"}} />
+    <col style={{width: "18%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>distributed</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable distributed streaming for multi-GPU setups. Automatically set to <code>true</code> for object storage paths on CUDA-like devices.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Auto-detected</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>concurrency</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>int</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Number of concurrent download streams. Higher values can improve throughput for large models.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>memory_limit</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>int</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Memory limit (in bytes) for the streaming buffer.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>System-dependent</td>
+    </tr>
+  </tbody>
+</table>
+
+## Performance Considerations
+
+### Distributed Streaming
+
+For multi-GPU setups, enable distributed streaming to parallelize weight loading across processes:
+
+```bash
+python -m sglang.launch_server \
+  --model-path s3://bucket/model/ \
+  --tp 8 \
+  --model-loader-extra-config '{"distributed": true}'
+```
+
+## Limitations
+
+- **Supported formats**: Only the `.safetensors` weight format is supported.
+- **Supported devices**: Distributed streaming is supported on CUDA-like devices; otherwise it falls back to non-distributed streaming.
+
+## See Also
+
+- [Runai model streamer documentation](https://github.com/run-ai/runai-model-streamer)

docs_new/docs/advanced_features/observability.mdx

@@ -0,0 +1,44 @@
+---
+title: "Observability"
+metatags:
+    description: "SGLang observability: Prometheus metrics, request logging, request dump and replay, crash dump debugging."
+---
+## Production Metrics
+SGLang exposes the following metrics via Prometheus. You can enable them by adding `--enable-metrics` when launching the server.
+You can query them by:
+```bash Command
+curl http://localhost:30000/metrics
+```
+
+See [Production Metrics](../references/production_metrics) and [Production Request Tracing](../references/production_request_trace) for more details.
+
+## Logging
+
+By default, SGLang does not log any request contents. You can log them by using `--log-requests`.
+You can control the verbosity by using `--log-request-level`.
+See [Logging](./server_arguments#logging) for more details.
+
+You can change verbosity at runtime:
+
+```bash Command
+python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --log-level=debug
+```
+
+## Request Dump and Replay
+
+You can dump all requests and replay them later for benchmarking or other purposes.
+
+To start dumping, use the following command to send a request to a server:
+```bash Command
+python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --dump-requests-folder /tmp/sglang_request_dump --dump-requests-threshold 100
+```
+The server will dump the requests into a pickle file for every 100 requests.
+
+To replay the request dump, use `scripts/playground/replay_request_dump.py`.
+
+## Crash Dump and Replay
+Sometimes the server might crash, and you may want to debug the cause of the crash.
+SGLang supports crash dumping, which will dump all requests from the 5 minutes before the crash, allowing you to replay the requests and debug the reason later.
+
+To enable crash dumping, use `--crash-dump-folder /tmp/crash_dump`.
+To replay the crash dump, use `scripts/playground/replay_request_dump.py`.

docs_new/docs/advanced_features/overview.mdx

@@ -0,0 +1,18 @@
+---
+title: Advanced Features
+description: Advanced configuration, optimization, and deployment features for SGLang.
+---
+
+- [Server Arguments](./server_arguments)
+- [Hyperparameter Tuning](./hyperparameter_tuning)
+- [Attention Backend](./attention_backend)
+- [Speculative Decoding](./speculative_decoding)
+- [Structured Outputs](./structured_outputs)
+- [Quantization](./quantization)
+- [Expert Parallelism](./expert_parallelism)
+- [LoRA](./lora)
+- [PD Disaggregation](./pd_disaggregation)
+- [Pipeline Parallelism](./pipeline_parallelism)
+- [HiCache](./hicache_best_practices)
+- [Observability](./observability)
+- [And more…](./server_arguments)

docs_new/docs/advanced_features/pd_disaggregation.mdx

@@ -0,0 +1,710 @@
+---
+title: "PD Disaggregation"
+metatags:
+    description: "SGLang PD disaggregation: separate prefill and decode phases, Mooncake and NIXL transfer engines, multi-node DeepSeek deployment."
+---
+## Why and What is PD Disaggregation?
+
+Large Language Model (LLM) inference comprises two distinct phases: **Prefill** and **Decode**. The Prefill phase is computation-intensive, processing the entire input sequence, while the Decode phase is memory-intensive, managing the Key-Value (KV) cache for token generation. Traditionally, these phases are handled within a unified engine, where combined scheduling of prefill and decode batches introduces inefficiencies. To address these challenges, we introduce **Prefill and Decoding (PD) Disaggregation** in SGLang.
+
+### Issues with Unified Scheduling
+
+The conventional unified engine, which processes prefill and decode batches together, results in two significant problems:
+
+1. **Prefill Interruption**: Incoming prefill batches frequently interrupt ongoing decode batches, causing substantial delays in token generation.
+2. **DP Attention Imbalance**: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency.
+
+PD Disaggregation resolves these by separating the two stages, enabling tailored optimizations for each.
+
+For the design details, please refer to [link](https://docs.google.com/document/d/1rQXJwKd5b9b1aOzLh98mnyMhBMhlxXA5ATZTHoQrwvc/edit?tab=t.0).
+
+Currently, we support Mooncake and NIXL as the transfer engine.
+
+## Profiling in PD Disaggregation Mode
+
+When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](../developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options.
+
+## Router Integration
+
+For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](./sgl_model_gateway#prefill-decode-disaggregation).
+
+
+## Mooncake
+### Requirements
+
+```bash
+uv pip install mooncake-transfer-engine
+```
+
+### IB Device Configuration
+
+`--disaggregation-ib-device` supports the following formats when using the Mooncake backend:
+
+1. Shared device list for all GPUs: `mlx5_0` or `mlx5_0,mlx5_1`
+2. Per-GPU JSON mapping: `{"0": "mlx5_0,mlx5_1", "1": "mlx5_2,mlx5_3"}`
+3. Path to a JSON file containing the same per-GPU mapping
+
+Each JSON value uses the same comma-separated device list format as the shared configuration.
+
+### Usage
+
+### Llama Single Node
+
+```bash
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --disaggregation-mode prefill \
+  --port 30000 \
+  --disaggregation-ib-device mlx5_roce0
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --disaggregation-mode decode \
+  --port 30001 \
+  --base-gpu-id 1 \
+  --disaggregation-ib-device mlx5_roce0
+python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
+```
+
+### DeepSeek Multi-Node
+
+```bash
+# prefill 0
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-ib-device ${device_name} \
+  --disaggregation-mode prefill \
+  --host ${local_ip} \
+  --port 30000 \
+  --trust-remote-code \
+  --dist-init-addr ${prefill_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 0 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8
+# prefill 1
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-ib-device ${device_name} \
+  --disaggregation-mode prefill \
+  --host ${local_ip} \
+  --port 30000 \
+  --trust-remote-code \
+  --dist-init-addr ${prefill_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 1 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8
+# decode 0
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-ib-device ${device_name} \
+  --disaggregation-mode decode \
+  --host ${local_ip} \
+  --port 30001 \
+  --trust-remote-code \
+  --dist-init-addr ${decode_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 0 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8 \
+  --max-running-requests 128
+# decode 1
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-ib-device ${device_name} \
+  --disaggregation-mode decode \
+  --host ${local_ip} \
+  --port 30001 \
+  --trust-remote-code \
+  --dist-init-addr ${decode_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 1 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8 \
+  --max-running-requests 128
+```
+### Advanced Configuration
+
+PD Disaggregation with Mooncake supports the following environment variables for fine-grained control over system behavior.
+
+#### NVLink Transport Configuration
+To enable NVLink transport for KV cache transfers with the mooncake backend (recommended for NVL72 deployments), set the following environment variables. Note that auxiliary data transfer will still use TCP as a temporary workaround.
+
+```bash Command
+export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK
+export MC_FORCE_MNNVL=True
+```
+
+To utilize Intra-Node NVLink for KV cache transfers with the Mooncake backend (recommended for A100, H20, H100, etc.), set the following environment variables. Please note that auxiliary data still needs to be transferred via TCP.
+
+```bash
+export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK
+export MC_INTRANODE_NVLINK=true
+```
+The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom memory pool. Supported values are `NVLINK` (or `True`), `BAREX`, and `INTRA_NODE_NVLINK`.
+
+#### Prefill Server Configuration
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.3%"}} />
+    <col style={{width: "33.3%"}} />
+    <col style={{width: "33.3%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variable</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Controls the total number of worker threads for KVCache transfer operations per TP rank</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>A dynamic value calculated by <code>int(0.75 * os.cpu_count()) // 8</code>, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_QUEUE_SIZE`**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to <code>1</code>, then we transfer requests one by one according to fcfs strategy</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`4`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT`**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout (seconds) for receiving destination KV indices during request initialization</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`300`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGGREGATION_BOOTSTRAP_ENTRY_CLEANUP_INTERVAL</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Interval (seconds) between cleanups of bootstrap entries</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>120</code></td>
+    </tr>
+</tbody>
+</table>
+
+If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600` (10 minutes) to relax the timeout condition.
+Please be aware that this setting will cause prefill instances to take a longer time to clean up the affected memory resources when a running decode node loses connection.
+
+#### Decode Server Configuration
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.3%"}} />
+    <col style={{width: "33.3%"}} />
+    <col style={{width: "33.3%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variable</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL`**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Interval (seconds) between health checks to prefill bootstrap servers</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`5.0`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE`**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Consecutive heartbeat failures before marking prefill server offline</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_WAITING_TIMEOUT`**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout (seconds) for receiving KV Cache after request initialization</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`300`</td>
+    </tr>
+  </tbody>
+</table>
+
+If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600` (10 minutes) to relax the timeout condition.
+
+
+## Heterogeneous TP with GPU Staging Buffer
+
+When prefill and decode use different tensor parallelism (TP) sizes (e.g., prefill TP=4, decode DP attention with TP=1), the KV cache memory layout differs between the two sides. The **GPU staging buffer** solves this by gathering KV head slices into a contiguous buffer on the prefill side, performing bulk RDMA transfer, then scattering into the correct KV cache pages on the decode side. This provides **2–5x throughput improvement** over the default per-token slice approach at high concurrency and matches homogeneous TP baselines within ~5%.
+
+Enable the staging buffer when prefill and decode use **different TP sizes** with the **Mooncake** transfer backend. When both sides use the same TP size, staging is automatically bypassed even if enabled.
+
+> **Note:** The staging buffer is designed for non-MLA models (e.g. GQA, MHA). MLA models (e.g. DeepSeek-V2/V3) should not enable this flag.
+
+### Environment Variables
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "30%"}} />
+    <col style={{width: "50%"}} />
+    <col style={{width: "20%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variable</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGG_STAGING_BUFFER</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable GPU staging buffer for heterogeneous TP KV transfer</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGG_STAGING_BUFFER_SIZE_MB</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Prefill-side per-worker staging buffer size in MB</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>64</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGG_STAGING_POOL_SIZE_MB</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Decode-side ring buffer pool total size in MB</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>4096</code></td>
+    </tr>
+  </tbody>
+</table>
+
+### Usage Example
+
+```bash Command
+# Set staging buffer environment variables on BOTH prefill and decode
+export SGLANG_DISAGG_STAGING_BUFFER=1
+export SGLANG_DISAGG_STAGING_BUFFER_SIZE_MB=64
+export SGLANG_DISAGG_STAGING_POOL_SIZE_MB=4096
+
+# Prefill with TP=4
+python -m sglang.launch_server \
+  --model-path $MODEL_PATH \
+  --disaggregation-mode prefill \
+  --port 30000 \
+  --tp 4 \
+  --trust-remote-code \
+  --disaggregation-ib-device mlx5_1,mlx5_2
+
+# Decode with TP=1 (or DP attention with effective attention TP=1)
+python -m sglang.launch_server \
+  --model-path $MODEL_PATH \
+  --disaggregation-mode decode \
+  --port 30001 \
+  --tp 4 \
+  --dp 4 \
+  --enable-dp-attention \
+  --trust-remote-code \
+  --disaggregation-ib-device mlx5_3,mlx5_4
+
+# Router
+python -m sglang_router.launch_router \
+  --pd-disaggregation \
+  --prefill http://127.0.0.1:30000 \
+  --decode http://127.0.0.1:30001 \
+  --host 0.0.0.0 --port 8000
+```
+
+## NIXL
+### Requirements
+
+Install via pip.
+
+```bash
+pip install nixl
+```
+
+Or build from source - may be required if you already have UCX installed.
+
+```bash
+git clone https://github.com/ai-dynamo/nixl.git
+cd nixl
+pip install . --config-settings=setup-args="-Ducx_path=/path/to/ucx"
+```
+
+
+### Usage
+
+### Llama Single Node
+
+```bash
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --disaggregation-mode prefill \
+  --port 30000 \
+  --disaggregation-transfer-backend nixl
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --disaggregation-mode decode \
+  --port 30001 \
+  --base-gpu-id 1 \
+  --disaggregation-transfer-backend nixl
+python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
+```
+
+### DeepSeek Multi-Node
+
+```bash
+# prefill 0
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-transfer-backend nixl \
+  --disaggregation-mode prefill \
+  --host ${local_ip} \
+  --port 30000 \
+  --trust-remote-code \
+  --dist-init-addr ${prefill_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 0 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8
+# prefill 1
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-transfer-backend nixl \
+  --disaggregation-mode prefill \
+  --host ${local_ip} \
+  --port 30000 \
+  --trust-remote-code \
+  --dist-init-addr ${prefill_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 1 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8
+# decode 0
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-transfer-backend nixl \
+  --disaggregation-mode decode \
+  --host ${local_ip} \
+  --port 30001 \
+  --trust-remote-code \
+  --dist-init-addr ${decode_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 0 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8 \
+  --max-running-requests 128
+# decode 1
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3-0324 \
+  --disaggregation-transfer-backend nixl \
+  --disaggregation-mode decode \
+  --host ${local_ip} \
+  --port 30001 \
+  --trust-remote-code \
+  --dist-init-addr ${decode_master_ip}:5000 \
+  --nnodes 2 \
+  --node-rank 1 \
+  --tp-size 16 \
+  --dp-size 8 \
+  --enable-dp-attention \
+  --moe-a2a-backend deepep \
+  --mem-fraction-static 0.8 \
+  --max-running-requests 128
+```
+
+### Advanced Configuration
+
+#### NIXL Backend Selection
+
+By default, NIXL uses the **UCX** backend for KV cache transfers. You can select a different NIXL plugin backend depending on your infrastructure using the environment variable `SGLANG_DISAGGREGATION_NIXL_BACKEND`.
+
+Example: `export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC`
+
+**Available backends:** UCX (default), LIBFABRIC, or any installed NIXL plugin.
+
+Example usage:
+```bash
+export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --disaggregation-mode prefill \
+  --disaggregation-transfer-backend nixl \
+  --port 30000
+```
+
+## ASCEND
+
+### Usage
+
+Use ascend backend with [memfabric_hybrid](https://gitcode.com/Ascend/memfabric_hybrid) and ASCEND_MF_STORE_URL being set
+
+```bash Command
+pip install memfabric-hybrid==1.0.0
+export ASCEND_MF_STORE_URL="tcp://xxx.xx.xxx.xxx:xxxx"
+```
+Use mooncake backend, more details can be found in mooncake section.
+```bash
+export ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE=true
+```
+ASCEND_NPU_PHY_ID need to be set in container env
+```bash
+export ASCEND_NPU_PHY_ID=xxx
+```
+
+
+### MIMO Single Node
+
+#### prefill
+
+```bash
+# high performance cpu
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+# bind cpu
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+# cann
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export STREAMS_PER_DEVICE=32
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
+export HCCL_BUFFSIZE=1600
+export HCCL_OP_EXPANSION_MODE=AIV
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export SGLANG_NPU_PROFILING=0
+export SGLANG_NPU_PROFILING_STAGE="prefill"
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
+export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669"
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
+export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export SGLANG_DEEPEP_BF16_DISPATCH=0
+export ASCEND_USE_FIA=1
+
+python3 -m sglang.launch_server \
+        --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \
+        --attention-backend ascend \
+        --device npu \
+        --tp-size 8 --nnodes 1 --node-rank 0 \
+        --chunked-prefill-size -1 \
+        --trust-remote-code --port 10000 \
+        --host 127.0.0.1 --max-running-requests 16 \
+        --mem-fraction-static 0.8 \
+        --disaggregation-mode prefill --disaggregation-transfer-backend ascend \
+        --disaggregation-bootstrap-port 8996 \
+        --base-gpu-id 0 \
+        --disable-radix-cache \
+        --disable-cuda-graph \
+        --moe-a2a-backend deepep --deepep-mode normal \
+        # 2>&1 | tee $SGLANG_LOG_PATH
+```
+
+#### decode
+
+``` bash
+# high performance cpu
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+# bind cpu
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+# export ASCEND_LAUNCH_BLOCKING=1
+
+# cann
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export STREAMS_PER_DEVICE=32
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256
+export HCCL_BUFFSIZE=1600
+export HCCL_OP_EXPANSION_MODE=AIV
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export SGLANG_NPU_PROFILING=0
+export SGLANG_NPU_PROFILING_STAGE="prefill"
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
+export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669"
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
+export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export SGLANG_DEEPEP_BF16_DISPATCH=0
+export ASCEND_USE_FIA=1
+
+export SGLANG_NPU_FUSED_MOE_MODE=2
+
+python3 -m sglang.launch_server \
+        --model-path  /path/to/MiMo-V2-Flash-w8a8-all-0512 \
+        --attention-backend ascend \
+        --device npu \
+        --tp-size 8 --nnodes 1 --node-rank 0 \
+        --trust-remote-code --port 10001 \
+        --host 127.0.0.1 --max-running-requests 16 \
+        --mem-fraction-static 0.8 \
+        --disaggregation-mode decode --disaggregation-transfer-backend ascend \
+        --disaggregation-bootstrap-port 8996 \
+        --base-gpu-id 8 \
+        --disable-radix-cache \
+        --cuda-graph-bs 1 2 4 8 10 12 14 16 \
+        --quantization modelslim \
+        --speculative-draft-model-quantization unquant \
+        --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+        --enable-multi-layer-eagle \
+        --moe-a2a-backend deepep --deepep-mode low_latency \
+        # 2>&1 | tee $SGLANG_LOG_PATH
+```
+
+
+#### router
+
+```bash
+python -m sglang_router.launch_router \
+    --pd-disaggregation \
+    --prefill http://127.0.0.1:10000 \
+    --decode  http://127.0.0.1:10001 \
+    --host 127.0.0.1 \
+    --port 9903 \
+    --health-check-interval-secs 3600 \
+    --mini-lb \
+```
+
+### DeepSeek Multi-Node
+
+#### Environment
+```bash
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH}
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+# IP set to p first node ip
+export ASCEND_MF_STORE_URL="tcp://XXXXXX:24670"
+
+# p node IP
+P_IP=('XXXXX')
+# D node IP
+D_IP=('XXXXX')
+
+
+# enable mlapo
+export SGLANG_NPU_USE_MLAPO=1
+export SGLANG_USE_FIA_NZ=1
+export ENABLE_MOE_NZ=1
+#export SGLANG_NPU_USE_MULTI_STREAM=1
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+```
+
+
+#### prefill
+
+```bash
+MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken
+for i in "${!P_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+    then
+        echo "${P_IP[$i]}"
+        export HCCL_BUFFSIZE=2600
+        export HCCL_SOCKET_IFNAME=lo
+        export GLOO_SOCKET_IFNAME=lo
+        python -m sglang.launch_server --model-path ${MODEL_PATH}  --disaggregation-mode prefill --host ${P_IP[$i]} \
+        --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
+        --tp-size 16 --mem-fraction-static 0.7 --attention-backend ascend --device npu --quantization modelslim \
+        --disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192  --disable-radix-cache \
+        --chunked-prefill-size -1 --max-prefill-tokens 10240 --moe-a2a-backend deepep --deepep-mode normal \
+        --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2  \
+        --dp-size 8 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16
+        NODE_RANK=$i
+        break
+    fi
+done
+```
+
+#### decode
+
+```bash
+MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken
+
+for i in "${!D_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+    then
+        echo "${D_IP[$i]}"
+        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+        export SGLANG_ENABLE_SPEC_V2=1
+        export HCCL_BUFFSIZE=900
+        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112
+        export TASK_QUEUE_ENABLE=1
+        export HCCL_SOCKET_IFNAME=data0.3001
+        export GLOO_SOCKET_IFNAME=data0.3001
+        python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
+        --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \
+        --mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \
+        --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \
+        --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4  \
+        --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
+       	--load-balance-method round_robin
+        NODE_RANK=$i
+        break
+    fi
+done
+```
+
+
+#### router
+
+```bash
+python -m  sglang_router.launch_router --prefill ${P_IP}:8000 \
+--decode ${D_IP}:8001 \
+--host ${D_IP} --port 6688 \
+--pd-disaggregation \
+--health-check-interval-secs 3600 \
+```

docs_new/docs/advanced_features/piecewise_cuda_graph.mdx

@@ -0,0 +1,299 @@
+---
+title: "Piecewise CUDA Graph"
+metatags:
+    description: "Use Piecewise CUDA Graph to reduce prefill and extend kernel launch overhead while supporting dynamic token shapes."
+---
+
+## Motivation
+
+Standard CUDA graphs capture the entire model forward pass as a single graph. This works well for decode (fixed batch size), but not for extend/prefill where the number of tokens varies across iterations.
+
+Piecewise CUDA Graph (PCG) solves this by splitting the model's computation graph into pieces (roughly one per layer) at "split points" (e.g., MoE dispatch ops). Each piece is captured as a separate CUDA graph for a set of pre-defined token lengths. At runtime, the input is padded to the nearest captured size, and each piece is replayed. This eliminates kernel launch overhead for prefill/extend while still supporting dynamic shapes.
+
+Recently we **enabled PCG by default**, which means that the old `--enable-piecewise-cuda-graph` flag is deprecated. Use `--disable-piecewise-cuda-graph` to turn it off.
+
+## Usage
+
+PCG is enabled by default for supported configurations. No extra flags needed:
+
+```bash
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct
+```
+
+### Disable PCG
+
+```bash
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --disable-piecewise-cuda-graph
+```
+
+### Custom capture sizes
+
+```bash
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --piecewise-cuda-graph-max-tokens 2048
+```
+
+### Server Args
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "32%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "48%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--disable-piecewise-cuda-graph</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Disable PCG for extend/prefill.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--enforce-piecewise-cuda-graph</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>False</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Force-enable PCG, skipping all auto-disable conditions. For testing only.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-max-tokens</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Maximum token count to capture. Defaults to <code>chunked_prefill_size</code> (non-MLA) or <code>2048</code> (MLA).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-tokens</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code> (auto)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Explicit list of token lengths to capture. Auto-generated if not set.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>--piecewise-cuda-graph-compiler</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>"eager"</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Compiler backend for the captured subgraphs. Choices: <code>eager</code>, <code>inductor</code>.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><del><code>--enable-piecewise-cuda-graph</code></del></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Deprecated.</strong> PCG is now enabled by default. Use <code>--enforce-piecewise-cuda-graph</code> to skip auto-disable conditions.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Bug Report
+
+PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--disable-piecewise-cuda-graph` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature.
+
+### For Users
+
+If you see an error message like the following during server startup, it is a PCG bug:
+
+```
+Piecewise CUDA Graph is enabled by default as an experimental feature.
+To work around this error, add --disable-piecewise-cuda-graph to your launch command.
+Please report this issue at https://github.com/sgl-project/sglang/issues/new/choose
+```
+
+To work around it, add `--disable-piecewise-cuda-graph` to your launch command. When filing a bug report, please include:
+1. The full error traceback
+2. Model name and quantization method
+3. Launch command with all arguments
+4. GPU type and driver version
+
+### For Developers
+
+Since PCG relies on `torch.compile` to trace the model's forward pass, newly developed CUDA kernels (both JIT kernels and sgl-kernels) are typically not compatible with `torch.compile` out of the box. The tracing will fail on untraceable operations such as JIT compilation, file I/O, or dynamic module loading inside the kernel.
+
+To make a kernel compatible with PCG, you need to register it as a custom op using `register_custom_op` from `sglang.srt.utils.custom_op`. This wraps the kernel as an opaque node in the compiled graph so that `torch.compile` will not trace inside it.
+
+**Example usage (JIT kernel):**
+
+```python
+from sglang.srt.utils.custom_op import register_custom_op
+
+# Inplace operator (no return value)
+@register_custom_op(mutates_args=["output_q", "output_s"])
+def per_token_group_quant_8bit(
+    input: torch.Tensor,
+    output_q: torch.Tensor,
+    output_s: torch.Tensor,
+) -> None:
+    # kernel implementation ...
+```
+
+**Example usage (operator with output):**
+
+```python
+# out_shape indicates which argument has the same shape as the output
+@register_custom_op(mutates_args=["x"], out_shape=0)
+def add(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor:
+    return x.add_(y)
+```
+
+For wrapping external library functions (e.g., FlashInfer kernels), use `register_custom_op_from_extern` instead. See `python/sglang/srt/utils/custom_op.py` for full API documentation.
+
+## How it works
+
+### Torch compile backend
+
+PCG uses `torch.compile` with a custom backend (`SGLangBackend`) to split and compile the model's forward pass. The flow is:
+
+```
+model.forward wrapper
+→ torch.compile(..., backend=SGLangBackend)
+→ FX graph
+→ split_graph() at registered split ops
+→ split_gm (top-level graph that chains the pieces)
+→ replace capturable submodules with CUDAPiecewiseBackend
+→ runtime dispatch: eager split ops + per-piece capture/replay
+```
+
+- **Install**: `install_torch_compiled()` replaces `model.forward` with a wrapper function. When `is_in_piecewise_cuda_graph()` returns True, the wrapper dispatches to the compiled callable; otherwise it falls back to the original forward. The first invocation through this path triggers Dynamo tracing and graph compilation — CUDA graph replay only happens after the capture phase completes.
+
+- **Split**: When `torch.compile` traces the model, `SGLangBackend` receives the FX graph and calls `split_graph()`. Ops listed in `CompilationConfig.split_ops` are treated as split points, so the graph is cut at each one. These split-op submodules are left to run eagerly at runtime, while the surrounding submodules are compiled and wrapped by `CUDAPiecewiseBackend`. The result is a top-level "stitching graph" (`split_gm`) with children such as `submod_0`, `submod_1`, … interleaving capturable subgraphs and eager split-op submodules.
+
+- **Replace**: `PiecewiseCompileInterpreter` iterates over each capturable submodule in `split_gm`, compiles it for general (dynamic) shapes, and replaces it in-place with a `CUDAPiecewiseBackend` instance. Split-op submodules (e.g., attention, all-reduce) are left as-is and run eagerly at runtime.
+
+- **Dispatch**: At runtime, calling `split_gm` executes the stitching graph, which calls each submodule in order. Split-op submodules run eagerly. Each `CUDAPiecewiseBackend` submodule goes through three phases:
+  - **Compile warmup** — runs the general-shape compiled path.
+  - **Capture** — for each capture size, runs one warmup pass then records a CUDA graph.
+  - **Steady-state replay** — replays the captured CUDA graph for each forward pass.
+
+### Piecewise cuda graph runner
+
+`PiecewiseCudaGraphRunner` orchestrates the full lifecycle through three phases:
+
+- **Compile** — Warms up JIT kernels with a dummy forward pass, then wraps the model with `torch.compile`, triggering Dynamo tracing to split the FX graph and create `CUDAPiecewiseBackend` instances for each subgraph piece.
+
+- **Capture** — Iterates over capture sizes in reverse order (largest first). For each size, runs the forward pass twice (one warmup, one CUDA graph capture).
+
+- **Replay** — At runtime, finds the smallest captured size >= actual token count via binary search, copies inputs into static buffers with zero-padding, replays the captured CUDA graphs, and slices outputs back to the actual token count.
+
+### Memory optimization
+
+The memory cost of PCG comes from two parts: **torch memory allocator** and **non-torch memory**.
+
+The torch memory allocator overhead is trivial thanks to several optimizations: a global shared memory pool is reused across all CUDA graph runners and capture sizes, capture is done in reverse order (large to small) so smaller graphs reuse memory allocated by larger ones, and output tensors of the last subgraph are stored as weak references to maximize memory reuse.
+
+The main memory overhead comes from non-torch memory — the CUDA graph objects themselves require GPU memory to store the recorded kernel launch parameters and internal state. This overhead scales with the number of captured sizes, which is why `piecewise_cuda_graph_max_tokens` is capped conservatively by default.
+
+### Shape configuration
+
+Piecewise CUDA graph pre-captures graphs for a set of token counts. At runtime, the actual token count is rounded up to the nearest captured size (via binary search), and the corresponding graph is replayed. If the token count exceeds the largest captured size, the runtime falls back to the normal (non-graph) forward path.
+
+The default capture schedule is auto-generated with increasing granularity:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Token range</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Step size</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>4 – 32</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>48 – 256</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>288 – 512</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>576 – 1024</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>1280 – 4096</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>256</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>4096+</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>512</td>
+    </tr>
+  </tbody>
+</table>
+
+For the auto-generated schedule, sizes are capped at `--piecewise-cuda-graph-max-tokens`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround.
+
+## Compatibility
+
+PCG is auto-disabled in the following scenarios. We are actively working on expanding compatibility — support for many of these will be coming soon.
+
+- Disabled model architectures (e.g., `DeepseekV32ForCausalLM`)
+- Speculative decoding
+- DP attention
+- Pipeline parallelism (`pp_size > 1`)
+- Non-CUDA hardware (AMD ROCm, Ascend NPU)
+- MoE A2A backend
+- LoRA
+- Multimodal / VLM models
+- DLLM (diffusion LLM)
+- Deterministic inference
+- PD disaggregation
+- Expert distribution recorder / EPLB
+
+Use `--enforce-piecewise-cuda-graph` to skip all auto-disable checks (for testing/debugging only).
+
+## Code Reference
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "48%"}} />
+    <col style={{width: "52%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>File</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/model_executor/runner_backend/tc_piecewise_cuda_graph_backend.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Backend implementation: compile, capture, replay</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/compile.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>install_torch_compiled</code> trampoline</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/backend.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>SGLangBackend</code>, graph splitting, piecewise compilation</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/cuda_piecewise_backend.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Per-subgraph CUDA graph capture/replay</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/piecewise_context_manager.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Global context flags and <code>ForwardContext</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/compilation/compilation_config.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Capture sizes, split ops, compiler config</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/utils/custom_op.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>register_custom_op</code> for torch.compile compatibility</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>python/sglang/srt/server_args.py</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Server arguments and auto-disable logic</td>
+    </tr>
+  </tbody>
+</table>

docs_new/docs/advanced_features/pipeline_parallelism.mdx

@@ -0,0 +1,119 @@
+---
+title: "Pipeline Parallelism for Long Context"
+metatags:
+    description: "SGLang pipeline parallelism: reduce TTFT for ultra-long sequences, dynamic chunking, async P2P communication, multi-node deployment."
+---
+## Why Pipeline Parallelism?
+
+As Large Language Models (LLMs) scale toward trillion-parameter architectures and "infinite" context windows, the underlying serving infrastructure must evolve toward more granular, cross-node parallelization strategies. While KV cache techniques effectively mitigate redundant computation, they cannot circumvent the prohibitive Time to First Token (TTFT) inherent in ultra-long sequences with extremely large initial Input Token Length (ITL). Although Tensor Parallelism (TP) remains the conventional approach for intra-node scaling, it frequently encounters communication bottlenecks during multi-node deployments. On the other hand, pipeline parallelism only requires cross-node communication at the boundaries of each pipeline stage, which can achieve better computation-communication overlap compared to a large TP. Therefore, it is also a promising parallelization strategy for improving throughput.
+
+Detailed analysis can be found in this [blog](https://lmsys.org/blog/2026-01-15-chunked-pipeline/).
+
+## Implementation Refactoring based on Async Communication
+With Dynamic Chunked Prefill, pipeline parallelism has the potential to reduce the TTFT of long-context inputs. For each request, its input tokens can be partitioned into multiple chunks, each no longer than the chunked prefill size. Different chunks of the same request can be processed simultaneously by different nodes, thus parallelizing the processing and reducing TTFT. SGLang has supported Pipeline Parallelism (#5724) for some time and made it compatible with the PD Disaggregation feature (#8846), but the implementation was not perfect and had significant room for performance improvements.
+
+To eliminate this performance hazard, SGLang implements a Micro-batching Event Loop with non-blocking asynchronous peer-to-peer (P2P) communication to overlap GPU computation with CPU metadata processing and PP communication. This ensures that while one micro-batch is being computed on the GPU, the next one is already being prepared and moved into position effectively, ensuring the pipeline remains as saturated as possible. This approach was first proposed in #7979 and has been redesigned and included in #11852.
+
+The key mechanisms of the implementation include:
+
+* **Decoupled Sync/Async Logic in the Event Loop:** The scheduler uses `async_send` in `_pp_send_pyobj_to_next_stage`. Instead of waiting for a transfer to complete, it returns a `P2PWork` handle. The actual synchronization (`P2PWork.work.wait()`) is deferred until `_pp_commit_comm_work` is called, allowing the CPU to perform other work—like scheduling the next batch or processing metadata—while data is in flight.
+* **Multi-Stream Execution:** In addition to the main `default_stream`, which serves as the synchronization stream, SGLang utilizes dedicated `forward_stream` and `copy_stream` to execute forward pass GPU computation and Data-to-Host (D2H) memory transfers separately for better overlapping. While `_pp_launch_batch` is executing the current micro-batch on the GPU for the current stage, the CPU processes the previous micro-batch's results using `_pp_process_batch_result`.
+
+## Guidance about Dynamic Chunking
+
+### Why Dynamic Chunking
+Chunked prefill with a fixed size can cause bubbles in the pipeline, especially when the pp size is large. The main reason behind this phenomenon is that the model has a non-uniform running time, even though each chunk size is identical (brought by the Transformer structure). The larger the prefix sequence length, the longer the running time of the chunk. And these bubbles will be propagated to the next stage, and will significantly degrade the scale efficiency of larger pp ranks.
+
+To address this issue, SGLang introduces a dynamic chunking mechanism to predict the optimal size for the next chunk such that it satisfies this condition:
+
+Runtime(L + Next Chunk Size) - Runtime(L) = Runtime(Initial Chunk Size)
+
+where ***L*** denotes the Prefix Sequence Length. By profiling a series of requests with different ITLs, we model the cumulative runtime as a quadratic function of sequence length. Using this model, we solve the optimal next chunk size for any given prefix length ***L***. Since the computation complexity of the Attention mechanism scales with ***L***, the next chunk size will be progressively reduced as ***L*** grows to maintain an aligned chunk execution time across pipeline stages.
+
+Based on this method, the scheduler can predict and dynamically reduce the chunk size during runtime to minimize the bubbles caused by the stage misalignment. To be noticed, the scheduler does not use the raw predicted value. To facilitate efficient KVCache memory management and ensure affinity with hardware execution efficiency, the value is aligned downward to the nearest multiple of max(`--page-size`, 64).
+
+
+### Chunked Prefill Size and Smoothing Factor
+
+When `--enable-dynamic-chunking` is enabled, each chunk size of a sequence is determined dynamically based on the quadratic model that predicts the next chunk size based on the estimated runtime of the initial chunk length. In this case, we use `--chunked-prefill-size` to set up the initial chunk size. When switching to the dynamic chunking mode, the initial chunk size (`--chunked-prefill-size`) should be set to a larger value comparable to the original chunked prefill size, so that there won't be too many chunks.
+
+**`SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR`** is an environmental variable that controls the smoothing factor for the dynamic chunking algorithm, defaulting to 0.75. It determines how much the chunk size can change during the prefill phase. A larger value means a more aggressive chunk size change, which may lead to better performance but also to greater chunk size changes (the chunk size at the end may become very small, which could lead to performance degradation) and more total chunks. When it is set to 1, the chunk size will be adjusted strictly based on the aforementioned quadratic model that predicts the next chunk size. A smaller value means a more conservative chunk size change, which may lead to smaller chunk size changes and fewer total chunks. When it is set to 0, the chunk size will not be adjusted dynamically, so it is identical to the traditional way with a fixed chunked prefill size.
+
+Due to the variation in hardware, models, and target workloads, a static configuration is seldom optimal across all scenarios. Consequently, achieving peak performance necessitates a degree of hyperparameter tuning when switching to the dynamic chunking mode.
+
+**Tuning Guidance for Dynamic Chunked Prefill**
+
+* **Step 1 \- Iterate to find the optimal fixed chunked prefill size for the targeted PP size**: Different PP sizes for targeted ITL may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling.
+* **Step 2 \- Initial Chunk Size Selection for Dynamic Chunking**: Set the initial size to 2× or 3× the optimal fixed chunked prefill size. This reduces the total number of chunks and prevents "tail chunks" from underutilizing hardware. To maintain efficiency for extremely large Input Token Lengths (ITL), the dynamic predictor automatically ensures subsequent chunks are at least 1/4 of this initial size. In addition, it is recommended to use a larger initial chunk size (e.g., 4× the optimal fixed chunked prefill size) for such cases as well.
+* **Step 3 \- Smooth Factor Adjustment**: This factor controls how strictly the chunk size adjusts the prediction given by the quadratic performance fitting model.
+  * 1.0: Follows the model strictly.
+  * **0.6 – 0.85 (Recommended)**: Typical range for the best balance between dynamic scaling and hardware stability. Through experiments, we find that a range between 0.6 and 0.85 typically yields the best performance for dynamic chunking.
+  * 0: Disables dynamic adjustment, reverting to traditional fixed-size chunking.
+* **Another small optimization tip:** Put the larger partition in the higher PP rank when the layers are not evenly divisible across ranks. It can increase the GPU utilization when a larger PP rank is waiting for the previous stage’s result, hence reducing the bubbles on higher PP ranks. If we take DeepSeek-V3.1 as an example, `SGLANG_PP_LAYER_PARTITION=15,15,15,16` usually performs better than `16,15,15,15`.
+
+## Best Practice for Long Context
+
+### Tuning the Chunked Prefill Size
+Optimizing the chunked prefill size is crucial for balancing pipeline efficiency and resource utilization. The ideal size depends on factors including model architecture, hardware configuration, and typical input lengths. We recommend starting with a small chunk size, such as 4K, and gradually increasing it until you find the optimal size for your specific use case (Different targeted ITL and PP Sizes may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling). Alternatively, you can analyze the hardware capacity and determine the optimal chunk size based on the roofline model.
+
+### Enable Dynamic Chunking and Adjust Smoothing Factor for Ultra-long ITL
+SGLang also offers a dynamic chunking solution that could further improve performance. This feature is currently an experimental feature that requires a certain amount of tuning experimentation and may not be suitable for all workloads. In addition, fine-tuning the smoothing factor can help optimize performance for specific workloads and model characteristics.
+
+### Case Study on NVIDIA H20
+
+When evaluating pipeline parallelism with fixed chunked prefill sizes from 2K to 16K, experiment results show that a 4K chunk size delivered optimal prefill TTFT performance for the DeepSeek-V3.1, and a 6K chunk size delivered optimal prefill TTFT performance for the Qwen3-235B-A22B-FP8.
+
+When enabling dynamic chunking, we first scale the optimal fixed chunked prefill size by a factor of 3 as the initial chunk size. Through experimentation, we found that a multiplier of 2-3 provides an appropriate balance—avoiding excessive initial pipeline bubbles while ensuring that subsequent chunks don't become too small as context length increases. With the default dynamic chunking smoothing factor of 0.75, we performed parameter tuning and determined that a value of 0.65 works optimally with the 12K initial chunk size for the DeepSeek-V3.1, while a value of 0.8 works optimally with the 18K initial chunk size for the Qwen3-235B-A22B-FP8.
+
+#### DeepSeek-V3.1 with 128K Input Token Length
+```bash Command
+# prefill node 0 (fixed chunked prefill size)
+python3 -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \
+  --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
+  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
+  --disable-radix-cache --mem-fraction-static 0.8  \
+  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
+  --max-running-requests 128 --chunked-prefill-size 4096
+```
+
+```bash Command
+# prefill node 0 (with dynamic chunking)
+export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.65
+python3 -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \
+  --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
+  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
+  --disable-radix-cache --mem-fraction-static 0.8  \
+  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
+  --max-running-requests 128 --chunked-prefill-size 12288 --enable-dynamic-chunking
+```
+
+#### Qwen3-235B-A22B-FP8 with 128K Input Token Length
+```bash Command
+# prefill node 0 (fixed chunked prefill size)
+python3 -m sglang.launch_server \
+  --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \
+  --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \
+  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
+  --disable-radix-cache --mem-fraction-static 0.8  \
+  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
+  --max-running-requests 128 --chunked-prefill-size 6144
+```
+
+```bash Command
+# prefill node 0 (with dynamic chunking)
+export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.8
+python3 -m sglang.launch_server \
+  --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \
+  --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \
+  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
+  --disable-radix-cache --mem-fraction-static 0.8  \
+  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
+  --max-running-requests 128 --chunked-prefill-size 18432 --enable-dynamic-chunking
+```
+
+Note: `--disable-radix-cache` is enabled only for reproducible benchmarking purposes. It is not recommended to use it in production.
+
+## Best Practice for Pipeline Parallelism with PD Disaggregation
+To be added. Stay tuned for the latest updates on Pipeline Parallelism with PD Disaggregation.

docs_new/docs/advanced_features/quantization.mdx

@@ -0,0 +1,950 @@
+---
+title: "Quantization"
+metatags:
+    description: "SGLang quantization: FP8, FP4, AWQ, GPTQ, ModelOpt, torchao. Offline and online quantization methods for efficient LLM inference."
+---
+SGLang supports various quantization methods, including offline quantization and online dynamic quantization.
+
+Offline quantization loads pre-quantized model weights directly during inference. This is required for quantization methods
+such as GPTQ and AWQ, which collect and pre-compute various statistics from the original weights using the calibration dataset.
+
+Online quantization dynamically computes scaling parameters—such as the maximum/minimum values of model weights—during runtime.
+Like NVIDIA FP8 training's [delayed scaling](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#Mixed-precision-training-with-FP8) mechanism, online quantization calculates the appropriate scaling factors
+on-the-fly to convert high-precision weights into a lower-precision format.
+
+**Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.**
+
+If you use a pre-quantized model, **do not add `--quantization` to enable online quantization at the same time**.
+For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [NVIDIA ModelOpt](https://huggingface.co/collections/nvidia/inference-optimized-checkpoints-with-model-optimizer)
+or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some
+popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization
+to guard against abnormal quantization loss regressions.
+
+## Platform Compatibility
+
+The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs.
+
+<table>
+  <thead>
+    <tr>
+      <th>Method</th>
+      <th>NVIDIA GPUs</th>
+      <th>AMD GPUs (MI300X/MI325X/MI350X)</th>
+      <th>Ascend NPUs (A2/A3/A5)</th>
+      <th>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>fp8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>WIP</td>
+      <td>Aiter or Triton backend on AMD</td>
+    </tr>
+    <tr>
+      <td><code>mxfp4</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>WIP</td>
+      <td>Requires CDNA3/CDNA4 with MXFP support; uses Aiter</td>
+    </tr>
+    <tr>
+      <td><code>mxfp8</code></td>
+      <td>No</td>
+      <td>No</td>
+      <td>Yes (A5 for Diffusion and LLM Dense Linear)</td>
+      <td>Ascend NPU only; online MXFP8 quantization for Diffusion models (e.g., Wan2.2) and LLM Dense Linear on A5 series; uses CANN <code>npu_dynamic_mx_quant</code> / <code>npu_quant_matmul</code> kernels</td>
+    </tr>
+    <tr>
+      <td><code>mxfp_w4a8</code></td>
+      <td>No</td>
+      <td>No</td>
+      <td>Yes (A5)</td>
+      <td>Ascend NPU only; online W4A8 for Qwen3 dense LLM (MXFP4 weights + MXFP8 activations) on A5 series; offline <code>W4A8_MXFP</code> checkpoints are auto-detected via <code>modelslim</code></td>
+    </tr>
+    <tr>
+      <td><code>blockwise_int8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>Triton-based, works on both platforms</td>
+    </tr>
+    <tr>
+      <td><code>w8a8_int8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>w8a8_fp8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>Aiter or Triton FP8 on AMD</td>
+    </tr>
+    <tr>
+      <td><code>awq</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Uses Triton dequantize on AMD (vs. optimized CUDA kernels on NVIDIA). Uses CANN kernels on Ascend</td>
+    </tr>
+    <tr>
+      <td><code>gptq</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Uses Triton or vLLM kernels on AMD. Uses CANN kernels on Ascend</td>
+    </tr>
+    <tr>
+      <td><code>compressed-tensors</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Partial</td>
+      <td>Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, <code>FP8</code> not supported yet</td>
+    </tr>
+    <tr>
+      <td><code>quark</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>AMD Quark quantization; Aiter GEMM paths on AMD</td>
+    </tr>
+    <tr>
+      <td><code>auto-round</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Partial</td>
+      <td>Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend</td>
+    </tr>
+    <tr>
+      <td><code>quark_int4fp8_moe</code></td>
+      <td>No</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4)</td>
+    </tr>
+    <tr>
+      <td><code>awq_marlin</code></td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>No</td>
+      <td>Marlin kernels are CUDA-only</td>
+    </tr>
+    <tr>
+      <td><code>gptq_marlin</code></td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>No</td>
+      <td>Marlin kernels are CUDA-only</td>
+    </tr>
+    <tr>
+      <td><code>gguf</code></td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>Yes</td>
+      <td>CUDA kernels in sgl-kernel; Ascend uses CPU pre-dequantization at load time</td>
+    </tr>
+    <tr>
+      <td><code>modelopt</code> / <code>modelopt_fp8</code></td>
+      <td>Yes (Hopper/SM90+)</td>
+      <td>No</td>
+      <td>No</td>
+      <td><a href="https://github.com/NVIDIA/Model-Optimizer">NVIDIA ModelOpt</a>; requires NVIDIA hardware</td>
+    </tr>
+    <tr>
+      <td><code>modelopt_fp4</code></td>
+      <td>Yes (SM80-SM90 via Marlin; SM100+ native FP4)</td>
+      <td>No</td>
+      <td>No</td>
+      <td><a href="https://github.com/NVIDIA/Model-Optimizer">NVIDIA ModelOpt</a>; use Marlin W4A16 fallback on Ampere/Hopper and native FP4 backends on Blackwell</td>
+    </tr>
+    <tr>
+      <td><code>nvfp4_online</code></td>
+      <td>Yes (Blackwell/SM100 or SM103)</td>
+      <td>No</td>
+      <td>No</td>
+      <td>Online MoE-only NVFP4 weight quantization with runtime per-token activation scaling for BF16/FP16/FP8 checkpoints; requires <code>flashinfer_trtllm</code> or <code>flashinfer_trtllm_routed</code></td>
+    </tr>
+    <tr>
+      <td><code>petit_nvfp4</code></td>
+      <td>No</td>
+      <td>Yes (MI250/MI300X/MI325X)</td>
+      <td>No</td>
+      <td>Enables NVFP4 on ROCm via <a href="https://github.com/causalflow-ai/petit-kernel">Petit</a>; use <code>modelopt_fp4</code> on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See <a href="https://lmsys.org/blog/2025-09-21-petit-amdgpu/">LMSYS blog</a> and <a href="https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html">AMD ROCm blog</a>.</td>
+    </tr>
+    <tr>
+      <td><code>bitsandbytes</code></td>
+      <td>Yes</td>
+      <td>Experimental</td>
+      <td>No</td>
+      <td>Depends on bitsandbytes ROCm support</td>
+    </tr>
+    <tr>
+      <td><code>torchao</code> (<code>int4wo</code>, etc.)</td>
+      <td>Yes</td>
+      <td>Partial</td>
+      <td>No</td>
+      <td><code>int4wo</code> not supported on AMD; other methods may work</td>
+    </tr>
+    <tr>
+      <td><code>modelslim</code></td>
+      <td>No</td>
+      <td>No</td>
+      <td>Yes</td>
+      <td>Ascend quantization; Uses CANN kernels</td>
+    </tr>
+
+  </tbody>
+</table>
+
+On AMD, several of these methods use [Aiter](https://github.com/ROCm/aiter) for acceleration -- set `SGLANG_USE_AITER=1` where noted. See [AMD GPU setup](../hardware-platforms/amd_gpu) for installation and configuration details.
+
+On Ascend, various layers quantization configurations are supported, see [Ascend NPU quantization](../hardware-platforms/ascend-npus/ascend_npu_quantization) for details.
+
+## GEMM Backends for FP4/FP8 Quantization
+
+<Note>
+Backend selection applies to **blockwise FP8**, **MXFP8** (dense linear), and **NVFP4** GEMM. When running offline or online FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`.
+</Note>
+
+### `--fp8-gemm-backend` (Blockwise FP8 GEMM)
+
+<table>
+  <thead>
+    <tr>
+      <th>Backend</th>
+      <th>Hardware</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>auto</code></td>
+      <td>All</td>
+      <td>Auto-selects based on hardware</td>
+    </tr>
+    <tr>
+      <td><code>deep_gemm</code></td>
+      <td>SM90, SM100</td>
+      <td>JIT-compiled; enabled when DeepGEMM is installed</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_trtllm</code></td>
+      <td>SM100</td>
+      <td>FlashInfer TensorRT-LLM backend; optimal for low-latency</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_cutlass</code></td>
+      <td>SM100/120</td>
+      <td>FlashInfer CUTLASS groupwise FP8 GEMM</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_deepgemm</code></td>
+      <td>SM90</td>
+      <td>Uses swapAB optimization for small M dimensions in decoding</td>
+    </tr>
+    <tr>
+      <td><code>cutlass</code></td>
+      <td>SM90, SM100/120</td>
+      <td>sgl-kernel CUTLASS</td>
+    </tr>
+    <tr>
+      <td><code>triton</code></td>
+      <td>All</td>
+      <td>Fallback; widely compatible</td>
+    </tr>
+    <tr>
+      <td><code>aiter</code></td>
+      <td>ROCm</td>
+      <td>AMD AITER backend</td>
+    </tr>
+  </tbody>
+</table>
+
+**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM90/SM100/120); 4) AITER (AMD); 5) Triton. **Exception:** SM120 always resolves to Triton.
+
+**MXFP8 dense linear:** `auto` uses `flashinfer_cutlass` on SM100 (else `triton`). `flashinfer_cutlass` is fastest on most shapes; `flashinfer_trtllm` is faster only at small M.
+
+### `--fp4-gemm-backend` (NVFP4 GEMM)
+
+<table>
+  <thead>
+    <tr>
+      <th>Backend</th>
+      <th>Hardware</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>auto</code></td>
+      <td>SM80+</td>
+      <td>Auto-selects: <code>flashinfer_cutedsl</code> on SM100; <code>marlin</code> on SM80-SM90; <code>flashinfer_cutlass</code> otherwise (including SM120)</td>
+    </tr>
+    <tr>
+      <td><code>cutlass</code></td>
+      <td>SM100/120</td>
+      <td>SGLang CUTLASS kernel</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_cutlass</code></td>
+      <td>SM100/120</td>
+      <td>FlashInfer CUTLASS backend</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_cudnn</code></td>
+      <td>SM100/120 (CUDA 13+, cuDNN 9.15+)</td>
+      <td>FlashInfer cuDNN backend</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_cutedsl</code></td>
+      <td>SM100</td>
+      <td>FlashInfer CuTe DSL backend</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_trtllm</code></td>
+      <td>SM100</td>
+      <td>FlashInfer TensorRT-LLM backend</td>
+    </tr>
+    <tr>
+      <td><code>marlin</code></td>
+      <td>SM80-SM90</td>
+      <td>Weight-only W4A16 fallback for NVFP4 checkpoints</td>
+    </tr>
+  </tbody>
+</table>
+
+On Blackwell, when FlashInfer is unavailable for NVFP4, the SGLang CUTLASS kernel is used as an automatic fallback. On SM80-SM90, `auto` selects Marlin for NVFP4.
+
+## Offline Quantization
+
+To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline,
+there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the
+downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.**
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
+    --port 30000 --host 0.0.0.0
+```
+
+Take note, if your model is **per-channel quantized (INT8 or FP8) with per-token dynamic quantization activation**, you can opt to include `--quantization w8a8_int8` or `--quantization w8a8_fp8` to invoke the corresponding CUTLASS int8_kernel or fp8_kernel in sgl-kernel. This action will ignore the Hugging Face config's quantization settings. For instance, with `neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic`, if you execute with `--quantization w8a8_fp8`, the system will use the `W8A8Fp8Config` from SGLang to invoke the sgl-kernel, rather than the `CompressedTensorsConfig` for vLLM kernels.
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic \
+    --quantization w8a8_fp8 \
+    --port 30000 --host 0.0.0.0
+```
+
+### Examples of Offline Model Quantization
+
+#### Using [Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide)
+
+We strongly suggest the use of Unsloth to quantize and load the model. Please refer to [SGLang Deployment & Inference Guide with Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide).
+
+#### Using [auto-round](https://github.com/intel/auto-round)
+
+```bash Command
+# Install
+pip install auto-round
+```
+
+- LLM quantization
+
+```py Example
+# for LLM
+from auto_round import AutoRound
+model_id = "meta-llama/Llama-3.2-1B-Instruct"
+quant_path = "Llama-3.2-1B-Instruct-autoround-4bit"
+# Scheme examples: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc.
+scheme = "W4A16"
+format = "auto_round"
+autoround = AutoRound(model_id, scheme=scheme)
+autoround.quantize_and_save(quant_path, format=format) # quantize and save
+
+```
+
+- VLM quantization
+```py Example
+# for VLMs
+from auto_round import AutoRoundMLLM
+model_name = "Qwen/Qwen2-VL-2B-Instruct"
+quant_path = "Qwen2-VL-2B-Instruct-autoround-4bit"
+scheme = "W4A16"
+format = "auto_round"
+autoround = AutoRoundMLLM(model_name, scheme)
+autoround.quantize_and_save(quant_path, format=format) # quantize and save
+
+```
+
+- Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)
+
+```bash Command
+auto-round \
+    --model meta-llama/Llama-3.2-1B-Instruct \
+    --bits 4 \
+    --group_size 128 \
+    --format "auto_round" \
+    --output_dir ./tmp_autoround
+```
+
+- SGlang API Usage (CPU/CUDA)
+
+```python Example
+from sglang.srt.configs.load_config import LoadConfig
+from sglang.srt.configs.model_config import ModelConfig
+from sglang.srt.model_loader.loader import get_model_loader
+from sglang.srt.configs.device_config import DeviceConfig
+
+# Configure model with inc quantization and saving
+model_config = ModelConfig(
+    model_path="meta-llama/Llama-3.2-3B-Instruct",
+    quantization="auto-round-int8",
+    trust_remote_code=True,
+)
+
+load_config = LoadConfig(
+    inc_save_path="./quantized_model",
+)
+device_config = DeviceConfig(device="cpu")
+
+# Load and quantize the model
+model_loader = get_model_loader(load_config, model_config)
+quantized_model = model_loader.load_model(
+    model_config=model_config,
+    device_config=device_config,
+)
+```
+
+- known issues
+
+Several limitations currently affect offline quantized model loading in sglang, These issues might be resolved in future updates of sglang. If you experience any problems, consider using Hugging Face Transformers as an alternative.
+
+1. Mixed-bit Quantization Limitations
+
+    Mixed-bit quantization is not fully supported. Due to vLLM's layer fusion (e.g., QKV fusion), applying different bit-widths to components within the same fused layer can lead to compatibility issues.
+
+
+2. Limited Support for Quantized MoE Models
+
+    Quantized MoE models may encounter inference issues due to kernel limitations (e.g., lack of support for mlp.gate layer quantization). please try to skip quantizing these layers to avoid such errors.
+
+
+3. Limited Support for Quantized VLMs
+    <Accordion title="Details">
+        {/* VLM failure cases */}
+
+    Qwen2.5-VL-7B
+
+    auto_round:auto_gptq format:  Accuracy is close to zero.
+
+    GPTQ format:  Fails with:
+    ```text Output
+    The output size is not aligned with the quantized weight shape
+    ```
+    auto_round:auto_awq and AWQ format:  These work as expected.
+    </Accordion>
+
+4. Limited Support for SGlang API Usage
+
+    SGlang API Usage only supports `auto-round-int8` quantization method now, more quantization methods are on the way.
+
+#### Using [GPTQModel](https://github.com/ModelCloud/GPTQModel)
+
+```bash Command
+# install
+pip install gptqmodel --no-build-isolation -v
+```
+
+```py Example
+from datasets import load_dataset
+from gptqmodel import GPTQModel, QuantizeConfig
+
+model_id = "meta-llama/Llama-3.2-1B-Instruct"
+quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"
+
+calibration_dataset = load_dataset(
+    "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz",
+    split="train"
+  ).select(range(1024))["text"]
+
+quant_config = QuantizeConfig(bits=4, group_size=128) # quantization config
+model = GPTQModel.load(model_id, quant_config) # load model
+
+model.quantize(calibration_dataset, batch_size=2) # quantize
+model.save(quant_path) # save model
+```
+
+#### Using [LLM Compressor](https://github.com/vllm-project/llm-compressor/)
+
+```bash Command
+# install
+pip install llmcompressor
+```
+
+Here, we take quantize `meta-llama/Meta-Llama-3-8B-Instruct` to `FP8` as an example to elaborate on how to do offline quantization.
+
+```python Example
+from transformers import AutoTokenizer
+from llmcompressor.transformers import SparseAutoModelForCausalLM
+from llmcompressor.transformers import oneshot
+from llmcompressor.modifiers.quantization import QuantizationModifier
+
+# Step 1: Load the original model.
+MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
+
+model = SparseAutoModelForCausalLM.from_pretrained(
+  MODEL_ID, device_map="auto", torch_dtype="auto")
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+
+# Step 2: Perform offline quantization.
+# Step 2.1: Configure the simple PTQ quantization.
+recipe = QuantizationModifier(
+  targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
+
+# Step 2.2: Apply the quantization algorithm.
+oneshot(model=model, recipe=recipe)
+
+# Step 3: Save the model.
+SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
+model.save_pretrained(SAVE_DIR)
+tokenizer.save_pretrained(SAVE_DIR)
+```
+
+Then, you can directly use the quantized model with `SGLang`, by using the following command:
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path $PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic \
+    --port 30000 --host 0.0.0.0
+```
+
+#### Using [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)
+
+NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware.
+
+**Offline vs. Online Quantization:**
+
+SGLang supports two modes for ModelOpt.
+
+* **Offline Quantization (pre-quantized):**
+    * **Usage:** Download a pre-quantized model from Hugging Face or run `hf_ptq.py` once to create a new quantized checkpoint. Then load this quantized checkpoint.
+    * **Pros:** Fast server startup, quantization can be validated before deployment, efficient resource usage.
+    * **Cons:** Requires an extra preparation step.
+
+* **Online Quantization (quant and serve):**
+    * **Usage:** Load a standard BF16/FP16 model and add a flag. The engine applies quantization *on startup*.
+    * **Pros:** Convenient (no new checkpoint needed).
+    * **Cons:** **High startup time**, increases VRAM usage during initialization (risk of OOM).
+
+The following sections guide you through using the Offline path: loading pre-quantized models or creating your own checkpoints.
+
+##### Using Pre-Quantized Checkpoints
+
+If a model is already quantized (e.g., from Hugging Face), you can load it directly.
+
+* **FP8 Models:**
+    Use `--quantization modelopt_fp8`.
+    ```bash Command
+    python3 -m sglang.launch_server \
+        --model-path nvidia/Llama-3.1-8B-Instruct-FP8 \
+        --quantization modelopt_fp8 \
+        --port 30000
+    ```
+
+* **FP4 Models:**
+    Use `--quantization modelopt_fp4`.
+    ```bash Command
+    python3 -m sglang.launch_server \
+        --model-path nvidia/Llama-3.3-70B-Instruct-NVFP4 \
+        --quantization modelopt_fp4 \
+        --port 30000
+    ```
+
+##### Creating Your Own Quantized Checkpoints
+
+If a pre-quantized checkpoint is not available for your model, you can create one using NVIDIA Model Optimizer's `hf_ptq.py` script.
+
+**Why quantize?**
+- Reduce VRAM usage
+- Higher throughput and lower latency
+- More flexible deployment (on smaller GPUs)
+
+**What can be quantized?**
+- The entire model
+- MLP layers only
+- KV cache
+
+**Key options in `hf_ptq.py`:**
+
+`--qformat`: Quantization formats `fp8`, `nvfp4`, `nvfp4_mlp_only`
+
+`--kv_cache_qformat`: KV cache quantization format (default: `fp8`)
+
+**Note:** The default `kv_cache_qformat` may not be optimal for all use cases. Consider setting this explicitly.
+
+**Hardware requirements:** Hopper and higher are recommended. Insufficient GPU memory may cause weight offloading, resulting in extremely long quantization time.
+
+For detailed usage and supported model architectures, see [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq).
+
+SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.
+
+##### Installation
+
+First, install ModelOpt:
+
+```bash Command
+pip install nvidia-modelopt
+```
+
+##### Quantization and Export Workflow
+
+SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow. Run from the SGLang repository root (see [modelopt_quantize_and_export.py](https://github.com/sgl-project/sglang/blob/main/examples/usage/modelopt_quantize_and_export.py)):
+
+```bash Command
+# Quantize and export a model using ModelOpt FP8 quantization
+python examples/usage/modelopt_quantize_and_export.py quantize \
+    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
+    --export-dir ./quantized_tinyllama_fp8 \
+    --quantization-method modelopt_fp8
+
+# For FP4 quantization (requires Blackwell GPU)
+python examples/usage/modelopt_quantize_and_export.py quantize \
+    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
+    --export-dir ./quantized_tinyllama_fp4 \
+    --quantization-method modelopt_fp4
+```
+
+##### Available Quantization Methods
+
+- `modelopt_fp8`: FP8 quantization with optimal performance on NVIDIA Hopper and Blackwell GPUs
+- `modelopt_fp4`: FP4 quantization with optimal performance on Nvidia Blackwell GPUs
+
+##### Python API Usage
+
+You can also use ModelOpt quantization programmatically:
+
+```python Example
+import sglang as sgl
+from sglang.srt.configs.device_config import DeviceConfig
+from sglang.srt.configs.load_config import LoadConfig
+from sglang.srt.configs.model_config import ModelConfig
+from sglang.srt.model_loader.loader import get_model_loader
+
+# Configure model with ModelOpt quantization and export
+model_config = ModelConfig(
+    model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+    quantization="modelopt_fp8",  # or "modelopt_fp4"
+    trust_remote_code=True,
+)
+
+load_config = LoadConfig(
+    modelopt_export_path="./exported_model",
+    modelopt_checkpoint_save_path="./checkpoint.pth",  # optional, fake quantized checkpoint
+)
+device_config = DeviceConfig(device="cuda")
+
+# Load and quantize the model (export happens automatically)
+model_loader = get_model_loader(load_config, model_config)
+quantized_model = model_loader.load_model(
+    model_config=model_config,
+    device_config=device_config,
+)
+```
+
+##### Deploying Quantized Models
+
+After quantization and export, you can deploy the model with SGLang:
+
+```bash Command
+# Deploy the exported quantized model
+python -m sglang.launch_server \
+    --model-path ./quantized_tinyllama_fp8 \
+    --quantization modelopt \
+    --port 30000 --host 0.0.0.0
+```
+
+Or using the Python API (use the same path as `modelopt_export_path` from the quantize step):
+
+```python Example
+import sglang as sgl
+
+def main():
+    # Deploy exported ModelOpt quantized model
+    # Path must match modelopt_export_path from quantize step (e.g., ./exported_model)
+    llm = sgl.Engine(
+        model_path="./exported_model",
+        quantization="modelopt",
+    )
+
+    # Run inference
+    prompts = [
+        "Hello, how are you?",
+        "What is the capital of France?",
+    ]
+    sampling_params = {
+        "temperature": 0.8,
+        "top_p": 0.95,
+        "max_new_tokens": 100,
+    }
+
+    outputs = llm.generate(prompts, sampling_params)
+
+    for i, output in enumerate(outputs):
+        print(f"Prompt: {prompts[i]}")
+        print(f"Output: {output['text']}")
+
+if __name__ == "__main__":
+    main()
+
+```
+
+##### Advanced Features
+
+**Checkpoint Management**: Save and restore fake quantized checkpoints for reuse:
+
+```bash Command
+# Save the fake quantized checkpoint during quantization
+python examples/usage/modelopt_quantize_and_export.py quantize \
+    --model-path meta-llama/Llama-3.2-1B-Instruct \
+    --export-dir ./quantized_model \
+    --quantization-method modelopt_fp8 \
+    --checkpoint-save-path ./my_checkpoint.pth
+
+# The checkpoint can be reused for future quantization runs and skip calibration
+```
+
+**Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly. See [LoadConfig](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/configs/load_config.py) for the full API:
+
+```python Example
+from sglang.srt.configs.device_config import DeviceConfig
+from sglang.srt.configs.load_config import LoadConfig
+from sglang.srt.configs.model_config import ModelConfig
+from sglang.srt.model_loader.loader import get_model_loader
+
+model_config = ModelConfig(
+    model_path="meta-llama/Llama-3.2-1B-Instruct",
+    quantization="modelopt_fp8",
+    trust_remote_code=True,
+)
+
+load_config = LoadConfig(
+    modelopt_checkpoint_restore_path="./my_checkpoint.pth",
+    modelopt_export_path="./exported_model",
+)
+
+# Load and export the model (DeviceConfig defaults to device="cuda")
+model_loader = get_model_loader(load_config, model_config)
+model_loader.load_model(model_config=model_config, device_config=DeviceConfig())
+```
+
+##### Benefits of ModelOpt
+
+- **Hardware Optimization**: Specifically optimized for NVIDIA GPU architectures
+- **Advanced Quantization**: Supports cutting-edge FP8 and FP4 quantization techniques
+- **Seamless Integration**: Automatic export to HuggingFace format for easy deployment
+- **Calibration-based**: Uses calibration datasets for optimal quantization quality
+- **Production Ready**: Enterprise-grade quantization with NVIDIA support
+
+#### Using [ModelSlim](https://gitcode.com/Ascend/msmodelslim)
+MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.
+
+- **Installation**
+
+    ```bash Command
+    # Clone repo and install msmodelslim:
+    git clone https://gitcode.com/Ascend/msmodelslim.git
+    cd msmodelslim
+    bash install.sh
+    ```
+
+- **LLM quantization**
+
+    Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card).
+    > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).
+
+    _Traditional quantification methods require the preparation of calibration data files (```.jsonl``` formats) for calibration in the quantification process._
+    ```bash Command
+    Qwen3-32B/      # floating-point model downloaded from official HF (or modelscope) repo
+    msmodelslim/    # msmodelslim repo
+      |----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones)
+          |----- some file (such as laos_calib.jsonl)
+      |----- lab_practice # best practice folder with configs for quantization
+          |----- model folder (such as qwen3_5_moe folder) # folder with quantization configs
+              |----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config
+      |----- another folders
+    output_folder/   # generated by below command
+      |----- quant_model_weights-00001-of-0001.safetensors # quantized weights
+      |----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.)
+      |----- another files (such as config.json, tokenizer.json, etc.)
+    ```
+    Run quantization using one-click quantization (recommended):
+    ```bash Command
+    msmodelslim quant \
+    --model_path ${MODEL_PATH} \
+    --save_path ${SAVE_PATH} \
+    --device npu:0,1 \
+    --model_type Qwen3-32B \
+    --quant_type w8a8 \
+    --trust_remote_code True
+    ```
+
+- **Usage Example**
+    ```bash Command
+    python3 -m sglang.launch_server \
+    --model-path $PWD/Qwen3-32B-w8a8 \
+    --port 30000 --host 0.0.0.0
+    ```
+
+- **Available Quantization Methods**:
+    - [x]  ```W4A4_DYNAMIC``` linear with online quantization of activations
+    - [x]  ```W8A8``` linear with offline quantization of activations
+    - [x]  ```W8A8_DYNAMIC``` linear with online quantization of activations
+    - [x]  ```W4A4_DYNAMIC``` MOE with online quantization of activations
+    - [x]  ```W4A8_DYNAMIC``` MOE with online quantization of activations
+    - [x]  ```W8A8_DYNAMIC``` MOE with online quantization of activations
+    - [ ]  ```W4A8``` linear TBD
+    - [ ]  ```W4A16``` linear TBD
+    - [ ]  ```W48A16``` linear TBD
+    - [ ]  ```W4A16``` MoE in progress
+    - [ ]  ```W8A16``` MoE in progress
+    - [ ]  ```KV Cache``` in progress
+    - [ ]  ```Attention``` in progress
+
+
+For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/README.md) section in ModelSLim repo.
+
+## Online Quantization
+
+To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`:
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --quantization fp8 \
+    --port 30000 --host 0.0.0.0
+```
+
+Our team is working on supporting more online quantization methods. SGLang will soon support methods including but not limited to `["awq", "gptq", "marlin", "gptq_marlin", "awq_marlin", "bitsandbytes", "gguf"]`.
+
+### torchao online quantization method
+
+SGLang also supports quantization methods based on [torchao](https://github.com/pytorch/ao). You can simply specify `--torchao-config` in the command line to support this feature. For example, if you want to enable `int4wo-128` for model `meta-llama/Meta-Llama-3.1-8B-Instruct`, you can launch the server with the following command:
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --torchao-config int4wo-128 \
+    --port 30000 --host 0.0.0.0
+```
+
+SGLang supports the following quantization methods based on torchao `["int8dq", "int8wo", "fp8wo", "fp8dq-per_tensor", "fp8dq-per_row", "int4wo-32", "int4wo-64", "int4wo-128", "int4wo-256"]`.
+
+Note: According to [this issue](https://github.com/sgl-project/sglang/issues/2219#issuecomment-2561890230), `"int8dq"` method currently has some bugs when using together with cuda graph capture. So we suggest to disable cuda graph capture when using `"int8dq"` method. Namely, please use the following command:
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --torchao-config int8dq \
+    --disable-cuda-graph \
+    --port 30000 --host 0.0.0.0
+```
+
+### `nvfp4_online` online quantization method
+
+Use `--quantization nvfp4_online` when you have a BF16, FP16, or FP8 MoE checkpoint and want SGLang to convert eligible MoE expert weights to NVFP4 while loading the model. This mode is for online conversion from higher-precision or FP8 checkpoints. It is not the serving path for already serialized NVFP4 checkpoints; use the existing ModelOpt FP4 path for those checkpoints.
+
+The design separates weight quantization from activation scaling:
+
+- **Weights:** SGLang quantizes each eligible MoE expert weight tensor as it is loaded, using standard 2D NVFP4 weight quantization. The generated NVFP4 weights use static E4M3 block scales plus static per-tensor FP32 scales derived from the weight amax. For gated MoE experts, the w1/w3 pair shares one per-tensor FP32 scale.
+- **Activations:** FlashInfer computes activation FP32 scales dynamically per token at runtime. Because activations are scaled per token, this mode does not need calibrated static activation FP32 scales from the checkpoint.
+- **FP8 checkpoints:** If an eligible expert weight is stored as FP8, SGLang first dequantizes that tensor with the checkpoint scale and then requantizes it to NVFP4 during loading.
+- **Other layers:** Dense linear layers stay in their source checkpoint precision or checkpoint quantization path.
+
+Only `--moe-runner-backend flashinfer_trtllm` and `--moe-runner-backend flashinfer_trtllm_routed` are supported. If `--moe-runner-backend` is omitted, SGLang selects `flashinfer_trtllm`. Tensor parallelism is supported; activation per-token scales are computed locally on each TP rank, while online weight quantization still uses the loaded expert tensor's per-tensor amax-derived FP32 scale.
+
+FlashInfer TRTLLM MoE backends disable shared-expert fusion, so online quantization applies to routed MoE experts while shared experts stay in the checkpoint precision. To keep specific routed MoE layers out of online FP4 conversion, include their module path in `SGLANG_FP4_IGNORED_LAYERS`; FP8 checkpoints keep those ignored experts in FP8 checkpoint precision.
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
+    --tp-size 2 \
+    --ep-size 2 \
+    --quantization nvfp4_online \
+    --port 30000 --host 0.0.0.0
+```
+
+### `quark_int4fp8_moe` online quantization method
+
+SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/v0.4.8/python/sglang/srt/layers/moe/fused_moe_triton/layer.py#L271) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.
+
+Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly.
+
+### `quark_mxfp4` online quantization method
+
+SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method `--quantization quark_mxfp4`, that will quantize BF16 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs.
+
+Example:
+
+```bash
+sglang serve --model-path Qwen/Qwen3-30B-A3B \
+    --tensor-parallel-size 1 \
+    --quantization quark_mxfp4
+```
+
+### Intel® Neural Compressor online quantization method
+
+SGLang supports quantization methods based on the advanced algorithm [auto-round](https://github.com/intel/auto-round) in [Intel® Neural Compressor](https://github.com/intel/neural-compressor). You can simply specify `--quantization auto-round-int8` to use this feature. It will quantize the model on the fly to target format. More online quantization methods are on the way.
+
+##### Available Quantization Methods
+
+| Quantization Method | Schemes | Validated Hardware Environment |
+|:--------------------|:--------|:-------------------------------|
+| auto-round-int8     |INT8 per-channel quantized weight <br /> INT8 per-token dynamic quantized activation | Intel Xeon Scalable processor <br /> Nvidia A100 GPU |
+
+
+## Diffusion Model Quantization on Ascend NPU
+
+SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend A5 NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI.
+
+**Requirements:** Ascend A5, CANN ≥ 8.0.RC3
+
+### Online MXFP8
+
+Pass `--quantization mxfp8` to dynamically quantize FP16/BF16 transformer weights to MXFP8 at load time:
+
+```bash
+sglang serve \
+  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
+  --quantization mxfp8 \
+  --num-gpus 4
+```
+
+### Offline MXFP8 (ModelSlim)
+
+Pre-quantize with [msModelSlim](https://gitcode.com/Ascend/msmodelslim) and load the checkpoint directly — the quantization scheme is auto-detected from `quant_model_description.json`:
+
+```bash
+sglang generate \
+  --model-path /path/to/wan2_2_mxfp8_diffusers \
+  --prompt "a beautiful sunset" \
+  --save-output
+```
+
+For the full quantization + format conversion workflow and a complete list of supported schemes, see [Diffusion Quantization on Ascend NPU](../hardware-platforms/ascend-npus/ascend_npu_quantization#diffusion-model-quantization-on-ascend-npu) and [SGLang-Diffusion Quantization](../sglang-diffusion/quantization#modelslim).
+
+## Reference
+
+- [GPTQModel](https://github.com/ModelCloud/GPTQModel)
+- [LLM Compressor](https://github.com/vllm-project/llm-compressor/)
+- [NVIDIA Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer)
+- [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq)
+- [Petit: NVFP4 on ROCm](https://github.com/causalflow-ai/petit-kernel) — [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/), [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html)
+- [Torchao: PyTorch Architecture Optimization](https://github.com/pytorch/ao)
+- [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/)
+- [auto-round](https://github.com/intel/auto-round)
+- [ModelSlim](https://gitcode.com/Ascend/msmodelslim)

docs_new/docs/advanced_features/quantized_kv_cache.mdx

@@ -0,0 +1,256 @@
+---
+title: "Quantized KV Cache"
+metatags:
+    description: "SGLang quantized KV cache: FP8 E4M3/E5M2 and FP4 E2M1 formats, memory savings up to 3.56x, scaling factors, accuracy benchmarks."
+---
+Quantized KV cache reduces the memory footprint of key-value cache storage by using lower-precision data types (FP8 or FP4) instead of the default model precision in BF16. During autoregressive generation, LLMs cache previously computed key-value pairs to avoid redundant calculations. The KV cache typically consumes a significant portion of GPU memory, especially for long sequences.
+
+Quantized KV cache is a memory optimization technique that primarily benefits throughput by allowing more tokens to be cached, but may introduce minimal accuracy degradation depending on the quantization format used.
+
+<Warning>
+**Performance Warning**: When quantized KV cache must be dequantized before use in attention operations, performance can be extremely slow if dequantization is not fused with the attention kernel. Always verify that your chosen attention backend supports quantized KV cache. Backends without fused support may experience significant throughput degradation, potentially negating the memory benefits.
+
+**Backend Support**: Not all attention backends support quantized KV cache. Refer to [Attention Backend](./attention_backend) for which backends support it.
+</Warning>
+
+## Supported Formats
+
+SGLang supports the following quantized KV cache formats:
+
+### FP8 Format
+
+[OCP (Open Compute Project)](https://www.opencompute.org) specifies two common 8-bit floating point formats:
+
+- **E5M2** (5 exponent bits, 2 mantissa bits): Larger dynamic range (±57344.0), lower precision
+- **E4M3** (4 exponent bits, 3 mantissa bits): Higher precision, smaller dynamic range (±240.0)
+
+### FP4 Format
+
+<Warning>
+FP4 quantization is currently experimental.
+</Warning>
+
+[OCP (Open Compute Project)](https://www.opencompute.org) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format:
+
+- **E2M1** (1 sign bit, 2 exponent bits, 1 mantissa bit): Uses block-based microscaling where tensors are divided into blocks of consecutive elements, with each block sharing a single 8-bit exponential scaling factor. While OCP specifies blocks of 32 elements, SGLang's current implementation uses blocks of 16 elements for KV cache quantization.
+
+## Usage
+
+### Enabling Quantized KV Cache
+
+To enable quantized KV cache, use the `--kv-cache-dtype` argument when launching the server:
+
+```bash Command
+# Enable FP8 E5M2 KV cache
+python3 -m sglang.launch_server \
+    --model-path deepseek-ai/DeepSeek-R1-0528 \
+    --kv-cache-dtype fp8_e5m2 \
+
+# Enable FP8 E4M3 KV cache
+python3 -m sglang.launch_server \
+    --model-path deepseek-ai/DeepSeek-R1-0528 \
+    --kv-cache-dtype fp8_e4m3 \
+
+# Enable FP4 E2M1 KV cache
+python3 -m sglang.launch_server \
+    --model-path nvidia/DeepSeek-R1-0528-NVFP4 \
+    --kv-cache-dtype fp4_e2m1 \
+```
+
+### Scaling Factors
+
+FP8 quantization requires scaling factors to properly quantize and dequantize the KV cache.
+
+<Note>
+Currently, only per-tensor (scalar) scaling factors are supported.
+</Note>
+
+Scaling factors can be:
+
+- **Loaded from checkpoints**: Pre-quantized models (e.g., ModelOpt) may include `k_scale` and `v_scale` parameters that are automatically loaded
+- **Provided via JSON**: Supply scaling factors via `--quantization-param-path`.
+
+The JSON file should follow this format:
+
+```json Config
+{
+  "kv_cache": {
+    "dtype": "float8_e4m3fn",
+    "scaling_factor": {
+      "0": {
+        "0": 1.0,
+        "1": 1.0
+      }
+    }
+  }
+}
+```
+
+Where the outer keys in `scaling_factor` are tensor parallel ranks and inner keys are layer indices.
+
+<Warning>
+If scaling factors are not provided and not found in the checkpoint, it will default to 1.0, which may cause accuracy issues.
+</Warning>
+
+<Tip>
+**FP4 (MXFP4)**: Unlike FP8, FP4 quantization handles scaling factors automatically on-the-fly during quantization and dequantization. No pre-quantized models or external scaling factor files are required—the block-based scaling factors are computed dynamically as needed.
+</Tip>
+
+## Performance Considerations
+
+### Memory Savings
+
+Quantized KV cache provides significant memory savings:
+- **BF16 → FP4**: Supports approximately 3.56× more tokens than BF16 (accounting for scaling factor overhead)
+
+<Note>
+FP4 and FP8 quantization require additional memory for block-based scaling factors, which reduces the effective memory savings compared to the raw bit-width reduction. FP4 with block size 16 supports approximately 1.78× more tokens than FP8, and approximately 3.56× more tokens than BF16. The relative token capacity between FP8 and BF16 can be derived from these ratios.
+</Note>
+
+This enables longer context lengths or more concurrent requests within the same memory budget.
+
+### Accuracy Impact
+
+#### FP8 Accuracy
+
+FP8 E4M3 quantization typically introduces minimal accuracy degradation. The impact depends on model architecture, sequence length, and quantization format (generally, E4M3 has better accuracy than E5M2).
+
+#### FP4 Accuracy
+
+FP4 (MXFP4) quantization provides significant memory savings with varying accuracy impact depending on model size and dataset complexity. Preliminary accuracy test results from [PR #10078](https://github.com/sgl-project/sglang/pull/10078) (MLA) and [PR #12612](https://github.com/sgl-project/sglang/pull/12612) (MHA) show:
+
+**Large Models (e.g., Qwen3-235B-A22B, DeepSeek-R1-0528)**
+
+On large-scale models, FP4 maintains accuracy close to FP8/BF16, especially on simpler datasets:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Dataset</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV16</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>KV8 (FP8 E4M3)</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV4 (FP4 E2M1)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gsm8k</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9168</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9181</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9186</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>aime25</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7733</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.7333</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6000</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gpqa_diamond</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7010</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.6899</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6778</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gsm8k</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9157</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9154</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9124</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>aime25</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.5067</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.4934</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.4000</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gpqa_diamond</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7707</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.7697</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7273</td>
+    </tr>
+  </tbody>
+</table>
+
+**Smaller Models (e.g., GPT-OSS-120B)**
+
+On smaller models, FP4 shows more pronounced accuracy drops, particularly on challenging datasets:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+    <col style={{width: "20%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Dataset</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV16</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>KV8 (FP8 E4M3)</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>KV4 (FP4 E2M1)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPT-OSS-120B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gsm8k</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9161</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.9163</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.9152</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPT-OSS-120B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>aime25</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.7533</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.7667</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.3533</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPT-OSS-120B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>gpqa_diamond</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.5081</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.5434</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.3202</td>
+    </tr>
+  </tbody>
+</table>
+
+**Key Observations:**
+
+- **Simple datasets (e.g., gsm8k)**: FP4 maintains accuracy close to FP8/BF16 across model sizes
+- **Model size matters**: Large models (200B+ parameters) generally tolerate FP4 quantization better than smaller models
+- **Context length**: Accuracy degradation may be more pronounced in long-context scenarios, as the accumulation of the quantization error may become significant.
+
+<Tip>
+Evaluate FP4 accuracy on your specific model and workload. Large models on simpler tasks typically show minimal degradation, while smaller models or complex reasoning tasks may require FP8 or BF16 for acceptable accuracy.
+</Tip>
+
+## Best Practices
+
+- **Use pre-quantized models**: Prefer models quantized offline with scaling factors included in the checkpoint.
+- **Choose the right format**: Use `fp8_e4m3` for better accuracy (recommended), `fp8_e5m2` for larger dynamic range, or `fp4_e2m1` for maximum memory savings (experimental)
+- **Check backend compatibility**: Verify that your chosen attention backend supports quantized KV cache
+
+<Note>
+See also:
+- [Quantization](./quantization)
+- [Attention Backend](./attention_backend)
+- [Server Arguments](./server_arguments)
+</Note>

docs_new/docs/advanced_features/rfork.mdx

@@ -0,0 +1,102 @@
+---
+title: "R-Fork"
+metatags:
+    description: "SGLang R-Fork: zero-copy GPU-to-GPU weight loading, reduce boot-up time from minutes to seconds. NCCL and TransferEngine backends."
+---
+R-Fork (Tensor Remote Fork) is a novel weight loading methodology that leverages efficient inter-node GPU-to-GPU data transfer path to load tensors from a running SGLang instance to a new instance with zero-copy. It can significantly optimize the SGLang instance boot-up time by reducing model weights loading from several minutes to mere seconds.
+
+To learn more details about R-Fork, please check **<a href="https://lmsys.org/blog/2025-12-10-rfork/"> R-Fork blog </a>**
+
+## Usage
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Usage</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>load-format</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>set to `remote_instance` to enable R-Fork.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-backend</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>nccl</code>, <code>transfer_engine</code>, or <code>modelexpress</code>. Default is <code>nccl</code>.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-seed-instance-ip</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>IP address of the seed instance who will provide the model weight. Used by <code>nccl</code> and <code>transfer_engine</code> backends.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-seed-instance-service-port</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the port that the seed instance's HTTP server is listening on. Used by <code>nccl</code> and <code>transfer_engine</code> backends.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-send-weights-group-ports</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the list of available ports on the seed instance that will be used to build NCCL communication groups between seed and client instance. Only needed by <code>nccl</code> backend.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-start-seed-via-transfer-engine</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>set to start seed service that supports TransferEngine as backend. Needed for seed instances when using <code>transfer_engine</code> as backend.</td>
+    </tr>
+<tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>modelexpress-config</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>JSON config for <code>modelexpress</code> backend. Keys: <code>"url"</code> (optional gRPC host:port override) and <code>"transport"</code> (<code>"nixl"</code> or <code>"transfer_engine"</code>, defaults to <code>"nixl"</code>).</td>
+    </tr>
+</tbody>
+</table>
+
+### NCCL as backend
+
+seed instance:
+```shell Command
+python -m sglang.launch_server [args]
+```
+
+client instance:
+```shell Command
+python -m sglang.launch_server [args] \
+  --load-format remote_instance \
+  --remote-instance-weight-loader-seed-instance-ip [seed_instance_ip] \
+  --remote-instance-weight-loader-seed-instance-service-port [seed_instance_service_port] \
+  --remote-instance-weight-loader-send-weights-group-ports [send_weights_nccl_group_ports_list]  \
+  --remote-instance-weight-loader-backend nccl
+```
+
+### TransferEngine as backend
+
+seed instance:
+```shell Command
+python -m sglang.launch_server [args] \
+  --remote-instance-weight-loader-start-seed-via-transfer-engine
+```
+
+```shell Command
+python -m sglang.launch_server [args] \
+  --load-format remote_instance \
+  --remote-instance-weight-loader-seed-instance-ip [seed_instance_ip] \
+  --remote-instance-weight-loader-seed-instance-service-port [seed_instance_service_port] \
+  --remote-instance-weight-loader-backend transfer_engine
+```
+
+### ModelExpress as backend
+
+[ModelExpress](https://github.com/ai-dynamo/modelexpress) is a coordination service that manages P2P weight transfer metadata. It removes the need for direct seed IP/port configuration by providing a centralized registry that instances publish to and discover from. The ModelExpress Python package must be installed in the SGLang image.
+
+A running ModelExpress server is required. See the [ModelExpress documentation](https://github.com/ai-dynamo/modelexpress) for setup instructions.
+
+server instance:
+```bash Command
+python -m sglang.launch_server [args] \
+  --load-format remote_instance \
+  --remote-instance-weight-loader-backend modelexpress \
+  --modelexpress-config '{"url": "[modelexpress_grpc_host:port]", "transport": "nixl"}'
+```
+
+All SGLang instances use the same command shape. If no ready source exists, the instance loads weights natively and publishes metadata to ModelExpress. If a compatible source exists, it loads weights through ModelExpress P2P transfer. Set <code>"transport": "transfer_engine"</code> to use Mooncake TransferEngine instead of the default NIXL transport.

docs_new/docs/advanced_features/separate_reasoning.ipynb

@@ -0,0 +1,381 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Reasoning Parser\n",
+    "\n",
+    "SGLang supports parsing reasoning content out from \"normal\" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).\n",
+    "\n",
+    "## Supported Models & Parsers\n",
+    "\n",
+    "| Model  |  Reasoning tags      | Parser | Notes |\n",
+    "|---------|-----------------------------|------------------|-------|\n",
+    "| [Apertus 2509 models](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509) | `<\\|inner_prefix\\|>` … `<\\|inner_suffix\\|>` | `apertus2509` | For tool use, also set `--tool-call-parser apertus2509`. |\n",
+    "| [DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) | `<think>` … `</think>` | `deepseek-r1` | Supports all variants (R1, R1-0528, R1-Distill) |\n",
+    "| [DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) | `<think>` … `</think>` | `deepseek-v3` | Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter |\n",
+    "| [Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | `<think>` … `</think>` | `qwen3` | Supports `enable_thinking` parameter |\n",
+    "| [Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507) | `<think>` … `</think>` | `qwen3` or `qwen3-thinking` | Always generates thinking content |\n",
+    "| [Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking) | `◁think▷` … `◁/think▷` | `kimi_k2` | Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use. |\n",
+    "| [GPT OSS](https://huggingface.co/openai/gpt-oss-120b) | `<\\|channel\\|>analysis<\\|message\\|>` … `<\\|end\\|>` | `gpt-oss` | N/A |\n",
+    "### Model-Specific Behaviors\n",
+    "\n",
+    "**Apertus 2509:**\n",
+    "- Uses `<|inner_prefix|>` and `<|inner_suffix|>` to delimit reasoning content. For agentic tool use, also specify `--tool-call-parser apertus2509`.\n",
+    "\n",
+    "**DeepSeek-R1 Family:**\n",
+    "- DeepSeek-R1: No `<think>` start tag, jumps directly to thinking content\n",
+    "- DeepSeek-R1-0528: Generates both `<think>` start and `</think>` end tags\n",
+    "- Both are handled by the same `deepseek-r1` parser\n",
+    "\n",
+    "**DeepSeek-V3 Family:**\n",
+    "- DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`)\n",
+    "\n",
+    "**Qwen3 Family:**\n",
+    "- Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates\n",
+    "- Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks\n",
+    "\n",
+    "**Kimi K2:**\n",
+    "- Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`.\n",
+    "\n",
+    "**GPT OSS:**\n",
+    "- GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Usage\n",
+    "\n",
+    "### Launching the Server"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Specify the `--reasoning-parser` option."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "from openai import OpenAI\n",
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that `--reasoning-parser` defines the parser used to interpret responses."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### OpenAI Compatible API\n",
+    "\n",
+    "Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1:\n",
+    "\n",
+    "- `reasoning_content`: The content of the CoT.\n",
+    "- `content`: The content of the final answer."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize OpenAI-like client\n",
+    "client = OpenAI(api_key=\"None\", base_url=f\"http://0.0.0.0:{port}/v1\")\n",
+    "model_name = client.models.list().data[0].id\n",
+    "\n",
+    "messages = [\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"What is 1+3?\",\n",
+    "    }\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Non-Streaming Request"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response_non_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0.6,\n",
+    "    top_p=0.95,\n",
+    "    stream=False,  # Non-streaming\n",
+    "    extra_body={\"separate_reasoning\": True},\n",
+    ")\n",
+    "print_highlight(\"==== Reasoning ====\")\n",
+    "print_highlight(response_non_stream.choices[0].message.reasoning_content)\n",
+    "\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(response_non_stream.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Streaming Request"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0.6,\n",
+    "    top_p=0.95,\n",
+    "    stream=True,  # Non-streaming\n",
+    "    extra_body={\"separate_reasoning\": True},\n",
+    ")\n",
+    "\n",
+    "reasoning_content = \"\"\n",
+    "content = \"\"\n",
+    "for chunk in response_stream:\n",
+    "    if chunk.choices[0].delta.content:\n",
+    "        content += chunk.choices[0].delta.content\n",
+    "    if chunk.choices[0].delta.reasoning_content:\n",
+    "        reasoning_content += chunk.choices[0].delta.reasoning_content\n",
+    "\n",
+    "print_highlight(\"==== Reasoning ====\")\n",
+    "print_highlight(reasoning_content)\n",
+    "\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0.6,\n",
+    "    top_p=0.95,\n",
+    "    stream=True,  # Non-streaming\n",
+    "    extra_body={\"separate_reasoning\": True, \"stream_reasoning\": False},\n",
+    ")\n",
+    "\n",
+    "reasoning_content = \"\"\n",
+    "content = \"\"\n",
+    "for chunk in response_stream:\n",
+    "    if chunk.choices[0].delta.content:\n",
+    "        content += chunk.choices[0].delta.content\n",
+    "    if chunk.choices[0].delta.reasoning_content:\n",
+    "        reasoning_content += chunk.choices[0].delta.reasoning_content\n",
+    "\n",
+    "print_highlight(\"==== Reasoning ====\")\n",
+    "print_highlight(reasoning_content)\n",
+    "\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The reasoning separation is enable by default when specify . \n",
+    "**To disable it, set the `separate_reasoning` option to `False` in request.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response_non_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0.6,\n",
+    "    top_p=0.95,\n",
+    "    stream=False,  # Non-streaming\n",
+    "    extra_body={\"separate_reasoning\": False},\n",
+    ")\n",
+    "\n",
+    "print_highlight(\"==== Original Output ====\")\n",
+    "print_highlight(response_non_stream.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### SGLang Native API "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
+    "input = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "\n",
+    "gen_url = f\"http://localhost:{port}/generate\"\n",
+    "gen_data = {\n",
+    "    \"text\": input,\n",
+    "    \"sampling_params\": {\n",
+    "        \"skip_special_tokens\": False,\n",
+    "        \"max_new_tokens\": 1024,\n",
+    "        \"temperature\": 0.6,\n",
+    "        \"top_p\": 0.95,\n",
+    "    },\n",
+    "}\n",
+    "gen_response = requests.post(gen_url, json=gen_data).json()[\"text\"]\n",
+    "\n",
+    "print_highlight(\"==== Original Output ====\")\n",
+    "print_highlight(gen_response)\n",
+    "\n",
+    "parse_url = f\"http://localhost:{port}/separate_reasoning\"\n",
+    "separate_reasoning_data = {\n",
+    "    \"text\": gen_response,\n",
+    "    \"reasoning_parser\": \"deepseek-r1\",\n",
+    "}\n",
+    "separate_reasoning_response_json = requests.post(\n",
+    "    parse_url, json=separate_reasoning_data\n",
+    ").json()\n",
+    "print_highlight(\"==== Reasoning ====\")\n",
+    "print_highlight(separate_reasoning_response_json[\"reasoning_text\"])\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(separate_reasoning_response_json[\"text\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Offline Engine API"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sglang as sgl\n",
+    "from sglang.srt.parser.reasoning_parser import ReasoningParser\n",
+    "from sglang.utils import print_highlight\n",
+    "\n",
+    "llm = sgl.Engine(model_path=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
+    "input = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "sampling_params = {\n",
+    "    \"max_new_tokens\": 1024,\n",
+    "    \"skip_special_tokens\": False,\n",
+    "    \"temperature\": 0.6,\n",
+    "    \"top_p\": 0.95,\n",
+    "}\n",
+    "result = llm.generate(prompt=input, sampling_params=sampling_params)\n",
+    "\n",
+    "generated_text = result[\"text\"]  # Assume there is only one prompt\n",
+    "\n",
+    "print_highlight(\"==== Original Output ====\")\n",
+    "print_highlight(generated_text)\n",
+    "\n",
+    "parser = ReasoningParser(\"deepseek-r1\")\n",
+    "reasoning_text, text = parser.parse_non_stream(generated_text)\n",
+    "print_highlight(\"==== Reasoning ====\")\n",
+    "print_highlight(reasoning_text)\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm.shutdown()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Supporting New Reasoning Model Schemas\n",
+    "\n",
+    "For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly."
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs_new/docs/advanced_features/separate_reasoning.mdx

@@ -0,0 +1,326 @@
+---
+title: "Reasoning Parser"
+metatags:
+    description: "SGLang reasoning parser: separate thinking content from output for DeepSeek R1, Qwen3, Kimi K2, GPT-OSS, and Apertus reasoning models."
+---
+SGLang supports parsing reasoning content out from "normal" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).
+
+## Supported Models & Parsers
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning tags</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parser</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Apertus 2509 models](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<|inner_prefix|>` … `<|inner_suffix|>`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`apertus2509`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Supports `enable_thinking` parameter</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseek-r1`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Supports all variants (R1, R1-0528, R1-Distill)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseek-v3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Supports `enable_thinking` parameter</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`<think>` … `</think>`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3` or `qwen3-thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Always generates thinking content</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`◁think▷` … `◁/think▷`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`kimi_k2`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[GPT OSS](https://huggingface.co/openai/gpt-oss-120b)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`&lt;|channel|&gt;analysis&lt;|message|&gt;` … `&lt;|end|&gt;`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`gpt-oss`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
+    </tr>
+  </tbody>
+</table>
+### Model-Specific Behaviors
+
+**Apertus 2509:**
+- Uses `<|inner_prefix|>` and `<|inner_suffix|>` to delimit reasoning content. For agentic tool use, also specify `--tool-call-parser apertus2509`.
+
+**DeepSeek-R1 Family:**
+- DeepSeek-R1: No `<think>` start tag, jumps directly to thinking content
+- DeepSeek-R1-0528: Generates both `<think>` start and `</think>` end tags
+- Both are handled by the same `deepseek-r1` parser
+
+**DeepSeek-V3 Family:**
+- DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`)
+
+**Qwen3 Family:**
+- Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates
+- Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks
+
+**Kimi K2:**
+- Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`.
+
+**GPT OSS:**
+- GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags
+
+
+## Usage
+
+### Launching the Server
+
+
+Specify the `--reasoning-parser` option.
+
+
+
+```python Example
+import requests
+from openai import OpenAI
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}")
+```
+
+Note that `--reasoning-parser` defines the parser used to interpret responses.
+
+
+### OpenAI Compatible API
+
+Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1:
+
+- `reasoning_content`: The content of the CoT.
+- `content`: The content of the final answer.
+
+
+
+```python Example
+# Initialize OpenAI-like client
+client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1")
+model_name = client.models.list().data[0].id
+
+messages = [
+    {
+        "role": "user",
+        "content": "What is 1+3?",
+    }
+]
+```
+
+#### Non-Streaming Request
+
+
+
+```python Example
+response_non_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0.6,
+    top_p=0.95,
+    stream=False,  # Non-streaming
+    extra_body={"separate_reasoning": True},
+)
+print_highlight("==== Reasoning ====")
+print_highlight(response_non_stream.choices[0].message.reasoning_content)
+
+print_highlight("==== Text ====")
+print_highlight(response_non_stream.choices[0].message.content)
+```
+
+#### Streaming Request
+
+
+
+```python Example
+response_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0.6,
+    top_p=0.95,
+    stream=True,  # Non-streaming
+    extra_body={"separate_reasoning": True},
+)
+
+reasoning_content = ""
+content = ""
+for chunk in response_stream:
+    if chunk.choices[0].delta.content:
+        content += chunk.choices[0].delta.content
+    if chunk.choices[0].delta.reasoning_content:
+        reasoning_content += chunk.choices[0].delta.reasoning_content
+
+print_highlight("==== Reasoning ====")
+print_highlight(reasoning_content)
+
+print_highlight("==== Text ====")
+print_highlight(content)
+```
+
+Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content).
+
+
+
+```python Example
+response_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0.6,
+    top_p=0.95,
+    stream=True,  # Non-streaming
+    extra_body={"separate_reasoning": True, "stream_reasoning": False},
+)
+
+reasoning_content = ""
+content = ""
+for chunk in response_stream:
+    if chunk.choices[0].delta.content:
+        content += chunk.choices[0].delta.content
+    if chunk.choices[0].delta.reasoning_content:
+        reasoning_content += chunk.choices[0].delta.reasoning_content
+
+print_highlight("==== Reasoning ====")
+print_highlight(reasoning_content)
+
+print_highlight("==== Text ====")
+print_highlight(content)
+```
+
+The reasoning separation is enable by default when specify .
+**To disable it, set the `separate_reasoning` option to `False` in request.**
+
+
+
+```python Example
+response_non_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0.6,
+    top_p=0.95,
+    stream=False,  # Non-streaming
+    extra_body={"separate_reasoning": False},
+)
+
+print_highlight("==== Original Output ====")
+print_highlight(response_non_stream.choices[0].message.content)
+```
+
+### SGLang Native API
+
+
+
+```python Example
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
+input = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+
+gen_url = f"http://localhost:{port}/generate"
+gen_data = {
+    "text": input,
+    "sampling_params": {
+        "skip_special_tokens": False,
+        "max_new_tokens": 1024,
+        "temperature": 0.6,
+        "top_p": 0.95,
+    },
+}
+gen_response = requests.post(gen_url, json=gen_data).json()["text"]
+
+print_highlight("==== Original Output ====")
+print_highlight(gen_response)
+
+parse_url = f"http://localhost:{port}/separate_reasoning"
+separate_reasoning_data = {
+    "text": gen_response,
+    "reasoning_parser": "deepseek-r1",
+}
+separate_reasoning_response_json = requests.post(
+    parse_url, json=separate_reasoning_data
+).json()
+print_highlight("==== Reasoning ====")
+print_highlight(separate_reasoning_response_json["reasoning_text"])
+print_highlight("==== Text ====")
+print_highlight(separate_reasoning_response_json["text"])
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+### Offline Engine API
+
+
+
+```python Example
+import sglang as sgl
+from sglang.srt.parser.reasoning_parser import ReasoningParser
+from sglang.utils import print_highlight
+
+llm = sgl.Engine(model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
+tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
+input = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+sampling_params = {
+    "max_new_tokens": 1024,
+    "skip_special_tokens": False,
+    "temperature": 0.6,
+    "top_p": 0.95,
+}
+result = llm.generate(prompt=input, sampling_params=sampling_params)
+
+generated_text = result["text"]  # Assume there is only one prompt
+
+print_highlight("==== Original Output ====")
+print_highlight(generated_text)
+
+parser = ReasoningParser("deepseek-r1")
+reasoning_text, text = parser.parse_non_stream(generated_text)
+print_highlight("==== Reasoning ====")
+print_highlight(reasoning_text)
+print_highlight("==== Text ====")
+print_highlight(text)
+```
+
+
+```python Example
+llm.shutdown()
+```
+
+## Supporting New Reasoning Model Schemas
+
+For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly.

docs_new/docs/advanced_features/sglang_for_rl.mdx

@@ -0,0 +1,655 @@
+---
+title: "SGLang for RL Systems"
+metatags:
+    description: "SGLang for RL: engine sleep/wake, weight refit, partial rollout, deterministic inference, cache-aware load balancing for RLHF."
+---
+This document is a practical guide for infrastructure teams integrating SGLang into RL and post-training systems. It focuses on the operational pain points in the loop (rollout, evaluation, training, weight sync) and maps them to concrete SGLang APIs, flags, and integration patterns. The focus is on maximizing rollout efficiency, accuracy and stability while keeping rollout-serving behavior aligned in production environments.
+
+## Why SGLang for RL Lifecycle?
+
+Let's embrace a guiding principle from early DeepMind's RL engineering:
+
+**Be a library, not a framework.**
+
+This philosophy empowers innovation by providing SGLang as flexible tools, not rigid structures. Here are five reasons to use SGLang for your RL lifecycle:
+
+* **Fine-Grained Engine Sleep and Wake Up**: facilitate maximum-powered rollout and training
+* **Open-To-Use Refit Functionality**: diverse methods for co-location or disaggregation
+* **Easy To Postpone Generation**: enable partial rollout and dedicated rollout control
+* **Deterministic Inference**: achieve deterministic inference to enable zero training-inference mismatch
+* **Load Balancing Router**: cache-aware load-balancing for high-throughput rollout
+
+The following sections cover these aspects in detail.
+
+## Fine-Grained Engine Sleep and Wake Up
+
+Rollout and training are both memory-intensive, and co-locating them on the same GPUs often leads to memory pressure and slow handoffs. SGLang provides a memory-aware sleep/wake mechanism that releases KV cache and weights while keeping the server process alive, then resumes them for rollout without a full restart. This avoids repeated disk I/O and CUDA graph recapture during each RL step.
+
+Under the hood, the RL team uses CUDA-graph-aware weight offload via [torch_memory_saver](https://github.com/fzyzcjy/torch_memory_saver) to preserve virtual memory addresses for graph replay. For details, see: [Efficient RL Training - Optimizing Memory Usage in verl](https://hebiao064.github.io/rl-memory-management).
+
+### Server flag
+
+Enable memory saver support when launching the server:
+
+```text Output
+--enable-memory-saver
+```
+
+### Release Memory
+
+**Endpoint:** `POST /release_memory_occupation`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`tags`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Which memory regions to release. If omitted, all are released.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str], values: `kv_cache`, `weights`</td>
+    </tr>
+  </tbody>
+</table>
+{/* python/sglang/srt/managers/io_struct.py#L1381 currently only supports `kv_cache`, `weights` */}
+**Behavior notes:**
+
+- This call asserts there are no ongoing requests. Ensure the engine is idle before calling it.
+- If `kv_cache` is released, SGLang flushes cache; subsequent requests will rebuild KV cache as needed.
+
+### Resume Memory
+
+**Endpoint:** `POST /resume_memory_occupation`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`tags`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Which memory regions to resume. If omitted, all are resumed.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str], values: `kv_cache`, `weights`</td>
+    </tr>
+  </tbody>
+</table>
+{/* python/sglang/srt/managers/io_struct.py#L1393 currently only supports `kv_cache`, `weights` */}
+
+## Open-To-Use Refit Functionality
+
+After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang's weight update utilities, see [RL System Deep Thinking: Weight Update Mechanisms](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN.md).
+
+**How to choose:**
+
+- **From disk** is simplest and best for elastic rollout scaling and checkpointing.
+- **From tensor** is best for co-located training/rollout when you can pass in-memory tensors.
+- **From distributed** is best for disaggregated training/rollout with dedicated communication groups (NCCL/IB).
+
+### Update Weights from Disk
+
+**When to use:**
+
+- Save checkpoint to disk and update weights from disk
+- Dynamic scaling (new rollout instances can load from the same checkpoint)
+
+**Why it works well:**
+
+This path trades some I/O overhead for simplicity and flexibility. It integrates naturally with checkpointing and makes it trivial to add new rollout engines: point them at the same checkpoint and call the API. It is also the safest option for high availability because the checkpoint itself is the source of truth.
+
+**Endpoint:** `POST /update_weights_from_disk`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`model_path`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The model path with the new weights.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`load_format`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The format to load the weights.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`abort_all_requests`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Abort all running requests before update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_version`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional weight version label tracked by the server.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`is_async`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Perform weight load asynchronously.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`torch_empty_cache`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Empty torch cache.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`keep_pause`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Keep scheduler paused after update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`recapture_cuda_graph`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Recapture CUDA graphs after update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`token_step`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Trainer step id for rollout bookkeeping.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flush_cache`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush KV cache after update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`True`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+  </tbody>
+</table>
+
+**Response body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`success`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Whether the update succeeded.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`message`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Status / error message.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`num_paused_requests`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of paused requests during update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
+    </tr>
+  </tbody>
+</table>
+
+**Python Engine API:** `engine.update_weights_from_disk(model_path, load_format=None)`
+
+**Diffusion engine (SGLang-Diffusion):** The diffusion engine exposes the same `POST /update_weights_from_disk` endpoint with the following behavior:
+
+- **All-or-nothing with rollback:** if any module fails to load, all previously updated modules are rolled back to the original weights by reloading from the original model path. No partial updates are left behind. If rollback itself fails, the exception propagates so the caller knows the model is in an inconsistent state.
+- **Offload-aware:** when layerwise offload (`--dit-layerwise-offload`) is enabled, the diffusion offload manager replaces GPU parameters with small `torch.empty((1,))` placeholders while real weights live in consolidated pinned CPU buffers. A naive `param.data.copy_()` would fail with a shape mismatch. Instead, the updater dynamically detects active offload managers and writes new weights directly into their CPU buffers, bypassing the placeholders entirely. For any layer that happens to be prefetched on GPU at update time, the live GPU tensor is also updated so the change takes effect immediately. This requires no extra GPU memory and does not disturb the offload state.
+- **DTensor-aware:** parameters distributed via `torch.distributed.tensor` (tensor parallelism) are updated through `distribute_tensor` so that each shard is correctly placed on the right device mesh.
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>model_path</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The model path with the new weights.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>flush_cache</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush TeaCache state after update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>True</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>target_modules</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List of module names to update (e.g. <code>["transformer"]</code>). If omitted, all <code>nn.Module</code> components are updated.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str]</td>
+    </tr>
+  </tbody>
+</table>
+
+**Response body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>success</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Whether the update succeeded.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>message</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Status / error message.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+  </tbody>
+</table>
+
+> **Note:** The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently.
+
+### Update Weights from Tensor
+
+**When to use:**
+
+- Co-located training and rollout, where training can provide tensors directly
+- Fast in-memory updates
+
+**Important constraints:**
+
+This strategy requires the training process and rollout engine to share access to the tensors. Co-located setups must keep the model on GPU; moving tensors to CPU will break the update path. For high-performance MoE or specialized attention kernels, co-location may limit some optimizations compared to disaggregated rollouts.
+
+**Endpoint:** `POST /update_weights_from_tensor`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>serialized_named_tensors</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Per-TP serialized tensor payloads.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str|bytes]</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>load_format</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional load format selector.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code>, <code>direct</code>, <code>flattened_bucket</code>, or a custom loader path string</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>flush_cache</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush KV cache after update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>True</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>abort_all_requests</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Abort all running requests before update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>weight_version</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional version label tracked by the server.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+  </tbody>
+</table>
+
+**Note:** The serialized tensor payloads must be created with `MultiprocessingSerializer.serialize(...)` and should be base64-safe strings.
+
+**Python Engine API:** `engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)`
+
+### Update Weights from Distributed Group
+
+**When to use:**
+
+- Disaggregated training and rollout
+- NCCL or IB-backed weight broadcast from training workers to rollout workers
+
+**How it works:**
+
+Training workers gather weights (typically on TP rank 0), broadcast them to the rollout group, and each rollout TP shard loads the parameters it needs. This avoids disk I/O and keeps training and rollout decoupled, at the cost of managing a dedicated communication group.
+
+**Initialize weight update group**
+
+**Endpoint:** `POST /init_weights_update_group`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`master_address`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group master address.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`master_port`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group master port.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`rank_offset`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Offset for local rank mapping.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`world_size`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Total world size.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`group_name`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group name.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_update_group`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`backend`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Communication backend.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`nccl`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+  </tbody>
+</table>
+
+**Update weight**
+
+**Endpoint:** `POST /update_weights_from_distributed`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`names`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameter names to update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str]</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`dtypes`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Dtype strings for each parameter.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str]</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`shapes`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor shapes.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[list[int]]</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`group_name`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group name.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_update_group`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flush_cache`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush KV cache after update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`True`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`abort_all_requests`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Abort all running requests before update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_version`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional version label.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`load_format`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional format selector.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None` or `flattened_bucket`</td>
+    </tr>
+  </tbody>
+</table>
+
+**Destroy weights update group**
+
+**Endpoint:** `POST /destroy_weights_update_group`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`group_name`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Group name.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_update_group`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+  </tbody>
+</table>
+
+**Python Engine APIs:**
+
+- `engine.init_weights_update_group(...)`
+- `engine.update_weights_from_distributed(names, dtypes, shapes, ...)`
+- `engine.destroy_weights_update_group(group_name)`
+
+## Easy To Postpone Generation
+
+Multi-turn RL rollouts often suffer from long-tail requests that block the entire batch. A small number of slow interactions can stall all GPUs, and the long-tail behavior makes profiling and monitoring difficult.
+
+SGLang exposes explicit pause/resume APIs so you can pause slow requests and continue them later. This pattern matches systems like [APRIL](https://arxiv.org/abs/2509.18521), terminate once enough responses are collected, and recycle incomplete responses in the next step. The result is higher GPU utilization without discarding partial work.
+
+`pause_generation` ---  update weights --- `continue_generation` is the correct execution flow when updating weights from training. An update can only happen when SGLang is not actively processing inference tasks.
+
+### Pause Generation
+
+**Endpoint:** `POST /pause_generation`
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`mode`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Pause mode.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`abort`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`abort`, `retract`, `in_place`</td>
+    </tr>
+  </tbody>
+</table>
+
+**Modes:**
+
+- `abort`: Default behavior, identical to `abort` endpoint with `abort_all` set. Pending requests from `waiting_queue` and `running_queue` will be returned immediately to the caller.
+- `retract`: Put engine in "paused" state.  Move running requests back to waiting queue. KV cache can be flushed and recomputed later.
+- `in_place`: Put engine in "paused" state without changing states of the requests. Running requests rely on availability of KV caches to continue, so any subsequent `flush_cache` call will be unsuccessful.
+
+### Continue Generation
+
+**Endpoint:** `POST /continue_generation`
+
+## Deterministic Inference
+
+In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem.
+
+SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: [True On-Policy](https://github.com/radixark/miles/tree/main/examples/true_on_policy) and [True On-Policy for VLM](https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm). For additional context, see the blog post [Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md).
+
+**Server flag:**
+
+```text Output
+--enable-deterministic-inference
+```
+
+For more details, see [Deterministic Inference](./deterministic_inference)
+
+## Load Balancing Router
+
+SGLang Model Gateway is the recommended control plane for large‑scale RL rollouts. It provides async, non‑blocking request handling, cache‑aware load balancing, and fault‑tolerant routing across rollout and reward servers. This lets you keep GPUs saturated while avoiding long‑tail stalls and brittle, engine‑local concurrency logic. It has been deployed in the training of GLM 4.5+ models and proven to be highly efficient in production-level large-scale RL workloads.
+
+Key benefits for RL infrastructure:
+
+- **Async non-blocking efficiency**: SGLang’s native async server/router architecture (HTTPS/gRPC) manages concurrency automatically. This guarantees maximum GPU saturation and effective continuous batching without requiring complex, manual implementation by engineers.
+- **Elasticity and fault tolerance**: By encapsulating the reward model and rollout as independent servers, SGLang decouples them logically and physically. This architecture provides robust disaster recovery for large-scale distributed training; if a server fails, the router automatically redirects traffic to healthy nodes, ensuring the training process continues without interruption.
+- **Training–Inference alignment**: Using the SGLang Model Gateway for both training and inference ensures "What You See Is What You Get." This eliminates score discrepancies and the painful backend alignment issues often caused by using different engines for training versus deployment.
+- **Dynamic load balancing and long-tail mitigation**: Unlike static partitioning, the SGLang Model Gateway enables request-level dynamic dispatching for multi-turn RL. It can distribute different turns of a conversation across different servers to balance workloads and eliminate long-tail latency caused by varying sequence lengths.
+
+For deployment and configuration, see: [SGLang Model Gateway](./sgl_model_gateway)

docs_new/docs/advanced_features/speculative_decoding.ipynb

@@ -0,0 +1,360 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Speculative Decoding\n",
+    "\n",
+    "SGLang now provides an EAGLE-based (EAGLE-2/EAGLE-3) speculative decoding option. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.\n",
+    "\n",
+    "### Performance Highlights\n",
+    "\n",
+    "Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding.\n",
+    "For further details please see the [EAGLE3 paper](https://arxiv.org/pdf/2503.01840).\n",
+    "\n",
+    "| Method | Throughput (tokens/s) |\n",
+    "|--------|----------------|\n",
+    "| SGLang (w/o speculative, 1x H100) | 158.34 tokens/s |\n",
+    "| SGLang + EAGLE-2 (1x H100) | 244.10 tokens/s |\n",
+    "| SGLang + EAGLE-3 (1x H100) | 373.25 tokens/s |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## EAGLE Decoding\n",
+    "\n",
+    "To enable EAGLE speculative decoding the following parameters are relevant:\n",
+    "* `speculative_draft_model_path`: Specifies draft model. This parameter is required.\n",
+    "* `speculative_num_steps`: Depth of autoregressive drafting. Increases speculation range but risks rejection cascades. Default is 5.\n",
+    "* `speculative_eagle_topk`: Branching factor per step. Improves candidate diversity, will lead to higher acceptance rate, but more lead to higher memory/compute consumption. Default is 4.\n",
+    "* `speculative_num_draft_tokens`: Maximum parallel verification capacity. Allows deeper tree evaluation but will lead to higher GPU memory usage. Default is 8.\n",
+    "\n",
+    "These parameters are the same for EAGLE-2 and EAGLE-3.\n",
+    "\n",
+    "You can find the best combinations of these parameters with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py).\n",
+    "\n",
+    "In the documentation below, we set `--cuda-graph-max-bs-decode` to be a small value for faster engine startup. For your own workloads, please tune the above parameters together with `--cuda-graph-max-bs-decode`, `--max-running-requests`, `--mem-fraction-static` for the best performance. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EAGLE-2 decoding\n",
+    "\n",
+    "You can enable EAGLE-2 decoding by setting `--speculative-algorithm EAGLE` and choosing an appropriate model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "import openai"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf  --speculative-algorithm EAGLE \\\n",
+    "    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 3 \\\n",
+    "    --speculative-eagle-topk 4 --speculative-num-draft-tokens 16 --cuda-graph-max-bs-decode 8 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Llama-2-7b-chat-hf\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EAGLE-2 Decoding with `torch.compile`\n",
+    "\n",
+    "You can also enable `torch.compile` for further optimizations and optionally set `--torch-compile-max-bs`:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model meta-llama/Llama-2-7b-chat-hf  --speculative-algorithm EAGLE \\\n",
+    "    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B --speculative-num-steps 5 \\\n",
+    "        --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.6 \\\n",
+    "            --enable-torch-compile --torch-compile-max-bs 2 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Llama-2-7b-chat-hf\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling\n",
+    "\n",
+    "By employing a truncated high-frequency token vocabulary in the draft model, Eagle speculative decoding reduces `lm_head` computational overhead while accelerating the pipeline without quality degradation. For more details, checkout [the paper](https://arxiv.org/pdf/arXiv:2502.14856).\n",
+    "\n",
+    "In our implementation, set `--speculative-token-map` to enable the optimization. You can get the high-frequency token in FR-Spec from [this model](https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec). Or you can obtain high-frequency token by directly downloading these token from [this repo](https://github.com/thunlp/FR-Spec/tree/main?tab=readme-ov-file#prepare-fr-spec-vocabulary-subset).\n",
+    "\n",
+    "Thanks for the contribution from [Weilin Zhao](https://github.com/Achazwl) and [Zhousx](https://github.com/Zhou-sx). "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model meta-llama/Meta-Llama-3-8B-Instruct --speculative-algorithm EAGLE \\\n",
+    "    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \\\n",
+    "    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \\\n",
+    "    --mem-fraction 0.7 --cuda-graph-max-bs-decode 2 --dtype float16  --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EAGLE-3 Decoding\n",
+    "\n",
+    "You can enable EAGLE-3 decoding by setting `--speculative-algorithm EAGLE3` and choosing an appropriate model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct  --speculative-algorithm EAGLE3 \\\n",
+    "    --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B --speculative-num-steps 5 \\\n",
+    "        --speculative-eagle-topk 8 --speculative-num-draft-tokens 32 --mem-fraction 0.6 \\\n",
+    "        --cuda-graph-max-bs-decode 2 --dtype float16 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multi Token Prediction\n",
+    "\n",
+    "We support [MTP(Multi-Token Prediction)](https://arxiv.org/pdf/2404.19737) in SGLang by using speculative decoding. We use Xiaomi/MiMo-7B-RL model as example here (deepseek mtp usage refer to [deepseek doc](../basic_usage/deepseek.md#multi-token-prediction))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "    python3 -m sglang.launch_server --model-path XiaomiMiMo/MiMo-7B-RL --host 0.0.0.0 --trust-remote-code \\\n",
+    "    --speculative-algorithm EAGLE --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \\\n",
+    "    --mem-fraction 0.5 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/chat/completions\"\n",
+    "\n",
+    "data = {\n",
+    "    \"model\": \"XiaomiMiMo/MiMo-7B-RL\",\n",
+    "    \"messages\": [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## References\n",
+    "\n",
+    "EAGLE process is as follows:\n",
+    "\n",
+    "- Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence $(f_1, ..., f_k)$ and the token sequence $(t_2, ..., t_{k+1})$. \n",
+    "- The next token is then sampled from $p_{k+2}=\\text{LMHead}(f_{k+1})$. Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the `speculative_eagle_topk` parameter—to ensure a more coherent connection of context, and are given as input again.\n",
+    "- EAGLE-2 additionally uses the draft model to evaluate how probable certain branches in the draft tree are, dynamically stopping the expansion of unlikely branches. After the expansion phase, reranking is employed to select only the top `speculative_num_draft_tokens` final nodes as draft tokens.\n",
+    "- EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner.\n",
+    "\n",
+    "This enhances drafting accuracy by operating on the features instead of tokens for more regular inputs and passing the tokens from the next timestep additionally to minimize randomness effects from sampling. Furthermore the dynamic adjustment of the draft tree and selection of reranked final nodes increases acceptance rate of draft tokens further. For more details see [EAGLE-2](https://arxiv.org/abs/2406.16858) and [EAGLE-3](https://arxiv.org/abs/2503.01840) paper.\n",
+    "\n",
+    "\n",
+    "For guidance how to train your own EAGLE model please see the [EAGLE repo](https://github.com/SafeAILab/EAGLE/tree/main?tab=readme-ov-file#train)."
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/advanced_features/structured_outputs.ipynb

@@ -0,0 +1,994 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Structured Outputs"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.\n",
+    "\n",
+    "SGLang supports three grammar backends:\n",
+    "\n",
+    "- [XGrammar](https://github.com/mlc-ai/xgrammar)(default): Supports JSON schema, regular expression, and EBNF constraints.\n",
+    "- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.\n",
+    "- [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints.\n",
+    "\n",
+    "We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).\n",
+    "\n",
+    "To use Outlines, simply add `--grammar-backend outlines` when launching the server.\n",
+    "To use llguidance, add `--grammar-backend llguidance`  when launching the server.\n",
+    "If no backend is specified, XGrammar will be used as the default.\n",
+    "\n",
+    "For better output quality, **It's advisable to explicitly include instructions in the prompt to guide the model to generate the desired format.** For example, you can specify, 'Please generate the output in the following JSON format: ...'.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## OpenAI Compatible API"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "import os\n",
+    "\n",
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
+    "\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### JSON\n",
+    "\n",
+    "you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Using Pydantic**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pydantic import BaseModel, Field\n",
+    "\n",
+    "\n",
+    "# Define the schema using Pydantic\n",
+    "class CapitalInfo(BaseModel):\n",
+    "    name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
+    "    population: int = Field(..., description=\"Population of the capital city\")\n",
+    "\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Please generate the information of the capital of France in the JSON format.\",\n",
+    "        },\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=128,\n",
+    "    response_format={\n",
+    "        \"type\": \"json_schema\",\n",
+    "        \"json_schema\": {\n",
+    "            \"name\": \"foo\",\n",
+    "            # convert the pydantic model to json schema\n",
+    "            \"schema\": CapitalInfo.model_json_schema(),\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "response_content = response.choices[0].message.content\n",
+    "# validate the JSON response by the pydantic model\n",
+    "capital_info = CapitalInfo.model_validate_json(response_content)\n",
+    "print_highlight(f\"Validated response: {capital_info.model_dump_json()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**JSON Schema Directly**\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "json_schema = json.dumps(\n",
+    "    {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
+    "            \"population\": {\"type\": \"integer\"},\n",
+    "        },\n",
+    "        \"required\": [\"name\", \"population\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Give me the information of the capital of France in the JSON format.\",\n",
+    "        },\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=128,\n",
+    "    response_format={\n",
+    "        \"type\": \"json_schema\",\n",
+    "        \"json_schema\": {\"name\": \"foo\", \"schema\": json.loads(json_schema)},\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EBNF"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ebnf_grammar = \"\"\"\n",
+    "root ::= city | description\n",
+    "city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\n",
+    "description ::= city \" is \" status\n",
+    "status ::= \"the capital of \" country\n",
+    "country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"\n",
+    "\"\"\"\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful geography bot.\"},\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Give me the information of the capital of France.\",\n",
+    "        },\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=32,\n",
+    "    extra_body={\"ebnf\": ebnf_grammar},\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Regular expression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"What is the capital of France?\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=128,\n",
+    "    extra_body={\"regex\": \"(Paris|London)\"},\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Structural Tag"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tool_get_current_weather = {\n",
+    "    \"type\": \"function\",\n",
+    "    \"function\": {\n",
+    "        \"name\": \"get_current_weather\",\n",
+    "        \"description\": \"Get the current weather in a given location\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"city\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
+    "                },\n",
+    "                \"state\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"the two-letter abbreviation for the state that the city is\"\n",
+    "                    \" in, e.g. 'CA' which would mean 'California'\",\n",
+    "                },\n",
+    "                \"unit\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"The unit to fetch the temperature in\",\n",
+    "                    \"enum\": [\"celsius\", \"fahrenheit\"],\n",
+    "                },\n",
+    "            },\n",
+    "            \"required\": [\"city\", \"state\", \"unit\"],\n",
+    "        },\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "tool_get_current_date = {\n",
+    "    \"type\": \"function\",\n",
+    "    \"function\": {\n",
+    "        \"name\": \"get_current_date\",\n",
+    "        \"description\": \"Get the current date and time for a given timezone\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"timezone\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"The timezone to fetch the current date and time for, e.g. 'America/New_York'\",\n",
+    "                }\n",
+    "            },\n",
+    "            \"required\": [\"timezone\"],\n",
+    "        },\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "schema_get_current_weather = tool_get_current_weather[\"function\"][\"parameters\"]\n",
+    "schema_get_current_date = tool_get_current_date[\"function\"][\"parameters\"]\n",
+    "\n",
+    "\n",
+    "def get_messages():\n",
+    "    return [\n",
+    "        {\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": f\"\"\"\n",
+    "# Tool Instructions\n",
+    "- Always execute python code in messages that you share.\n",
+    "- When looking for real time information use relevant functions if available else fallback to brave_search\n",
+    "You have access to the following functions:\n",
+    "Use the function 'get_current_weather' to: Get the current weather in a given location\n",
+    "{tool_get_current_weather[\"function\"]}\n",
+    "Use the function 'get_current_date' to: Get the current date and time for a given timezone\n",
+    "{tool_get_current_date[\"function\"]}\n",
+    "If a you choose to call a function ONLY reply in the following format:\n",
+    "<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}\n",
+    "where\n",
+    "start_tag => `<function`\n",
+    "parameters => a JSON dict with the function argument name as key and function argument value as value.\n",
+    "end_tag => `</function>`\n",
+    "Here is an example,\n",
+    "<function=example_function_name>{{\"example_name\": \"example_value\"}}</function>\n",
+    "Reminder:\n",
+    "- Function calls MUST follow the specified format\n",
+    "- Required parameters MUST be specified\n",
+    "- Only call one function at a time\n",
+    "- Put the entire function call reply on one line\n",
+    "- Always add your sources when using search results to answer the user query\n",
+    "You are a helpful assistant.\"\"\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"You are in New York. Please get the current date and time, and the weather.\",\n",
+    "        },\n",
+    "    ]\n",
+    "\n",
+    "\n",
+    "messages = get_messages()\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
+    "    messages=messages,\n",
+    "    response_format={\n",
+    "        \"type\": \"structural_tag\",\n",
+    "        \"structures\": [\n",
+    "            {\n",
+    "                \"begin\": \"<function=get_current_weather>\",\n",
+    "                \"schema\": schema_get_current_weather,\n",
+    "                \"end\": \"</function>\",\n",
+    "            },\n",
+    "            {\n",
+    "                \"begin\": \"<function=get_current_date>\",\n",
+    "                \"schema\": schema_get_current_date,\n",
+    "                \"end\": \"</function>\",\n",
+    "            },\n",
+    "        ],\n",
+    "        \"triggers\": [\"<function=\"],\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Support for XGrammar latest structural tag format\n",
+    "# <https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html>\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
+    "    messages=messages,\n",
+    "    response_format={\n",
+    "        \"type\": \"structural_tag\",\n",
+    "        \"format\": {\n",
+    "            \"type\": \"triggered_tags\",\n",
+    "            \"triggers\": [\"<function=\"],\n",
+    "            \"tags\": [\n",
+    "                {\n",
+    "                    \"begin\": \"<function=get_current_weather>\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"json_schema\",\n",
+    "                        \"json_schema\": schema_get_current_weather,\n",
+    "                    },\n",
+    "                    \"end\": \"</function>\",\n",
+    "                },\n",
+    "                {\n",
+    "                    \"begin\": \"<function=get_current_date>\",\n",
+    "                    \"content\": {\n",
+    "                        \"type\": \"json_schema\",\n",
+    "                        \"json_schema\": schema_get_current_date,\n",
+    "                    },\n",
+    "                    \"end\": \"</function>\",\n",
+    "                },\n",
+    "            ],\n",
+    "            \"at_least_one\": False,\n",
+    "            \"stop_after_first\": False,\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Native API and SGLang Runtime (SRT)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### JSON"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Using Pydantic**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "import json\n",
+    "from pydantic import BaseModel, Field\n",
+    "\n",
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3.1-8B-Instruct\")\n",
+    "\n",
+    "\n",
+    "# Define the schema using Pydantic\n",
+    "class CapitalInfo(BaseModel):\n",
+    "    name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
+    "    population: int = Field(..., description=\"Population of the capital city\")\n",
+    "\n",
+    "\n",
+    "# Make API request\n",
+    "messages = [\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Here is the information of the capital of France in the JSON format.\\n\",\n",
+    "    }\n",
+    "]\n",
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": text,\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 64,\n",
+    "            \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "print_highlight(response.json())\n",
+    "\n",
+    "\n",
+    "response_data = json.loads(response.json()[\"text\"])\n",
+    "# validate the response by the pydantic model\n",
+    "capital_info = CapitalInfo.model_validate(response_data)\n",
+    "print_highlight(f\"Validated response: {capital_info.model_dump_json()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**JSON Schema Directly**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "json_schema = json.dumps(\n",
+    "    {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
+    "            \"population\": {\"type\": \"integer\"},\n",
+    "        },\n",
+    "        \"required\": [\"name\", \"population\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# JSON\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": text,\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 64,\n",
+    "            \"json_schema\": json_schema,\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EBNF"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "messages = [\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Give me the information of the capital of France.\",\n",
+    "    }\n",
+    "]\n",
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": text,\n",
+    "        \"sampling_params\": {\n",
+    "            \"max_new_tokens\": 128,\n",
+    "            \"temperature\": 0,\n",
+    "            \"n\": 3,\n",
+    "            \"ebnf\": (\n",
+    "                \"root ::= city | description\\n\"\n",
+    "                'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
+    "                'description ::= city \" is \" status\\n'\n",
+    "                'status ::= \"the capital of \" country\\n'\n",
+    "                'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
+    "            ),\n",
+    "        },\n",
+    "        \"stream\": False,\n",
+    "        \"return_logprob\": False,\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Regular expression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "messages = [\n",
+    "    {\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": \"Paris is the capital of\",\n",
+    "    }\n",
+    "]\n",
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": text,\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 64,\n",
+    "            \"regex\": \"(France|England)\",\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Structural Tag"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "# generate an answer\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3.1-8B-Instruct\")\n",
+    "\n",
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "payload = {\n",
+    "    \"text\": text,\n",
+    "    \"sampling_params\": {\n",
+    "        \"structural_tag\": json.dumps(\n",
+    "            {\n",
+    "                \"type\": \"structural_tag\",\n",
+    "                \"structures\": [\n",
+    "                    {\n",
+    "                        \"begin\": \"<function=get_current_weather>\",\n",
+    "                        \"schema\": schema_get_current_weather,\n",
+    "                        \"end\": \"</function>\",\n",
+    "                    },\n",
+    "                    {\n",
+    "                        \"begin\": \"<function=get_current_date>\",\n",
+    "                        \"schema\": schema_get_current_date,\n",
+    "                        \"end\": \"</function>\",\n",
+    "                    },\n",
+    "                ],\n",
+    "                \"triggers\": [\"<function=\"],\n",
+    "            }\n",
+    "        )\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "\n",
+    "# Send POST request to the API endpoint\n",
+    "response = requests.post(f\"http://localhost:{port}/generate\", json=payload)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Support for XGrammar latest structural tag format\n",
+    "# <https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html>\n",
+    "payload = {\n",
+    "    \"text\": text,\n",
+    "    \"sampling_params\": {\n",
+    "        \"structural_tag\": json.dumps(\n",
+    "            {\n",
+    "                \"type\": \"structural_tag\",\n",
+    "                \"format\": {\n",
+    "                    \"type\": \"triggered_tags\",\n",
+    "                    \"triggers\": [\"<function=\"],\n",
+    "                    \"tags\": [\n",
+    "                        {\n",
+    "                            \"begin\": \"<function=get_current_weather>\",\n",
+    "                            \"content\": {\n",
+    "                                \"type\": \"json_schema\",\n",
+    "                                \"json_schema\": schema_get_current_weather,\n",
+    "                            },\n",
+    "                            \"end\": \"</function>\",\n",
+    "                        },\n",
+    "                        {\n",
+    "                            \"begin\": \"<function=get_current_date>\",\n",
+    "                            \"content\": {\n",
+    "                                \"type\": \"json_schema\",\n",
+    "                                \"json_schema\": schema_get_current_date,\n",
+    "                            },\n",
+    "                            \"end\": \"</function>\",\n",
+    "                        },\n",
+    "                    ],\n",
+    "                    \"at_least_one\": False,\n",
+    "                    \"stop_after_first\": False,\n",
+    "                },\n",
+    "            }\n",
+    "        )\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "\n",
+    "# Send POST request to the API endpoint\n",
+    "response = requests.post(f\"http://localhost:{port}/generate\", json=payload)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Offline Engine API"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sglang as sgl\n",
+    "\n",
+    "llm = sgl.Engine(\n",
+    "    model_path=\"meta-llama/Meta-Llama-3.1-8B-Instruct\", grammar_backend=\"xgrammar\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### JSON"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Using Pydantic**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from pydantic import BaseModel, Field\n",
+    "\n",
+    "prompts = [\n",
+    "    \"Give me the information of the capital of China in the JSON format.\",\n",
+    "    \"Give me the information of the capital of France in the JSON format.\",\n",
+    "    \"Give me the information of the capital of Ireland in the JSON format.\",\n",
+    "]\n",
+    "\n",
+    "\n",
+    "# Define the schema using Pydantic\n",
+    "class CapitalInfo(BaseModel):\n",
+    "    name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
+    "    population: int = Field(..., description=\"Population of the capital city\")\n",
+    "\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.1,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
+    "}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print_highlight(\"===============================\")\n",
+    "    print_highlight(f\"Prompt: {prompt}\")  # validate the output by the pydantic model\n",
+    "    capital_info = CapitalInfo.model_validate_json(output[\"text\"])\n",
+    "    print_highlight(f\"Validated output: {capital_info.model_dump_json()}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**JSON Schema Directly**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Give me the information of the capital of China in the JSON format.\",\n",
+    "    \"Give me the information of the capital of France in the JSON format.\",\n",
+    "    \"Give me the information of the capital of Ireland in the JSON format.\",\n",
+    "]\n",
+    "\n",
+    "json_schema = json.dumps(\n",
+    "    {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
+    "            \"population\": {\"type\": \"integer\"},\n",
+    "        },\n",
+    "        \"required\": [\"name\", \"population\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.1, \"top_p\": 0.95, \"json_schema\": json_schema}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print_highlight(\"===============================\")\n",
+    "    print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EBNF\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Give me the information of the capital of France.\",\n",
+    "    \"Give me the information of the capital of Germany.\",\n",
+    "    \"Give me the information of the capital of Italy.\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.8,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"ebnf\": (\n",
+    "        \"root ::= city | description\\n\"\n",
+    "        'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
+    "        'description ::= city \" is \" status\\n'\n",
+    "        'status ::= \"the capital of \" country\\n'\n",
+    "        'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
+    "    ),\n",
+    "}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print_highlight(\"===============================\")\n",
+    "    print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Regular expression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Please provide information about London as a major global city:\",\n",
+    "    \"Please provide information about Paris as a major global city:\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95, \"regex\": \"(France|England)\"}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print_highlight(\"===============================\")\n",
+    "    print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Structural Tag"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "prompts = [text]\n",
+    "\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.8,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"structural_tag\": json.dumps(\n",
+    "        {\n",
+    "            \"type\": \"structural_tag\",\n",
+    "            \"structures\": [\n",
+    "                {\n",
+    "                    \"begin\": \"<function=get_current_weather>\",\n",
+    "                    \"schema\": schema_get_current_weather,\n",
+    "                    \"end\": \"</function>\",\n",
+    "                },\n",
+    "                {\n",
+    "                    \"begin\": \"<function=get_current_date>\",\n",
+    "                    \"schema\": schema_get_current_date,\n",
+    "                    \"end\": \"</function>\",\n",
+    "                },\n",
+    "            ],\n",
+    "            \"triggers\": [\"<function=\"],\n",
+    "        }\n",
+    "    ),\n",
+    "}\n",
+    "\n",
+    "\n",
+    "# Send POST request to the API endpoint\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print_highlight(\"===============================\")\n",
+    "    print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Support for XGrammar latest structural tag format\n",
+    "# <https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html>\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.8,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"structural_tag\": json.dumps(\n",
+    "        {\n",
+    "            \"type\": \"structural_tag\",\n",
+    "            \"format\": {\n",
+    "                \"type\": \"triggered_tags\",\n",
+    "                \"triggers\": [\"<function=\"],\n",
+    "                \"tags\": [\n",
+    "                    {\n",
+    "                        \"begin\": \"<function=get_current_weather>\",\n",
+    "                        \"content\": {\n",
+    "                            \"type\": \"json_schema\",\n",
+    "                            \"json_schema\": schema_get_current_weather,\n",
+    "                        },\n",
+    "                        \"end\": \"</function>\",\n",
+    "                    },\n",
+    "                    {\n",
+    "                        \"begin\": \"<function=get_current_date>\",\n",
+    "                        \"content\": {\n",
+    "                            \"type\": \"json_schema\",\n",
+    "                            \"json_schema\": schema_get_current_date,\n",
+    "                        },\n",
+    "                        \"end\": \"</function>\",\n",
+    "                    },\n",
+    "                ],\n",
+    "                \"at_least_one\": False,\n",
+    "                \"stop_after_first\": False,\n",
+    "            },\n",
+    "        }\n",
+    "    ),\n",
+    "}\n",
+    "\n",
+    "\n",
+    "# Send POST request to the API endpoint\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print_highlight(\"===============================\")\n",
+    "    print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm.shutdown()"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/advanced_features/structured_outputs.mdx

@@ -0,0 +1,803 @@
+---
+title: "Structured Outputs"
+metatags:
+    description: "SGLang structured outputs: JSON schema, regex, EBNF constraints. XGrammar, Outlines, Llguidance backends for guaranteed output format."
+---
+You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.
+
+SGLang supports three grammar backends:
+
+- [XGrammar](https://github.com/mlc-ai/xgrammar)(default): Supports JSON schema, regular expression, and EBNF constraints.
+- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.
+- [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints.
+
+We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).
+
+To use Outlines, simply add `--grammar-backend outlines` when launching the server.
+To use llguidance, add `--grammar-backend llguidance`  when launching the server.
+If no backend is specified, XGrammar will be used as the default.
+
+For better output quality, **It's advisable to explicitly include instructions in the prompt to guide the model to generate the desired format.** For example, you can specify, 'Please generate the output in the following JSON format: ...'.
+
+
+
+## OpenAI Compatible API
+
+
+
+```python Example
+import openai
+import os
+
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+
+
+server_process, port = launch_server_cmd(
+    "python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}")
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+```
+
+### JSON
+
+you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response.
+
+
+**Using Pydantic**
+
+
+
+```python Example
+from pydantic import BaseModel, Field
+
+
+# Define the schema using Pydantic
+class CapitalInfo(BaseModel):
+    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
+    population: int = Field(..., description="Population of the capital city")
+
+
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
+    messages=[
+        {
+            "role": "user",
+            "content": "Please generate the information of the capital of France in the JSON format.",
+        },
+    ],
+    temperature=0,
+    max_tokens=128,
+    response_format={
+        "type": "json_schema",
+        "json_schema": {
+            "name": "foo",
+            # convert the pydantic model to json schema
+            "schema": CapitalInfo.model_json_schema(),
+        },
+    },
+)
+
+response_content = response.choices[0].message.content
+# validate the JSON response by the pydantic model
+capital_info = CapitalInfo.model_validate_json(response_content)
+print_highlight(f"Validated response: {capital_info.model_dump_json()}")
+```
+
+**JSON Schema Directly**
+
+
+
+
+```python Example
+import json
+
+json_schema = json.dumps(
+    {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string", "pattern": "^[\\w]+$"},
+            "population": {"type": "integer"},
+        },
+        "required": ["name", "population"],
+    }
+)
+
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
+    messages=[
+        {
+            "role": "user",
+            "content": "Give me the information of the capital of France in the JSON format.",
+        },
+    ],
+    temperature=0,
+    max_tokens=128,
+    response_format={
+        "type": "json_schema",
+        "json_schema": {"name": "foo", "schema": json.loads(json_schema)},
+    },
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+### EBNF
+
+
+
+```python Example
+ebnf_grammar = """
+root ::= city | description
+city ::= "London" | "Paris" | "Berlin" | "Rome"
+description ::= city " is " status
+status ::= "the capital of " country
+country ::= "England" | "France" | "Germany" | "Italy"
+"""
+
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful geography bot."},
+        {
+            "role": "user",
+            "content": "Give me the information of the capital of France.",
+        },
+    ],
+    temperature=0,
+    max_tokens=32,
+    extra_body={"ebnf": ebnf_grammar},
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+### Regular expression
+
+
+
+```python Example
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
+    messages=[
+        {"role": "user", "content": "What is the capital of France?"},
+    ],
+    temperature=0,
+    max_tokens=128,
+    extra_body={"regex": "(Paris|London)"},
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+### Structural Tag
+
+
+
+```python Example
+tool_get_current_weather = {
+    "type": "function",
+    "function": {
+        "name": "get_current_weather",
+        "description": "Get the current weather in a given location",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "city": {
+                    "type": "string",
+                    "description": "The city to find the weather for, e.g. 'San Francisco'",
+                },
+                "state": {
+                    "type": "string",
+                    "description": "the two-letter abbreviation for the state that the city is"
+                    " in, e.g. 'CA' which would mean 'California'",
+                },
+                "unit": {
+                    "type": "string",
+                    "description": "The unit to fetch the temperature in",
+                    "enum": ["celsius", "fahrenheit"],
+                },
+            },
+            "required": ["city", "state", "unit"],
+        },
+    },
+}
+
+tool_get_current_date = {
+    "type": "function",
+    "function": {
+        "name": "get_current_date",
+        "description": "Get the current date and time for a given timezone",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "timezone": {
+                    "type": "string",
+                    "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'",
+                }
+            },
+            "required": ["timezone"],
+        },
+    },
+}
+
+schema_get_current_weather = tool_get_current_weather["function"]["parameters"]
+schema_get_current_date = tool_get_current_date["function"]["parameters"]
+
+
+def get_messages():
+    return [
+        {
+            "role": "system",
+            "content": f"""
+# Tool Instructions
+- Always execute python code in messages that you share.
+- When looking for real time information use relevant functions if available else fallback to brave_search
+You have access to the following functions:
+Use the function 'get_current_weather' to: Get the current weather in a given location
+{tool_get_current_weather["function"]}
+Use the function 'get_current_date' to: Get the current date and time for a given timezone
+{tool_get_current_date["function"]}
+If a you choose to call a function ONLY reply in the following format:
+<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}
+where
+start_tag => `<function`
+parameters => a JSON dict with the function argument name as key and function argument value as value.
+end_tag => `</function>`
+Here is an example,
+<function=example_function_name>{{"example_name": "example_value"}}</function>
+Reminder:
+- Function calls MUST follow the specified format
+- Required parameters MUST be specified
+- Only call one function at a time
+- Put the entire function call reply on one line
+- Always add your sources when using search results to answer the user query
+You are a helpful assistant.""",
+        },
+        {
+            "role": "user",
+            "content": "You are in New York. Please get the current date and time, and the weather.",
+        },
+    ]
+
+
+messages = get_messages()
+
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
+    messages=messages,
+    response_format={
+        "type": "structural_tag",
+        "structures": [
+            {
+                "begin": "<function=get_current_weather>",
+                "schema": schema_get_current_weather,
+                "end": "</function>",
+            },
+            {
+                "begin": "<function=get_current_date>",
+                "schema": schema_get_current_date,
+                "end": "</function>",
+            },
+        ],
+        "triggers": ["<function="],
+    },
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+
+```python Example
+# Support for XGrammar latest structural tag format
+# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html
+
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
+    messages=messages,
+    response_format={
+        "type": "structural_tag",
+        "format": {
+            "type": "triggered_tags",
+            "triggers": ["<function="],
+            "tags": [
+                {
+                    "begin": "<function=get_current_weather>",
+                    "content": {
+                        "type": "json_schema",
+                        "json_schema": schema_get_current_weather,
+                    },
+                    "end": "</function>",
+                },
+                {
+                    "begin": "<function=get_current_date>",
+                    "content": {
+                        "type": "json_schema",
+                        "json_schema": schema_get_current_date,
+                    },
+                    "end": "</function>",
+                },
+            ],
+            "at_least_one": False,
+            "stop_after_first": False,
+        },
+    },
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+## Native API and SGLang Runtime (SRT)
+
+
+### JSON
+
+
+**Using Pydantic**
+
+
+
+```python Example
+import requests
+import json
+from pydantic import BaseModel, Field
+
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
+
+
+# Define the schema using Pydantic
+class CapitalInfo(BaseModel):
+    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
+    population: int = Field(..., description="Population of the capital city")
+
+
+# Make API request
+messages = [
+    {
+        "role": "user",
+        "content": "Here is the information of the capital of France in the JSON format.\n",
+    }
+]
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": text,
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "json_schema": json.dumps(CapitalInfo.model_json_schema()),
+        },
+    },
+)
+print_highlight(response.json())
+
+
+response_data = json.loads(response.json()["text"])
+# validate the response by the pydantic model
+capital_info = CapitalInfo.model_validate(response_data)
+print_highlight(f"Validated response: {capital_info.model_dump_json()}")
+```
+
+**JSON Schema Directly**
+
+
+
+```python Example
+json_schema = json.dumps(
+    {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string", "pattern": "^[\\w]+$"},
+            "population": {"type": "integer"},
+        },
+        "required": ["name", "population"],
+    }
+)
+
+# JSON
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": text,
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "json_schema": json_schema,
+        },
+    },
+)
+
+print_highlight(response.json())
+```
+
+### EBNF
+
+
+
+```python Example
+messages = [
+    {
+        "role": "user",
+        "content": "Give me the information of the capital of France.",
+    }
+]
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": text,
+        "sampling_params": {
+            "max_new_tokens": 128,
+            "temperature": 0,
+            "n": 3,
+            "ebnf": (
+                "root ::= city | description\n"
+                'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
+                'description ::= city " is " status\n'
+                'status ::= "the capital of " country\n'
+                'country ::= "England" | "France" | "Germany" | "Italy"'
+            ),
+        },
+        "stream": False,
+        "return_logprob": False,
+    },
+)
+
+print_highlight(response.json())
+```
+
+### Regular expression
+
+
+
+```python Example
+messages = [
+    {
+        "role": "user",
+        "content": "Paris is the capital of",
+    }
+]
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": text,
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "regex": "(France|England)",
+        },
+    },
+)
+print_highlight(response.json())
+```
+
+### Structural Tag
+
+
+
+```python Example
+from transformers import AutoTokenizer
+
+# generate an answer
+tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
+
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+payload = {
+    "text": text,
+    "sampling_params": {
+        "structural_tag": json.dumps(
+            {
+                "type": "structural_tag",
+                "structures": [
+                    {
+                        "begin": "<function=get_current_weather>",
+                        "schema": schema_get_current_weather,
+                        "end": "</function>",
+                    },
+                    {
+                        "begin": "<function=get_current_date>",
+                        "schema": schema_get_current_date,
+                        "end": "</function>",
+                    },
+                ],
+                "triggers": ["<function="],
+            }
+        )
+    },
+}
+
+
+# Send POST request to the API endpoint
+response = requests.post(f"http://localhost:{port}/generate", json=payload)
+print_highlight(response.json())
+```
+
+
+```python Example
+# Support for XGrammar latest structural tag format
+# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html
+
+payload = {
+    "text": text,
+    "sampling_params": {
+        "structural_tag": json.dumps(
+            {
+                "type": "structural_tag",
+                "format": {
+                    "type": "triggered_tags",
+                    "triggers": ["<function="],
+                    "tags": [
+                        {
+                            "begin": "<function=get_current_weather>",
+                            "content": {
+                                "type": "json_schema",
+                                "json_schema": schema_get_current_weather,
+                            },
+                            "end": "</function>",
+                        },
+                        {
+                            "begin": "<function=get_current_date>",
+                            "content": {
+                                "type": "json_schema",
+                                "json_schema": schema_get_current_date,
+                            },
+                            "end": "</function>",
+                        },
+                    ],
+                    "at_least_one": False,
+                    "stop_after_first": False,
+                },
+            }
+        )
+    },
+}
+
+
+# Send POST request to the API endpoint
+response = requests.post(f"http://localhost:{port}/generate", json=payload)
+print_highlight(response.json())
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+## Offline Engine API
+
+
+
+```python Example
+import sglang as sgl
+
+llm = sgl.Engine(
+    model_path="meta-llama/Meta-Llama-3.1-8B-Instruct", grammar_backend="xgrammar"
+)
+```
+
+### JSON
+
+
+**Using Pydantic**
+
+
+
+```python Example
+import json
+from pydantic import BaseModel, Field
+
+
+prompts = [
+    "Give me the information of the capital of China in the JSON format.",
+    "Give me the information of the capital of France in the JSON format.",
+    "Give me the information of the capital of Ireland in the JSON format.",
+]
+
+
+# Define the schema using Pydantic
+class CapitalInfo(BaseModel):
+    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
+    population: int = Field(..., description="Population of the capital city")
+
+
+sampling_params = {
+    "temperature": 0.1,
+    "top_p": 0.95,
+    "json_schema": json.dumps(CapitalInfo.model_json_schema()),
+}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print_highlight("===============================")
+    print_highlight(f"Prompt: {prompt}")  # validate the output by the pydantic model
+    capital_info = CapitalInfo.model_validate_json(output["text"])
+    print_highlight(f"Validated output: {capital_info.model_dump_json()}")
+```
+
+**JSON Schema Directly**
+
+
+
+```python Example
+prompts = [
+    "Give me the information of the capital of China in the JSON format.",
+    "Give me the information of the capital of France in the JSON format.",
+    "Give me the information of the capital of Ireland in the JSON format.",
+]
+
+json_schema = json.dumps(
+    {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string", "pattern": "^[\\w]+$"},
+            "population": {"type": "integer"},
+        },
+        "required": ["name", "population"],
+    }
+)
+
+sampling_params = {"temperature": 0.1, "top_p": 0.95, "json_schema": json_schema}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print_highlight("===============================")
+    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+### EBNF
+
+
+
+
+```python Example
+prompts = [
+    "Give me the information of the capital of France.",
+    "Give me the information of the capital of Germany.",
+    "Give me the information of the capital of Italy.",
+]
+
+sampling_params = {
+    "temperature": 0.8,
+    "top_p": 0.95,
+    "ebnf": (
+        "root ::= city | description\n"
+        'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
+        'description ::= city " is " status\n'
+        'status ::= "the capital of " country\n'
+        'country ::= "England" | "France" | "Germany" | "Italy"'
+    ),
+}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print_highlight("===============================")
+    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+### Regular expression
+
+
+
+```python Example
+prompts = [
+    "Please provide information about London as a major global city:",
+    "Please provide information about Paris as a major global city:",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print_highlight("===============================")
+    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+### Structural Tag
+
+
+
+```python Example
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+prompts = [text]
+
+
+sampling_params = {
+    "temperature": 0.8,
+    "top_p": 0.95,
+    "structural_tag": json.dumps(
+        {
+            "type": "structural_tag",
+            "structures": [
+                {
+                    "begin": "<function=get_current_weather>",
+                    "schema": schema_get_current_weather,
+                    "end": "</function>",
+                },
+                {
+                    "begin": "<function=get_current_date>",
+                    "schema": schema_get_current_date,
+                    "end": "</function>",
+                },
+            ],
+            "triggers": ["<function="],
+        }
+    ),
+}
+
+
+# Send POST request to the API endpoint
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print_highlight("===============================")
+    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+
+```python Example
+# Support for XGrammar latest structural tag format
+# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html
+
+sampling_params = {
+    "temperature": 0.8,
+    "top_p": 0.95,
+    "structural_tag": json.dumps(
+        {
+            "type": "structural_tag",
+            "format": {
+                "type": "triggered_tags",
+                "triggers": ["<function="],
+                "tags": [
+                    {
+                        "begin": "<function=get_current_weather>",
+                        "content": {
+                            "type": "json_schema",
+                            "json_schema": schema_get_current_weather,
+                        },
+                        "end": "</function>",
+                    },
+                    {
+                        "begin": "<function=get_current_date>",
+                        "content": {
+                            "type": "json_schema",
+                            "json_schema": schema_get_current_date,
+                        },
+                        "end": "</function>",
+                    },
+                ],
+                "at_least_one": False,
+                "stop_after_first": False,
+            },
+        }
+    ),
+}
+
+
+# Send POST request to the API endpoint
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print_highlight("===============================")
+    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+
+```python Example
+llm.shutdown()
+```

docs_new/docs/advanced_features/structured_outputs_for_reasoning_models.ipynb

@@ -0,0 +1,841 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Structured Outputs For Reasoning Models\n",
+    "\n",
+    "When working with reasoning models that use special tokens like `<think>...</think>` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output.\n",
+    "\n",
+    "SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output.\n",
+    "\n",
+    "To enable this feature, use the `--reasoning-parser` flag which decide the think_end_token, such as `</think>`, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag.\n",
+    "\n",
+    "## Supported Models\n",
+    "\n",
+    "Currently, SGLang supports the following reasoning models:\n",
+    "- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `<think>` and `</think>` tags.\n",
+    "- [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `<think>` and `</think>` tags.\n",
+    "\n",
+    "\n",
+    "## Usage\n",
+    "\n",
+    "## OpenAI Compatible API"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Specify the `--grammar-backend`, `--reasoning-parser` option."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "import os\n",
+    "\n",
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
+    "\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### JSON\n",
+    "\n",
+    "you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Using Pydantic**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pydantic import BaseModel, Field\n",
+    "\n",
+    "\n",
+    "# Define the schema using Pydantic\n",
+    "class CapitalInfo(BaseModel):\n",
+    "    name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
+    "    population: int = Field(..., description=\"Population of the capital city\")\n",
+    "\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
+    "        },\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=2048,\n",
+    "    response_format={\n",
+    "        \"type\": \"json_schema\",\n",
+    "        \"json_schema\": {\n",
+    "            \"name\": \"foo\",\n",
+    "            # convert the pydantic model to json schema\n",
+    "            \"schema\": CapitalInfo.model_json_schema(),\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(\n",
+    "    f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**JSON Schema Directly**\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "\n",
+    "json_schema = json.dumps(\n",
+    "    {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
+    "            \"population\": {\"type\": \"integer\"},\n",
+    "        },\n",
+    "        \"required\": [\"name\", \"population\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
+    "        },\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=2048,\n",
+    "    response_format={\n",
+    "        \"type\": \"json_schema\",\n",
+    "        \"json_schema\": {\"name\": \"foo\", \"schema\": json.loads(json_schema)},\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(\n",
+    "    f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EBNF"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ebnf_grammar = \"\"\"\n",
+    "root ::= city | description\n",
+    "city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\n",
+    "description ::= city \" is \" status\n",
+    "status ::= \"the capital of \" country\n",
+    "country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"\n",
+    "\"\"\"\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"system\", \"content\": \"You are a helpful geography bot.\"},\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
+    "        },\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=2048,\n",
+    "    extra_body={\"ebnf\": ebnf_grammar},\n",
+    ")\n",
+    "\n",
+    "print_highlight(\n",
+    "    f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Regular expression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.chat.completions.create(\n",
+    "    model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"assistant\", \"content\": \"What is the capital of France?\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=2048,\n",
+    "    extra_body={\"regex\": \"(Paris|London)\"},\n",
+    ")\n",
+    "\n",
+    "print_highlight(\n",
+    "    f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Structural Tag"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tool_get_current_weather = {\n",
+    "    \"type\": \"function\",\n",
+    "    \"function\": {\n",
+    "        \"name\": \"get_current_weather\",\n",
+    "        \"description\": \"Get the current weather in a given location\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"city\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
+    "                },\n",
+    "                \"state\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"the two-letter abbreviation for the state that the city is\"\n",
+    "                    \" in, e.g. 'CA' which would mean 'California'\",\n",
+    "                },\n",
+    "                \"unit\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"The unit to fetch the temperature in\",\n",
+    "                    \"enum\": [\"celsius\", \"fahrenheit\"],\n",
+    "                },\n",
+    "            },\n",
+    "            \"required\": [\"city\", \"state\", \"unit\"],\n",
+    "        },\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "tool_get_current_date = {\n",
+    "    \"type\": \"function\",\n",
+    "    \"function\": {\n",
+    "        \"name\": \"get_current_date\",\n",
+    "        \"description\": \"Get the current date and time for a given timezone\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"timezone\": {\n",
+    "                    \"type\": \"string\",\n",
+    "                    \"description\": \"The timezone to fetch the current date and time for, e.g. 'America/New_York'\",\n",
+    "                }\n",
+    "            },\n",
+    "            \"required\": [\"timezone\"],\n",
+    "        },\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "schema_get_current_weather = tool_get_current_weather[\"function\"][\"parameters\"]\n",
+    "schema_get_current_date = tool_get_current_date[\"function\"][\"parameters\"]\n",
+    "\n",
+    "\n",
+    "def get_messages():\n",
+    "    return [\n",
+    "        {\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": f\"\"\"\n",
+    "# Tool Instructions\n",
+    "- Always execute python code in messages that you share.\n",
+    "- When looking for real time information use relevant functions if available else fallback to brave_search\n",
+    "You have access to the following functions:\n",
+    "Use the function 'get_current_weather' to: Get the current weather in a given location\n",
+    "{tool_get_current_weather[\"function\"]}\n",
+    "Use the function 'get_current_date' to: Get the current date and time for a given timezone\n",
+    "{tool_get_current_date[\"function\"]}\n",
+    "If a you choose to call a function ONLY reply in the following format:\n",
+    "<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}\n",
+    "where\n",
+    "start_tag => `<function`\n",
+    "parameters => a JSON dict with the function argument name as key and function argument value as value.\n",
+    "end_tag => `</function>`\n",
+    "Here is an example,\n",
+    "<function=example_function_name>{{\"example_name\": \"example_value\"}}</function>\n",
+    "Reminder:\n",
+    "- Function calls MUST follow the specified format\n",
+    "- Required parameters MUST be specified\n",
+    "- Only call one function at a time\n",
+    "- Put the entire function call reply on one line\n",
+    "- Always add your sources when using search results to answer the user query\n",
+    "You are a helpful assistant.\"\"\",\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"You are in New York. Please get the current date and time, and the weather.\",\n",
+    "        },\n",
+    "    ]\n",
+    "\n",
+    "\n",
+    "messages = get_messages()\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
+    "    messages=messages,\n",
+    "    response_format={\n",
+    "        \"type\": \"structural_tag\",\n",
+    "        \"max_new_tokens\": 2048,\n",
+    "        \"structures\": [\n",
+    "            {\n",
+    "                \"begin\": \"<function=get_current_weather>\",\n",
+    "                \"schema\": schema_get_current_weather,\n",
+    "                \"end\": \"</function>\",\n",
+    "            },\n",
+    "            {\n",
+    "                \"begin\": \"<function=get_current_date>\",\n",
+    "                \"schema\": schema_get_current_date,\n",
+    "                \"end\": \"</function>\",\n",
+    "            },\n",
+    "        ],\n",
+    "        \"triggers\": [\"<function=\"],\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(\n",
+    "    f\"reasoing_content: {response.choices[0].message.reasoning_content}\\n\\ncontent: {response.choices[0].message.content}\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Native API and SGLang Runtime (SRT)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> Note: For native API, as a work-around, you need to set `require_reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### JSON"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Using Pydantic**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "from pydantic import BaseModel, Field\n",
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\")\n",
+    "\n",
+    "\n",
+    "# Define the schema using Pydantic\n",
+    "class CapitalInfo(BaseModel):\n",
+    "    name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
+    "    population: int = Field(..., description=\"Population of the capital city\")\n",
+    "\n",
+    "\n",
+    "messages = [\n",
+    "    {\n",
+    "        \"role\": \"assistant\",\n",
+    "        \"content\": \"Give me the information and population of the capital of France in the JSON format.\",\n",
+    "    },\n",
+    "]\n",
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "# Make API request\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": text,\n",
+    "        \"require_reasoning\": True,\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 2048,\n",
+    "            \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "print(response.json())\n",
+    "\n",
+    "\n",
+    "reasoing_content = response.json()[\"text\"].split(\"</think>\")[0]\n",
+    "content = response.json()[\"text\"].split(\"</think>\")[1]\n",
+    "print_highlight(f\"reasoing_content: {reasoing_content}\\n\\ncontent: {content}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**JSON Schema Directly**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "json_schema = json.dumps(\n",
+    "    {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
+    "            \"population\": {\"type\": \"integer\"},\n",
+    "        },\n",
+    "        \"required\": [\"name\", \"population\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# JSON\n",
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": text,\n",
+    "        \"require_reasoning\": True,\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 2048,\n",
+    "            \"json_schema\": json_schema,\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EBNF"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": \"Give me the information of the capital of France.\",\n",
+    "        \"require_reasoning\": True,\n",
+    "        \"sampling_params\": {\n",
+    "            \"max_new_tokens\": 2048,\n",
+    "            \"temperature\": 0,\n",
+    "            \"n\": 3,\n",
+    "            \"ebnf\": (\n",
+    "                \"root ::= city | description\\n\"\n",
+    "                'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
+    "                'description ::= city \" is \" status\\n'\n",
+    "                'status ::= \"the capital of \" country\\n'\n",
+    "                'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
+    "            ),\n",
+    "        },\n",
+    "        \"stream\": False,\n",
+    "        \"return_logprob\": False,\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Regular expression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": \"Paris is the capital of\",\n",
+    "        \"require_reasoning\": True,\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 2048,\n",
+    "            \"regex\": \"(France|England)\",\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "print(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Structural Tag"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "payload = {\n",
+    "    \"text\": text,\n",
+    "    \"require_reasoning\": True,\n",
+    "    \"sampling_params\": {\n",
+    "        \"max_new_tokens\": 2048,\n",
+    "        \"structural_tag\": json.dumps(\n",
+    "            {\n",
+    "                \"type\": \"structural_tag\",\n",
+    "                \"structures\": [\n",
+    "                    {\n",
+    "                        \"begin\": \"<function=get_current_weather>\",\n",
+    "                        \"schema\": schema_get_current_weather,\n",
+    "                        \"end\": \"</function>\",\n",
+    "                    },\n",
+    "                    {\n",
+    "                        \"begin\": \"<function=get_current_date>\",\n",
+    "                        \"schema\": schema_get_current_date,\n",
+    "                        \"end\": \"</function>\",\n",
+    "                    },\n",
+    "                ],\n",
+    "                \"triggers\": [\"<function=\"],\n",
+    "            }\n",
+    "        ),\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "\n",
+    "# Send POST request to the API endpoint\n",
+    "response = requests.post(f\"http://localhost:{port}/generate\", json=payload)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Offline Engine API"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sglang as sgl\n",
+    "\n",
+    "llm = sgl.Engine(\n",
+    "    model_path=\"deepseek-ai/DeepSeek-R1-Distill-Qwen-7B\",\n",
+    "    reasoning_parser=\"deepseek-r1\",\n",
+    "    grammar_backend=\"xgrammar\",\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### JSON"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Using Pydantic**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from pydantic import BaseModel, Field\n",
+    "\n",
+    "prompts = [\n",
+    "    \"Give me the information of the capital of China in the JSON format.\",\n",
+    "    \"Give me the information of the capital of France in the JSON format.\",\n",
+    "    \"Give me the information of the capital of Ireland in the JSON format.\",\n",
+    "]\n",
+    "\n",
+    "\n",
+    "# Define the schema using Pydantic\n",
+    "class CapitalInfo(BaseModel):\n",
+    "    name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
+    "    population: int = Field(..., description=\"Population of the capital city\")\n",
+    "\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"max_new_tokens\": 2048,\n",
+    "    \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
+    "}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print(\"===============================\")\n",
+    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**JSON Schema Directly**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Give me the information of the capital of China in the JSON format.\",\n",
+    "    \"Give me the information of the capital of France in the JSON format.\",\n",
+    "    \"Give me the information of the capital of Ireland in the JSON format.\",\n",
+    "]\n",
+    "\n",
+    "json_schema = json.dumps(\n",
+    "    {\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\n",
+    "            \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
+    "            \"population\": {\"type\": \"integer\"},\n",
+    "        },\n",
+    "        \"required\": [\"name\", \"population\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0, \"max_new_tokens\": 2048, \"json_schema\": json_schema}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print(\"===============================\")\n",
+    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### EBNF\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Give me the information of the capital of France.\",\n",
+    "    \"Give me the information of the capital of Germany.\",\n",
+    "    \"Give me the information of the capital of Italy.\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.8,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"ebnf\": (\n",
+    "        \"root ::= city | description\\n\"\n",
+    "        'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
+    "        'description ::= city \" is \" status\\n'\n",
+    "        'status ::= \"the capital of \" country\\n'\n",
+    "        'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
+    "    ),\n",
+    "}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print(\"===============================\")\n",
+    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Regular expression"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Please provide information about London as a major global city:\",\n",
+    "    \"Please provide information about Paris as a major global city:\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95, \"regex\": \"(France|England)\"}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print(\"===============================\")\n",
+    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "text = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, return_dict=False\n",
+    ")\n",
+    "prompts = [text]\n",
+    "\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.8,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"max_new_tokens\": 2048,\n",
+    "    \"structural_tag\": json.dumps(\n",
+    "        {\n",
+    "            \"type\": \"structural_tag\",\n",
+    "            \"structures\": [\n",
+    "                {\n",
+    "                    \"begin\": \"<function=get_current_weather>\",\n",
+    "                    \"schema\": schema_get_current_weather,\n",
+    "                    \"end\": \"</function>\",\n",
+    "                },\n",
+    "                {\n",
+    "                    \"begin\": \"<function=get_current_date>\",\n",
+    "                    \"schema\": schema_get_current_date,\n",
+    "                    \"end\": \"</function>\",\n",
+    "                },\n",
+    "            ],\n",
+    "            \"triggers\": [\"<function=\"],\n",
+    "        }\n",
+    "    ),\n",
+    "}\n",
+    "\n",
+    "\n",
+    "# Send POST request to the API endpoint\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print(\"===============================\")\n",
+    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm.shutdown()"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/advanced_features/structured_outputs_for_reasoning_models.mdx

@@ -0,0 +1,663 @@
+---
+title: "Structured Outputs For Reasoning Models"
+metatags:
+    description: "SGLang structured outputs for reasoning models: free-form thinking with constrained final output for DeepSeek R1, QwQ models."
+---
+When working with reasoning models that use special tokens like `<think>...</think>` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output.
+
+SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output.
+
+To enable this feature, use the `--reasoning-parser` flag which decide the think_end_token, such as `</think>`, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag.
+
+## Supported Models
+
+Currently, SGLang supports the following reasoning models:
+- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `<think>` and `</think>` tags.
+- [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `<think>` and `</think>` tags.
+
+
+## Usage
+
+## OpenAI Compatible API
+
+
+Specify the `--grammar-backend`, `--reasoning-parser` option.
+
+
+
+```python Example
+import openai
+import os
+
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+
+
+server_process, port = launch_server_cmd(
+    "python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}")
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+```
+
+### JSON
+
+you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response.
+
+
+**Using Pydantic**
+
+
+
+```python Example
+from pydantic import BaseModel, Field
+
+
+# Define the schema using Pydantic
+class CapitalInfo(BaseModel):
+    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
+    population: int = Field(..., description="Population of the capital city")
+
+
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
+    messages=[
+        {
+            "role": "assistant",
+            "content": "Give me the information and population of the capital of France in the JSON format.",
+        },
+    ],
+    temperature=0,
+    max_tokens=2048,
+    response_format={
+        "type": "json_schema",
+        "json_schema": {
+            "name": "foo",
+            # convert the pydantic model to json schema
+            "schema": CapitalInfo.model_json_schema(),
+        },
+    },
+)
+
+print_highlight(
+    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
+)
+```
+
+**JSON Schema Directly**
+
+
+
+
+```python Example
+import json
+
+json_schema = json.dumps(
+    {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string", "pattern": "^[\\w]+$"},
+            "population": {"type": "integer"},
+        },
+        "required": ["name", "population"],
+    }
+)
+
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
+    messages=[
+        {
+            "role": "assistant",
+            "content": "Give me the information and population of the capital of France in the JSON format.",
+        },
+    ],
+    temperature=0,
+    max_tokens=2048,
+    response_format={
+        "type": "json_schema",
+        "json_schema": {"name": "foo", "schema": json.loads(json_schema)},
+    },
+)
+
+print_highlight(
+    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
+)
+```
+
+### EBNF
+
+
+
+```python Example
+ebnf_grammar = """
+root ::= city | description
+city ::= "London" | "Paris" | "Berlin" | "Rome"
+description ::= city " is " status
+status ::= "the capital of " country
+country ::= "England" | "France" | "Germany" | "Italy"
+"""
+
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
+    messages=[
+        {"role": "system", "content": "You are a helpful geography bot."},
+        {
+            "role": "assistant",
+            "content": "Give me the information and population of the capital of France in the JSON format.",
+        },
+    ],
+    temperature=0,
+    max_tokens=2048,
+    extra_body={"ebnf": ebnf_grammar},
+)
+
+print_highlight(
+    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
+)
+```
+
+### Regular expression
+
+
+
+```python Example
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
+    messages=[
+        {"role": "assistant", "content": "What is the capital of France?"},
+    ],
+    temperature=0,
+    max_tokens=2048,
+    extra_body={"regex": "(Paris|London)"},
+)
+
+print_highlight(
+    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
+)
+```
+
+### Structural Tag
+
+
+
+```python Example
+tool_get_current_weather = {
+    "type": "function",
+    "function": {
+        "name": "get_current_weather",
+        "description": "Get the current weather in a given location",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "city": {
+                    "type": "string",
+                    "description": "The city to find the weather for, e.g. 'San Francisco'",
+                },
+                "state": {
+                    "type": "string",
+                    "description": "the two-letter abbreviation for the state that the city is"
+                    " in, e.g. 'CA' which would mean 'California'",
+                },
+                "unit": {
+                    "type": "string",
+                    "description": "The unit to fetch the temperature in",
+                    "enum": ["celsius", "fahrenheit"],
+                },
+            },
+            "required": ["city", "state", "unit"],
+        },
+    },
+}
+
+tool_get_current_date = {
+    "type": "function",
+    "function": {
+        "name": "get_current_date",
+        "description": "Get the current date and time for a given timezone",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "timezone": {
+                    "type": "string",
+                    "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'",
+                }
+            },
+            "required": ["timezone"],
+        },
+    },
+}
+
+schema_get_current_weather = tool_get_current_weather["function"]["parameters"]
+schema_get_current_date = tool_get_current_date["function"]["parameters"]
+
+
+def get_messages():
+    return [
+        {
+            "role": "system",
+            "content": f"""
+# Tool Instructions
+- Always execute python code in messages that you share.
+- When looking for real time information use relevant functions if available else fallback to brave_search
+You have access to the following functions:
+Use the function 'get_current_weather' to: Get the current weather in a given location
+{tool_get_current_weather["function"]}
+Use the function 'get_current_date' to: Get the current date and time for a given timezone
+{tool_get_current_date["function"]}
+If a you choose to call a function ONLY reply in the following format:
+<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}
+where
+start_tag => `<function`
+parameters => a JSON dict with the function argument name as key and function argument value as value.
+end_tag => `</function>`
+Here is an example,
+<function=example_function_name>{{"example_name": "example_value"}}</function>
+Reminder:
+- Function calls MUST follow the specified format
+- Required parameters MUST be specified
+- Only call one function at a time
+- Put the entire function call reply on one line
+- Always add your sources when using search results to answer the user query
+You are a helpful assistant.""",
+        },
+        {
+            "role": "assistant",
+            "content": "You are in New York. Please get the current date and time, and the weather.",
+        },
+    ]
+
+
+messages = get_messages()
+
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
+    messages=messages,
+    response_format={
+        "type": "structural_tag",
+        "max_new_tokens": 2048,
+        "structures": [
+            {
+                "begin": "<function=get_current_weather>",
+                "schema": schema_get_current_weather,
+                "end": "</function>",
+            },
+            {
+                "begin": "<function=get_current_date>",
+                "schema": schema_get_current_date,
+                "end": "</function>",
+            },
+        ],
+        "triggers": ["<function="],
+    },
+)
+
+print_highlight(
+    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
+)
+```
+
+## Native API and SGLang Runtime (SRT)
+
+
+> Note: For native API, as a work-around, you need to set `require_reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API.
+
+
+### JSON
+
+
+**Using Pydantic**
+
+
+
+```python Example
+import requests
+from pydantic import BaseModel, Field
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
+
+
+# Define the schema using Pydantic
+class CapitalInfo(BaseModel):
+    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
+    population: int = Field(..., description="Population of the capital city")
+
+
+messages = [
+    {
+        "role": "assistant",
+        "content": "Give me the information and population of the capital of France in the JSON format.",
+    },
+]
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+# Make API request
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": text,
+        "require_reasoning": True,
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 2048,
+            "json_schema": json.dumps(CapitalInfo.model_json_schema()),
+        },
+    },
+)
+print(response.json())
+
+
+reasoing_content = response.json()["text"].split("</think>")[0]
+content = response.json()["text"].split("</think>")[1]
+print_highlight(f"reasoing_content: {reasoing_content}\n\ncontent: {content}")
+```
+
+**JSON Schema Directly**
+
+
+
+```python Example
+json_schema = json.dumps(
+    {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string", "pattern": "^[\\w]+$"},
+            "population": {"type": "integer"},
+        },
+        "required": ["name", "population"],
+    }
+)
+
+# JSON
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": text,
+        "require_reasoning": True,
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 2048,
+            "json_schema": json_schema,
+        },
+    },
+)
+
+print_highlight(response.json())
+```
+
+### EBNF
+
+
+
+```python Example
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": "Give me the information of the capital of France.",
+        "require_reasoning": True,
+        "sampling_params": {
+            "max_new_tokens": 2048,
+            "temperature": 0,
+            "n": 3,
+            "ebnf": (
+                "root ::= city | description\n"
+                'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
+                'description ::= city " is " status\n'
+                'status ::= "the capital of " country\n'
+                'country ::= "England" | "France" | "Germany" | "Italy"'
+            ),
+        },
+        "stream": False,
+        "return_logprob": False,
+    },
+)
+
+print(response.json())
+```
+
+### Regular expression
+
+
+
+```python Example
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": "Paris is the capital of",
+        "require_reasoning": True,
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 2048,
+            "regex": "(France|England)",
+        },
+    },
+)
+print(response.json())
+```
+
+### Structural Tag
+
+
+
+```python Example
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+payload = {
+    "text": text,
+    "require_reasoning": True,
+    "sampling_params": {
+        "max_new_tokens": 2048,
+        "structural_tag": json.dumps(
+            {
+                "type": "structural_tag",
+                "structures": [
+                    {
+                        "begin": "<function=get_current_weather>",
+                        "schema": schema_get_current_weather,
+                        "end": "</function>",
+                    },
+                    {
+                        "begin": "<function=get_current_date>",
+                        "schema": schema_get_current_date,
+                        "end": "</function>",
+                    },
+                ],
+                "triggers": ["<function="],
+            }
+        ),
+    },
+}
+
+
+# Send POST request to the API endpoint
+response = requests.post(f"http://localhost:{port}/generate", json=payload)
+print_highlight(response.json())
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+## Offline Engine API
+
+
+
+```python Example
+import sglang as sgl
+
+llm = sgl.Engine(
+    model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
+    reasoning_parser="deepseek-r1",
+    grammar_backend="xgrammar",
+)
+```
+
+### JSON
+
+
+**Using Pydantic**
+
+
+
+```python Example
+import json
+from pydantic import BaseModel, Field
+
+
+prompts = [
+    "Give me the information of the capital of China in the JSON format.",
+    "Give me the information of the capital of France in the JSON format.",
+    "Give me the information of the capital of Ireland in the JSON format.",
+]
+
+
+# Define the schema using Pydantic
+class CapitalInfo(BaseModel):
+    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
+    population: int = Field(..., description="Population of the capital city")
+
+
+sampling_params = {
+    "temperature": 0,
+    "top_p": 0.95,
+    "max_new_tokens": 2048,
+    "json_schema": json.dumps(CapitalInfo.model_json_schema()),
+}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print("===============================")
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+**JSON Schema Directly**
+
+
+
+```python Example
+prompts = [
+    "Give me the information of the capital of China in the JSON format.",
+    "Give me the information of the capital of France in the JSON format.",
+    "Give me the information of the capital of Ireland in the JSON format.",
+]
+
+json_schema = json.dumps(
+    {
+        "type": "object",
+        "properties": {
+            "name": {"type": "string", "pattern": "^[\\w]+$"},
+            "population": {"type": "integer"},
+        },
+        "required": ["name", "population"],
+    }
+)
+
+sampling_params = {"temperature": 0, "max_new_tokens": 2048, "json_schema": json_schema}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print("===============================")
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+### EBNF
+
+
+
+
+```python Example
+prompts = [
+    "Give me the information of the capital of France.",
+    "Give me the information of the capital of Germany.",
+    "Give me the information of the capital of Italy.",
+]
+
+sampling_params = {
+    "temperature": 0.8,
+    "top_p": 0.95,
+    "ebnf": (
+        "root ::= city | description\n"
+        'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
+        'description ::= city " is " status\n'
+        'status ::= "the capital of " country\n'
+        'country ::= "England" | "France" | "Germany" | "Italy"'
+    ),
+}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print("===============================")
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+### Regular expression
+
+
+
+```python Example
+prompts = [
+    "Please provide information about London as a major global city:",
+    "Please provide information about Paris as a major global city:",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print("===============================")
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+
+```python Example
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, return_dict=False
+)
+prompts = [text]
+
+
+sampling_params = {
+    "temperature": 0.8,
+    "top_p": 0.95,
+    "max_new_tokens": 2048,
+    "structural_tag": json.dumps(
+        {
+            "type": "structural_tag",
+            "structures": [
+                {
+                    "begin": "<function=get_current_weather>",
+                    "schema": schema_get_current_weather,
+                    "end": "</function>",
+                },
+                {
+                    "begin": "<function=get_current_date>",
+                    "schema": schema_get_current_date,
+                    "end": "</function>",
+                },
+            ],
+            "triggers": ["<function="],
+        }
+    ),
+}
+
+
+# Send POST request to the API endpoint
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print("===============================")
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+
+```python Example
+llm.shutdown()
+```

docs_new/docs/advanced_features/tool_parser.ipynb

@@ -0,0 +1,857 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tool Parser\n",
+    "\n",
+    "This guide demonstrates how to use SGLang’s [Function calling](https://platform.openai.com/docs/guides/function-calling) functionality."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Currently supported parsers:\n",
+    "\n",
+    "| Parser | Supported Models | Notes |\n",
+    "|---|---|---|\n",
+    "| `apertus2509` | Apertus 2509 (e.g., `swiss-ai/Apertus-{8,70}B-Instruct-2509`) | Tool calls are emitted as a JSON list of single-key objects: `<\\|tools_prefix\\|>[{\"tool\": {...}}]<\\|tools_suffix\\|>`. |\n",
+    "| `deepseekv3` | DeepSeek-v3 (e.g., `deepseek-ai/DeepSeek-V3-0324`) | Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` to launch command. |\n",
+    "| `deepseekv31` | DeepSeek-V3.1 and DeepSeek-V3.2-Exp (e.g. `deepseek-ai/DeepSeek-V3.1`, `deepseek-ai/DeepSeek-V3.2-Exp`) | Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja` (Or ..deepseekv32.jinja for DeepSeek-V3.2) to launch command. |\n",
+    "| `deepseekv32` | DeepSeek-V3.2 (`deepseek-ai/DeepSeek-V3.2`) | |\n",
+    "| `glm` | GLM series (e.g. `zai-org/GLM-4.6`) | |\n",
+    "| `gpt-oss` | GPT-OSS (e.g., `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `lmsys/gpt-oss-120b-bf16`, `lmsys/gpt-oss-20b-bf16`) | The gpt-oss tool parser filters out analysis channel events and only preserves normal text. This can cause the content to be empty when explanations are in the analysis channel. To work around this, complete the tool round by returning tool results as `role=\"tool\"` messages, which enables the model to generate the final content. |\n",
+    "| `kimi_k2` | `moonshotai/Kimi-K2-Instruct` | |\n",
+    "| `llama3` | Llama 3.1 / 3.2 / 3.3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.3-70B-Instruct`) | |\n",
+    "| `llama4` | Llama 4 (e.g. `meta-llama/Llama-4-Scout-17B-16E-Instruct`) | |\n",
+    "| `mistral` | Mistral (e.g. `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mistral-Nemo-Instruct-2407`, `mistralai/Mistral-7B-v0.3`) | |\n",
+    "| `pythonic` | Llama-3.2 / Llama-3.3 / Llama-4 | Model outputs function calls as Python code. Requires `--tool-call-parser pythonic` and is recommended to use with a specific chat template. |\n",
+    "| `qwen` | Qwen series (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-VL-30B-A3B-Thinking`) except Qwen3-Coder| |\n",
+    "| `qwen3_coder` | Qwen3-Coder (e.g. `Qwen/Qwen3-Coder-30B-A3B-Instruct`) | |\n",
+    "| `step3` | Step-3 | |\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## OpenAI Compatible API"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Launching the Server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning\"  # qwen25\n",
+    ")\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that `--tool-call-parser` defines the parser used to interpret responses."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Define Tools for Function Call\n",
+    "Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Define tools\n",
+    "tools = [\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"get_current_weather\",\n",
+    "            \"description\": \"Get the current weather in a given location\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"city\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
+    "                    },\n",
+    "                    \"state\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"the two-letter abbreviation for the state that the city is\"\n",
+    "                        \" in, e.g. 'CA' which would mean 'California'\",\n",
+    "                    },\n",
+    "                    \"unit\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The unit to fetch the temperature in\",\n",
+    "                        \"enum\": [\"celsius\", \"fahrenheit\"],\n",
+    "                    },\n",
+    "                },\n",
+    "                \"required\": [\"city\", \"state\", \"unit\"],\n",
+    "            },\n",
+    "        },\n",
+    "    }\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Define Messages"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def get_messages():\n",
+    "    return [\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.\",\n",
+    "        }\n",
+    "    ]\n",
+    "\n",
+    "\n",
+    "messages = get_messages()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Initialize the Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Initialize OpenAI-like client\n",
+    "client = OpenAI(api_key=\"None\", base_url=f\"http://0.0.0.0:{port}/v1\")\n",
+    "model_name = client.models.list().data[0].id"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "###  Non-Streaming Request"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Non-streaming mode test\n",
+    "response_non_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0,\n",
+    "    top_p=0.95,\n",
+    "    max_tokens=1024,\n",
+    "    stream=False,  # Non-streaming\n",
+    "    tools=tools,\n",
+    ")\n",
+    "print_highlight(\"Non-stream response:\")\n",
+    "print_highlight(response_non_stream)\n",
+    "print_highlight(\"==== content ====\")\n",
+    "print_highlight(response_non_stream.choices[0].message.content)\n",
+    "print_highlight(\"==== tool_calls ====\")\n",
+    "print_highlight(response_non_stream.choices[0].message.tool_calls)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Handle Tools\n",
+    "When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name\n",
+    "arguments_non_stream = (\n",
+    "    response_non_stream.choices[0].message.tool_calls[0].function.arguments\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Final streamed function call name: {name_non_stream}\")\n",
+    "print_highlight(f\"Final streamed function call arguments: {arguments_non_stream}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming Request"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Streaming mode test\n",
+    "print_highlight(\"Streaming response:\")\n",
+    "response_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0,\n",
+    "    top_p=0.95,\n",
+    "    max_tokens=1024,\n",
+    "    stream=True,  # Enable streaming\n",
+    "    tools=tools,\n",
+    ")\n",
+    "\n",
+    "texts = \"\"\n",
+    "tool_calls = []\n",
+    "name = \"\"\n",
+    "arguments = \"\"\n",
+    "for chunk in response_stream:\n",
+    "    if chunk.choices[0].delta.content:\n",
+    "        texts += chunk.choices[0].delta.content\n",
+    "    if chunk.choices[0].delta.tool_calls:\n",
+    "        tool_calls.append(chunk.choices[0].delta.tool_calls[0])\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(texts)\n",
+    "\n",
+    "print_highlight(\"==== Tool Call ====\")\n",
+    "for tool_call in tool_calls:\n",
+    "    print_highlight(tool_call)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Handle Tools\n",
+    "When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Parse and combine function call arguments\n",
+    "arguments = []\n",
+    "for tool_call in tool_calls:\n",
+    "    if tool_call.function.name:\n",
+    "        print_highlight(f\"Streamed function call name: {tool_call.function.name}\")\n",
+    "\n",
+    "    if tool_call.function.arguments:\n",
+    "        arguments.append(tool_call.function.arguments)\n",
+    "\n",
+    "# Combine all fragments into a single JSON string\n",
+    "full_arguments = \"\".join(arguments)\n",
+    "print_highlight(f\"streamed function call arguments: {full_arguments}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Define a Tool Function"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This is a demonstration, define real function according to your usage.\n",
+    "def get_current_weather(city: str, state: str, unit: \"str\"):\n",
+    "    return (\n",
+    "        f\"The weather in {city}, {state} is 85 degrees {unit}. It is \"\n",
+    "        \"partly cloudly, with highs in the 90's.\"\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "available_tools = {\"get_current_weather\": get_current_weather}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "### Execute the Tool"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "messages.append(response_non_stream.choices[0].message)\n",
+    "\n",
+    "# Call the corresponding tool function\n",
+    "tool_call = messages[-1].tool_calls[0]\n",
+    "tool_name = tool_call.function.name\n",
+    "tool_to_call = available_tools[tool_name]\n",
+    "result = tool_to_call(**(json.loads(tool_call.function.arguments)))\n",
+    "print_highlight(f\"Function call result: {result}\")\n",
+    "# messages.append({\"role\": \"tool\", \"content\": result, \"name\": tool_name})\n",
+    "messages.append(\n",
+    "    {\n",
+    "        \"role\": \"tool\",\n",
+    "        \"tool_call_id\": tool_call.id,\n",
+    "        \"content\": str(result),\n",
+    "        \"name\": tool_name,\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Updated message history: {messages}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Send Results Back to Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "final_response = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0,\n",
+    "    top_p=0.95,\n",
+    "    stream=False,\n",
+    "    tools=tools,\n",
+    ")\n",
+    "print_highlight(\"Non-stream response:\")\n",
+    "print_highlight(final_response)\n",
+    "\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(final_response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Native API and SGLang Runtime (SRT)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "import requests\n",
+    "\n",
+    "# generate an answer\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-7B-Instruct\")\n",
+    "\n",
+    "messages = get_messages()\n",
+    "\n",
+    "input = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=False, add_generation_prompt=True, tools=tools, return_dict=False\n",
+    ")\n",
+    "\n",
+    "gen_url = f\"http://localhost:{port}/generate\"\n",
+    "gen_data = {\n",
+    "    \"text\": input,\n",
+    "    \"sampling_params\": {\n",
+    "        \"skip_special_tokens\": False,\n",
+    "        \"max_new_tokens\": 1024,\n",
+    "        \"temperature\": 0,\n",
+    "        \"top_p\": 0.95,\n",
+    "    },\n",
+    "}\n",
+    "gen_response = requests.post(gen_url, json=gen_data).json()[\"text\"]\n",
+    "print_highlight(\"==== Response ====\")\n",
+    "print_highlight(gen_response)\n",
+    "\n",
+    "# parse the response\n",
+    "parse_url = f\"http://localhost:{port}/parse_function_call\"\n",
+    "\n",
+    "function_call_input = {\n",
+    "    \"text\": gen_response,\n",
+    "    \"tool_call_parser\": \"qwen25\",\n",
+    "    \"tools\": tools,\n",
+    "}\n",
+    "\n",
+    "function_call_response = requests.post(parse_url, json=function_call_input)\n",
+    "function_call_response_json = function_call_response.json()\n",
+    "\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print(function_call_response_json[\"normal_text\"])\n",
+    "print_highlight(\"==== Calls ====\")\n",
+    "print(\"function name: \", function_call_response_json[\"calls\"][0][\"name\"])\n",
+    "print(\"function arguments: \", function_call_response_json[\"calls\"][0][\"parameters\"])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Offline Engine API"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sglang as sgl\n",
+    "from sglang.srt.function_call.function_call_parser import FunctionCallParser\n",
+    "from sglang.srt.managers.io_struct import Tool, Function\n",
+    "\n",
+    "llm = sgl.Engine(model_path=\"Qwen/Qwen2.5-7B-Instruct\")\n",
+    "tokenizer = llm.tokenizer_manager.tokenizer\n",
+    "input_ids = tokenizer.apply_chat_template(\n",
+    "    messages, tokenize=True, add_generation_prompt=True, tools=tools, return_dict=False\n",
+    ")\n",
+    "\n",
+    "# Note that for gpt-oss tool parser, adding \"no_stop_trim\": True\n",
+    "# to make sure the tool call token <call> is not trimmed.\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"max_new_tokens\": 1024,\n",
+    "    \"temperature\": 0,\n",
+    "    \"top_p\": 0.95,\n",
+    "    \"skip_special_tokens\": False,\n",
+    "}\n",
+    "\n",
+    "# 1) Offline generation\n",
+    "result = llm.generate(input_ids=input_ids, sampling_params=sampling_params)\n",
+    "generated_text = result[\"text\"]  # Assume there is only one prompt\n",
+    "\n",
+    "print_highlight(\"=== Offline Engine Output Text ===\")\n",
+    "print_highlight(generated_text)\n",
+    "\n",
+    "\n",
+    "# 2) Parse using FunctionCallParser\n",
+    "def convert_dict_to_tool(tool_dict: dict) -> Tool:\n",
+    "    function_dict = tool_dict.get(\"function\", {})\n",
+    "    return Tool(\n",
+    "        type=tool_dict.get(\"type\", \"function\"),\n",
+    "        function=Function(\n",
+    "            name=function_dict.get(\"name\"),\n",
+    "            description=function_dict.get(\"description\"),\n",
+    "            parameters=function_dict.get(\"parameters\"),\n",
+    "        ),\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools]\n",
+    "\n",
+    "parser = FunctionCallParser(tools=tools, tool_call_parser=\"qwen25\")\n",
+    "normal_text, calls = parser.parse_non_stream(generated_text)\n",
+    "\n",
+    "print_highlight(\"=== Parsing Result ===\")\n",
+    "print(\"Normal text portion:\", normal_text)\n",
+    "print_highlight(\"Function call portion:\")\n",
+    "for call in calls:\n",
+    "    # call: ToolCallItem\n",
+    "    print_highlight(f\"  - tool name: {call.name}\")\n",
+    "    print_highlight(f\"    parameters: {call.parameters}\")\n",
+    "\n",
+    "# 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm.shutdown()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Tool Choice Mode\n",
+    "\n",
+    "SGLang supports OpenAI's `tool_choice` parameter to control when and which tools the model should call. This feature is implemented using EBNF (Extended Backus-Naur Form) grammar to ensure reliable tool calling behavior.\n",
+    "\n",
+    "### Supported Tool Choice Options\n",
+    "\n",
+    "- **`tool_choice=\"required\"`**: Forces the model to call at least one tool\n",
+    "- **`tool_choice={\"type\": \"function\", \"function\": {\"name\": \"specific_function\"}}`**: Forces the model to call a specific function\n",
+    "\n",
+    "### Backend Compatibility\n",
+    "\n",
+    "Tool choice is fully supported with the **Xgrammar backend**, which is the default grammar backend (`--grammar-backend xgrammar`). However, it may not be fully supported with other backends such as `outlines`.\n",
+    "\n",
+    "### Example: Required Tool Choice"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "\n",
+    "# Start a new server session for tool choice examples\n",
+    "server_process_tool_choice, port_tool_choice = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0  --log-level warning\"\n",
+    ")\n",
+    "wait_for_server(\n",
+    "    f\"http://localhost:{port_tool_choice}\", process=server_process_tool_choice\n",
+    ")\n",
+    "\n",
+    "# Initialize client for tool choice examples\n",
+    "client_tool_choice = OpenAI(\n",
+    "    api_key=\"None\", base_url=f\"http://0.0.0.0:{port_tool_choice}/v1\"\n",
+    ")\n",
+    "model_name_tool_choice = client_tool_choice.models.list().data[0].id\n",
+    "\n",
+    "# Example with tool_choice=\"required\" - forces the model to call a tool\n",
+    "messages_required = [\n",
+    "    {\"role\": \"user\", \"content\": \"Hello, what is the capital of France?\"}\n",
+    "]\n",
+    "\n",
+    "# Define tools\n",
+    "tools = [\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"get_current_weather\",\n",
+    "            \"description\": \"Get the current weather in a given location\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"city\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
+    "                    },\n",
+    "                    \"unit\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The unit to fetch the temperature in\",\n",
+    "                        \"enum\": [\"celsius\", \"fahrenheit\"],\n",
+    "                    },\n",
+    "                },\n",
+    "                \"required\": [\"city\", \"unit\"],\n",
+    "            },\n",
+    "        },\n",
+    "    }\n",
+    "]\n",
+    "\n",
+    "response_required = client_tool_choice.chat.completions.create(\n",
+    "    model=model_name_tool_choice,\n",
+    "    messages=messages_required,\n",
+    "    temperature=0,\n",
+    "    max_tokens=1024,\n",
+    "    tools=tools,\n",
+    "    tool_choice=\"required\",  # Force the model to call a tool\n",
+    ")\n",
+    "\n",
+    "print_highlight(\"Response with tool_choice='required':\")\n",
+    "print(\"Content:\", response_required.choices[0].message.content)\n",
+    "print(\"Tool calls:\", response_required.choices[0].message.tool_calls)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Example: Specific Function Choice\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example with specific function choice - forces the model to call a specific function\n",
+    "messages_specific = [\n",
+    "    {\"role\": \"user\", \"content\": \"What are the most attactive places in France?\"}\n",
+    "]\n",
+    "\n",
+    "response_specific = client_tool_choice.chat.completions.create(\n",
+    "    model=model_name_tool_choice,\n",
+    "    messages=messages_specific,\n",
+    "    temperature=0,\n",
+    "    max_tokens=1024,\n",
+    "    tools=tools,\n",
+    "    tool_choice={\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\"name\": \"get_current_weather\"},\n",
+    "    },  # Force the model to call the specific get_current_weather function\n",
+    ")\n",
+    "\n",
+    "print_highlight(\"Response with specific function choice:\")\n",
+    "print(\"Content:\", response_specific.choices[0].message.content)\n",
+    "print(\"Tool calls:\", response_specific.choices[0].message.tool_calls)\n",
+    "\n",
+    "if response_specific.choices[0].message.tool_calls:\n",
+    "    tool_call = response_specific.choices[0].message.tool_calls[0]\n",
+    "    print_highlight(f\"Called function: {tool_call.function.name}\")\n",
+    "    print_highlight(f\"Arguments: {tool_call.function.arguments}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process_tool_choice)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4)\n",
+    "\n",
+    "Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a \"pythonic\" tool call format, where the model outputs function calls as Python code, e.g.:\n",
+    "\n",
+    "```python\n",
+    "[get_current_weather(city=\"San Francisco\", state=\"CA\", unit=\"celsius\")]\n",
+    "```\n",
+    "\n",
+    "- The output is a Python list of function calls, with arguments as Python literals (not JSON).\n",
+    "- Multiple tool calls can be returned in the same list:\n",
+    "```python\n",
+    "[get_current_weather(city=\"San Francisco\", state=\"CA\", unit=\"celsius\"),\n",
+    " get_current_weather(city=\"New York\", state=\"NY\", unit=\"fahrenheit\")]\n",
+    "```\n",
+    "\n",
+    "For more information, refer to Meta’s documentation on  [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message).\n",
+    "\n",
+    "Note that this feature is still under development on Blackwell.\n",
+    "\n",
+    "### How to enable\n",
+    "- Launch the server with `--tool-call-parser pythonic`\n",
+    "- You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`).\n",
+    "This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results.\n",
+    "\n",
+    "#### Forcing Pythonic Tool Call Output Without a Chat Template\n",
+    "If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \" python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1  --log-level warning\"  # llama-3.2-1b-instruct\n",
+    ")\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
+    "\n",
+    "tools = [\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"get_weather\",\n",
+    "            \"description\": \"Get the current weather for a given location.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"location\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The name of the city or location.\",\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"location\"],\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "    {\n",
+    "        \"type\": \"function\",\n",
+    "        \"function\": {\n",
+    "            \"name\": \"get_tourist_attractions\",\n",
+    "            \"description\": \"Get a list of top tourist attractions for a given city.\",\n",
+    "            \"parameters\": {\n",
+    "                \"type\": \"object\",\n",
+    "                \"properties\": {\n",
+    "                    \"city\": {\n",
+    "                        \"type\": \"string\",\n",
+    "                        \"description\": \"The name of the city to find attractions for.\",\n",
+    "                    }\n",
+    "                },\n",
+    "                \"required\": [\"city\"],\n",
+    "            },\n",
+    "        },\n",
+    "    },\n",
+    "]\n",
+    "\n",
+    "\n",
+    "def get_messages():\n",
+    "    return [\n",
+    "        {\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": (\n",
+    "                \"You are a travel assistant. \"\n",
+    "                \"When asked to call functions, ALWAYS respond ONLY with a python list of function calls, \"\n",
+    "                \"using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. \"\n",
+    "                \"Do NOT use JSON, do NOT use variables, do NOT use any other format. \"\n",
+    "                \"Here is an example:\\n\"\n",
+    "                '[get_weather(location=\"Paris\"), get_tourist_attractions(city=\"Paris\")]'\n",
+    "            ),\n",
+    "        },\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": (\n",
+    "                \"I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? \"\n",
+    "                \"Propose parallel tool calls at once, using the python list of function calls format as shown above.\"\n",
+    "            ),\n",
+    "        },\n",
+    "    ]\n",
+    "\n",
+    "\n",
+    "messages = get_messages()\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://localhost:{port}/v1\", api_key=\"xxxxxx\")\n",
+    "model_name = client.models.list().data[0].id\n",
+    "\n",
+    "\n",
+    "response_non_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0,\n",
+    "    top_p=0.9,\n",
+    "    stream=False,  # Non-streaming\n",
+    "    tools=tools,\n",
+    ")\n",
+    "print_highlight(\"Non-stream response:\")\n",
+    "print_highlight(response_non_stream)\n",
+    "\n",
+    "response_stream = client.chat.completions.create(\n",
+    "    model=model_name,\n",
+    "    messages=messages,\n",
+    "    temperature=0,\n",
+    "    top_p=0.9,\n",
+    "    stream=True,\n",
+    "    tools=tools,\n",
+    ")\n",
+    "texts = \"\"\n",
+    "tool_calls = []\n",
+    "name = \"\"\n",
+    "arguments = \"\"\n",
+    "\n",
+    "for chunk in response_stream:\n",
+    "    if chunk.choices[0].delta.content:\n",
+    "        texts += chunk.choices[0].delta.content\n",
+    "    if chunk.choices[0].delta.tool_calls:\n",
+    "        tool_calls.append(chunk.choices[0].delta.tool_calls[0])\n",
+    "\n",
+    "print_highlight(\"Streaming Response:\")\n",
+    "print_highlight(\"==== Text ====\")\n",
+    "print_highlight(texts)\n",
+    "\n",
+    "print_highlight(\"==== Tool Call ====\")\n",
+    "for tool_call in tool_calls:\n",
+    "    print_highlight(tool_call)\n",
+    "\n",
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> **Note:**  \n",
+    "> The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## How to support a new model?\n",
+    "1. Update the TOOLS_TAG_LIST in sglang/srt/function_call_parser.py with the model’s tool tags. Currently supported tags include:\n",
+    "```\n",
+    "\tTOOLS_TAG_LIST = [\n",
+    "\t    “<|plugin|>“,\n",
+    "\t    “<function=“,\n",
+    "\t    “<tool_call>“,\n",
+    "\t    “<|python_tag|>“,\n",
+    "\t    “[TOOL_CALLS]”\n",
+    "\t]\n",
+    "```\n",
+    "2. Create a new detector class in sglang/srt/function_call_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example:\n",
+    "```\n",
+    "    class NewModelDetector(BaseFormatDetector):\n",
+    "```\n",
+    "3. Add the new detector to the MultiFormatParser class that manages all the format detectors."
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs_new/docs/advanced_features/tool_parser.mdx

@@ -0,0 +1,745 @@
+---
+title: "Tool Parser"
+metatags:
+    description: "SGLang function calling: tool parsers for DeepSeek, Llama, Qwen, Mistral, GLM, Kimi K2, and Apertus. OpenAI-compatible tool use API."
+---
+This guide demonstrates how to use SGLang’s [Function calling](https://platform.openai.com/docs/guides/function-calling) functionality.
+
+
+## Currently supported parsers:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parser</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Models</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`apertus2509`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Apertus 2509 (e.g., `swiss-ai/Apertus-{8,70}B-Instruct-2509`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Tool calls are emitted as a JSON list of single-key objects: `<|tools_prefix|>[{"tool": {...}}]<|tools_suffix|>`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseekv3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-v3 (e.g., `deepseek-ai/DeepSeek-V3-0324`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` to launch command.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseekv31`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-V3.1 and DeepSeek-V3.2-Exp (e.g. `deepseek-ai/DeepSeek-V3.1`, `deepseek-ai/DeepSeek-V3.2-Exp`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja` (Or ..deepseekv32.jinja for DeepSeek-V3.2) to launch command.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`deepseekv32`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-V3.2 (`deepseek-ai/DeepSeek-V3.2`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`glm`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM series (e.g. `zai-org/GLM-4.6`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`gpt-oss`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GPT-OSS (e.g., `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `lmsys/gpt-oss-120b-bf16`, `lmsys/gpt-oss-20b-bf16`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The gpt-oss tool parser filters out analysis channel events and only preserves normal text. This can cause the content to be empty when explanations are in the analysis channel. To work around this, complete the tool round by returning tool results as `role="tool"` messages, which enables the model to generate the final content.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`kimi_k2`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`moonshotai/Kimi-K2-Instruct`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`llama3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama 3.1 / 3.2 / 3.3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.3-70B-Instruct`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`llama4`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama 4 (e.g. `meta-llama/Llama-4-Scout-17B-16E-Instruct`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`mistral`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral (e.g. `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mistral-Nemo-Instruct-2407`, `mistralai/Mistral-7B-v0.3`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`pythonic`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama-3.2 / Llama-3.3 / Llama-4</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Model outputs function calls as Python code. Requires `--tool-call-parser pythonic` and is recommended to use with a specific chat template.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen series (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-VL-30B-A3B-Thinking`) except Qwen3-Coder</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`qwen3_coder`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-Coder (e.g. `Qwen/Qwen3-Coder-30B-A3B-Instruct`)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`step3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Step-3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
+    </tr>
+  </tbody>
+</table>
+
+
+
+## OpenAI Compatible API
+
+
+### Launching the Server
+
+
+
+```python Example
+import json
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+from openai import OpenAI
+
+server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning"  # qwen25
+)
+wait_for_server(f"http://localhost:{port}")
+```
+
+Note that `--tool-call-parser` defines the parser used to interpret responses.
+
+
+### Define Tools for Function Call
+Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters.
+
+
+
+```python Example
+# Define tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_current_weather",
+            "description": "Get the current weather in a given location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "city": {
+                        "type": "string",
+                        "description": "The city to find the weather for, e.g. 'San Francisco'",
+                    },
+                    "state": {
+                        "type": "string",
+                        "description": "the two-letter abbreviation for the state that the city is"
+                        " in, e.g. 'CA' which would mean 'California'",
+                    },
+                    "unit": {
+                        "type": "string",
+                        "description": "The unit to fetch the temperature in",
+                        "enum": ["celsius", "fahrenheit"],
+                    },
+                },
+                "required": ["city", "state", "unit"],
+            },
+        },
+    }
+]
+```
+
+### Define Messages
+
+
+
+```python Example
+def get_messages():
+    return [
+        {
+            "role": "user",
+            "content": "What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.",
+        }
+    ]
+
+
+messages = get_messages()
+```
+
+### Initialize the Client
+
+
+
+```python Example
+# Initialize OpenAI-like client
+client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1")
+model_name = client.models.list().data[0].id
+```
+
+###  Non-Streaming Request
+
+
+
+```python Example
+# Non-streaming mode test
+response_non_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0,
+    top_p=0.95,
+    max_tokens=1024,
+    stream=False,  # Non-streaming
+    tools=tools,
+)
+print_highlight("Non-stream response:")
+print_highlight(response_non_stream)
+print_highlight("==== content ====")
+print_highlight(response_non_stream.choices[0].message.content)
+print_highlight("==== tool_calls ====")
+print_highlight(response_non_stream.choices[0].message.tool_calls)
+```
+
+#### Handle Tools
+When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly.
+
+
+
+```python Example
+name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name
+arguments_non_stream = (
+    response_non_stream.choices[0].message.tool_calls[0].function.arguments
+)
+
+print_highlight(f"Final streamed function call name: {name_non_stream}")
+print_highlight(f"Final streamed function call arguments: {arguments_non_stream}")
+```
+
+### Streaming Request
+
+
+
+```python Example
+# Streaming mode test
+print_highlight("Streaming response:")
+response_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0,
+    top_p=0.95,
+    max_tokens=1024,
+    stream=True,  # Enable streaming
+    tools=tools,
+)
+
+texts = ""
+tool_calls = []
+name = ""
+arguments = ""
+for chunk in response_stream:
+    if chunk.choices[0].delta.content:
+        texts += chunk.choices[0].delta.content
+    if chunk.choices[0].delta.tool_calls:
+        tool_calls.append(chunk.choices[0].delta.tool_calls[0])
+print_highlight("==== Text ====")
+print_highlight(texts)
+
+print_highlight("==== Tool Call ====")
+for tool_call in tool_calls:
+    print_highlight(tool_call)
+```
+
+#### Handle Tools
+When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly.
+
+
+
+```python Example
+# Parse and combine function call arguments
+arguments = []
+for tool_call in tool_calls:
+    if tool_call.function.name:
+        print_highlight(f"Streamed function call name: {tool_call.function.name}")
+
+    if tool_call.function.arguments:
+        arguments.append(tool_call.function.arguments)
+
+# Combine all fragments into a single JSON string
+full_arguments = "".join(arguments)
+print_highlight(f"streamed function call arguments: {full_arguments}")
+```
+
+### Define a Tool Function
+
+
+
+```python Example
+# This is a demonstration, define real function according to your usage.
+def get_current_weather(city: str, state: str, unit: "str"):
+    return (
+        f"The weather in {city}, {state} is 85 degrees {unit}. It is "
+        "partly cloudly, with highs in the 90's."
+    )
+
+
+available_tools = {"get_current_weather": get_current_weather}
+```
+
+
+### Execute the Tool
+
+
+
+```python Example
+messages.append(response_non_stream.choices[0].message)
+
+# Call the corresponding tool function
+tool_call = messages[-1].tool_calls[0]
+tool_name = tool_call.function.name
+tool_to_call = available_tools[tool_name]
+result = tool_to_call(**(json.loads(tool_call.function.arguments)))
+print_highlight(f"Function call result: {result}")
+# messages.append({"role": "tool", "content": result, "name": tool_name})
+messages.append(
+    {
+        "role": "tool",
+        "tool_call_id": tool_call.id,
+        "content": str(result),
+        "name": tool_name,
+    }
+)
+
+print_highlight(f"Updated message history: {messages}")
+```
+
+### Send Results Back to Model
+
+
+
+```python Example
+final_response = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0,
+    top_p=0.95,
+    stream=False,
+    tools=tools,
+)
+print_highlight("Non-stream response:")
+print_highlight(final_response)
+
+print_highlight("==== Text ====")
+print_highlight(final_response.choices[0].message.content)
+```
+
+## Native API and SGLang Runtime (SRT)
+
+
+
+```python Example
+from transformers import AutoTokenizer
+import requests
+
+# generate an answer
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
+
+messages = get_messages()
+
+input = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True, tools=tools, return_dict=False
+)
+
+gen_url = f"http://localhost:{port}/generate"
+gen_data = {
+    "text": input,
+    "sampling_params": {
+        "skip_special_tokens": False,
+        "max_new_tokens": 1024,
+        "temperature": 0,
+        "top_p": 0.95,
+    },
+}
+gen_response = requests.post(gen_url, json=gen_data).json()["text"]
+print_highlight("==== Response ====")
+print_highlight(gen_response)
+
+# parse the response
+parse_url = f"http://localhost:{port}/parse_function_call"
+
+function_call_input = {
+    "text": gen_response,
+    "tool_call_parser": "qwen25",
+    "tools": tools,
+}
+
+function_call_response = requests.post(parse_url, json=function_call_input)
+function_call_response_json = function_call_response.json()
+
+print_highlight("==== Text ====")
+print(function_call_response_json["normal_text"])
+print_highlight("==== Calls ====")
+print("function name: ", function_call_response_json["calls"][0]["name"])
+print("function arguments: ", function_call_response_json["calls"][0]["parameters"])
+```
+
+
+```python Example
+terminate_process(server_process)
+```
+
+## Offline Engine API
+
+
+
+```python Example
+import sglang as sgl
+from sglang.srt.function_call.function_call_parser import FunctionCallParser
+from sglang.srt.managers.io_struct import Tool, Function
+
+llm = sgl.Engine(model_path="Qwen/Qwen2.5-7B-Instruct")
+tokenizer = llm.tokenizer_manager.tokenizer
+input_ids = tokenizer.apply_chat_template(
+    messages, tokenize=True, add_generation_prompt=True, tools=tools, return_dict=False
+)
+
+# Note that for gpt-oss tool parser, adding "no_stop_trim": True
+# to make sure the tool call token <call> is not trimmed.
+
+sampling_params = {
+    "max_new_tokens": 1024,
+    "temperature": 0,
+    "top_p": 0.95,
+    "skip_special_tokens": False,
+}
+
+# 1) Offline generation
+result = llm.generate(input_ids=input_ids, sampling_params=sampling_params)
+generated_text = result["text"]  # Assume there is only one prompt
+
+print_highlight("=== Offline Engine Output Text ===")
+print_highlight(generated_text)
+
+
+# 2) Parse using FunctionCallParser
+def convert_dict_to_tool(tool_dict: dict) -> Tool:
+    function_dict = tool_dict.get("function", {})
+    return Tool(
+        type=tool_dict.get("type", "function"),
+        function=Function(
+            name=function_dict.get("name"),
+            description=function_dict.get("description"),
+            parameters=function_dict.get("parameters"),
+        ),
+    )
+
+
+tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools]
+
+parser = FunctionCallParser(tools=tools, tool_call_parser="qwen25")
+normal_text, calls = parser.parse_non_stream(generated_text)
+
+print_highlight("=== Parsing Result ===")
+print("Normal text portion:", normal_text)
+print_highlight("Function call portion:")
+for call in calls:
+    # call: ToolCallItem
+    print_highlight(f"  - tool name: {call.name}")
+    print_highlight(f"    parameters: {call.parameters}")
+
+# 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc.
+```
+
+
+```python Example
+llm.shutdown()
+```
+
+## Tool Choice Mode
+
+SGLang supports OpenAI's `tool_choice` parameter to control when and which tools the model should call. This feature is implemented using EBNF (Extended Backus-Naur Form) grammar to ensure reliable tool calling behavior.
+
+### Supported Tool Choice Options
+
+- **`tool_choice="required"`**: Forces the model to call at least one tool
+- **`tool_choice={"type": "function", "function": {"name": "specific_function"}}`**: Forces the model to call a specific function
+
+### Backend Compatibility
+
+Tool choice is fully supported with the **Xgrammar backend**, which is the default grammar backend (`--grammar-backend xgrammar`). However, it may not be fully supported with other backends such as `outlines`.
+
+### Example: Required Tool Choice
+
+
+
+```python Example
+from openai import OpenAI
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+from sglang.test.doc_patch import launch_server_cmd
+
+# Start a new server session for tool choice examples
+server_process_tool_choice, port_tool_choice = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0  --log-level warning"
+)
+wait_for_server(f"http://localhost:{port_tool_choice}")
+
+# Initialize client for tool choice examples
+client_tool_choice = OpenAI(
+    api_key="None", base_url=f"http://0.0.0.0:{port_tool_choice}/v1"
+)
+model_name_tool_choice = client_tool_choice.models.list().data[0].id
+
+# Example with tool_choice="required" - forces the model to call a tool
+messages_required = [
+    {"role": "user", "content": "Hello, what is the capital of France?"}
+]
+
+# Define tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_current_weather",
+            "description": "Get the current weather in a given location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "city": {
+                        "type": "string",
+                        "description": "The city to find the weather for, e.g. 'San Francisco'",
+                    },
+                    "unit": {
+                        "type": "string",
+                        "description": "The unit to fetch the temperature in",
+                        "enum": ["celsius", "fahrenheit"],
+                    },
+                },
+                "required": ["city", "unit"],
+            },
+        },
+    }
+]
+
+response_required = client_tool_choice.chat.completions.create(
+    model=model_name_tool_choice,
+    messages=messages_required,
+    temperature=0,
+    max_tokens=1024,
+    tools=tools,
+    tool_choice="required",  # Force the model to call a tool
+)
+
+print_highlight("Response with tool_choice='required':")
+print("Content:", response_required.choices[0].message.content)
+print("Tool calls:", response_required.choices[0].message.tool_calls)
+```
+
+### Example: Specific Function Choice
+
+
+
+
+```python Example
+# Example with specific function choice - forces the model to call a specific function
+messages_specific = [
+    {"role": "user", "content": "What are the most attactive places in France?"}
+]
+
+response_specific = client_tool_choice.chat.completions.create(
+    model=model_name_tool_choice,
+    messages=messages_specific,
+    temperature=0,
+    max_tokens=1024,
+    tools=tools,
+    tool_choice={
+        "type": "function",
+        "function": {"name": "get_current_weather"},
+    },  # Force the model to call the specific get_current_weather function
+)
+
+print_highlight("Response with specific function choice:")
+print("Content:", response_specific.choices[0].message.content)
+print("Tool calls:", response_specific.choices[0].message.tool_calls)
+
+if response_specific.choices[0].message.tool_calls:
+    tool_call = response_specific.choices[0].message.tool_calls[0]
+    print_highlight(f"Called function: {tool_call.function.name}")
+    print_highlight(f"Arguments: {tool_call.function.arguments}")
+```
+
+
+```python Example
+terminate_process(server_process_tool_choice)
+```
+
+## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4)
+
+Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a "pythonic" tool call format, where the model outputs function calls as Python code, e.g.:
+
+```python Example
+[get_current_weather(city="San Francisco", state="CA", unit="celsius")]
+```
+
+- The output is a Python list of function calls, with arguments as Python literals (not JSON).
+- Multiple tool calls can be returned in the same list:
+```python Example
+[get_current_weather(city="San Francisco", state="CA", unit="celsius"),
+ get_current_weather(city="New York", state="NY", unit="fahrenheit")]
+```
+
+For more information, refer to Meta’s documentation on  [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message).
+
+Note that this feature is still under development on Blackwell.
+
+### How to enable
+- Launch the server with `--tool-call-parser pythonic`
+- You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`).
+This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results.
+
+#### Forcing Pythonic Tool Call Output Without a Chat Template
+If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need:
+
+
+
+```python Example
+import openai
+
+server_process, port = launch_server_cmd(
+    " python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1  --log-level warning"  # llama-3.2-1b-instruct
+)
+wait_for_server(f"http://localhost:{port}")
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a given location.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The name of the city or location.",
+                    }
+                },
+                "required": ["location"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "get_tourist_attractions",
+            "description": "Get a list of top tourist attractions for a given city.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "city": {
+                        "type": "string",
+                        "description": "The name of the city to find attractions for.",
+                    }
+                },
+                "required": ["city"],
+            },
+        },
+    },
+]
+
+
+def get_messages():
+    return [
+        {
+            "role": "system",
+            "content": (
+                "You are a travel assistant. "
+                "When asked to call functions, ALWAYS respond ONLY with a python list of function calls, "
+                "using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. "
+                "Do NOT use JSON, do NOT use variables, do NOT use any other format. "
+                "Here is an example:\n"
+                '[get_weather(location="Paris"), get_tourist_attractions(city="Paris")]'
+            ),
+        },
+        {
+            "role": "user",
+            "content": (
+                "I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? "
+                "Propose parallel tool calls at once, using the python list of function calls format as shown above."
+            ),
+        },
+    ]
+
+
+messages = get_messages()
+
+client = openai.Client(base_url=f"http://localhost:{port}/v1", api_key="xxxxxx")
+model_name = client.models.list().data[0].id
+
+
+response_non_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0,
+    top_p=0.9,
+    stream=False,  # Non-streaming
+    tools=tools,
+)
+print_highlight("Non-stream response:")
+print_highlight(response_non_stream)
+
+response_stream = client.chat.completions.create(
+    model=model_name,
+    messages=messages,
+    temperature=0,
+    top_p=0.9,
+    stream=True,
+    tools=tools,
+)
+texts = ""
+tool_calls = []
+name = ""
+arguments = ""
+
+for chunk in response_stream:
+    if chunk.choices[0].delta.content:
+        texts += chunk.choices[0].delta.content
+    if chunk.choices[0].delta.tool_calls:
+        tool_calls.append(chunk.choices[0].delta.tool_calls[0])
+
+print_highlight("Streaming Response:")
+print_highlight("==== Text ====")
+print_highlight(texts)
+
+print_highlight("==== Tool Call ====")
+for tool_call in tool_calls:
+    print_highlight(tool_call)
+
+terminate_process(server_process)
+```
+
+> **Note:**
+> The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template.
+
+
+## How to support a new model?
+1. Update the TOOLS_TAG_LIST in sglang/srt/function_call_parser.py with the model’s tool tags. Currently supported tags include:
+```text Output
+	TOOLS_TAG_LIST = [
+	    “<|plugin|>“,
+	    “<function=“,
+	    “<tool_call>“,
+	    “<|python_tag|>“,
+	    “[TOOL_CALLS]”
+	]
+```
+2. Create a new detector class in sglang/srt/function_call_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example:
+```text Output
+    class NewModelDetector(BaseFormatDetector):
+```
+3. Add the new detector to the MultiFormatParser class that manages all the format detectors.

docs_new/docs/advanced_features/vlm_query.ipynb

@@ -0,0 +1,379 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# Query VLM with Offline Engine\n",
+    "\n",
+    "This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches:\n",
+    "\n",
+    "1. **Basic Call**: Directly pass images and text.\n",
+    "2. **Processor Output**: Use HuggingFace processor for data preprocessing.\n",
+    "3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {},
+   "source": [
+    "## Understanding the Three Input Formats\n",
+    "\n",
+    "SGLang supports three ways to pass visual data, each optimized for different scenarios:\n",
+    "\n",
+    "### 1. **Raw Images** - Simplest approach\n",
+    "- Pass PIL Images, file paths, URLs, or base64 strings directly\n",
+    "- SGLang handles all preprocessing automatically\n",
+    "- Best for: Quick prototyping, simple applications\n",
+    "\n",
+    "### 2. **Processor Output** - For custom preprocessing\n",
+    "- Pre-process images with HuggingFace processor\n",
+    "- Pass the complete processor output dict with `format: \"processor_output\"`\n",
+    "- Best for: Custom image transformations, integration with existing pipelines\n",
+    "- Requirement: Must use `input_ids` instead of text prompt\n",
+    "\n",
+    "### 3. **Precomputed Embeddings** - For maximum performance\n",
+    "- Pre-calculate visual embeddings using the vision encoder\n",
+    "- Pass embeddings with `format: \"precomputed_embedding\"`\n",
+    "- Best for: Repeated queries on same images, caching, high-throughput serving\n",
+    "- Performance gain: Avoids redundant vision encoder computation (30-50% speedup)\n",
+    "\n",
+    "**Key Rule**: Within a single request, use only one format for all images. Don't mix formats.\n",
+    "\n",
+    "The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {},
+   "source": [
+    "## Querying Qwen2.5-VL Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import nest_asyncio\n",
+    "\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "import sglang.test.doc_patch  # noqa: F401\n",
+    "\n",
+    "model_path = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n",
+    "chat_template = \"qwen2-vl\"\n",
+    "example_image_url = \"https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from io import BytesIO\n",
+    "import requests\n",
+    "from PIL import Image\n",
+    "\n",
+    "from sglang.srt.parser.conversation import chat_templates\n",
+    "\n",
+    "image = Image.open(BytesIO(requests.get(example_image_url).content))\n",
+    "\n",
+    "conv = chat_templates[chat_template].copy()\n",
+    "conv.append_message(conv.roles[0], f\"What's shown here: {conv.image_token}?\")\n",
+    "conv.append_message(conv.roles[1], \"\")\n",
+    "conv.image_data = [image]\n",
+    "\n",
+    "print(\"Generated prompt text:\")\n",
+    "print(conv.get_prompt())\n",
+    "print(f\"\\nImage size: {image.size}\")\n",
+    "image"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5",
+   "metadata": {},
+   "source": [
+    "### Basic Offline Engine API Call"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang import Engine\n",
+    "\n",
+    "llm = Engine(model_path=model_path, chat_template=chat_template, log_level=\"warning\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
+    "print(\"Model response:\")\n",
+    "print(out[\"text\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8",
+   "metadata": {},
+   "source": [
+    "### Call with Processor Output\n",
+    "\n",
+    "Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoProcessor\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
+    "processor_output = processor(\n",
+    "    images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
+    ")\n",
+    "\n",
+    "out = llm.generate(\n",
+    "    input_ids=processor_output[\"input_ids\"][0].detach().cpu().tolist(),\n",
+    "    image_data=[dict(processor_output, format=\"processor_output\")],\n",
+    ")\n",
+    "print(\"Response using processor output:\")\n",
+    "print(out[\"text\"])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10",
+   "metadata": {},
+   "source": [
+    "### Call with Precomputed Embeddings\n",
+    "\n",
+    "You can pre-calculate image features to avoid repeated visual encoding processes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoProcessor\n",
+    "from transformers import Qwen2_5_VLForConditionalGeneration\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
+    "model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval()\n",
+    "vision = model.model.visual.cuda()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "12",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "processor_output = processor(\n",
+    "    images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
+    ")\n",
+    "\n",
+    "input_ids = processor_output[\"input_ids\"][0].detach().cpu().tolist()\n",
+    "\n",
+    "precomputed_embeddings = vision(\n",
+    "    processor_output[\"pixel_values\"].cuda(), processor_output[\"image_grid_thw\"].cuda()\n",
+    ")\n",
+    "precomputed_embeddings = precomputed_embeddings.pooler_output\n",
+    "\n",
+    "multi_modal_item = dict(\n",
+    "    processor_output,\n",
+    "    format=\"precomputed_embedding\",\n",
+    "    feature=precomputed_embeddings,\n",
+    ")\n",
+    "\n",
+    "out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item])\n",
+    "print(\"Response using precomputed embeddings:\")\n",
+    "print(out[\"text\"])\n",
+    "\n",
+    "llm.shutdown()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "13",
+   "metadata": {},
+   "source": [
+    "## Querying Llama 4 Vision Model\n",
+    "\n",
+    "```python\n",
+    "model_path = \"meta-llama/Llama-4-Scout-17B-16E-Instruct\"\n",
+    "chat_template = \"llama-4\"\n",
+    "\n",
+    "from io import BytesIO\n",
+    "import requests\n",
+    "from PIL import Image\n",
+    "\n",
+    "from sglang.srt.parser.conversation import chat_templates\n",
+    "\n",
+    "# Download the same example image\n",
+    "image = Image.open(BytesIO(requests.get(example_image_url).content))\n",
+    "\n",
+    "conv = chat_templates[chat_template].copy()\n",
+    "conv.append_message(conv.roles[0], f\"What's shown here: {conv.image_token}?\")\n",
+    "conv.append_message(conv.roles[1], \"\")\n",
+    "conv.image_data = [image]\n",
+    "\n",
+    "print(\"Llama 4 generated prompt text:\")\n",
+    "print(conv.get_prompt())\n",
+    "print(f\"Image size: {image.size}\")\n",
+    "\n",
+    "image\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "14",
+   "metadata": {},
+   "source": [
+    "### Llama 4 Basic Call\n",
+    "\n",
+    "Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp_size=4) and larger context length.\n",
+    "\n",
+    "```python\n",
+    "llm = Engine(\n",
+    "    model_path=model_path,\n",
+    "    enable_multimodal=True,\n",
+    "    attention_backend=\"fa3\",\n",
+    "    tp_size=4,\n",
+    "    context_length=65536,\n",
+    ")\n",
+    "\n",
+    "out = llm.generate(prompt=conv.get_prompt(), image_data=[image])\n",
+    "print(\"Llama 4 response:\")\n",
+    "print(out[\"text\"])\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15",
+   "metadata": {},
+   "source": [
+    "### Call with Processor Output\n",
+    "\n",
+    "Using HuggingFace processor to preprocess data can reduce computational overhead during inference.\n",
+    "\n",
+    "```python\n",
+    "from transformers import AutoProcessor\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
+    "processor_output = processor(\n",
+    "    images=[image], text=conv.get_prompt(), return_tensors=\"pt\"\n",
+    ")\n",
+    "\n",
+    "out = llm.generate(\n",
+    "    input_ids=processor_output[\"input_ids\"][0].detach().cpu().tolist(),\n",
+    "    image_data=[dict(processor_output, format=\"processor_output\")],\n",
+    ")\n",
+    "print(\"Response using processor output:\")\n",
+    "print(out)\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "16",
+   "metadata": {},
+   "source": [
+    "### Call with Precomputed Embeddings\n",
+    "\n",
+    "```python\n",
+    "from transformers import AutoProcessor\n",
+    "from transformers import Llama4ForConditionalGeneration\n",
+    "\n",
+    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
+    "model = Llama4ForConditionalGeneration.from_pretrained(\n",
+    "    model_path, torch_dtype=\"auto\"\n",
+    ").eval()\n",
+    "\n",
+    "vision = model.vision_model.cuda()\n",
+    "multi_modal_projector = model.multi_modal_projector.cuda()\n",
+    "\n",
+    "print(f'Image pixel values shape: {processor_output[\"pixel_values\"].shape}')\n",
+    "input_ids = processor_output[\"input_ids\"][0].detach().cpu().tolist()\n",
+    "\n",
+    "# Process image through vision encoder\n",
+    "image_outputs = vision(\n",
+    "    processor_output[\"pixel_values\"].to(\"cuda\"), \n",
+    "    aspect_ratio_ids=processor_output[\"aspect_ratio_ids\"].to(\"cuda\"),\n",
+    "    aspect_ratio_mask=processor_output[\"aspect_ratio_mask\"].to(\"cuda\"),\n",
+    "    output_hidden_states=False\n",
+    ")\n",
+    "image_features = image_outputs.last_hidden_state\n",
+    "\n",
+    "# Flatten image features and pass through multimodal projector\n",
+    "vision_flat = image_features.view(-1, image_features.size(-1))\n",
+    "precomputed_embeddings = multi_modal_projector(vision_flat)\n",
+    "\n",
+    "# Build precomputed embedding data item\n",
+    "mm_item = dict(\n",
+    "    processor_output, \n",
+    "    format=\"precomputed_embedding\", \n",
+    "    feature=precomputed_embeddings\n",
+    ")\n",
+    "\n",
+    "# Use precomputed embeddings for efficient inference\n",
+    "out = llm.generate(input_ids=input_ids, image_data=[mm_item])\n",
+    "print(\"Llama 4 precomputed embedding response:\")\n",
+    "print(out[\"text\"])\n",
+    "```"
+   ]
+  }
+ ],
+ "metadata": {
+  "jupytext": {
+   "cell_metadata_filter": "-all",
+   "custom_cell_magics": "kql",
+   "encoding": "# -*- coding: utf-8 -*-",
+   "text_representation": {
+    "extension": ".py",
+    "format_name": "light",
+    "format_version": "1.5",
+    "jupytext_version": "1.16.1"
+   }
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs_new/docs/advanced_features/vlm_query.mdx

@@ -0,0 +1,249 @@
+---
+title: "Query VLM with Offline Engine"
+metatags:
+    description: "SGLang VLM offline engine: raw images, processor output, precomputed embeddings. Qwen2.5-VL and Llama 4 examples."
+---
+This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches:
+
+1. **Basic Call**: Directly pass images and text.
+2. **Processor Output**: Use HuggingFace processor for data preprocessing.
+3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency.
+
+## Understanding the Three Input Formats
+
+SGLang supports three ways to pass visual data, each optimized for different scenarios:
+
+### 1. **Raw Images** - Simplest approach
+- Pass PIL Images, file paths, URLs, or base64 strings directly
+- SGLang handles all preprocessing automatically
+- Best for: Quick prototyping, simple applications
+
+### 2. **Processor Output** - For custom preprocessing
+- Pre-process images with HuggingFace processor
+- Pass the complete processor output dict with `format: "processor_output"`
+- Best for: Custom image transformations, integration with existing pipelines
+- Requirement: Must use `input_ids` instead of text prompt
+
+### 3. **Precomputed Embeddings** - For maximum performance
+- Pre-calculate visual embeddings using the vision encoder
+- Pass embeddings with `format: "precomputed_embedding"`
+- Best for: Repeated queries on same images, caching, high-throughput serving
+- Performance gain: Avoids redundant vision encoder computation (30-50% speedup)
+
+**Key Rule**: Within a single request, use only one format for all images. Don't mix formats.
+
+The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models.
+
+## Querying Qwen2.5-VL Model
+
+```python Example
+import nest_asyncio
+
+nest_asyncio.apply()
+
+import sglang.test.doc_patch  # noqa: F401
+
+model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
+chat_template = "qwen2-vl"
+example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
+```
+
+```python Example
+from io import BytesIO
+import requests
+from PIL import Image
+
+from sglang.srt.parser.conversation import chat_templates
+
+image = Image.open(BytesIO(requests.get(example_image_url).content))
+
+conv = chat_templates[chat_template].copy()
+conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
+conv.append_message(conv.roles[1], "")
+conv.image_data = [image]
+
+print("Generated prompt text:")
+print(conv.get_prompt())
+print(f"\nImage size: {image.size}")
+image
+```
+
+### Basic Offline Engine API Call
+
+```python Example
+from sglang import Engine
+
+llm = Engine(model_path=model_path, chat_template=chat_template, log_level="warning")
+```
+
+```python Example
+out = llm.generate(prompt=conv.get_prompt(), image_data=[image])
+print("Model response:")
+print(out["text"])
+```
+
+### Call with Processor Output
+
+Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`.
+
+```python Example
+from transformers import AutoProcessor
+
+processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
+processor_output = processor(
+    images=[image], text=conv.get_prompt(), return_tensors="pt"
+)
+
+out = llm.generate(
+    input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
+    image_data=[dict(processor_output, format="processor_output")],
+)
+print("Response using processor output:")
+print(out["text"])
+```
+
+### Call with Precomputed Embeddings
+
+You can pre-calculate image features to avoid repeated visual encoding processes.
+
+```python Example
+from transformers import AutoProcessor
+from transformers import Qwen2_5_VLForConditionalGeneration
+
+processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval()
+vision = model.model.visual.cuda()
+```
+
+```python Example
+processor_output = processor(
+    images=[image], text=conv.get_prompt(), return_tensors="pt"
+)
+
+input_ids = processor_output["input_ids"][0].detach().cpu().tolist()
+
+precomputed_embeddings = vision(
+    processor_output["pixel_values"].cuda(), processor_output["image_grid_thw"].cuda()
+)
+precomputed_embeddings = precomputed_embeddings.pooler_output
+
+multi_modal_item = dict(
+    processor_output,
+    format="precomputed_embedding",
+    feature=precomputed_embeddings,
+)
+
+out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item])
+print("Response using precomputed embeddings:")
+print(out["text"])
+
+llm.shutdown()
+```
+
+## Querying Llama 4 Vision Model
+
+```python Example
+model_path = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
+chat_template = "llama-4"
+
+from io import BytesIO
+import requests
+from PIL import Image
+
+from sglang.srt.parser.conversation import chat_templates
+
+# Download the same example image
+image = Image.open(BytesIO(requests.get(example_image_url).content))
+
+conv = chat_templates[chat_template].copy()
+conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
+conv.append_message(conv.roles[1], "")
+conv.image_data = [image]
+
+print("Llama 4 generated prompt text:")
+print(conv.get_prompt())
+print(f"Image size: {image.size}")
+
+image
+```
+
+### Llama 4 Basic Call
+
+Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp_size=4) and larger context length.
+
+```python Example
+llm = Engine(
+    model_path=model_path,
+    enable_multimodal=True,
+    attention_backend="fa3",
+    tp_size=4,
+    context_length=65536,
+)
+
+out = llm.generate(prompt=conv.get_prompt(), image_data=[image])
+print("Llama 4 response:")
+print(out["text"])
+```
+
+### Call with Processor Output
+
+Using HuggingFace processor to preprocess data can reduce computational overhead during inference.
+
+```python Example
+from transformers import AutoProcessor
+
+processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
+processor_output = processor(
+    images=[image], text=conv.get_prompt(), return_tensors="pt"
+)
+
+out = llm.generate(
+    input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
+    image_data=[dict(processor_output, format="processor_output")],
+)
+print("Response using processor output:")
+print(out)
+```
+
+### Call with Precomputed Embeddings
+
+```python Example
+from transformers import AutoProcessor
+from transformers import Llama4ForConditionalGeneration
+
+processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
+model = Llama4ForConditionalGeneration.from_pretrained(
+    model_path, torch_dtype="auto"
+).eval()
+
+vision = model.vision_model.cuda()
+multi_modal_projector = model.multi_modal_projector.cuda()
+
+print(f'Image pixel values shape: {processor_output["pixel_values"].shape}')
+input_ids = processor_output["input_ids"][0].detach().cpu().tolist()
+
+# Process image through vision encoder
+image_outputs = vision(
+    processor_output["pixel_values"].to("cuda"),
+    aspect_ratio_ids=processor_output["aspect_ratio_ids"].to("cuda"),
+    aspect_ratio_mask=processor_output["aspect_ratio_mask"].to("cuda"),
+    output_hidden_states=False
+)
+image_features = image_outputs.last_hidden_state
+
+# Flatten image features and pass through multimodal projector
+vision_flat = image_features.view(-1, image_features.size(-1))
+precomputed_embeddings = multi_modal_projector(vision_flat)
+
+# Build precomputed embedding data item
+mm_item = dict(
+    processor_output,
+    format="precomputed_embedding",
+    feature=precomputed_embeddings
+)
+
+# Use precomputed embeddings for efficient inference
+out = llm.generate(input_ids=input_ids, image_data=[mm_item])
+print("Llama 4 precomputed embedding response:")
+print(out["text"])
+```

docs_new/docs/basic_usage/anthropic_api.mdx

@@ -0,0 +1,287 @@
+---
+title: "Anthropic-Compatible API"
+description: "Use the Anthropic Messages API (/v1/messages) with SGLang, including Claude Code integration and prefix-cache tuning."
+---
+SGLang ships an Anthropic-compatible `/v1/messages` endpoint so any client built for the Anthropic
+Messages API — including the Anthropic SDKs and agentic CLIs such as Claude Code — can talk to a
+self-hosted SGLang server without changes. A complete reference for the API is available in the
+[Anthropic API Reference](https://docs.anthropic.com/en/api/messages).
+
+The endpoint is registered automatically on every SGLang server; no extra flag is required to enable it.
+It reuses the same model, chat template, and reasoning / tool-call parsers as the OpenAI-compatible
+endpoint, and supports both non-streaming and streaming responses, tool use, and a `count_tokens` route.
+
+This tutorial covers:
+
+- `POST /v1/messages` (non-streaming and streaming)
+- `POST /v1/messages/count_tokens`
+- Pointing **Claude Code** at the server, including the `CLAUDE_CODE_ATTRIBUTION_HEADER` setting that is
+  required for good prefix-cache reuse.
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize. The Anthropic `/v1/messages` endpoint
+is registered automatically — no extra flag is required beyond the usual server launch. The example below
+is a single-node GLM-5.2-FP8 config; see the
+[GLM-5.2 cookbook](/cookbook/autoregressive/GLM/GLM-5.2) for verified commands
+across hardware and quantizations.
+
+```bash Command
+sglang serve \
+    --model-path zai-org/GLM-5.2-FP8 \
+    --tp 8 \
+    --speculative-algorithm EAGLE \
+    --speculative-num-steps 5 \
+    --speculative-eagle-topk 1 \
+    --speculative-num-draft-tokens 6 \
+    --reasoning-parser glm45 \
+    --tool-call-parser glm47 \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+<Note>
+- **The endpoint is model-agnostic.** The `/v1/messages` route is on by default for any model; GLM-5.2 is
+  used here because its reasoning + tool-use output is where Claude Code integration shines, but any model
+  works.
+- **Model name and `[1m]`.** SGLang does not validate the request `model` field, so Claude Code can send
+  any name. The `[1m]` suffix is a **client-side hint**: Claude Code only enables its 1M-context beta when
+  the model name ends in `[1m]` — without it, context is capped. Set the same `glm-5.2[1m]` in the
+  `ANTHROPIC_DEFAULT_*_MODEL` env vars below.
+- **`--reasoning-parser` / `--tool-call-parser` are optional.** Add them when the model emits reasoning
+  content (GLM-5.2, Qwen3, DeepSeek-R1, …) or when you want tool calls parsed into structured `tool_use`
+  blocks. Without a tool-call parser, tool schemas are still accepted but the model's tool calls come back
+  as raw text, and Claude Code cannot execute them.
+- **Context length** defaults to the model's own (1M for GLM-5.2); pass `--context-length` only to cap it.
+</Note>
+
+## Send A Message
+
+### Non-Streaming
+
+Use the Anthropic Python SDK pointed at the server. Unlike the OpenAI SDK, the Anthropic SDK appends
+`/v1/messages` itself, so `base_url` is the server root **without** a `/v1` suffix.
+
+```python Example
+from anthropic import Anthropic
+
+client = Anthropic(
+    base_url="http://127.0.0.1:30000",
+    api_key="EMPTY",  # SGLang does not require a real key by default
+)
+
+message = client.messages.create(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    messages=[{"role": "user", "content": "List 3 countries and their capitals."}],
+)
+# A reasoning model may emit a `thinking` block before the `text` block —
+# pick the text block rather than assuming content[0].
+print(next(b.text for b in message.content if b.type == "text"))
+```
+
+**Example Output:**
+```text Output
+Here are 3 countries and their capitals:
+
+1. **France** - Paris
+2. **Japan** - Tokyo
+3. **Brazil** - Brasília
+```
+
+### Streaming
+
+Set `stream=True` to receive Server-Sent Events as they are produced.
+
+```python Example
+with client.messages.stream(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    messages=[{"role": "user", "content": "Say this is a test"}],
+) as stream:
+    for text in stream.text_stream:
+        print(text, end="", flush=True)
+```
+
+**Example Output:**
+```text Output
+This is a test.
+```
+
+### System Prompt
+
+The top-level `system` field is accepted as a string or as a list of text blocks, matching the Anthropic
+API shape:
+
+```python Example
+message = client.messages.create(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    system="You are a helpful assistant that answers concisely.",
+    messages=[{"role": "user", "content": "What is the capital of France?"}],
+)
+print(next(b.text for b in message.content if b.type == "text"))
+```
+
+**Example Output:**
+```text Output
+The capital of France is Paris.
+```
+
+### Tool Use
+
+Tool definitions follow the Anthropic `tools` schema. When the server is launched with a
+`--tool-call-parser`, the model's tool calls are returned as `tool_use` content blocks:
+
+```python Example
+message = client.messages.create(
+    model="zai-org/GLM-5.2-FP8",
+    max_tokens=512,
+    tools=[
+        {
+            "name": "get_weather",
+            "description": "Get the weather for a city",
+            "input_schema": {
+                "type": "object",
+                "properties": {"city": {"type": "string"}},
+                "required": ["city"],
+            },
+        }
+    ],
+    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
+)
+print(message.stop_reason)
+print([b for b in message.content if b.type == "tool_use"])
+```
+
+**Example Output:**
+```text Output
+tool_use
+[ToolUseBlock(type='tool_use', id='toolu_01XXXX', name='get_weather', input={'city': 'Paris'})]
+```
+
+### Counting Tokens
+
+`POST /v1/messages/count_tokens` returns the tokenized length of a request without generating a
+response. It reuses the same request conversion as `/v1/messages`, so system prompts, tools, and
+multi-turn history are all accounted for.
+
+```python Example
+resp = client.messages.count_tokens(
+    model="zai-org/GLM-5.2-FP8",
+    messages=[{"role": "user", "content": "Hello, world"}],
+)
+print(resp.input_tokens)
+```
+
+**Example Output:**
+```text Output
+15
+```
+
+## Using Claude Code
+
+Claude Code can be pointed at an SGLang server by setting a few env vars in the shell that starts it.
+With the server already running on `:30000`, export the full set and launch `claude`:
+
+```bash Command
+export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
+export ANTHROPIC_AUTH_TOKEN="dummy"                 # required by Claude Code; any non-empty string works
+export API_TIMEOUT_MS="3000000"                     # long timeout — reasoning + 1M-context turns are slow
+export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"     # let auto-compact use the full 1M window
+export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1   # drop autoupdater/telemetry/error-reporting noise
+export CLAUDE_CODE_ATTRIBUTION_HEADER=0             # required for prefix-cache reuse — see below
+export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"    # [1m] suffix enables Claude Code's 1M-context beta
+export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"  # [1m] suffix enables Claude Code's 1M-context beta
+export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"     # [1m] suffix enables Claude Code's 1M-context beta
+claude
+```
+
+Each var matters:
+
+- **`ANTHROPIC_BASE_URL`** — points Claude Code at your SGLang server instead of the Anthropic API.
+- **`ANTHROPIC_AUTH_TOKEN`** — Claude Code requires a non-empty auth token; SGLang accepts any value
+  when launched without `--api-key`.
+- **`API_TIMEOUT_MS`** — raise it; reasoning models with long outputs and 1M-context turns routinely
+  exceed the default timeout.
+- **`ANTHROPIC_DEFAULT_{HAIKU,SONNET,OPUS}_MODEL`** — the model name Claude Code sends for each tier.
+  SGLang does not validate this field, so any name works. Use `glm-5.2[1m]`: the `[1m]` suffix is a
+  client-side hint that enables Claude Code's 1M-context beta (without it, context is capped).
+- **`CLAUDE_CODE_AUTO_COMPACT_WINDOW`** — set to `1000000` so auto-compaction uses the full 1M window
+  instead of the default, keeping long sessions alive.
+
+<Tip>
+Instead of exporting these in every shell, persist them in `~/.claude/settings.json` under the `env` key
+— they apply to all Claude Code sessions:
+
+```json
+{
+  "env": {
+    "ANTHROPIC_BASE_URL": "http://127.0.0.1:30000",
+    "ANTHROPIC_AUTH_TOKEN": "dummy",
+    "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
+    "CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
+    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "glm-5.2[1m]",
+    "ANTHROPIC_DEFAULT_SONNET_MODEL": "glm-5.2[1m]",
+    "ANTHROPIC_DEFAULT_OPUS_MODEL": "glm-5.2[1m]"
+  }
+}
+```
+</Tip>
+
+### Required: `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for prefix-cache reuse
+
+<Note>
+**Set this whenever Claude Code routes through SGLang (or any non-Anthropic gateway).** Without it,
+multi-turn conversations re-prefill the whole history every turn.
+</Note>
+
+Claude Code prepends a per-request attribution block to the start of the system prompt, of the form
+`x-anthropic-billing-header: cc_version=<ver>.<per-request-hash>; cc_entrypoint=...; cch=<hash>;`. The
+per-request hash is the **first token to differ between turns**, so the radix prefix cache can only reuse
+the short prefix before that hash and re-prefills the system prompt plus the entire conversation history
+on every turn.
+
+Setting `CLAUDE_CODE_ATTRIBUTION_HEADER=0` removes the whole attribution line from the system prompt.
+This is a documented Claude Code env var whose explicit purpose is to "improve prompt-cache hit rates when
+routing through an [LLM gateway](https://code.claude.com/docs/en/llm-gateway)" (see the [Claude Code env-vars reference](https://code.claude.com/docs/en/env-vars)).
+
+<Note>
+`CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` does **not** remove the attribution block — it only covers
+autoupdater/telemetry/error reporting. The attribution header is a separate code path; use
+`CLAUDE_CODE_ATTRIBUTION_HEADER=0` for it.
+</Note>
+
+## Troubleshooting
+
+**Connection refused / `fetch failed`** — Ensure the server is up and the port in `ANTHROPIC_BASE_URL`
+matches `--port` (default 30000). If you set `ANTHROPIC_BASE_URL` to a remote host, confirm it's reachable
+and not behind a proxy that blocks the connection.
+
+**`Model not found` / 404 from the server** — SGLang does not validate the request `model` field and
+serves whatever model was loaded at startup, so a 404 usually means the request did not reach the
+`/v1/messages` route at all. Confirm `ANTHROPIC_BASE_URL` points at the server (not missing the port) and
+that the server finished loading.
+
+**Tool calls not working / returned as raw text** — Launch the server with the correct
+`--tool-call-parser` for your model (e.g. `glm47`, `qwen3`). Without it the `tools` field is still accepted
+but the model's tool calls come back as text instead of `tool_use` blocks, and Claude Code cannot execute
+them.
+
+**Slow / re-prefills the whole history every turn** — You are missing
+`CLAUDE_CODE_ATTRIBUTION_HEADER=0`. Claude Code's per-request attribution hash in the system prompt
+defeats radix prefix-cache reuse; see the section above.
+
+**Context capped below 1M** — The model name must end in `[1m]` for Claude Code to enable its 1M-context
+beta. Verify `ANTHROPIC_DEFAULT_*_MODEL` uses the `[1m]` suffix, and that the loaded model's native context
+is 1M (GLM-5.2 is 1048576; pass `--context-length` only to cap it, not to extend).
+
+## Parameters
+
+The `/v1/messages` endpoint accepts the standard Anthropic Messages API parameters. Refer to the
+[Anthropic Messages API reference](https://docs.anthropic.com/en/api/messages) for the full list.
+
+Reasoning models are supported through the same `--reasoning-parser` mechanism as the OpenAI-compatible
+endpoint; pass the model's reasoning kwarg via the request (e.g. `thinking` for DeepSeek-V3-style models,
+`enable_thinking` for Qwen3-style models). See [OpenAI APIs - Completions](./openai_api_completions) for
+the reasoning-parser / chat-template mapping.

docs_new/docs/basic_usage/aws_sagemaker.mdx

@@ -0,0 +1,142 @@
+---
+title: "Amazon SageMaker AI"
+description: "Deploy SGLang on Amazon SageMaker AI endpoints using the AWS Deep Learning Container."
+---
+
+Deploy SGLang on [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/) endpoints using the
+[AWS Deep Learning Container (DLC)](https://aws.github.io/deep-learning-containers/sglang/) for SGLang.
+The SageMaker image variant accepts model configuration via environment variables and serves on port 8080.
+
+This guide uses the pre-built DLC image. To build and deploy your own container instead, see
+[Method 7: Run on AWS SageMaker](/docs/get-started/install#more-3) in the installation guide.
+
+## Container image
+
+AWS publishes pre-built, security-patched SGLang DLCs. The SageMaker GPU image is available from the
+Amazon ECR registry (account `763104351884`) in each supported region. For example, in `us-west-2`:
+
+```text
+763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0
+```
+
+For the full list of image tags, see the
+[Available DLC Images](https://aws.github.io/deep-learning-containers/reference/available_images/) reference,
+and for region-specific account IDs and supported regions, see
+[Region Availability](https://aws.github.io/deep-learning-containers/reference/region_availability/).
+
+## Specifying the model
+
+The SageMaker image resolves the model in this order:
+
+1. **`SM_SGLANG_MODEL_PATH` environment variable** — explicit Hugging Face ID or path.
+2. **`/opt/ml/model`** — when SageMaker mounts model artifacts via `ModelDataUrl` or `ModelDataSource`,
+   the entrypoint uses this path by default.
+
+For gated models, also pass `HF_TOKEN`.
+
+Any `SM_SGLANG_*` environment variable is converted to a `--<name>` SGLang server argument
+(for example, `SM_SGLANG_CONTEXT_LENGTH=4096` becomes `--context-length 4096`).
+
+## Deploy with the SageMaker Python SDK
+
+```python
+from sagemaker.model import Model
+from sagemaker.predictor import Predictor
+from sagemaker.serializers import JSONSerializer
+
+model = Model(
+    image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0",
+    role="arn:aws:iam::<account_id>:role/<role_name>",
+    predictor_cls=Predictor,
+    env={"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"},
+)
+
+predictor = model.deploy(
+    instance_type="ml.g5.2xlarge",
+    initial_instance_count=1,
+    inference_ami_version="al2023-ami-sagemaker-inference-gpu-4-1",
+    serializer=JSONSerializer(),
+)
+
+response = predictor.predict({
+    "model": "openai/gpt-oss-20b",
+    "messages": [{"role": "user", "content": "What is deep learning?"}],
+    "max_tokens": 256,
+})
+print(response)
+
+# Cleanup
+predictor.delete_model()
+predictor.delete_endpoint(delete_endpoint_config=True)
+```
+
+## Deploy with Boto3
+
+```python
+import json
+import boto3
+
+sm = boto3.client("sagemaker")
+smrt = boto3.client("sagemaker-runtime")
+
+sm.create_model(
+    ModelName="sglang-model",
+    PrimaryContainer={
+        "Image": "763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0",
+        "Environment": {"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"},
+    },
+    ExecutionRoleArn="arn:aws:iam::<account_id>:role/<role_name>",
+)
+
+sm.create_endpoint_config(
+    EndpointConfigName="sglang-config",
+    ProductionVariants=[{
+        "VariantName": "default",
+        "ModelName": "sglang-model",
+        "InstanceType": "ml.g5.2xlarge",
+        "InitialInstanceCount": 1,
+        "InferenceAmiVersion": "al2023-ami-sagemaker-inference-gpu-4-1",
+    }],
+)
+
+sm.create_endpoint(EndpointName="sglang-endpoint", EndpointConfigName="sglang-config")
+sm.get_waiter("endpoint_in_service").wait(EndpointName="sglang-endpoint")
+
+resp = smrt.invoke_endpoint(
+    EndpointName="sglang-endpoint",
+    ContentType="application/json",
+    Body=json.dumps({
+        "model": "openai/gpt-oss-20b",
+        "messages": [{"role": "user", "content": "What is deep learning?"}],
+        "max_tokens": 256,
+    }),
+)
+print(json.loads(resp["Body"].read()))
+
+# Cleanup
+sm.delete_endpoint(EndpointName="sglang-endpoint")
+sm.delete_endpoint_config(EndpointConfigName="sglang-config")
+sm.delete_model(ModelName="sglang-model")
+```
+
+## Model artifacts
+
+When `ModelDataUrl` (or `ModelDataSource`) points to a tarball or S3 prefix, SageMaker mounts the contents
+at `/opt/ml/model`. The entrypoint defaults `--model-path` to that location, so `SM_SGLANG_MODEL_PATH`
+can be omitted:
+
+```text
+model.tar.gz
+├── config.json              # standard model files (Hugging Face layout)
+├── tokenizer.json
+└── *.safetensors
+```
+
+## Notes
+
+- GPU deployments require `inference_ami_version` — the default SageMaker host AMI has incompatible NVIDIA
+  drivers for CUDA 13 images. See the
+  [ProductionVariant API reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html)
+  for valid values.
+- The endpoint exposes an OpenAI-compatible API, so the request body matches the SGLang server's
+  `/v1/chat/completions` schema.

docs_new/docs/basic_usage/native_api.ipynb

@@ -0,0 +1,675 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# SGLang Native APIs\n",
+    "\n",
+    "Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:\n",
+    "\n",
+    "- `/generate` (text generation model)\n",
+    "- `/get_model_info`\n",
+    "- `/server_info`\n",
+    "- `/health`\n",
+    "- `/health_generate`\n",
+    "- `/flush_cache`\n",
+    "- `/update_weights`\n",
+    "- `/encode`(embedding model)\n",
+    "- `/v1/rerank`(cross encoder rerank model)\n",
+    "- `/v1/score`(decoder-only scoring)\n",
+    "- `/classify`(reward model)\n",
+    "- `/start_expert_distribution_record`\n",
+    "- `/stop_expert_distribution_record`\n",
+    "- `/dump_expert_distribution_record`\n",
+    "- `/tokenize`\n",
+    "- `/detokenize`\n",
+    "- A full list of these APIs can be found at [http_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py)\n",
+    "\n",
+    "We mainly use `requests` to test these APIs in the following examples. You can also use `curl`.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Generate (text generation model)\n",
+    "Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](sampling_params.md)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = f\"http://localhost:{port}/generate\"\n",
+    "data = {\"text\": \"What is the capital of France?\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Get Model Info\n",
+    "\n",
+    "Get the information of the model.\n",
+    "\n",
+    "- `model_path`: The path/name of the model.\n",
+    "- `is_generation`: Whether the model is used as generation model or embedding model.\n",
+    "- `tokenizer_path`: The path/name of the tokenizer.\n",
+    "- `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args.\n",
+    "- `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.\n",
+    "- `has_image_understanding`: Whether the model has image-understanding capability.\n",
+    "- `has_audio_understanding`: Whether the model has audio-understanding capability.\n",
+    "- `model_type`: The model type from the HuggingFace config (e.g., \"qwen2\", \"llama\").\n",
+    "- `architectures`: The model architectures from the HuggingFace config (e.g., [\"Qwen2ForCausalLM\"])."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/get_model_info\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "response_json = response.json()\n",
+    "print_highlight(response_json)\n",
+    "assert response_json[\"model_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
+    "assert response_json[\"is_generation\"] is True\n",
+    "assert response_json[\"tokenizer_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
+    "assert response_json[\"preferred_sampling_params\"] is None\n",
+    "assert response_json.keys() == {\n",
+    "    \"model_path\",\n",
+    "    \"is_generation\",\n",
+    "    \"tokenizer_path\",\n",
+    "    \"preferred_sampling_params\",\n",
+    "    \"weight_version\",\n",
+    "    \"has_image_understanding\",\n",
+    "    \"has_audio_understanding\",\n",
+    "    \"model_type\",\n",
+    "    \"architectures\",\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Get Server Info\n",
+    "Gets the server information including CLI arguments, token limits, and memory pool sizes.\n",
+    "- Note: `get_server_info` merges the following deprecated endpoints:\n",
+    "  - `get_server_args`\n",
+    "  - `get_memory_pool_size`\n",
+    "  - `get_max_total_num_tokens`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/server_info\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Health Check\n",
+    "- `/health`: Check the health of the server.\n",
+    "- `/health_generate`: Check the health of the server by generating one token."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/health_generate\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/health\"\n",
+    "\n",
+    "response = requests.get(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Flush Cache\n",
+    "\n",
+    "Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API.\n",
+    "\n",
+    "Parameters:\n",
+    "- `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors.\n",
+    "\n",
+    "```bash\n",
+    "# With timeout (wait up to 30s for idle state)\n",
+    "curl -s -X POST \"http://127.0.0.1:30000/flush_cache?timeout=30\"\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = f\"http://localhost:{port}/flush_cache\"\n",
+    "\n",
+    "response = requests.post(url)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Update Weights From Disk\n",
+    "\n",
+    "Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.\n",
+    "\n",
+    "SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# successful update with same architecture and size\n",
+    "\n",
+    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
+    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.text)\n",
+    "assert response.json()[\"success\"] is True\n",
+    "assert response.json()[\"message\"] == \"Succeeded to update model weights.\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# failed update with different parameter size or wrong name\n",
+    "\n",
+    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
+    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct-wrong\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "print_highlight(response_json)\n",
+    "assert response_json[\"success\"] is False\n",
+    "assert response_json[\"message\"] == (\n",
+    "    \"Failed to get weights iterator: \"\n",
+    "    \"qwen/qwen2.5-0.5b-instruct-wrong\"\n",
+    "    \" (repository not found).\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Encode (embedding model)\n",
+    "\n",
+    "Encode text into embeddings. Note that this API is only available for [embedding models](openai_api_embeddings.ipynb) and will raise an error for generation models.\n",
+    "Therefore, we launch a new server to server an embedding model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "embedding_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
+    "    --host 0.0.0.0 --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# successful encode for embedding model\n",
+    "\n",
+    "url = f\"http://localhost:{port}/encode\"\n",
+    "data = {\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"text\": \"Once upon a time\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "print_highlight(f\"Text embedding (first 10): {response_json['embedding'][:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## v1/rerank (cross encoder rerank model)\n",
+    "Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "reranker_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \\\n",
+    "    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=reranker_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# compute rerank scores for query and documents\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/rerank\"\n",
+    "data = {\n",
+    "    \"model\": \"BAAI/bge-reranker-v2-m3\",\n",
+    "    \"query\": \"what is panda?\",\n",
+    "    \"documents\": [\n",
+    "        \"hi\",\n",
+    "        \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\",\n",
+    "    ],\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "for item in response_json:\n",
+    "    print_highlight(f\"Score: {item['score']:.2f} - Document: '{item['document']}'\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(reranker_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## v1/score (decoder-only scoring)\n",
+    "\n",
+    "Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities.\n",
+    "\n",
+    "Parameters:\n",
+    "- `query`: Query text\n",
+    "- `items`: Item text(s) to score\n",
+    "- `label_token_ids`: Token IDs to compute probabilities for\n",
+    "- `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False)\n",
+    "- `item_first`: Whether items come first in concatenation order (default: False)\n",
+    "- `model`: Model name\n",
+    "\n",
+    "The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "score_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \\\n",
+    "    --host 0.0.0.0 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=score_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Score the probability of different completions given a query\n",
+    "query = \"The capital of France is\"\n",
+    "items = [\"Paris\", \"London\", \"Berlin\"]\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/score\"\n",
+    "data = {\n",
+    "    \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    \"query\": query,\n",
+    "    \"items\": items,\n",
+    "    \"label_token_ids\": [9454, 2753],  # e.g. \"Yes\" and \"No\" token ids\n",
+    "    \"apply_softmax\": True,  # Normalize probabilities to sum to 1\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "response_json = response.json()\n",
+    "\n",
+    "# Display scores for each item\n",
+    "for item, scores in zip(items, response_json[\"scores\"]):\n",
+    "    print_highlight(f\"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(score_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Classify (reward model)\n",
+    "\n",
+    "SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Note that SGLang now treats embedding models and reward models as the same type of models.\n",
+    "# This will be updated in the future.\n",
+    "\n",
+    "reward_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=reward_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "PROMPT = (\n",
+    "    \"What is the range of the numeric output of a sigmoid node in a neural network?\"\n",
+    ")\n",
+    "\n",
+    "RESPONSE1 = \"The output of a sigmoid node is bounded between -1 and 1.\"\n",
+    "RESPONSE2 = \"The output of a sigmoid node is bounded between 0 and 1.\"\n",
+    "\n",
+    "CONVS = [\n",
+    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE1}],\n",
+    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE2}],\n",
+    "]\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\")\n",
+    "prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)\n",
+    "\n",
+    "url = f\"http://localhost:{port}/classify\"\n",
+    "data = {\"model\": \"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\", \"text\": prompts}\n",
+    "\n",
+    "responses = requests.post(url, json=data).json()\n",
+    "for response in responses:\n",
+    "    print_highlight(f\"reward: {response['embedding'][0]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(reward_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Capture expert selection distribution in MoE models\n",
+    "\n",
+    "SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.\n",
+    "\n",
+    "*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "expert_record_server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=expert_record_server_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = requests.post(f\"http://localhost:{port}/start_expert_distribution_record\")\n",
+    "print_highlight(response)\n",
+    "\n",
+    "url = f\"http://localhost:{port}/generate\"\n",
+    "data = {\"text\": \"What is the capital of France?\"}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.json())\n",
+    "\n",
+    "response = requests.post(f\"http://localhost:{port}/stop_expert_distribution_record\")\n",
+    "print_highlight(response)\n",
+    "\n",
+    "response = requests.post(f\"http://localhost:{port}/dump_expert_distribution_record\")\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(expert_record_server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Tokenize/Detokenize Example (Round Trip)\n",
+    "\n",
+    "This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer_free_server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=tokenizer_free_server_process)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "from sglang.utils import print_highlight\n",
+    "\n",
+    "base_url = f\"http://localhost:{port}\"\n",
+    "tokenize_url = f\"{base_url}/tokenize\"\n",
+    "detokenize_url = f\"{base_url}/detokenize\"\n",
+    "\n",
+    "model_name = \"qwen/qwen2.5-0.5b-instruct\"\n",
+    "input_text = \"SGLang provides efficient tokenization endpoints.\"\n",
+    "print_highlight(f\"Original Input Text:\\n'{input_text}'\")\n",
+    "\n",
+    "# --- tokenize the input text ---\n",
+    "tokenize_payload = {\n",
+    "    \"model\": model_name,\n",
+    "    \"prompt\": input_text,\n",
+    "    \"add_special_tokens\": False,\n",
+    "}\n",
+    "try:\n",
+    "    tokenize_response = requests.post(tokenize_url, json=tokenize_payload)\n",
+    "    tokenize_response.raise_for_status()\n",
+    "    tokenization_result = tokenize_response.json()\n",
+    "    token_ids = tokenization_result.get(\"tokens\")\n",
+    "\n",
+    "    if not token_ids:\n",
+    "        raise ValueError(\"Tokenization returned empty tokens.\")\n",
+    "\n",
+    "    print_highlight(f\"\\nTokenized Output (IDs):\\n{token_ids}\")\n",
+    "    print_highlight(f\"Token Count: {tokenization_result.get('count')}\")\n",
+    "    print_highlight(f\"Max Model Length: {tokenization_result.get('max_model_len')}\")\n",
+    "\n",
+    "    # --- detokenize the obtained token IDs ---\n",
+    "    detokenize_payload = {\n",
+    "        \"model\": model_name,\n",
+    "        \"tokens\": token_ids,\n",
+    "        \"skip_special_tokens\": True,\n",
+    "    }\n",
+    "\n",
+    "    detokenize_response = requests.post(detokenize_url, json=detokenize_payload)\n",
+    "    detokenize_response.raise_for_status()\n",
+    "    detokenization_result = detokenize_response.json()\n",
+    "    reconstructed_text = detokenization_result.get(\"text\")\n",
+    "\n",
+    "    print_highlight(f\"\\nDetokenized Output (Text):\\n'{reconstructed_text}'\")\n",
+    "\n",
+    "    if input_text == reconstructed_text:\n",
+    "        print_highlight(\n",
+    "            \"\\nRound Trip Successful: Original and reconstructed text match.\"\n",
+    "        )\n",
+    "    else:\n",
+    "        print_highlight(\n",
+    "            \"\\nRound Trip Mismatch: Original and reconstructed text differ.\"\n",
+    "        )\n",
+    "\n",
+    "except requests.exceptions.RequestException as e:\n",
+    "    print_highlight(f\"\\nHTTP Request Error: {e}\")\n",
+    "except Exception as e:\n",
+    "    print_highlight(f\"\\nAn error occurred: {e}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(tokenizer_free_server_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs_new/docs/basic_usage/native_api.mdx

@@ -0,0 +1,448 @@
+---
+title: "SGLang Native APIs"
+metatags:
+    description: "SGLang native server APIs for text generation, embedding, reranking, model info, cache management, and more."
+---
+Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:
+
+- `/generate` (text generation model)
+- `/get_model_info`
+- `/server_info`
+- `/health`
+- `/health_generate`
+- `/flush_cache`
+- `/update_weights`
+- `/encode`(embedding model)
+- `/v1/rerank`(cross encoder rerank model)
+- `/v1/score`(decoder-only scoring)
+- `/classify`(reward model)
+- `/start_expert_distribution_record`
+- `/stop_expert_distribution_record`
+- `/dump_expert_distribution_record`
+- `/tokenize`
+- `/detokenize`
+- A full list of these APIs can be found at [http_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py)
+
+We mainly use `requests` to test these APIs in the following examples. You can also use `curl`.
+
+## Launch A Server
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}", process=server_process)
+```
+
+## Generate (text generation model)
+Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params).
+
+```python Example
+import requests
+
+url = f"http://localhost:{port}/generate"
+data = {"text": "What is the capital of France?"}
+
+response = requests.post(url, json=data)
+print_highlight(response.json())
+```
+
+## Get Model Info
+
+Get the information of the model.
+
+- `model_path`: The path/name of the model.
+- `is_generation`: Whether the model is used as generation model or embedding model.
+- `tokenizer_path`: The path/name of the tokenizer.
+- `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args.
+- `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.
+- `has_image_understanding`: Whether the model has image-understanding capability.
+- `has_audio_understanding`: Whether the model has audio-understanding capability.
+- `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
+- `architectures`: The model architectures from the HuggingFace config (e.g., ["Qwen2ForCausalLM"]).
+
+```python Example
+url = f"http://localhost:{port}/get_model_info"
+
+response = requests.get(url)
+response_json = response.json()
+print_highlight(response_json)
+assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct"
+assert response_json["is_generation"] is True
+assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
+assert response_json["preferred_sampling_params"] is None
+assert response_json.keys() == {
+    "model_path",
+    "is_generation",
+    "tokenizer_path",
+    "preferred_sampling_params",
+    "weight_version",
+    "has_image_understanding",
+    "has_audio_understanding",
+    "model_type",
+    "architectures",
+}
+```
+
+## Get Server Info
+Gets the server information including CLI arguments, token limits, and memory pool sizes.
+- Note: `get_server_info` merges the following deprecated endpoints:
+  - `get_server_args`
+  - `get_memory_pool_size`
+  - `get_max_total_num_tokens`
+
+```python Example
+url = f"http://localhost:{port}/server_info"
+
+response = requests.get(url)
+print_highlight(response.text)
+```
+
+## Health Check
+- `/health`: Check the health of the server.
+- `/health_generate`: Check the health of the server by generating one token.
+
+```python Example
+url = f"http://localhost:{port}/health_generate"
+
+response = requests.get(url)
+print_highlight(response.text)
+```
+
+```python Example
+url = f"http://localhost:{port}/health"
+
+response = requests.get(url)
+print_highlight(response.text)
+```
+
+## Flush Cache
+
+Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API.
+
+Parameters:
+- `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors.
+
+```bash Command
+# With timeout (wait up to 30s for idle state)
+curl -s -X POST "http://127.0.0.1:30000/flush_cache?timeout=30"
+```
+
+```python Example
+url = f"http://localhost:{port}/flush_cache"
+
+response = requests.post(url)
+print_highlight(response.text)
+```
+
+## Update Weights From Disk
+
+Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.
+
+SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).
+
+```python Example
+# successful update with same architecture and size
+
+url = f"http://localhost:{port}/update_weights_from_disk"
+data = {"model_path": "qwen/qwen2.5-0.5b-instruct"}
+
+response = requests.post(url, json=data)
+print_highlight(response.text)
+assert response.json()["success"] is True
+assert response.json()["message"] == "Succeeded to update model weights."
+```
+
+```python Example
+# failed update with different parameter size or wrong name
+
+url = f"http://localhost:{port}/update_weights_from_disk"
+data = {"model_path": "qwen/qwen2.5-0.5b-instruct-wrong"}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+print_highlight(response_json)
+assert response_json["success"] is False
+assert response_json["message"] == (
+    "Failed to get weights iterator: "
+    "qwen/qwen2.5-0.5b-instruct-wrong"
+    " (repository not found)."
+)
+```
+
+```python Example
+terminate_process(server_process)
+```
+
+## Encode (embedding model)
+
+Encode text into embeddings. Note that this API is only available for [embedding models](./openai_api_embeddings) and will raise an error for generation models.
+Therefore, we launch a new server to server an embedding model.
+
+```python Example
+embedding_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
+    --host 0.0.0.0 --is-embedding --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=embedding_process)
+```
+
+```python Example
+# successful encode for embedding model
+
+url = f"http://localhost:{port}/encode"
+data = {"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "text": "Once upon a time"}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}")
+```
+
+```python Example
+terminate_process(embedding_process)
+```
+
+## v1/rerank (cross encoder rerank model)
+Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.
+
+```python Example
+reranker_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \
+    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=reranker_process)
+```
+
+```python Example
+# compute rerank scores for query and documents
+
+url = f"http://localhost:{port}/v1/rerank"
+data = {
+    "model": "BAAI/bge-reranker-v2-m3",
+    "query": "what is panda?",
+    "documents": [
+        "hi",
+        "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.",
+    ],
+}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+for item in response_json:
+    print_highlight(f"Score: {item['score']:.2f} - Document: '{item['document']}'")
+```
+
+```python Example
+terminate_process(reranker_process)
+```
+
+## v1/score (decoder-only scoring)
+
+Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities.
+
+Parameters:
+- `query`: Query text
+- `items`: Item text(s) to score
+- `label_token_ids`: Token IDs to compute probabilities for
+- `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False)
+- `item_first`: Whether items come first in concatenation order (default: False)
+- `model`: Model name
+
+The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`.
+
+```python Example
+score_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
+    --host 0.0.0.0 --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=score_process)
+```
+
+```python Example
+# Score the probability of different completions given a query
+query = "The capital of France is"
+items = ["Paris", "London", "Berlin"]
+
+url = f"http://localhost:{port}/v1/score"
+data = {
+    "model": "qwen/qwen2.5-0.5b-instruct",
+    "query": query,
+    "items": items,
+    "label_token_ids": [9454, 2753],  # e.g. "Yes" and "No" token ids
+    "apply_softmax": True,  # Normalize probabilities to sum to 1
+}
+
+response = requests.post(url, json=data)
+response_json = response.json()
+
+# Display scores for each item
+for item, scores in zip(items, response_json["scores"]):
+    print_highlight(f"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}")
+```
+
+```python Example
+terminate_process(score_process)
+```
+
+## Classify (reward model)
+
+SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations.
+
+```python Example
+# Note that SGLang now treats embedding models and reward models as the same type of models.
+# This will be updated in the future.
+
+reward_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=reward_process)
+```
+
+```python Example
+from transformers import AutoTokenizer
+
+PROMPT = (
+    "What is the range of the numeric output of a sigmoid node in a neural network?"
+)
+
+RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1."
+RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1."
+
+CONVS = [
+    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}],
+    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}],
+]
+
+tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2")
+prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)
+
+url = f"http://localhost:{port}/classify"
+data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts}
+
+responses = requests.post(url, json=data).json()
+for response in responses:
+    print_highlight(f"reward: {response['embedding'][0]}")
+```
+
+```python Example
+terminate_process(reward_process)
+```
+
+## Capture expert selection distribution in MoE models
+
+SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.
+
+*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*
+
+```python Example
+expert_record_server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}", process=expert_record_server_process)
+```
+
+```python Example
+response = requests.post(f"http://localhost:{port}/start_expert_distribution_record")
+print_highlight(response)
+
+url = f"http://localhost:{port}/generate"
+data = {"text": "What is the capital of France?"}
+
+response = requests.post(url, json=data)
+print_highlight(response.json())
+
+response = requests.post(f"http://localhost:{port}/stop_expert_distribution_record")
+print_highlight(response)
+
+response = requests.post(f"http://localhost:{port}/dump_expert_distribution_record")
+print_highlight(response)
+```
+
+```python Example
+terminate_process(expert_record_server_process)
+```
+
+## Tokenize/Detokenize Example (Round Trip)
+
+This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization.
+
+```python Example
+tokenizer_free_server_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct
+""")
+
+wait_for_server(f"http://localhost:{port}", process=tokenizer_free_server_process)
+```
+
+```python Example
+import requests
+from sglang.utils import print_highlight
+
+base_url = f"http://localhost:{port}"
+tokenize_url = f"{base_url}/tokenize"
+detokenize_url = f"{base_url}/detokenize"
+
+model_name = "qwen/qwen2.5-0.5b-instruct"
+input_text = "SGLang provides efficient tokenization endpoints."
+print_highlight(f"Original Input Text:\n'{input_text}'")
+
+# --- tokenize the input text ---
+tokenize_payload = {
+    "model": model_name,
+    "prompt": input_text,
+    "add_special_tokens": False,
+}
+try:
+    tokenize_response = requests.post(tokenize_url, json=tokenize_payload)
+    tokenize_response.raise_for_status()
+    tokenization_result = tokenize_response.json()
+    token_ids = tokenization_result.get("tokens")
+
+    if not token_ids:
+        raise ValueError("Tokenization returned empty tokens.")
+
+    print_highlight(f"\nTokenized Output (IDs):\n{token_ids}")
+    print_highlight(f"Token Count: {tokenization_result.get('count')}")
+    print_highlight(f"Max Model Length: {tokenization_result.get('max_model_len')}")
+
+    # --- detokenize the obtained token IDs ---
+    detokenize_payload = {
+        "model": model_name,
+        "tokens": token_ids,
+        "skip_special_tokens": True,
+    }
+
+    detokenize_response = requests.post(detokenize_url, json=detokenize_payload)
+    detokenize_response.raise_for_status()
+    detokenization_result = detokenize_response.json()
+    reconstructed_text = detokenization_result.get("text")
+
+    print_highlight(f"\nDetokenized Output (Text):\n'{reconstructed_text}'")
+
+    if input_text == reconstructed_text:
+        print_highlight(
+            "\nRound Trip Successful: Original and reconstructed text match."
+        )
+    else:
+        print_highlight(
+            "\nRound Trip Mismatch: Original and reconstructed text differ."
+        )
+
+except requests.exceptions.RequestException as e:
+    print_highlight(f"\nHTTP Request Error: {e}")
+except Exception as e:
+    print_highlight(f"\nAn error occurred: {e}")
+```
+
+```python Example
+terminate_process(tokenizer_free_server_process)
+```

docs_new/docs/basic_usage/offline_engine_api.ipynb

@@ -0,0 +1,291 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Offline Engine API\n",
+    "\n",
+    "SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:\n",
+    "\n",
+    "- Offline Batch Inference\n",
+    "- Custom Server on Top of the Engine\n",
+    "\n",
+    "This document focuses on the offline batch inference, demonstrating four different inference modes:\n",
+    "\n",
+    "- Non-streaming synchronous generation\n",
+    "- Streaming synchronous generation\n",
+    "- Non-streaming asynchronous generation\n",
+    "- Streaming asynchronous generation\n",
+    "\n",
+    "Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Nest Asyncio\n",
+    "Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code:\n",
+    "```python\n",
+    "import nest_asyncio\n",
+    "\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Advanced Usage\n",
+    "\n",
+    "The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/blob/main/examples/runtime/hidden_states). \n",
+    "\n",
+    "Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Ray Integration\n",
+    "\n",
+    "When running in a Ray cluster, you can use `RayEngine` with a custom placement group for fine-grained GPU placement control.\n",
+    "\n",
+    "### Custom Placement Groups\n",
+    "\n",
+    "Pass a `placement_group` with 1-GPU-per-bundle bundles to control exactly which GPUs are used. Each bundle should have exactly 1 GPU for deterministic mapping.\n",
+    "\n",
+    "```python\n",
+    "import ray\n",
+    "from ray.util.placement_group import placement_group\n",
+    "from sglang.srt.ray.engine import RayEngine\n",
+    "\n",
+    "ray.init()\n",
+    "\n",
+    "# Create placement group with specific GPU bundles\n",
+    "pg = placement_group(\n",
+    "    [{\"GPU\": 1} for _ in range(4)],  # 4 bundles, each with 1 GPU\n",
+    "    strategy=\"STRICT_PACK\",\n",
+    ")\n",
+    "ray.get(pg.ready())\n",
+    "\n",
+    "# Launch RayEngine on custom placement group\n",
+    "engine = RayEngine(\n",
+    "    model_path=\"meta-llama/Meta-Llama-3-8B-Instruct\",\n",
+    "    tp_size=4,\n",
+    "    use_ray=True,\n",
+    "    placement_group=pg,\n",
+    ")\n",
+    "\n",
+    "# Optional: specify exact bundle indices via environment variable\n",
+    "# export SGLANG_RAY_BUNDLE_INDICES=\"0,1,2,3\"\n",
+    "```\n",
+    "\n",
+    "### Bundle Index Control\n",
+    "\n",
+    "Use `SGLANG_RAY_BUNDLE_INDICES` environment variable to specify which placement group bundles to use for each worker rank. This enables:\n",
+    "- Skipping unhealthy GPUs\n",
+    "- Topology-aware placement (e.g., NVLink-connected GPUs)\n",
+    "- Non-sequential bundle assignment\n",
+    "\n",
+    "```bash\n",
+    "# Use bundles 0,1,2,7 (skip bundles 3-6) for tp_size=4\n",
+    "export SGLANG_RAY_BUNDLE_INDICES=\"0,1,2,7\"\n",
+    "\n",
+    "# Place workers on NVLink-connected GPUs\n",
+    "export SGLANG_RAY_BUNDLE_INDICES=\"0,1,2,3\"\n",
+    "```\n",
+    "\n",
+    "The number of indices must match `world_size` (`tp_size * pp_size * dp_size`, or `tp_size * pp_size` when `enable_dp_attention=True`)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Offline Batch Inference\n",
+    "\n",
+    "SGLang offline engine supports batch inference with efficient scheduling."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# launch the offline engine\n",
+    "import asyncio\n",
+    "\n",
+    "import sglang as sgl\n",
+    "import sglang.test.doc_patch  # noqa: F401\n",
+    "from sglang.utils import async_stream_and_merge, stream_and_merge\n",
+    "\n",
+    "llm = sgl.Engine(model_path=\"qwen/qwen2.5-0.5b-instruct\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Non-streaming Synchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Hello, my name is\",\n",
+    "    \"The president of the United States is\",\n",
+    "    \"The capital of France is\",\n",
+    "    \"The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
+    "\n",
+    "outputs = llm.generate(prompts, sampling_params)\n",
+    "for prompt, output in zip(prompts, outputs):\n",
+    "    print(\"===============================\")\n",
+    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming Synchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
+    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
+    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\n",
+    "    \"temperature\": 0.2,\n",
+    "    \"top_p\": 0.9,\n",
+    "}\n",
+    "\n",
+    "print(\"\\n=== Testing synchronous streaming generation with overlap removal ===\\n\")\n",
+    "\n",
+    "for prompt in prompts:\n",
+    "    print(f\"Prompt: {prompt}\")\n",
+    "    merged_output = stream_and_merge(llm, prompt, sampling_params)\n",
+    "    print(\"Generated text:\", merged_output)\n",
+    "    print()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Non-streaming Asynchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
+    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
+    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
+    "\n",
+    "print(\"\\n=== Testing asynchronous batch generation ===\")\n",
+    "\n",
+    "\n",
+    "async def main():\n",
+    "    outputs = await llm.async_generate(prompts, sampling_params)\n",
+    "\n",
+    "    for prompt, output in zip(prompts, outputs):\n",
+    "        print(f\"\\nPrompt: {prompt}\")\n",
+    "        print(f\"Generated text: {output['text']}\")\n",
+    "\n",
+    "\n",
+    "asyncio.run(main())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming Asynchronous Generation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "prompts = [\n",
+    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
+    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
+    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
+    "]\n",
+    "\n",
+    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
+    "\n",
+    "print(\"\\n=== Testing asynchronous streaming generation (no repeats) ===\")\n",
+    "\n",
+    "\n",
+    "async def main():\n",
+    "    for prompt in prompts:\n",
+    "        print(f\"\\nPrompt: {prompt}\")\n",
+    "        print(\"Generated text: \", end=\"\", flush=True)\n",
+    "\n",
+    "        # Replace direct calls to async_generate with our custom overlap-aware version\n",
+    "        async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):\n",
+    "            print(cleaned_chunk, end=\"\", flush=True)\n",
+    "\n",
+    "        print()  # New line after each prompt\n",
+    "\n",
+    "\n",
+    "asyncio.run(main())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm.shutdown()"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/offline_engine_api.mdx

@@ -0,0 +1,143 @@
+---
+title: "Offline Engine API"
+metatags:
+    description: "Use SGLang's offline engine for direct batch inference without HTTP server overhead. Supports sync/async and streaming modes."
+---
+SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:
+
+- Offline Batch Inference
+- Custom Server on Top of the Engine
+
+This document focuses on the offline batch inference, demonstrating four different inference modes:
+
+- Non-streaming synchronous generation
+- Streaming synchronous generation
+- Non-streaming asynchronous generation
+- Streaming asynchronous generation
+
+Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).
+
+## Nest Asyncio
+Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code:
+```python Example
+import nest_asyncio
+
+nest_asyncio.apply()
+
+```
+
+## Advanced Usage
+
+The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/tree/main/examples/runtime/hidden_states).
+
+Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases.
+
+## Offline Batch Inference
+
+SGLang offline engine supports batch inference with efficient scheduling.
+
+```python Example
+# launch the offline engine
+import asyncio
+
+import sglang as sgl
+import sglang.test.doc_patch
+from sglang.utils import async_stream_and_merge, stream_and_merge
+
+llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
+```
+
+### Non-streaming Synchronous Generation
+
+```python Example
+prompts = [
+    "Hello, my name is",
+    "The president of the United States is",
+    "The capital of France is",
+    "The future of AI is",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95}
+
+outputs = llm.generate(prompts, sampling_params)
+for prompt, output in zip(prompts, outputs):
+    print("===============================")
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
+```
+
+### Streaming Synchronous Generation
+
+```python Example
+prompts = [
+    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+    "Provide a concise factual statement about France’s capital city. The capital of France is",
+    "Explain possible future trends in artificial intelligence. The future of AI is",
+]
+
+sampling_params = {
+    "temperature": 0.2,
+    "top_p": 0.9,
+}
+
+print("\n=== Testing synchronous streaming generation with overlap removal ===\n")
+
+for prompt in prompts:
+    print(f"Prompt: {prompt}")
+    merged_output = stream_and_merge(llm, prompt, sampling_params)
+    print("Generated text:", merged_output)
+    print()
+```
+
+### Non-streaming Asynchronous Generation
+
+```python Example
+prompts = [
+    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+    "Provide a concise factual statement about France’s capital city. The capital of France is",
+    "Explain possible future trends in artificial intelligence. The future of AI is",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95}
+
+print("\n=== Testing asynchronous batch generation ===")
+
+async def main():
+    outputs = await llm.async_generate(prompts, sampling_params)
+
+    for prompt, output in zip(prompts, outputs):
+        print(f"\nPrompt: {prompt}")
+        print(f"Generated text: {output['text']}")
+
+asyncio.run(main())
+```
+
+### Streaming Asynchronous Generation
+
+```python Example
+prompts = [
+    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+    "Provide a concise factual statement about France’s capital city. The capital of France is",
+    "Explain possible future trends in artificial intelligence. The future of AI is",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95}
+
+print("\n=== Testing asynchronous streaming generation (no repeats) ===")
+
+async def main():
+    for prompt in prompts:
+        print(f"\nPrompt: {prompt}")
+        print("Generated text: ", end="", flush=True)
+
+        # Replace direct calls to async_generate with our custom overlap-aware version
+        async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):
+            print(cleaned_chunk, end="", flush=True)
+
+        print()  # New line after each prompt
+
+asyncio.run(main())
+```
+
+```python Example
+llm.shutdown()
+```

docs_new/docs/basic_usage/ollama_api.mdx

@@ -0,0 +1,157 @@
+---
+title: "Ollama-Compatible API"
+metatags:
+    description: "SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend."
+---
+SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend.
+
+## Prerequisites
+
+<CodeGroup>
+```bash Command
+# Install the Ollama Python library (for Python client usage)
+pip install ollama
+```
+</CodeGroup>
+
+<Note>You don't need the Ollama server installed - SGLang acts as the backend. You only need the `ollama` CLI or Python library as the client.</Note>
+
+## Endpoints
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Endpoint</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Method</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GET, HEAD</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Health check for Ollama CLI</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/tags`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GET</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>List available models</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/chat`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Chat completions (streaming & non-streaming)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/generate`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Text generation (streaming & non-streaming)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`/api/show`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>POST</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Model information</td>
+    </tr>
+  </tbody>
+</table>
+
+## Quick Start
+
+### 1. Launch SGLang Server
+
+<CodeGroup>
+```bash Command
+python -m sglang.launch_server \
+    --model Qwen/Qwen2.5-1.5B-Instruct \
+    --port 30001 \
+    --host 0.0.0.0
+```
+</CodeGroup>
+
+<Note>The model name used with `ollama run` must match exactly what you passed to `--model`.</Note>
+
+### 2. Use Ollama CLI
+
+<CodeGroup>
+```bash Command
+# List available models
+OLLAMA_HOST=http://localhost:30001 ollama list
+
+# Interactive chat
+OLLAMA_HOST=http://localhost:30001 ollama run "Qwen/Qwen2.5-1.5B-Instruct"
+```
+</CodeGroup>
+
+If connecting to a remote server behind a firewall:
+
+<CodeGroup>
+```bash Command
+# SSH tunnel
+ssh -L 30001:localhost:30001 user@gpu-server -N &
+
+# Then use Ollama CLI as above
+OLLAMA_HOST=http://localhost:30001 ollama list
+```
+</CodeGroup>
+
+### 3. Use Ollama Python Library
+
+```python Example
+import ollama
+
+client = ollama.Client(host='http://localhost:30001')
+
+# Non-streaming
+response = client.chat(
+    model='Qwen/Qwen2.5-1.5B-Instruct',
+    messages=[{'role': 'user', 'content': 'Hello!'}]
+)
+print(response['message']['content'])
+
+# Streaming
+stream = client.chat(
+    model='Qwen/Qwen2.5-1.5B-Instruct',
+    messages=[{'role': 'user', 'content': 'Tell me a story'}],
+    stream=True
+)
+for chunk in stream:
+    print(chunk['message']['content'], end='', flush=True)
+```
+
+## Smart Router
+
+For intelligent routing between local Ollama (fast) and remote SGLang (powerful) using an LLM judge, see the [Smart Router documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/ollama/README).
+
+## Summary
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Ollama API**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Familiar CLI/API that developers already know</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**SGLang Backend**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>High-performance inference engine</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Smart Router**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Intelligent routing - fast local for simple tasks, powerful remote for complex tasks</td>
+    </tr>
+  </tbody>
+</table>

docs_new/docs/basic_usage/openai_api.mdx

@@ -0,0 +1,9 @@
+---
+title: "OpenAI-Compatible APIs"
+description: "Documentation for OpenAI-Compatible APIs"
+---
+- [Openai Api Completions](./openai_api_completions)
+- [Openai Api Vision](./openai_api_vision)
+- [Openai Api Embeddings](./openai_api_embeddings)
+
+For the Anthropic-compatible `/v1/messages` endpoint (including Claude Code integration), see [Anthropic-Compatible API](./anthropic_api).

docs_new/docs/basic_usage/openai_api.rst

@@ -0,0 +1,9 @@
+OpenAI-Compatible APIs
+======================
+
+.. toctree::
+   :maxdepth: 1
+
+   openai_api_completions.ipynb
+   openai_api_vision.ipynb
+   openai_api_embeddings.ipynb

docs_new/docs/basic_usage/openai_api_completions.ipynb

@@ -0,0 +1,552 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# OpenAI APIs - Completions\n",
+    "\n",
+    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
+    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).\n",
+    "\n",
+    "This tutorial covers the following popular APIs:\n",
+    "\n",
+    "- `chat/completions`\n",
+    "- `completions`\n",
+    "\n",
+    "Check out other tutorials to learn about [vision APIs](openai_api_vision.ipynb) for vision-language models and [embedding APIs](openai_api_embeddings.ipynb) for embedding models."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server\n",
+    "\n",
+    "Launch the server in your terminal and wait for it to initialize."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\n",
+    "    \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning\"\n",
+    ")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
+    "print(f\"Server started on http://localhost:{port}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Chat Completions\n",
+    "\n",
+    "### Usage\n",
+    "\n",
+    "The server fully implements the OpenAI API.\n",
+    "It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.\n",
+    "You can also specify a custom chat template with `--chat-template` when launching the server."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Model Thinking/Reasoning Support\n",
+    "\n",
+    "Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers.\n",
+    "\n",
+    "#### Supported Models and Configuration\n",
+    "\n",
+    "| Model Family | Chat Template Parameter | Reasoning Parser | Notes |\n",
+    "|--------------|------------------------|------------------|--------|\n",
+    "| DeepSeek-R1 (R1, R1-0528, R1-Distill) | `enable_thinking` | `--reasoning-parser deepseek-r1` | Standard reasoning models |\n",
+    "| DeepSeek-V3.1 | `thinking` | `--reasoning-parser deepseek-v3` | Hybrid model (thinking/non-thinking modes) |\n",
+    "| Qwen3 (standard) | `enable_thinking` | `--reasoning-parser qwen3` | Hybrid model (thinking/non-thinking modes) |\n",
+    "| Qwen3-Thinking | N/A (always enabled) | `--reasoning-parser qwen3-thinking` | Always generates reasoning |\n",
+    "| Kimi | N/A (always enabled) | `--reasoning-parser kimi` | Kimi thinking models |\n",
+    "| Gpt-Oss | N/A (always enabled) | `--reasoning-parser gpt-oss` | Gpt-Oss thinking models |\n",
+    "\n",
+    "#### Basic Usage\n",
+    "\n",
+    "To enable reasoning output, you need to:\n",
+    "1. Launch the server with the appropriate reasoning parser\n",
+    "2. Set the model-specific parameter in `chat_template_kwargs`\n",
+    "3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`)\n",
+    "\n",
+    "**Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Example: Qwen3 Models\n",
+    "\n",
+    "```python\n",
+    "# Launch server:\n",
+    "# python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3\n",
+    "\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=\"EMPTY\",\n",
+    "    base_url=f\"http://127.0.0.1:30000/v1\",\n",
+    ")\n",
+    "\n",
+    "model = \"Qwen/Qwen3-4B\"\n",
+    "messages = [{\"role\": \"user\", \"content\": \"How many r's are in 'strawberry'?\"}]\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=model,\n",
+    "    messages=messages,\n",
+    "    extra_body={\n",
+    "        \"chat_template_kwargs\": {\"enable_thinking\": True},\n",
+    "        \"separate_reasoning\": True\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "print(\"Reasoning:\", response.choices[0].message.reasoning_content)\n",
+    "print(\"-\"*100)\n",
+    "print(\"Answer:\", response.choices[0].message.content)\n",
+    "```\n",
+    "\n",
+    "**ExampleOutput:**\n",
+    "```\n",
+    "Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down.\n",
+    "\n",
+    "Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y. \n",
+    "...\n",
+    "Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right.\n",
+    "\n",
+    "----------------------------------------------------------------------------------------------------\n",
+    "Answer: The word \"strawberry\" contains **three** letters 'r'. Here's the breakdown:\n",
+    "\n",
+    "1. **S-T-R-A-W-B-E-R-R-Y**  \n",
+    "   - The **third letter** is 'R'.  \n",
+    "   - The **eighth and ninth letters** are also 'R's.  \n",
+    "\n",
+    "Thus, the total count is **3**.  \n",
+    "\n",
+    "**Answer:** 3.\n",
+    "```\n",
+    "\n",
+    "**Note:** Setting `\"enable_thinking\": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Logit Bias Support\n",
+    "\n",
+    "SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where:\n",
+    "\n",
+    "- **Positive values** (0 to 100) increase the likelihood of the token being selected\n",
+    "- **Negative values** (-100 to 0) decrease the likelihood of the token being selected\n",
+    "- **-100** effectively prevents the token from being generated\n",
+    "\n",
+    "The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Getting Token IDs\n",
+    "\n",
+    "To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs:\n",
+    "\n",
+    "```python\n",
+    "# Get tokenizer to find token IDs\n",
+    "import tiktoken\n",
+    "\n",
+    "# For OpenAI models, use the appropriate encoding\n",
+    "tokenizer = tiktoken.encoding_for_model(\"gpt-3.5-turbo\")  # or your model\n",
+    "\n",
+    "# Get token IDs for specific words\n",
+    "word = \"sunny\"\n",
+    "token_ids = tokenizer.encode(word)\n",
+    "print(f\"Token IDs for '{word}': {token_ids}\")\n",
+    "\n",
+    "# For SGLang models, you can access the tokenizer through the client\n",
+    "# and get token IDs for bias\n",
+    "```\n",
+    "\n",
+    "**Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Example: DeepSeek-V3 Models\n",
+    "\n",
+    "DeepSeek-V3 models support thinking mode through the `thinking` parameter:\n",
+    "\n",
+    "```python\n",
+    "# Launch server:\n",
+    "# python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8  --reasoning-parser deepseek-v3\n",
+    "\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(\n",
+    "    api_key=\"EMPTY\",\n",
+    "    base_url=f\"http://127.0.0.1:30000/v1\",\n",
+    ")\n",
+    "\n",
+    "model = \"deepseek-ai/DeepSeek-V3.1\"\n",
+    "messages = [{\"role\": \"user\", \"content\": \"How many r's are in 'strawberry'?\"}]\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=model,\n",
+    "    messages=messages,\n",
+    "    extra_body={\n",
+    "        \"chat_template_kwargs\": {\"thinking\": True},\n",
+    "        \"separate_reasoning\": True\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "print(\"Reasoning:\", response.choices[0].message.reasoning_content)\n",
+    "print(\"-\"*100)\n",
+    "print(\"Answer:\", response.choices[0].message.content)\n",
+    "```\n",
+    "\n",
+    "**Example Output:**\n",
+    "```\n",
+    "Reasoning: First, the question is: \"How many r's are in 'strawberry'?\"\n",
+    "\n",
+    "I need to count the number of times the letter 'r' appears in the word \"strawberry\".\n",
+    "\n",
+    "Let me write out the word: S-T-R-A-W-B-E-R-R-Y.\n",
+    "\n",
+    "Now, I'll go through each letter and count the 'r's.\n",
+    "...\n",
+    "So, I have three 'r's in \"strawberry\".\n",
+    "\n",
+    "I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct.\n",
+    "\n",
+    "Therefore, the answer should be 3.\n",
+    "----------------------------------------------------------------------------------------------------\n",
+    "Answer: The word \"strawberry\" contains **3** instances of the letter \"r\". Here's a breakdown for clarity:\n",
+    "\n",
+    "- The word is spelled: S-T-R-A-W-B-E-R-R-Y\n",
+    "- The \"r\" appears at the 3rd, 8th, and 9th positions.\n",
+    "```\n",
+    "\n",
+    "**Note:** DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example with logit_bias parameter\n",
+    "# Note: You need to get the actual token IDs from your tokenizer\n",
+    "# For demonstration, we'll use some example token IDs\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"Complete this sentence: The weather today is\"}\n",
+    "    ],\n",
+    "    temperature=0.7,\n",
+    "    max_tokens=20,\n",
+    "    logit_bias={\n",
+    "        \"12345\": 50,  # Increase likelihood of token ID 12345\n",
+    "        \"67890\": -50,  # Decrease likelihood of token ID 67890\n",
+    "        \"11111\": 25,  # Slightly increase likelihood of token ID 11111\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response with logit bias: {response.choices[0].message.content}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Parameters\n",
+    "\n",
+    "The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.\n",
+    "\n",
+    "SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"system\",\n",
+    "            \"content\": \"You are a knowledgeable historian who provides concise responses.\",\n",
+    "        },\n",
+    "        {\"role\": \"user\", \"content\": \"Tell me about ancient Rome\"},\n",
+    "        {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Ancient Rome was a civilization centered in Italy.\",\n",
+    "        },\n",
+    "        {\"role\": \"user\", \"content\": \"What were their major achievements?\"},\n",
+    "    ],\n",
+    "    temperature=0.3,  # Lower temperature for more focused responses\n",
+    "    max_tokens=128,  # Reasonable length for a concise response\n",
+    "    top_p=0.95,  # Slightly higher for better fluency\n",
+    "    presence_penalty=0.2,  # Mild penalty to avoid repetition\n",
+    "    frequency_penalty=0.2,  # Mild penalty for more natural language\n",
+    "    n=1,  # Single response is usually more stable\n",
+    "    seed=42,  # Keep for reproducibility\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Streaming mode is also supported."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Logit Bias Support\n",
+    "\n",
+    "The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "stream = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[{\"role\": \"user\", \"content\": \"Say this is a test\"}],\n",
+    "    stream=True,\n",
+    ")\n",
+    "for chunk in stream:\n",
+    "    if chunk.choices[0].delta.content is not None:\n",
+    "        print(chunk.choices[0].delta.content, end=\"\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Returning Routed Experts (MoE Models)\n",
+    "\n",
+    "For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Example with logit_bias parameter for completions API\n",
+    "# Note: You need to get the actual token IDs from your tokenizer\n",
+    "# For demonstration, we'll use some example token IDs\n",
+    "response = client.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    prompt=\"The best programming language for AI is\",\n",
+    "    temperature=0.7,\n",
+    "    max_tokens=20,\n",
+    "    logit_bias={\n",
+    "        \"12345\": 75,  # Strongly favor token ID 12345\n",
+    "        \"67890\": -100,  # Completely avoid token ID 67890\n",
+    "        \"11111\": -25,  # Slightly discourage token ID 11111\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response with logit bias: {response.choices[0].text}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Completions\n",
+    "\n",
+    "### Usage\n",
+    "Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    prompt=\"List 3 countries and their capitals.\",\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    "    n=1,\n",
+    "    stop=None,\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Parameters\n",
+    "\n",
+    "The completions API accepts OpenAI Completions API's parameters.  Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.\n",
+    "\n",
+    "Here is an example of a detailed completions request:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    prompt=\"Write a short story about a space explorer.\",\n",
+    "    temperature=0.7,  # Moderate temperature for creative writing\n",
+    "    max_tokens=150,  # Longer response for a story\n",
+    "    top_p=0.9,  # Balanced diversity in word choice\n",
+    "    stop=[\"\\n\\n\", \"THE END\"],  # Multiple stop sequences\n",
+    "    presence_penalty=0.3,  # Encourage novel elements\n",
+    "    frequency_penalty=0.3,  # Reduce repetitive phrases\n",
+    "    n=1,  # Generate one completion\n",
+    "    seed=123,  # For reproducible results\n",
+    ")\n",
+    "\n",
+    "print_highlight(f\"Response: {response}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Returning Routed Experts (MoE Models)\n",
+    "\n",
+    "For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Structured Outputs (JSON, Regex, EBNF)\n",
+    "\n",
+    "For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs.ipynb) for more details.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using LoRA Adapters\n",
+    "\n",
+    "SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax.\n",
+    "\n",
+    "**Server Setup:**\n",
+    "```bash\n",
+    "python -m sglang.launch_server \\\n",
+    "    --model-path qwen/qwen2.5-0.5b-instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b\n",
+    "```\n",
+    "\n",
+    "For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora.ipynb).\n",
+    "\n",
+    "**API Call:**\n",
+    "\n",
+    "(Recommended) Use the `model:adapter` syntax to specify which adapter to use:\n",
+    "```python\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct:adapter_a\",  # ← base-model:adapter-name\n",
+    "    messages=[{\"role\": \"user\", \"content\": \"Convert to SQL: show all users\"}],\n",
+    "    max_tokens=50,\n",
+    ")\n",
+    "```\n",
+    "\n",
+    "**Backward Compatible: Using `extra_body`**\n",
+    "\n",
+    "The old `extra_body` method is still supported for backward compatibility:\n",
+    "```python\n",
+    "# Backward compatible method\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[{\"role\": \"user\", \"content\": \"Convert to SQL: show all users\"}],\n",
+    "    extra_body={\"lora_path\": \"adapter_a\"},  # ← old method\n",
+    "    max_tokens=50,\n",
+    ")\n",
+    "```\n",
+    "**Note:** When both `model:adapter` and `extra_body[\"lora_path\"]` are specified, the `model:adapter` syntax takes precedence."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/openai_api_completions.mdx

@@ -0,0 +1,456 @@
+---
+title: "OpenAI APIs - Completions"
+metatags:
+    description: "This tutorial covers the following popular APIs: 'chat/completions' and 'completions'"
+---
+SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
+A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).
+
+This tutorial covers the following popular APIs:
+
+- `chat/completions`
+- `completions`
+
+Check out other tutorials to learn about [vision APIs](./openai_api_vision) for vision-language models and [embedding APIs](./openai_api_embeddings) for embedding models.
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize.
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+server_process, port = launch_server_cmd(
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
+)
+
+wait_for_server(f"http://localhost:{port}")
+print(f"Server started on http://localhost:{port}")
+```
+
+## Chat Completions
+
+### Usage
+
+The server fully implements the OpenAI API.
+It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.
+You can also specify a custom chat template with `--chat-template` when launching the server.
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+)
+
+print_highlight(f"Response: {response}")
+```
+
+### Model Thinking/Reasoning Support
+
+Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers.
+
+#### Supported Models and Configuration
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Family</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Chat Template Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Reasoning Parser</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1 (R1, R1-0528, R1-Distill)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`enable_thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser deepseek-r1`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Standard reasoning models</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser deepseek-v3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hybrid model (thinking/non-thinking modes)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3 (standard)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`enable_thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser qwen3`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hybrid model (thinking/non-thinking modes)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-Thinking</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser qwen3-thinking`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Always generates reasoning</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser kimi`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi thinking models</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Gpt-Oss</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A (always enabled)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser gpt-oss`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gpt-Oss thinking models</td>
+    </tr>
+  </tbody>
+</table>
+
+#### Basic Usage
+
+To enable reasoning output, you need to:
+1. Launch the server with the appropriate reasoning parser
+2. Set the model-specific parameter in `chat_template_kwargs`
+3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`)
+
+<Note>
+**Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content.
+</Note>
+
+#### Example: Qwen3 Models
+
+```python Example
+# Launch server:
+# python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3
+
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url=f"http://127.0.0.1:30000/v1",
+)
+
+model = "Qwen/Qwen3-4B"
+messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    extra_body={
+        "chat_template_kwargs": {"enable_thinking": True},
+        "separate_reasoning": True
+    }
+)
+
+print("Reasoning:", response.choices[0].message.reasoning_content)
+print("-"*100)
+print("Answer:", response.choices[0].message.content)
+```
+
+**ExampleOutput:**
+```text Output
+Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down.
+
+Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y.
+...
+Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right.
+
+----------------------------------------------------------------------------------------------------
+Answer: The word "strawberry" contains **three** letters 'r'. Here's the breakdown:
+
+1. **S-T-R-A-W-B-E-R-R-Y**
+   - The **third letter** is 'R'.
+   - The **eighth and ninth letters** are also 'R's.
+
+Thus, the total count is **3**.
+
+**Answer:** 3.
+```
+<Note>
+Setting `"enable_thinking": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter.
+</Note>
+
+#### Logit Bias Support
+
+SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where:
+
+- **Positive values** (0 to 100) increase the likelihood of the token being selected
+- **Negative values** (-100 to 0) decrease the likelihood of the token being selected
+- **-100** effectively prevents the token from being generated
+
+The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats).
+
+#### Getting Token IDs
+
+To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs:
+
+```python Example
+# Get tokenizer to find token IDs
+import tiktoken
+
+# For OpenAI models, use the appropriate encoding
+tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo")  # or your model
+
+# Get token IDs for specific words
+word = "sunny"
+token_ids = tokenizer.encode(word)
+print(f"Token IDs for '{word}': {token_ids}")
+
+# For SGLang models, you can access the tokenizer through the client
+# and get token IDs for bias
+```
+<Tip>
+**Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words.
+</Tip>
+
+#### Example: DeepSeek-V3 Models
+
+DeepSeek-V3 models support thinking mode through the `thinking` parameter:
+
+```python Example
+# Launch server:
+# python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8  --reasoning-parser deepseek-v3
+
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url=f"http://127.0.0.1:30000/v1",
+)
+
+model = "deepseek-ai/DeepSeek-V3.1"
+messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]
+
+response = client.chat.completions.create(
+    model=model,
+    messages=messages,
+    extra_body={
+        "chat_template_kwargs": {"thinking": True},
+        "separate_reasoning": True
+    }
+)
+
+print("Reasoning:", response.choices[0].message.reasoning_content)
+print("-"*100)
+print("Answer:", response.choices[0].message.content)
+```
+
+**Example Output:**
+```text Output
+Reasoning: First, the question is: "How many r's are in 'strawberry'?"
+
+I need to count the number of times the letter 'r' appears in the word "strawberry".
+
+Let me write out the word: S-T-R-A-W-B-E-R-R-Y.
+
+Now, I'll go through each letter and count the 'r's.
+...
+So, I have three 'r's in "strawberry".
+
+I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct.
+
+Therefore, the answer should be 3.
+----------------------------------------------------------------------------------------------------
+Answer: The word "strawberry" contains **3** instances of the letter "r". Here's a breakdown for clarity:
+
+- The word is spelled: S-T-R-A-W-B-E-R-R-Y
+- The "r" appears at the 3rd, 8th, and 9th positions.
+```
+<Note>
+DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output.
+</Note>
+
+```python Example
+# Example with logit_bias parameter
+# Note: You need to get the actual token IDs from your tokenizer
+# For demonstration, we'll use some example token IDs
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "Complete this sentence: The weather today is"}
+    ],
+    temperature=0.7,
+    max_tokens=20,
+    logit_bias={
+        "12345": 50,  # Increase likelihood of token ID 12345
+        "67890": -50,  # Decrease likelihood of token ID 67890
+        "11111": 25,  # Slightly increase likelihood of token ID 11111
+    },
+)
+
+print_highlight(f"Response with logit bias: {response.choices[0].message.content}")
+```
+
+### Parameters
+
+The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.
+
+SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor.
+
+```python Example
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {
+            "role": "system",
+            "content": "You are a knowledgeable historian who provides concise responses.",
+        },
+        {"role": "user", "content": "Tell me about ancient Rome"},
+        {
+            "role": "assistant",
+            "content": "Ancient Rome was a civilization centered in Italy.",
+        },
+        {"role": "user", "content": "What were their major achievements?"},
+    ],
+    temperature=0.3,  # Lower temperature for more focused responses
+    max_tokens=128,  # Reasonable length for a concise response
+    top_p=0.95,  # Slightly higher for better fluency
+    presence_penalty=0.2,  # Mild penalty to avoid repetition
+    frequency_penalty=0.2,  # Mild penalty for more natural language
+    n=1,  # Single response is usually more stable
+    seed=42,  # Keep for reproducibility
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+Streaming mode is also supported.
+
+#### Logit Bias Support
+
+The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above.
+
+```python Example
+stream = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[{"role": "user", "content": "Say this is a test"}],
+    stream=True,
+)
+for chunk in stream:
+    if chunk.choices[0].delta.content is not None:
+        print(chunk.choices[0].delta.content, end="")
+```
+
+#### Returning Routed Experts (MoE Models)
+
+For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks.
+
+```python Example
+# Example with logit_bias parameter for completions API
+# Note: You need to get the actual token IDs from your tokenizer
+# For demonstration, we'll use some example token IDs
+response = client.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    prompt="The best programming language for AI is",
+    temperature=0.7,
+    max_tokens=20,
+    logit_bias={
+        "12345": 75,  # Strongly favor token ID 12345
+        "67890": -100,  # Completely avoid token ID 67890
+        "11111": -25,  # Slightly discourage token ID 11111
+    },
+)
+
+print_highlight(f"Response with logit bias: {response.choices[0].text}")
+```
+
+## Completions
+
+### Usage
+Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates.
+
+```python Example
+response = client.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    prompt="List 3 countries and their capitals.",
+    temperature=0,
+    max_tokens=64,
+    n=1,
+    stop=None,
+)
+
+print_highlight(f"Response: {response}")
+```
+
+### Parameters
+
+The completions API accepts OpenAI Completions API's parameters.  Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.
+
+Here is an example of a detailed completions request:
+
+```python Example
+response = client.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    prompt="Write a short story about a space explorer.",
+    temperature=0.7,  # Moderate temperature for creative writing
+    max_tokens=150,  # Longer response for a story
+    top_p=0.9,  # Balanced diversity in word choice
+    stop=["\n\n", "THE END"],  # Multiple stop sequences
+    presence_penalty=0.3,  # Encourage novel elements
+    frequency_penalty=0.3,  # Reduce repetitive phrases
+    n=1,  # Generate one completion
+    seed=123,  # For reproducible results
+)
+
+print_highlight(f"Response: {response}")
+```
+
+#### Returning Routed Experts (MoE Models)
+
+For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks.
+
+## Structured Outputs (JSON, Regex, EBNF)
+
+For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs) for more details.
+
+## Using LoRA Adapters
+
+SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax.
+
+**Server Setup:**
+```bash Command
+python -m sglang.launch_server \
+    --model-path qwen/qwen2.5-0.5b-instruct \
+    --enable-lora \
+    --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b
+```
+
+For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora).
+
+**API Call:**
+
+(Recommended) Use the `model:adapter` syntax to specify which adapter to use:
+```python Example
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct:adapter_a",  # ← base-model:adapter-name
+    messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
+    max_tokens=50,
+)
+```
+
+**Backward Compatible: Using `extra_body`**
+
+The old `extra_body` method is still supported for backward compatibility:
+```python Example
+# Backward compatible method
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
+    extra_body={"lora_path": "adapter_a"},  # ← old method
+    max_tokens=50,
+)
+```
+**Note:** When both `model:adapter` and `extra_body["lora_path"]` are specified, the `model:adapter` syntax takes precedence.
+
+```python Example
+terminate_process(server_process)
+```

docs_new/docs/basic_usage/openai_api_embeddings.ipynb

@@ -0,0 +1,193 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# OpenAI APIs - Embedding\n",
+    "\n",
+    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
+    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).\n",
+    "\n",
+    "This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported_models/retrieval_ranking/embedding_models.md)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server\n",
+    "\n",
+    "Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "embedding_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
+    "    --host 0.0.0.0 --is-embedding --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using cURL"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, json\n",
+    "\n",
+    "text = \"Once upon a time\"\n",
+    "\n",
+    "curl_text = f\"\"\"curl -s http://localhost:{port}/v1/embeddings \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -d '{{\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": \"{text}\"}}'\"\"\"\n",
+    "\n",
+    "result = subprocess.check_output(curl_text, shell=True)\n",
+    "\n",
+    "print(result)\n",
+    "\n",
+    "text_embedding = json.loads(result)[\"data\"][0][\"embedding\"]\n",
+    "\n",
+    "print_highlight(f\"Text embedding (first 10): {text_embedding[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Python Requests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "text = \"Once upon a time\"\n",
+    "\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/v1/embeddings\",\n",
+    "    json={\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": text},\n",
+    ")\n",
+    "\n",
+    "text_embedding = response.json()[\"data\"][0][\"embedding\"]\n",
+    "\n",
+    "print_highlight(f\"Text embedding (first 10): {text_embedding[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using OpenAI Python Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "# Text embedding example\n",
+    "response = client.embeddings.create(\n",
+    "    model=\"Alibaba-NLP/gte-Qwen2-1.5B-instruct\",\n",
+    "    input=text,\n",
+    ")\n",
+    "\n",
+    "embedding = response.data[0].embedding[:10]\n",
+    "print_highlight(f\"Text embedding (first 10): {embedding}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Input IDs\n",
+    "\n",
+    "SGLang also supports `input_ids` as input to get the embedding."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import os\n",
+    "from transformers import AutoTokenizer\n",
+    "\n",
+    "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(\"Alibaba-NLP/gte-Qwen2-1.5B-instruct\")\n",
+    "input_ids = tokenizer.encode(text)\n",
+    "\n",
+    "curl_ids = f\"\"\"curl -s http://localhost:{port}/v1/embeddings \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -d '{{\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"input\": {json.dumps(input_ids)}}}'\"\"\"\n",
+    "\n",
+    "input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))[\"data\"][\n",
+    "    0\n",
+    "][\"embedding\"]\n",
+    "\n",
+    "print_highlight(f\"Input IDs embedding (first 10): {input_ids_embedding[:10]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(embedding_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multi-Modal Embedding Model\n",
+    "Please refer to [Multi-Modal Embedding Model](../supported_models/retrieval_ranking/embedding_models.md)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/openai_api_embeddings.mdx

@@ -0,0 +1,126 @@
+---
+title: "OpenAI APIs - Embedding"
+metatags:
+    description: "This tutorial covers the embedding APIs for embedding models."
+---
+SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
+A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).
+
+This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported-models)
+
+
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize. Remember to add `--is-embedding` to the command.
+
+
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+embedding_process, port = launch_server_cmd(
+    """
+python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
+    --host 0.0.0.0 --is-embedding --log-level warning
+"""
+)
+
+wait_for_server(f"http://localhost:{port}")
+```
+
+## Using cURL
+
+
+
+```python Example
+import subprocess, json
+
+text = "Once upon a time"
+
+curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'"""
+
+result = subprocess.check_output(curl_text, shell=True)
+
+print(result)
+
+text_embedding = json.loads(result)["data"][0]["embedding"]
+
+print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
+```
+
+## Using Python Requests
+
+
+
+```python Example
+import requests
+
+text = "Once upon a time"
+
+response = requests.post(
+    f"http://localhost:{port}/v1/embeddings",
+    json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text},
+)
+
+text_embedding = response.json()["data"][0]["embedding"]
+
+print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
+```
+
+## Using OpenAI Python Client
+
+
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+# Text embedding example
+response = client.embeddings.create(
+    model="Alibaba-NLP/gte-Qwen2-1.5B-instruct",
+    input=text,
+)
+
+embedding = response.data[0].embedding[:10]
+print_highlight(f"Text embedding (first 10): {embedding}")
+```
+
+## Using Input IDs
+
+SGLang also supports `input_ids` as input to get the embedding.
+
+
+
+```python Example
+import json
+import os
+from transformers import AutoTokenizer
+
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+
+tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct")
+input_ids = tokenizer.encode(text)
+
+curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \
+  -H "Content-Type: application/json" \
+  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'"""
+
+input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][
+    0
+]["embedding"]
+
+print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
+```
+
+
+```python Example
+terminate_process(embedding_process)
+```
+
+## Multi-Modal Embedding Model
+Please refer to [Multi-Modal Embedding Model](../supported-models)

docs_new/docs/basic_usage/openai_api_vision.ipynb

@@ -0,0 +1,253 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# OpenAI APIs - Vision\n",
+    "\n",
+    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
+    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).\n",
+    "This tutorial covers the vision APIs for vision language models.\n",
+    "\n",
+    "SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported_models/text_generation/multimodal_language_models.md).\n",
+    "\n",
+    "As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server\n",
+    "\n",
+    "Launch the server in your terminal and wait for it to initialize."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "example_image_url = \"https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png\"\n",
+    "logo_image_url = (\n",
+    "    \"https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png\"\n",
+    ")\n",
+    "\n",
+    "vision_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=vision_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using cURL\n",
+    "\n",
+    "Once the server is up, you can send test requests using curl or requests."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess\n",
+    "\n",
+    "curl_command = f\"\"\"\n",
+    "curl -s http://localhost:{port}/v1/chat/completions \\\\\n",
+    "  -H \"Content-Type: application/json\" \\\\\n",
+    "  -d '{{\n",
+    "    \"model\": \"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    \"messages\": [\n",
+    "      {{\n",
+    "        \"role\": \"user\",\n",
+    "        \"content\": [\n",
+    "          {{\n",
+    "            \"type\": \"text\",\n",
+    "            \"text\": \"What’s in this image?\"\n",
+    "          }},\n",
+    "          {{\n",
+    "            \"type\": \"image_url\",\n",
+    "            \"image_url\": {{\n",
+    "              \"url\": \"{example_image_url}\"\n",
+    "            }}\n",
+    "          }}\n",
+    "        ]\n",
+    "      }}\n",
+    "    ],\n",
+    "    \"max_tokens\": 300\n",
+    "  }}'\n",
+    "\"\"\"\n",
+    "\n",
+    "response = subprocess.check_output(curl_command, shell=True).decode()\n",
+    "print_highlight(response)\n",
+    "\n",
+    "\n",
+    "response = subprocess.check_output(curl_command, shell=True).decode()\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Python Requests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/chat/completions\"\n",
+    "\n",
+    "data = {\n",
+    "    \"model\": \"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    \"messages\": [\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\"type\": \"text\", \"text\": \"What’s in this image?\"},\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\"url\": example_image_url},\n",
+    "                },\n",
+    "            ],\n",
+    "        }\n",
+    "    ],\n",
+    "    \"max_tokens\": 300,\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using OpenAI Python Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(base_url=f\"http://localhost:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\n",
+    "                    \"type\": \"text\",\n",
+    "                    \"text\": \"What is in this image?\",\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\"url\": example_image_url},\n",
+    "                },\n",
+    "            ],\n",
+    "        }\n",
+    "    ],\n",
+    "    max_tokens=300,\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Multiple-Image Inputs\n",
+    "\n",
+    "The server also supports multiple images and interleaved text and images if the model supports it."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openai import OpenAI\n",
+    "\n",
+    "client = OpenAI(base_url=f\"http://localhost:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"Qwen/Qwen2.5-VL-7B-Instruct\",\n",
+    "    messages=[\n",
+    "        {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": [\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\n",
+    "                        \"url\": example_image_url,\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"image_url\",\n",
+    "                    \"image_url\": {\n",
+    "                        \"url\": logo_image_url,\n",
+    "                    },\n",
+    "                },\n",
+    "                {\n",
+    "                    \"type\": \"text\",\n",
+    "                    \"text\": \"I have two very different images. They are not related at all. \"\n",
+    "                    \"Please describe the first image in one sentence, and then describe the second image in another sentence.\",\n",
+    "                },\n",
+    "            ],\n",
+    "        }\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.choices[0].message.content)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(vision_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/openai_api_vision.mdx

@@ -0,0 +1,176 @@
+---
+title: "OpenAI APIs - Vision"
+metatags:
+    description: "This tutorial covers the vision APIs for vision language models."
+---
+SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
+A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).
+This tutorial covers the vision APIs for vision language models.
+
+SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported-models/multimodal_language_models).
+
+As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py).
+
+## Launch A Server
+
+Launch the server in your terminal and wait for it to initialize.
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, print_highlight, terminate_process
+
+example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
+logo_image_url = (
+    "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"
+)
+
+vision_process, port = launch_server_cmd("""
+python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning
+""")
+
+wait_for_server(f"http://localhost:{port}", process=vision_process)
+```
+
+## Using cURL
+
+Once the server is up, you can send test requests using curl or requests.
+
+```python Example
+import subprocess
+
+curl_command = f"""
+curl -s http://localhost:{port}/v1/chat/completions \\
+  -H "Content-Type: application/json" \\
+  -d '{{
+    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
+    "messages": [
+      {{
+        "role": "user",
+        "content": [
+          {{
+            "type": "text",
+            "text": "What’s in this image?"
+          }},
+          {{
+            "type": "image_url",
+            "image_url": {{
+              "url": "{example_image_url}"
+            }}
+          }}
+        ]
+      }}
+    ],
+    "max_tokens": 300
+  }}'
+"""
+
+response = subprocess.check_output(curl_command, shell=True).decode()
+print_highlight(response)
+
+
+response = subprocess.check_output(curl_command, shell=True).decode()
+print_highlight(response)
+```
+
+## Using Python Requests
+
+```python Example
+import requests
+
+url = f"http://localhost:{port}/v1/chat/completions"
+
+data = {
+    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "What’s in this image?"},
+                {
+                    "type": "image_url",
+                    "image_url": {"url": example_image_url},
+                },
+            ],
+        }
+    ],
+    "max_tokens": 300,
+}
+
+response = requests.post(url, json=data)
+print_highlight(response.text)
+```
+
+## Using OpenAI Python Client
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen2.5-VL-7B-Instruct",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "text",
+                    "text": "What is in this image?",
+                },
+                {
+                    "type": "image_url",
+                    "image_url": {"url": example_image_url},
+                },
+            ],
+        }
+    ],
+    max_tokens=300,
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+## Multiple-Image Inputs
+
+The server also supports multiple images and interleaved text and images if the model supports it.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen2.5-VL-7B-Instruct",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": example_image_url,
+                    },
+                },
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": logo_image_url,
+                    },
+                },
+                {
+                    "type": "text",
+                    "text": "I have two very different images. They are not related at all. "
+                    "Please describe the first image in one sentence, and then describe the second image in another sentence.",
+                },
+            ],
+        }
+    ],
+    temperature=0,
+)
+
+print_highlight(response.choices[0].message.content)
+```
+
+```python Example
+terminate_process(vision_process)
+```

docs_new/docs/basic_usage/overview.mdx

@@ -0,0 +1,12 @@
+---
+title: Basic Usage
+description: Core APIs and common usage patterns for SGLang.
+---
+
+- [OpenAI-Compatible APIs](./openai_api_completions) — Chat completions, vision, and embeddings
+- [Anthropic-Compatible API](./anthropic_api) — `/v1/messages`, including Claude Code integration
+- [Ollama API](./ollama_api)
+- [Offline Engine API](./offline_engine_api)
+- [Native API](./native_api)
+- [Sampling Parameters](./sampling_params)
+- [Popular Model Usage](/cookbook/autoregressive/intro) — DeepSeek, GLM, Qwen, Llama, and more

docs_new/docs/basic_usage/sampling_params.mdx

@@ -0,0 +1,581 @@
+---
+title: "Sampling Parameters"
+metatags:
+    description: "Complete reference for SGLang sampling parameters: temperature, top_p, top_k, frequency penalty, stop tokens, and more."
+---
+This doc describes the sampling parameters of the SGLang Runtime. It is the low-level endpoint of the runtime.
+If you want a high-level endpoint that can automatically handle chat templates, consider using the [OpenAI Compatible API](./openai_api_completions).
+
+## `/generate` Endpoint
+
+The `/generate` endpoint accepts the following parameters in JSON format. For detailed usage, see the [native API doc](./native_api). The object is defined at `io_struct.py::GenerateReqInput`. You can also read the source code to find more arguments and docs.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>text</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[str], str]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The input prompt. Can be a single prompt or a batch of prompts.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>input_ids</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[int]], List[int]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The token IDs for text; one can specify either text or input_ids.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>input_embeds</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[List[float]]], List[List[float]]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The embeddings for input_ids; one can specify either text, input_ids, or input_embeds.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>image_data</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The image input. Supports three formats: (1) **Raw images**: PIL Image, file path, URL, or base64 string; (2) **Processor output**: Dict with `format: "processor_output"` containing HuggingFace processor outputs; (3) **Precomputed embeddings**: Dict with `format: "precomputed_embedding"` and `feature` containing pre-calculated visual embeddings. Can be a single image, list of images, or list of lists of images. See [Multimodal Input Formats](#multimodal-input-formats) for details.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>audio_data</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[AudioDataItem], AudioDataItem]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The audio input. Can be a file name, URL, or base64 encoded string.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>sampling_params</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Dict], Dict]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The sampling parameters as described in the sections below.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>rid</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[str], str]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The request ID.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_logprob</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[bool], bool]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return log probabilities for tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>logprob_start_len</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[int], int]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the start location in the prompt for returning logprobs. Default is "-1", which returns logprobs for output tokens only.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_logprobs_num</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[int], int]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the number of top logprobs to return at each position.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>token_ids_logprob</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[List[int]], List[int]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If return_logprob, the token IDs to return logprob for.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_text_in_logprobs</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to detokenize tokens in text in the returned logprobs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stream</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to stream output.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lora_path</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Optional[str]], Optional[str]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The path to the LoRA.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>custom_logit_processor</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[List[Optional[str]], str]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom logit processor for advanced sampling control. Must be a serialized instance of `CustomLogitProcessor` using its `to_str()` method. For usage see below.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_hidden_states</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Union[List[bool], bool] = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return hidden states.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>return_routed_experts</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether to return routed experts for MoE models. Requires `--enable-return-routed-experts` server flag. With the default `routed_experts_start_len=0`, returns the full available sequence `[0, seqlen - 1)` because RL workflows need routed experts for the full sequence. The result is base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>routed_experts_start_len</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>If `return_routed_experts`, the absolute start position for returned routed-experts rows. `0` preserves the default full sequence; set it to an accumulated prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks. Must be in `[0, prompt_tokens]`.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Sampling parameters
+
+The object is defined at `sampling_params.py::SamplingParams`. You can also read the source code to find more arguments and docs.
+
+### Note on defaults
+
+By default, SGLang initializes several sampling parameters from the model's `generation_config.json` (when the server is launched with `--sampling-defaults model`, which is the default). To use SGLang/OpenAI constant defaults instead, start the server with `--sampling-defaults openai`. You can always override any parameter per request via `sampling_params`.
+
+```bash Command
+# Use model-provided defaults from generation_config.json (default behavior)
+python -m sglang.launch_server --model-path <MODEL> --sampling-defaults model
+
+# Use SGLang/OpenAI constant defaults instead
+python -m sglang.launch_server --model-path <MODEL> --sampling-defaults openai
+```
+
+### Core parameters
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>max_new_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 128`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The maximum output length measured in tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[str, List[str]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>One or multiple [stop words](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stop). Generation will stop if one of these words is sampled.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop_token_ids</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[List[int]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Provide stop words in the form of token IDs. Generation will stop if one of these token IDs is sampled.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stop_regex</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[Union[str, List[str]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Stop when hitting any of the regex patterns in this list</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>temperature</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 1.0)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Temperature](https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature) when sampling the next token. `temperature = 0` corresponds to greedy sampling, a higher temperature leads to more diversity.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_p</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 1.0)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Top-p](https://platform.openai.com/docs/api-reference/chat/create#chat-create-top_p) selects tokens from the smallest sorted set whose cumulative probability exceeds `top_p`. When `top_p = 1`, this reduces to unrestricted sampling from all tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>top_k</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int (model default; fallback -1)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Top-k](https://developer.nvidia.com/blog/how-to-get-better-outputs-from-your-large-language-model/#predictability_vs_creativity) randomly selects from the `k` highest-probability tokens.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>min_p</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float (model default; fallback 0.0)`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Min-p](https://github.com/huggingface/transformers/issues/27670) samples from tokens with probability larger than `min_p * highest_token_probability`.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Penalizers
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>frequency_penalty</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 0.0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Penalizes tokens based on their frequency in generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of penalization grows linearly with each appearance of a token.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>presence_penalty</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 0.0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Penalizes tokens if they appeared in the generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of the penalization is constant if a token occurred.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>repetition_penalty</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`float = 1.0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values < 1) repetition. Valid range is `(0, 2]`; `1.0` leaves probabilities unchanged.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>min_new_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 0`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Forces the model to generate at least `min_new_tokens` until a stop word or EOS token is sampled. Note that this might lead to unintended behavior, for example, if the distribution is highly skewed towards these tokens.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Constrained decoding
+
+Please refer to our dedicated guide on [constrained decoding](../advanced_features/structured_outputs) for the following parameters.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>json_schema</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>JSON schema for structured outputs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>regex</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Regex for structured outputs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ebnf</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>EBNF for structured outputs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>structural_tag</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[str] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The structal tag for structured outputs.</td>
+    </tr>
+  </tbody>
+</table>
+
+### Other options
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Type/Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>n</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`int = 1`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Specifies the number of output sequences to generate per request. (Generating multiple outputs in one request (n > 1) is discouraged; repeating the same prompts several times offers better control and efficiency.)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ignore_eos</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Don't stop generation when EOS token is sampled.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>skip_special_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = True`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Remove special tokens during decoding.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>spaces_between_special_tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = True`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Whether or not to add spaces between special tokens during detokenization.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>no_stop_trim</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`bool = False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Don't trim stop words or EOS token from the generated text.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>custom_params</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`Optional[List[Optional[Dict[str, Any]]]] = None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Used when employing `CustomLogitProcessor`. For usage, see below.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Examples
+
+### Normal
+
+Launch a server:
+
+```bash Command
+python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000
+```
+
+Send a request:
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+    },
+)
+print(response.json())
+```
+
+Detailed example in [send request](./send_request).
+
+### Streaming
+
+Send a request and stream the output:
+
+```python Example
+import requests, json
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+        "stream": True,
+    },
+    stream=True,
+)
+
+prev = 0
+for chunk in response.iter_lines(decode_unicode=False):
+    chunk = chunk.decode("utf-8")
+    if chunk and chunk.startswith("data:"):
+        if chunk == "data: [DONE]":
+            break
+        data = json.loads(chunk[5:].strip("\n"))
+        output = data["text"].strip()
+        print(output[prev:], end="", flush=True)
+        prev = len(output)
+print("")
+```
+
+Detailed example in [openai compatible api](./openai_api_completions).
+
+### Multimodal
+
+Launch a server:
+
+```bash Command
+python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-7b-ov
+```
+
+Download an image:
+
+```bash Command
+curl -o example_image.png -L https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true
+```
+
+Send a request:
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
+                "<|im_start|>user\n<image>\nDescribe this image in a very short sentence.<|im_end|>\n"
+                "<|im_start|>assistant\n",
+        "image_data": "example_image.png",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+    },
+)
+print(response.json())
+```
+
+The `image_data` can be a file name, a URL, or a base64 encoded string. See also `python/sglang/srt/utils.py:load_image`.
+
+Streaming is supported in a similar manner as [above](#streaming).
+
+Detailed example in [OpenAI API Vision](./openai_api_vision).
+
+### Structured Outputs (JSON, Regex, EBNF)
+
+You can specify a JSON schema, regular expression or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.
+
+SGLang supports two grammar backends:
+
+- [XGrammar](https://github.com/mlc-ai/xgrammar) (default): Supports JSON schema, regular expression, and EBNF constraints.
+  - XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README).
+- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.
+
+If instead you want to initialize the Outlines backend, you can use `--grammar-backend outlines` flag:
+
+```bash Command
+python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+--port 30000 --host 0.0.0.0 --grammar-backend [xgrammar|outlines] # xgrammar or outlines (default: xgrammar)
+```
+
+```python Example
+import json
+import requests
+
+json_schema = json.dumps({
+    "type": "object",
+    "properties": {
+        "name": {"type": "string", "pattern": "^[\\w]+$"},
+        "population": {"type": "integer"},
+    },
+    "required": ["name", "population"],
+})
+
+# JSON (works with both Outlines and XGrammar)
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "Here is the information of the capital of France in the JSON format.\n",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "json_schema": json_schema,
+        },
+    },
+)
+print(response.json())
+
+# Regular expression (Outlines backend only)
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "Paris is the capital of",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "regex": "(France|England)",
+        },
+    },
+)
+print(response.json())
+
+# EBNF (XGrammar backend only)
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "Write a greeting.",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 64,
+            "ebnf": 'root ::= "Hello" | "Hi" | "Hey"',
+        },
+    },
+)
+print(response.json())
+```
+
+Detailed example in [structured outputs](../advanced_features/structured_outputs).
+
+### Custom logit processor
+
+Launch a server with `--enable-custom-logit-processor` flag on.
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path meta-llama/Meta-Llama-3-8B-Instruct \
+  --port 30000 \
+  --enable-custom-logit-processor
+```
+
+Define a custom logit processor that will always sample a specific token id.
+
+```python Example
+from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor
+
+class DeterministicLogitProcessor(CustomLogitProcessor):
+    """A dummy logit processor that changes the logits to always
+    sample the given token id.
+    """
+
+    def __call__(self, logits, custom_param_list):
+        # Check that the number of logits matches the number of custom parameters
+        assert logits.shape[0] == len(custom_param_list)
+        key = "token_id"
+
+        for i, param_dict in enumerate(custom_param_list):
+            # Mask all other tokens
+            logits[i, :] = -float("inf")
+            # Assign highest probability to the specified token
+            logits[i, param_dict[key]] = 0.0
+        return logits
+```
+
+Send a request:
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
+        "sampling_params": {
+            "temperature": 0.0,
+            "max_new_tokens": 32,
+            "custom_params": {"token_id": 5},
+        },
+    },
+)
+print(response.json())
+```
+
+Send an OpenAI chat completion request:
+
+```python Example
+import openai
+from sglang.utils import print_highlight
+
+client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="meta-llama/Meta-Llama-3-8B-Instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0.0,
+    max_tokens=32,
+    extra_body={
+        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
+        "custom_params": {"token_id": 5},
+    },
+)
+
+print_highlight(f"Response: {response}")
+```

docs_new/docs/basic_usage/send_request.ipynb

@@ -0,0 +1,251 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Sending Requests\n",
+    "This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one.\n",
+    "\n",
+    "- For Vision Language Models, see [OpenAI APIs - Vision](openai_api_vision.ipynb).\n",
+    "- For Embedding Models, see [OpenAI APIs - Embedding](openai_api_embeddings.ipynb) and [Encode (embedding model)](native_api.html#Encode-(embedding-model)).\n",
+    "- For Reward Models, see [Classify (reward model)](native_api.html#Classify-(reward-model))."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Launch A Server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sglang.test.doc_patch import launch_server_cmd\n",
+    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
+    "\n",
+    "# This is equivalent to running the following command in your terminal\n",
+    "# python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0\n",
+    "\n",
+    "server_process, port = launch_server_cmd(\"\"\"\n",
+    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \\\n",
+    " --host 0.0.0.0 --log-level warning\n",
+    "\"\"\")\n",
+    "\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using cURL\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, json\n",
+    "\n",
+    "curl_command = f\"\"\"\n",
+    "curl -s http://localhost:{port}/v1/chat/completions \\\n",
+    "  -H \"Content-Type: application/json\" \\\n",
+    "  -d '{{\"model\": \"qwen/qwen2.5-0.5b-instruct\", \"messages\": [{{\"role\": \"user\", \"content\": \"What is the capital of France?\"}}]}}'\n",
+    "\"\"\"\n",
+    "\n",
+    "response = json.loads(subprocess.check_output(curl_command, shell=True))\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Python Requests"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "url = f\"http://localhost:{port}/v1/chat/completions\"\n",
+    "\n",
+    "data = {\n",
+    "    \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    \"messages\": [{\"role\": \"user\", \"content\": \"What is the capital of France?\"}],\n",
+    "}\n",
+    "\n",
+    "response = requests.post(url, json=data)\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using OpenAI Python Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    ")\n",
+    "print_highlight(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openai\n",
+    "\n",
+    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
+    "\n",
+    "# Use stream=True for streaming responses\n",
+    "response = client.chat.completions.create(\n",
+    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
+    "    messages=[\n",
+    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
+    "    ],\n",
+    "    temperature=0,\n",
+    "    max_tokens=64,\n",
+    "    stream=True,\n",
+    ")\n",
+    "\n",
+    "# Handle the streaming output\n",
+    "for chunk in response:\n",
+    "    if chunk.choices[0].delta.content:\n",
+    "        print(chunk.choices[0].delta.content, end=\"\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Using Native Generation APIs\n",
+    "\n",
+    "You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](sampling_params.md)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
+    "\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": \"The capital of France is\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 32,\n",
+    "        },\n",
+    "    },\n",
+    ")\n",
+    "\n",
+    "print_highlight(response.json())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Streaming"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests, json\n",
+    "\n",
+    "response = requests.post(\n",
+    "    f\"http://localhost:{port}/generate\",\n",
+    "    json={\n",
+    "        \"text\": \"The capital of France is\",\n",
+    "        \"sampling_params\": {\n",
+    "            \"temperature\": 0,\n",
+    "            \"max_new_tokens\": 32,\n",
+    "        },\n",
+    "        \"stream\": True,\n",
+    "    },\n",
+    "    stream=True,\n",
+    ")\n",
+    "\n",
+    "prev = 0\n",
+    "for chunk in response.iter_lines(decode_unicode=False):\n",
+    "    chunk = chunk.decode(\"utf-8\")\n",
+    "    if chunk and chunk.startswith(\"data:\"):\n",
+    "        if chunk == \"data: [DONE]\":\n",
+    "            break\n",
+    "        data = json.loads(chunk[5:].strip(\"\\n\"))\n",
+    "        output = data[\"text\"]\n",
+    "        print(output[prev:], end=\"\", flush=True)\n",
+    "        prev = len(output)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs_new/docs/basic_usage/send_request.mdx

@@ -0,0 +1,172 @@
+---
+title: "Tutorial: Sending a request"
+metatags:
+    description: "This notebook provides a quick-start guide to use SGLang in chat completions after installation. "
+---
+This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one.
+
+- For Vision Language Models, see [OpenAI APIs - Vision](./openai_api_vision).
+- For Embedding Models, see [OpenAI APIs - Embedding](./openai_api_embeddings) and [Encode (embedding model)](./native_api#encode-embedding-model).
+- For Reward Models, see [Classify (reward model)](./native_api#classify-reward-model).
+
+
+## Launch A Server
+
+
+
+```python Example
+from sglang.test.doc_patch import launch_server_cmd
+from sglang.utils import wait_for_server, terminate_process
+
+# This is equivalent to running the following command in your terminal
+# python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0
+
+server_process, port = launch_server_cmd(
+    """
+python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
+ --host 0.0.0.0 --log-level warning
+"""
+)
+
+wait_for_server(f"http://localhost:{port}")
+```
+
+## Using cURL
+
+
+
+
+```python Example
+import subprocess, json
+
+curl_command = f"""
+curl -s http://localhost:{port}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{{"role": "user", "content": "What is the capital of France?"}}]}}'
+"""
+
+response = json.loads(subprocess.check_output(curl_command, shell=True))
+print(response)
+```
+
+## Using Python Requests
+
+
+
+```python Example
+import requests
+
+url = f"http://localhost:{port}/v1/chat/completions"
+
+data = {
+    "model": "qwen/qwen2.5-0.5b-instruct",
+    "messages": [{"role": "user", "content": "What is the capital of France?"}],
+}
+
+response = requests.post(url, json=data)
+print(response.json())
+```
+
+## Using OpenAI Python Client
+
+
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+)
+print(response)
+```
+
+### Streaming
+
+
+
+```python Example
+import openai
+
+client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
+
+# Use stream=True for streaming responses
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+    stream=True,
+)
+
+# Handle the streaming output
+for chunk in response:
+    if chunk.choices[0].delta.content:
+        print(chunk.choices[0].delta.content, end="", flush=True)
+```
+
+## Using Native Generation APIs
+
+You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](./sampling_params).
+
+
+
+```python Example
+import requests
+
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+    },
+)
+
+print(response.json())
+```
+### Streaming
+
+
+
+```python Example
+import requests, json
+
+response = requests.post(
+    f"http://localhost:{port}/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+        "stream": True,
+    },
+    stream=True,
+)
+
+prev = 0
+for chunk in response.iter_lines(decode_unicode=False):
+    chunk = chunk.decode("utf-8")
+    if chunk and chunk.startswith("data:"):
+        if chunk == "data: [DONE]":
+            break
+        data = json.loads(chunk[5:].strip("\n"))
+        output = data["text"]
+        print(output[prev:], end="", flush=True)
+        prev = len(output)
+```
+
+```python Example
+terminate_process(server_process)
+```

docs_new/docs/developer_guide/bench_serving.mdx

@@ -0,0 +1,447 @@
+---
+title: "Bench Serving Guide"
+metatags:
+    description: "SGLang bench_serving: benchmark throughput, TTFT, ITL with random/sharegpt/image datasets. Multi-backend support."
+---
+This guide explains how to benchmark online serving throughput and latency using `python -m sglang.bench_serving`. It supports multiple inference backends via OpenAI-compatible and native endpoints, and produces both console metrics and optional JSONL outputs.
+
+### What it does
+
+- Generates synthetic or dataset-driven prompts and submits them to a target serving endpoint
+- Measures throughput, time-to-first-token (TTFT), inter-token latency (ITL), per-request end-to-end latency, and more
+- Supports streaming or non-streaming modes, rate control, and concurrency limits
+
+### Supported backends and endpoints
+
+- `sglang` / `sglang-native`: `POST /generate`
+- `sglang-oai`, `vllm`, `lmdeploy`: `POST /v1/completions`
+- `sglang-oai-chat`, `vllm-chat`, `lmdeploy-chat`: `POST /v1/chat/completions`
+- `trt` (TensorRT-LLM): `POST /v2/models/ensemble/generate_stream`
+- `gserver`: Custom server (Not Implemented yet in this script)
+- `truss`: `POST /v1/models/model:predict`
+
+If `--base-url` is provided, requests are sent to it. Otherwise, `--host` and `--port` are used. When `--model` is not provided, the script will attempt to query `GET /v1/models` for an available model ID (OpenAI-compatible endpoints).
+
+### Prerequisites
+
+- Python 3.10+
+- Dependencies typically used by this script: `aiohttp`, `numpy`, `requests`, `tqdm`, `transformers`, and for some datasets `datasets`, `pillow`, `pybase64`. Install as needed.
+- An inference server running and reachable via the endpoints above
+- If your server requires authentication, set environment variable `OPENAI_API_KEY` (used as `Authorization: Bearer <key>`)
+
+### Quick start
+
+Run a basic benchmark against an sglang server exposing `/generate`:
+
+```bash Command
+python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
+```
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --num-prompts 1000 \
+  --model meta-llama/Llama-3.1-8B-Instruct
+```
+
+Or, using an OpenAI-compatible endpoint (completions):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend vllm \
+  --base-url http://127.0.0.1:8000 \
+  --num-prompts 1000 \
+  --model meta-llama/Llama-3.1-8B-Instruct
+```
+
+### Datasets
+
+Select with `--dataset-name`:
+
+- `sharegpt` (default): loads ShareGPT-style pairs; optionally restrict with `--sharegpt-context-len` and override outputs with `--sharegpt-output-len`
+- `random`: random text lengths; sampled from ShareGPT token space
+- `random-ids`: random token ids (can lead to gibberish)
+- `image`: generates images and wraps them in chat messages; supports custom resolutions, multiple formats, and different content types
+- `generated-shared-prefix`: synthetic dataset with shared long system prompts and short questions
+- `mmmu`: samples from MMMU (Math split) and includes images
+- `speed-bench`: [SPEED-Bench](https://huggingface.co/datasets/nvidia/SPEED-Bench) (**SPEculative Evaluation Dataset**) — a unified benchmark for evaluating [Speculative Decoding (SD)](https://arxiv.org/abs/2604.09557) algorithms. Uses the Throughput split, which provides fixed-length input sequences (1K–32K tokens) grouped into three output-entropy categories (`low_entropy`, `mixed`, `high_entropy`). Requires a pre-downloaded JSONL file passed via `--dataset-path`.
+- `agentic-trace`: replays pre-built multi-turn agentic traces (e.g. OpenHands / SWE-smith). Each conversation is replayed round by round, feeding the server's real assistant reply back into the next round's history. Requires a chat backend (`--backend sglang-oai-chat`) and a trace JSON passed via `--dataset-path`.
+
+Common dataset flags:
+
+- `--num-prompts N`: number of requests
+- `--random-input-len`, `--random-output-len`, `--random-range-ratio`: for random/random-ids/image
+- `--image-count`: Number of images per request (for `image` dataset).
+
+- `--apply-chat-template`: apply tokenizer chat template when constructing prompts
+- `--dataset-path PATH`: file path for ShareGPT json; if blank and missing, it will be downloaded and cached
+
+Generated Shared Prefix flags (for `generated-shared-prefix`):
+
+- `--gsp-num-groups`
+- `--gsp-prompts-per-group`
+- `--gsp-system-prompt-len`
+- `--gsp-question-len`
+- `--gsp-output-len`
+- `--gsp-group-distribution {uniform,zipf}`: per-request prefix-group sampling distribution (default: `uniform`). With `zipf`, each request's group is sampled by rank with `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)`; rank starts at 1 and group index 0 is the hottest. The on-disk dataset cache uses a distinct key per `(group_distribution, zipf_alpha)`, so uniform-mode caches are never mixed with zipf-mode caches.
+- `--gsp-zipf-alpha FLOAT`: Zipf exponent for `--gsp-group-distribution=zipf`. Must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups. Required when the distribution is `zipf`; must be omitted otherwise.
+
+Image dataset flags (for `image`):
+
+- `--image-count`: Number of images per request
+- `--image-resolution`: Image resolution; supports presets (4k, 1080p, 720p, 360p) or custom 'heightxwidth' format (e.g., 1080x1920, 512x768)
+- `--image-format`: Image format (jpeg or png)
+- `--image-content`: Image content type (random or blank)
+
+Agentic trace flags (for `agentic-trace`):
+
+- `--dataset-path`: path to the pre-built trace JSON
+- `--sharegpt-output-len`: per-turn output length (default: 220)
+- `--dataset-offset`: rotate the conversation list by this many entries before sampling, so successive sweep steps start on fresh conversations
+- `--agentic-max-turns`: cap each conversation to at most this many turns (useful for small, fast profiling runs)
+
+SPEED-Bench flags (for `speed-bench`):
+
+- `--dataset-path PATH`: path to the pre-downloaded SPEED-Bench Throughput JSONL (e.g., `throughput_1k.jsonl`). Use the [SPEED-Bench measurement framework](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/specdec_bench) to generate it.
+- `--speed-bench-category`: filter to one entropy category: `low_entropy`, `mixed`, or `high_entropy` (default: all)
+- `--speed-bench-output-len`: fixed number of output tokens per request (default: 512)
+
+### Examples
+
+1. To benchmark image dataset with 3 images per request, 500 prompts, 512 input length, and 512 output length, you can run:
+
+```bash Command
+python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-3B-Instruct --disable-radix-cache
+```
+
+```bash Command
+python -m sglang.bench_serving \
+    --backend sglang-oai-chat \
+    --dataset-name image \
+    --num-prompts 500 \
+    --image-count 3 \
+    --image-resolution 720p \
+    --random-input-len 512 \
+    --random-output-len 512
+```
+
+2. To benchmark random dataset with 3000 prompts, 1024 input length, and 1024 output length, you can run:
+
+```bash Command
+python -m sglang.launch_server --model-path Qwen/Qwen2.5-3B-Instruct
+```
+
+```bash Command
+python3 -m sglang.bench_serving \
+    --backend sglang \
+    --dataset-name random \
+    --num-prompts 3000 \
+    --random-input 1024 \
+    --random-output 1024 \
+    --random-range-ratio 0.5
+```
+
+3. To benchmark speculative decoding throughput using SPEED-Bench (mixed-entropy category, 1K ISL), you can run:
+
+```bash Command
+python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --speculative-algorithm EAGLE --speculative-draft-model-path <draft-model-path>
+```
+
+```bash Command
+python3 -m sglang.bench_serving \
+    --backend sglang \
+    --dataset-name speed-bench \
+    --dataset-path /path/to/throughput_1k.jsonl \
+    --speed-bench-category mixed \
+    --speed-bench-output-len 512 \
+    --num-prompts 512
+```
+
+### Choosing model and tokenizer
+
+- `--model` is required unless the backend exposes `GET /v1/models`, in which case the first model ID is auto-selected.
+- `--tokenizer` defaults to `--model`. Both can be HF model IDs or local paths.
+- For ModelScope workflows, setting `SGLANG_USE_MODELSCOPE=true` enables fetching via ModelScope (weights are skipped for speed).
+- If your tokenizer lacks a chat template, the script warns because token counting can be less robust for gibberish outputs.
+
+### Rate, concurrency, and streaming
+
+- `--request-rate`: requests per second. `inf` sends all immediately (burst). Non-infinite rate uses a Poisson process for arrival times.
+- `--max-concurrency`: caps concurrent in-flight requests regardless of arrival rate.
+- `--disable-stream`: switch to non-streaming mode when supported; TTFT then equals total latency for chat completions.
+
+### Other key options
+
+- `--output-file FILE.jsonl`: append JSONL results to file; auto-named if unspecified
+- `--output-details`: include per-request arrays (generated texts, errors, ttfts, itls, input/output lens)
+- `--extra-request-body '&#123;"top_p":0.9,"temperature":0.6&#125;'`: merged into payload (sampling params, etc.)
+- `--disable-ignore-eos`: pass through EOS behavior (varies by backend)
+- `--warmup-requests N`: run warmup requests with short output first (default 1)
+- `--flush-cache`: call `/flush_cache` (sglang) before main run
+- `--profile`: call `/start_profile` and `/stop_profile` (requires server to enable profiling, e.g., `SGLANG_TORCH_PROFILER_DIR`)
+- `--lora-name name1 name2 ...`: randomly pick one per request and pass to backend (e.g., `lora_path` for sglang)
+- `--tokenize-prompt`: send integer IDs instead of text (currently supports `--backend sglang` only)
+
+### Authentication
+
+If your target endpoint requires OpenAI-style auth, set:
+
+```bash Command
+export OPENAI_API_KEY=sk-...yourkey...
+```
+
+The script will add `Authorization: Bearer $OPENAI_API_KEY` automatically for OpenAI-compatible routes.
+
+### Metrics explained
+
+Printed after each run:
+
+- Request throughput (req/s)
+- Input token throughput (tok/s) - includes both text and vision tokens
+- Output token throughput (tok/s)
+- Total token throughput (tok/s) - includes both text and vision tokens
+- Total input text tokens and Total input vision tokens - per-modality breakdown
+- Concurrency: aggregate time of all requests divided by wall time
+- End-to-End Latency (ms): mean/median/std/p99 per-request total latency
+- Time to First Token (TTFT, ms): mean/median/std/p99 for streaming mode
+- Inter-Token Latency (ITL, ms): mean/median/std/p95/p99/max between tokens
+- TPOT (ms): Token processing time after first token, i.e., `(latency - ttft)/(tokens-1)`
+- Accept length (sglang-only, if available): speculative decoding accept length
+
+The script also retokenizes generated text with the configured tokenizer and reports "retokenized" counts.
+
+### JSONL output format
+
+When `--output-file` is set, one JSON object is appended per run. Base fields:
+
+- Arguments summary: backend, dataset, request_rate, max_concurrency, etc.
+- Duration and totals: completed, total_input_tokens, total_output_tokens, retokenized totals
+- Throughputs and latency statistics as printed in the console
+- `accept_length` when available (sglang)
+
+With `--output-details`, an extended object also includes arrays:
+
+- `input_lens`, `output_lens`
+- `ttfts`, `itls` (per request: ITL arrays)
+- `generated_texts`, `errors`
+
+### End-to-end examples
+
+1) sglang native `/generate` (streaming):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --dataset-name random \
+  --random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.5 \
+  --num-prompts 2000 \
+  --request-rate 100 \
+  --max-concurrency 512 \
+  --output-file sglang_random.jsonl --output-details
+```
+
+2) OpenAI-compatible Completions (e.g., vLLM):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend vllm \
+  --base-url http://127.0.0.1:8000 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --dataset-name sharegpt \
+  --num-prompts 1000 \
+  --sharegpt-output-len 256
+```
+
+3) OpenAI-compatible Chat Completions (streaming):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend vllm-chat \
+  --base-url http://127.0.0.1:8000 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --dataset-name random \
+  --num-prompts 500 \
+  --apply-chat-template
+```
+
+4) Images (VLM) with chat template:
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model your-vlm-model \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 --random-output-len 256 \
+  --num-prompts 200 \
+  --apply-chat-template
+```
+
+4a) Images with custom resolution:
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model your-vlm-model \
+  --dataset-name image \
+  --image-count 1 \
+  --image-resolution 512x768 \
+  --random-input-len 64 --random-output-len 128 \
+  --num-prompts 100 \
+  --apply-chat-template
+```
+
+4b) 1080p images with PNG format and blank content:
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model your-vlm-model \
+  --dataset-name image \
+  --image-count 1 \
+  --image-resolution 1080p \
+  --image-format png \
+  --image-content blank \
+  --random-input-len 64 --random-output-len 128 \
+  --num-prompts 100 \
+  --apply-chat-template
+```
+
+5) Generated shared prefix (long system prompts + short questions):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --dataset-name generated-shared-prefix \
+  --gsp-num-groups 64 --gsp-prompts-per-group 16 \
+  --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \
+  --num-prompts 1024
+```
+
+Zipfian / power-law prefix popularity (opt-in via `--gsp-group-distribution=zipf`):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --dataset-name generated-shared-prefix \
+  --gsp-num-groups 64 --gsp-prompts-per-group 16 \
+  --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \
+  --gsp-group-distribution zipf --gsp-zipf-alpha 1.2 \
+  --seed 42
+```
+
+`zipf` mode samples each request's prefix group from the rank-based distribution `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)` with rank starting at 1, so group index 0 is the hottest. The total request count stays `num_groups * prompts_per_group` — identical to `uniform` mode — and only the per-request group assignment changes. `alpha` must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups.
+
+The on-disk dataset cache at `~/.cache/sglang/benchmark/gen_shared_prefix_*.pkl` includes `group_distribution` and `zipf_alpha` in its key, so uniform-mode and zipf-mode runs (or two zipf runs with different alpha) never share a cache file. Uniform-mode filenames are unchanged from the legacy format, so existing caches remain valid.
+
+This flag controls prefix-popularity shape only. It does not by itself reproduce any production trace or guarantee an observed cache-hit rate for a given engine.
+
+6) Tokenized prompts (ids) for strict length control (sglang only):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --dataset-name random \
+  --tokenize-prompt \
+  --random-input-len 2048 --random-output-len 256 --random-range-ratio 0.2
+```
+
+7) Profiling and cache flush (sglang):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --profile \
+  --flush-cache
+```
+
+8) TensorRT-LLM streaming endpoint:
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend trt \
+  --base-url http://127.0.0.1:8000 \
+  --model your-trt-llm-model \
+  --dataset-name random \
+  --num-prompts 100 \
+  --disable-ignore-eos
+```
+
+9) Evaluating large-scale KVCache sharing with mooncake trace (sglang only):
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30000 \
+  --model model-name \
+  --dataset-name mooncake \
+  --mooncake-slowdown-factor 1.0 \
+  --mooncake-num-rounds 1000 \
+  --mooncake-workload conversation|mooncake|agent|synthetic
+  --use-trace-timestamps true \
+  --random-output-len 256
+```
+
+10) Fake decode stress testing (PD disaggregation, decode-only):
+
+When benchmarking pure decode performance in a PD disaggregation setup, you can bypass the prefill node entirely by using `--fake-prefill`. This requires the decode server to be started with `--disaggregation-transfer-backend fake`:
+
+```bash Command
+# Step 1: Start a decode-only server with fake transfer backend
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --disaggregation-mode decode \
+  --disaggregation-transfer-backend fake \
+  --port 30001
+
+# Step 2: Run bench_serving with --fake-prefill
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 --port 30001 \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --dataset-name random \
+  --num-prompts 500 \
+  --random-input-len 1024 --random-output-len 256 \
+  --fake-prefill
+```
+
+Similarly, `bench_one_batch_server` also supports `--fake-prefill`:
+
+```bash Command
+python3 -m sglang.bench_one_batch_server \
+  --base-url http://127.0.0.1:30001 \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --batch-size 32 --input-len 1024 --output-len 256 \
+  --fake-prefill
+```
+
+The `--fake-prefill` flag automatically injects special sentinel values into each request, telling the decode server to skip real KV transfer and generate fake KV data locally.
+
+### Troubleshooting
+
+- All requests failed: verify `--backend`, server URL/port, `--model`, and authentication. Check warmup errors printed by the script.
+- Throughput seems too low: adjust `--request-rate` and `--max-concurrency`; verify server batch size/scheduling; ensure streaming is enabled if appropriate.
+- Token counts look odd: prefer chat/instruct models with proper chat templates; otherwise tokenization of gibberish may be inconsistent.
+- Image/MMMU datasets: ensure you installed extra deps (`pillow`, `datasets`, `pybase64`).
+- Authentication errors (401/403): set `OPENAI_API_KEY` or disable auth on your server.
+
+### Notes
+
+- The script raises the file descriptor soft limit (`RLIMIT_NOFILE`) to help with many concurrent connections.
+- For sglang, `/server_info` is queried post-run to report speculative decoding accept length when available.

docs_new/docs/developer_guide/benchmark_and_profiling.mdx

@@ -0,0 +1,506 @@
+---
+title: "Benchmark and Profiling"
+metatags:
+    description: "SGLang benchmarking and profiling: PyTorch Profiler, Nsight Systems, layerwise NVTX, PD disaggregation profiling."
+---
+## Benchmark
+
+SGLang provides four benchmark tools that operate at different levels of the stack. The table below summarizes their key differences:
+
+<table>
+  <thead>
+    <tr>
+      <th>Tool</th>
+      <th>HTTP Server</th>
+      <th>Scheduler</th>
+      <th>Use Case</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>bench_serving</code></td>
+      <td>Yes (async HTTP client to a running server)</td>
+      <td>Yes (indirectly, via server)</td>
+      <td>Realistic online serving benchmarks with latency metrics (TTFT, TPOT, ITL)</td>
+    </tr>
+    <tr>
+      <td><code>bench_one_batch_server</code></td>
+      <td>Yes (sends HTTP requests to a running server)</td>
+      <td>Yes (indirectly, via server)</td>
+      <td>End-to-end single-batch latency including HTTP and scheduler overhead</td>
+    </tr>
+    <tr>
+      <td><code>bench_offline_throughput</code></td>
+      <td>No</td>
+      <td>Yes (directly uses <code>Engine</code> in-process)</td>
+      <td>Maximum throughput measurement without HTTP overhead</td>
+    </tr>
+    <tr>
+      <td><code>bench_one_batch</code></td>
+      <td>No</td>
+      <td>No (directly calls <code>ModelRunner</code>)</td>
+      <td>Kernel-level latency profiling of a single static batch</td>
+    </tr>
+  </tbody>
+</table>
+
+Use `bench_serving` by default unless there are specific needs.
+
+**`bench_serving`** is an async HTTP load-testing client that sends requests at controlled rates with configurable concurrency to a running server. It measures realistic online serving metrics including time-to-first-token (TTFT), time-per-output-token (TPOT), inter-token latency (ITL), and throughput. Use `num-prompts >= 5 * max-concurrency` to measure steady-state performance. Launch a server with `sglang.launch_server` first.
+
+  ```bash Command
+  python3 -m sglang.bench_serving --backend sglang --max-concurrency 16 --num-prompts 80 --random-input-len 256 --random-output-len 32 --dataset-name random
+  ```
+
+**`bench_one_batch_server`** sends a single batch as one HTTP request to a running server. Due to only having a single batch, the server is never in a steady-state and metrics will be biased. Launch a server with `sglang.launch_server` first.
+
+  ```bash Command
+  python3 -m sglang.bench_one_batch_server --base-url http://127.0.0.1:30000 --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32
+  ```
+
+  - Pass `--enable-multi-batch` and set `--batch-size` to a multiple of the server's `--max-running-requests` to stabilize throughput measurements. Surplus requests are queued by the scheduler and promoted batch-by-batch, amortizing per-request prefill and first-step transients into steady-state decode. Under this flag, only `overall_throughput` is authoritative; `input_throughput`, `output_throughput`, `last_ttft`, and ITL include cross-batch queueing in their denominators and should be treated as informational.
+  - Pass `--lora-name <name>` to route every prompt through a pre-loaded LoRA adapter. Requires the server to be launched with `--enable-lora --lora-paths <name>=<path>`.
+
+**`bench_offline_throughput`** directly instantiates the `Engine` object in-process (no HTTP server) and submits all requests at once via `engine.generate()`. The engine's scheduler handles batching and execution. This measures maximum achievable throughput without any network overhead.
+
+  ```bash Command
+  python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10
+  ```
+
+**`bench_one_batch`** is the lowest-level tool. It directly instantiates a `ModelRunner` and calls `extend()` / `decode()` on a fixed static batch, bypassing the scheduler entirely. The prefill and decode phases are run separately, making profiling easier but rendering the metrics unrealistic. Because there is no dynamic batching, it may run out of memory for batch sizes that a real server can handle (a real server chunks prefill into smaller batches). This is best suited for profiling individual kernel performance.
+
+  ```bash Command
+  python3 -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32
+  ```
+
+## Profile with PyTorch Profiler
+
+[Pytorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) is a convenient basic tool to inspect kernel execution time, call stack, and kernel overlap and occupancy.
+
+### Profile a server with `sglang.bench_serving`
+
+```bash Command
+# set trace path
+export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log
+
+# start server
+python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
+
+# send profiling request from client
+python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile
+```
+
+For `bench_serving --profile`, the output directory is selected on the client side from `--profile-output-dir` or `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`), then sent in the `/start_profile` request.
+If you call `/start_profile` directly and do not provide `output_dir`, the server uses its own `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`).
+
+Setting `SGLANG_TORCH_PROFILER_DIR` on both server and client is still recommended to avoid confusion about where traces are written.
+
+For more details, please refer to [Bench Serving Guide](./bench_serving).
+
+### Profile In PD Disaggregation Mode
+
+When profiling in PD disaggregation mode, prefill and decode workers **must be profiled separately** due to torch profiler limitations. The `bench_serving` command provides dedicated options for this:
+
+#### Profile Prefill Workers
+
+```bash Command
+# set trace path
+export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log
+
+# start prefill and decode servers (see PD disaggregation docs for setup)
+python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode prefill
+python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode decode --port 30001 --base-gpu-id 1
+
+# start router
+python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
+
+# send profiling request targeting prefill workers
+python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000
+```
+
+#### Profile Decode Workers
+
+```bash Command
+# send profiling request targeting decode workers
+python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001
+```
+
+#### Important Notes
+
+- `--profile-prefill-url` and `--profile-decode-url` are **mutually exclusive** - you cannot profile both at the same time
+- Both options support multiple worker URLs for multi-instance setups:
+  ```bash Command
+  # Profile multiple prefill workers
+  python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000 http://127.0.0.1:30002
+
+  # Profile multiple decode workers
+  python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001 http://127.0.0.1:30003
+  ```
+- Make sure `SGLANG_TORCH_PROFILER_DIR` is set on all worker nodes before starting the servers
+- For more details on setting up PD disaggregation, see [PD Disaggregation Guide](../advanced_features/pd_disaggregation)
+
+### Profile a server with `sglang.bench_offline_throughput`
+```bash Command
+export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log
+
+# profile one batch with bench_one_batch.py
+# batch size can be controlled with --batch argument
+python3 -m sglang.bench_one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile
+
+# profile multiple batches with bench_offline_throughput.py
+python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8
+```
+
+### Profile a server with `sglang.profiler`
+
+When the server is running (e.g., processing a decoding request), you can start live profiling immediately by sending a profile request to the server.
+
+You can do this by running `python3 -m sglang.profiler`. For example:
+
+```text Output
+# Terminal 1: Send a generation request
+python3 -m sglang.test.send_one
+
+# Terminal 2: Before the above request finishes, quickly launch the following command in a separate terminal.
+# It will generate a profile of the above request for several decoding batches.
+python3 -m sglang.profiler
+```
+
+You can also combine the above operations into a single command
+
+```text Output
+python3 -m sglang.test.send_one --profile
+```
+
+### Profile a server with HTTP API endpoints
+
+SGLang provides HTTP API endpoints to control profiling on a running server. This allows you to start and stop profiling programmatically, which is useful for capturing specific workload patterns.
+
+#### Using `/start_profile` endpoint
+
+The `/start_profile` endpoint starts profiling on the server. You can control when profiling begins and how long it runs using the following parameters:
+
+**Basic usage:**
+
+```bash Command
+# Start profiling immediately for 10 steps
+curl -X POST http://127.0.0.1:30000/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{
+    "num_steps": 10
+  }'
+```
+
+**Parameters:**
+
+- `output_dir` (optional): Directory where profile traces will be saved. If not specified, uses `SGLANG_TORCH_PROFILER_DIR` environment variable, or `/tmp` as the default
+- `num_steps` (optional): Number of steps to profile. If not specified, profiling continues until manually stopped with `/stop_profile`
+- `start_step` (optional): Step number at which to start profiling (inclusive). Useful for skipping warmup iterations
+- `activities` (optional): List of activities to profile, e.g., `["CPU", "GPU"]`. Default is `["CPU", "GPU"]`
+- `merge_profiles` (optional): Whether to merge distributed traces. Default is `false`
+
+**Note on step ranges:** Profiling starts at `start_step` (inclusive) and continues for `num_steps` iterations. For example, with `start_step=3` and `num_steps=10`, profiling captures steps 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 (10 steps total, starting from step 3).
+
+**Advanced usage with `start_step`:**
+
+```bash Command
+# Wait 5 steps (warmup), then profile for 10 steps
+curl -X POST http://127.0.0.1:30000/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{
+    "output_dir": "/tmp/profiles",
+    "start_step": 5,
+    "num_steps": 10,
+    "activities": ["CPU", "GPU"]
+  }'
+```
+
+**Continuous profiling (manual stop):**
+
+```bash Command
+# Start profiling without num_steps - must manually stop with /stop_profile
+curl -X POST http://127.0.0.1:30000/start_profile
+```
+
+#### Using `/stop_profile` endpoint
+
+The `/stop_profile` endpoint stops an ongoing profiling session and saves the trace file.
+
+```bash Command
+# Stop profiling and save traces
+curl -X POST http://127.0.0.1:30000/stop_profile
+```
+
+This is only needed when you start profiling without specifying `num_steps`. If `num_steps` is specified, profiling will automatically stop after that many steps.
+
+#### Example workflow
+
+```bash Command
+# Terminal 1: Start the server
+export SGLANG_TORCH_PROFILER_DIR=/tmp/profiles
+python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
+
+# Terminal 2: Start continuous profiling
+curl -X POST http://127.0.0.1:30000/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{
+    "start_step": 3
+  }'
+
+# Terminal 3: Send requests to generate load
+python -m sglang.bench_serving --backend sglang --num-prompts 100
+
+# Terminal 2: Stop profiling when done
+curl -X POST http://127.0.0.1:30000/stop_profile
+```
+
+### Profiler Trace Merger for Distributed Traces
+
+SGLang now supports automatic merging of profiling traces from distributed setups with multiple parallelism types (TP, DP, PP, EP). This feature is particularly useful for analyzing performance across distributed runs.
+
+#### Multi-Node Profiling and Shared Storage Considerations
+
+Single-node profiler output merging is completely supported. When profiling in distributed environments spanning multiple nodes, shared storage (e.g., NFS, Lustre) should be accessible by all nodes for the output directory to enable merging of trace files.
+
+If there is no shared storage accessible across nodes, automatic merging of trace files during profiling is not supported directly as of now.
+
+#### HTTP API Usage
+
+```bash Command
+# Start profiling with automatic trace merging enabled
+curl -X POST <BASE_URL>/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{
+    "output_dir": "/tmp/profiles", # where to store profile traces
+    "num_steps": 10,
+    "activities": ["CPU", "GPU"],
+    "merge_profiles": true # optional argument to merge profile traces (default=False)
+  }'
+```
+
+#### Command Line Usage
+
+```bash Command
+# Start profiling with merge enabled
+python -m sglang.profiler \
+  --num-steps 10 \
+  --cpu \
+  --gpu \
+  --output-dir /tmp/profiles \
+  --merge-profiles # optional argument to merge profile traces (default=False)
+```
+
+#### Output Files
+
+The profile merger generates:
+- Individual rank trace files: `&#123;profile_id&#125;-TP-&#123;tp&#125;-DP-&#123;dp&#125;-PP-&#123;pp&#125;-EP-&#123;ep&#125;.trace.json.gz`
+- Merged trace file: `merged-&#123;profile_id&#125;.trace.json.gz`
+
+### Possible PyTorch bugs
+If in any cases you encounter the following error (for example, using qwen 2.5 VL):
+```bash Command
+RuntimeError: !stack.empty() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/autograd/profiler_python.cpp":983, please report a bug to PyTorch. Python replay stack is empty.
+```
+This is likely a PyTorch Bug reported in [Bug: vLLM Profiler](https://github.com/vllm-project/vllm/issues/18240) and [Bug: torch.profiler.profile](https://github.com/pytorch/pytorch/issues/101632). As a workaround, you may disable `with_stack` with an environment variable such as follows:
+```bash Command
+export SGLANG_PROFILE_WITH_STACK=False
+python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8
+```
+
+### View traces
+
+Trace files can be loaded and visualized from:
+
+1. https://ui.perfetto.dev/ (any browser)
+2. chrome://tracing (Chrome browser only)
+
+If browser cannot open trace file due to its large size,
+client can generate a small trace file (&lt;100MB) by controlling number of prompts and lengths of prompt outputs.
+For example, when profiling a server,
+
+```bash Command
+python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 2 --sharegpt-output-len 100 --profile
+```
+
+This command sets the number of prompts to 2 with `--num-prompts` argument and limits the length of output sequences to 100 with `--sharegpt-output-len` argument, which can generate a small trace file for browser to open smoothly.
+
+Additionally, if you want to locate the SGLang Python source code through the cuda kernel in Trace, you need to disable CUDA Graph when starting the service. This can be done by using the `--disable-cuda-graph` parameter in the command to start the service.
+
+## Profile with Nsight
+
+[Nsight systems](https://docs.nvidia.com/nsight-systems/) is an advanced tool that exposes more profiling details, such as register and shared memory usage, annotated code regions and low-level CUDA APIs and events.
+
+1. Prerequisite:
+
+   Install using apt, or run inside a [NVIDIA Docker container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) or [SGLang Docker container](https://github.com/sgl-project/sglang/tree/main/docker).
+
+   ```bash Command
+   # install nsys
+   # https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html
+   apt update
+   apt install -y --no-install-recommends gnupg
+   echo "deb http://developer.download.nvidia.com/devtools/repos/ubuntu$(source /etc/lsb-release; echo "$DISTRIB_RELEASE" | tr -d .)/$(dpkg --print-architecture) /" | tee /etc/apt/sources.list.d/nvidia-devtools.list
+   apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
+   apt update
+   apt install nsight-systems-cli
+   ```
+
+2. To profile a single batch, use
+
+   ```bash Command
+   nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 64 --input-len 512
+   ```
+
+3. To profile a server, e.g.
+
+   ```bash Command
+   # launch the server, set the delay and duration times according to needs
+   # after the duration time has been used up, server will be killed by nsys
+
+   nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node -o sglang.out --delay 60 --duration 70 python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disable-radix-cache
+
+   # client
+   python3 -m sglang.bench_serving --backend sglang --num-prompts 1000 --dataset-name random --random-input 1024 --random-output 512
+   ```
+
+   In practice, we recommend users to set `--duration` argument to a large value. Whenever user wants the server to stop profiling. Firstly run:
+
+   ```bash Command
+   nsys sessions list
+   ```
+
+   to get the session id in the form of `profile-XXXXX`, then run:
+
+   ```bash Command
+   nsys stop --session=profile-XXXXX
+   ```
+
+   to manually kill the profiler and generate `nsys-rep` files instantly.
+
+4. Use NVTX to annotate code regions, e.g. to see their execution time.
+
+   ```bash Command
+   # install nvtx
+   pip install nvtx
+   ```
+
+   ```python Example
+   # code snippets
+   import nvtx
+   with nvtx.annotate("description", color="color"):
+       # some critical code
+   ```
+
+### Layer-wise NVTX Profiling with Nsight Systems
+
+SGLang provides built-in layerwise NVTX annotations that can be combined with the CUDA Profiler for detailed per-layer profiling in Nsight Systems. This is particularly useful for identifying performance bottlenecks at the layer level.
+
+#### Using `--enable-layerwise-nvtx-marker` with Nsight Systems and `/start_profile`
+
+The `--enable-layerwise-nvtx-marker` flag automatically adds NVTX markers to every layer in your model. This is particularly powerful when combined with Nsight Systems profiling to see detailed per-layer performance.
+
+**Method 1: Using `/start_profile` with CUDA_PROFILER (for programmatic control)**
+
+This method allows you to control exactly when profiling starts/stops via HTTP API while Nsight Systems is running.
+
+1. Launch the server with layerwise NVTX enabled under Nsight Systems:
+
+   ```bash Command
+   # Terminal 1: Start server with nsys and capture-range option
+   nsys profile --trace-fork-before-exec=true \
+     --cuda-graph-trace=node \
+     --capture-range=cudaProfilerApi \
+     --capture-range-end=stop \
+     -o layerwise_profile \
+     python -m sglang.launch_server \
+       --model-path meta-llama/Llama-3.1-8B-Instruct \
+       --enable-layerwise-nvtx-marker \
+       --disable-cuda-graph
+   ```
+
+   Note: NVTX markers are not emitted for kernel launches captured by CUDA graphs. Use `--disable-cuda-graph` to ensure all layerwise NVTX markers are emitted in the trace.
+
+2. In another terminal, control profiling via `/start_profile` with `CUDA_PROFILER` activity:
+
+   ```bash Command
+   # Terminal 2: Wait for server to be ready, then start CUDA profiling
+   # Wait 3 steps for warmup, then profile for 10 steps
+   curl -X POST http://127.0.0.1:30000/start_profile \
+     -H "Content-Type: application/json" \
+     -d '{
+       "start_step": 3,
+       "num_steps": 10,
+       "activities": ["CUDA_PROFILER"]
+     }'
+   ```
+
+3. Send requests to generate load:
+
+   ```bash Command
+   # Terminal 3: Generate workload
+   python -m sglang.bench_serving --backend sglang --num-prompts 100
+   ```
+
+4. Profiling will automatically stop after 10 steps (due to `num_steps: 10`). If you hadn't specified `num_steps`, you would need to manually stop it:
+
+   ```bash Command
+   # Terminal 2: Only needed if num_steps was not specified
+   curl -X POST http://127.0.0.1:30000/stop_profile
+   ```
+
+The `--capture-range=cudaProfilerApi` option tells Nsight Systems to only capture data between `cudaProfilerStart()` and `cudaProfilerStop()` calls (triggered by `/start_profile` and `/stop_profile`), reducing overhead and file size. The `start_step` parameter skips the first 3 steps to avoid capturing warmup overhead.
+
+**Method 2: Simpler approach without `/start_profile` API**
+
+For simpler use cases where you don't need fine-grained control over profiling start/stop, you can profile with Nsight Systems capturing the entire workload:
+
+```bash Command
+# Terminal 1: Start server with layerwise NVTX
+# Note: --disable-cuda-graph ensures all NVTX markers are emitted
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --enable-layerwise-nvtx-marker \
+  --disable-cuda-graph
+
+# Terminal 2: Profile the benchmarking client
+nsys profile --trace-fork-before-exec=true \
+  --cuda-graph-trace=node \
+  -o layerwise_profile \
+  python -m sglang.bench_serving --backend sglang --num-prompts 10
+```
+
+This approach profiles the entire client execution, including all server interactions. The layerwise NVTX markers will be visible in the Nsight Systems timeline.
+
+**Viewing the profiling results:**
+
+Open the generated `.qdrep` file with Nsight Systems:
+
+```bash Command
+nsys-ui layerwise_profile.qdrep
+```
+
+In the Nsight Systems GUI, you'll see:
+- **NVTX ranges**: Each layer appears as a labeled range in the timeline with detailed information in the marker metadata
+- **CUDA kernels**: All GPU kernels are shown alongside the layer annotations
+- **Layer hierarchy**: The full module path (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct.model.layers.0.self_attn.qkv_proj`) helps identify specific layers. The prefix uses the full model path from `--model-path`.
+- **Tensor shapes**: Input/output dimensions and parameter shapes are included in the NVTX marker data
+
+**Benefits of layerwise NVTX profiling:**
+
+- **Granular visibility**: See exactly which layers are taking the most time
+- **Memory tracking**: Identify layers with large memory allocations
+- **Bottleneck identification**: Quickly locate inefficient operations
+- **Communication overhead**: In multi-GPU setups, see per-layer communication costs
+- **Development debugging**: Validate that model architecture changes have the expected performance impact
+
+## Other tips
+
+1. You can benchmark a model using dummy weights by only providing the config.json file. This allows for quick testing of model variants without training. To do so, add `--load-format dummy` to the above commands and then you only need a correct `config.json` under the checkpoint folder.
+2. You can benchmark a model with modified configs (e.g., less layers) by using `--json-model-override-args`. For example, you can benchmark a model with only 2 layers and 2 kv heads using:
+
+   ```bash Command
+   python -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32 --load-format dummy --json-model-override-args '{"num_hidden_layers": 1, "num_key_value_heads": 1}'
+   ```
+
+3. You can use `--python-backtrace=cuda` to see python call stack for all CUDA kernels, as in PyTorch Profiler. (Caveat: this can cause inaccurately long kernel runtimes for CUDA event based timing)
+4. For more arguments see [Nsight Systems User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).

docs_new/docs/developer_guide/contribution_guide.mdx

@@ -0,0 +1,190 @@
+---
+title: "Contribution Guide"
+mode: wide
+metatags:
+    description: "SGLang contribution guide: source install, pre-commit, unit tests, CI triggers, code style, sgl-kernel updates."
+---
+Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.
+
+## Install SGLang from Source
+
+### Fork and clone the repository
+
+**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.
+
+```bash
+git clone https://github.com/<your_user_name>/sglang.git
+```
+
+### Build from source
+
+Refer to [Install SGLang from Source](../get-started/install#method-2-from-source).
+
+## Format code with pre-commit
+
+We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:
+
+```bash
+pip3 install pre-commit
+pre-commit install
+pre-commit run --all-files
+```
+
+- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
+- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
+- Link checking with lychee is **enforced in CI**. By default, it is not blocking local commits.
+- To run local link checks manually, use: `pre-commit run --hook-stage manual lychee --all-files`.
+
+## Run and add unit tests
+
+If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression.
+
+### Unit tests (no server required)
+
+Unit tests live under [`test/registered/unit/`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit), organized to mirror the `python/sglang/srt/` source tree. These tests validate component logic **without** launching a server or loading real model weights.
+SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework with [pytest](https://docs.pytest.org/) as the test runner.
+
+**When to add a unit test:** If you modify a file under `python/sglang/srt/`, check whether a corresponding test exists in `test/registered/unit/` and add coverage for your changes. For example:
+
+```
+srt/mem_cache/radix_cache.py   →  unit/mem_cache/test_radix_cache.py
+srt/sampling/sampling_params.py →  unit/sampling/test_sampling_params.py
+```
+
+**Run unit tests locally:**
+
+```bash Command
+pytest test/registered/unit/ -v                # all unit tests
+pytest test/registered/unit/mem_cache/ -v      # one module
+```
+
+**Run with coverage:**
+
+```bash Command
+pytest test/registered/unit/ --cov --cov-config=.coveragerc -v
+```
+
+For conventions on CI registration, test structure, and examples, see [`test/registered/unit/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit/README.md).
+
+### E2E tests (server required)
+
+For tests that require launching a server, refer to [`test/registered/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/README.md) for guidance on where to place your test.
+
+For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
+
+## Write documentations
+
+We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
+For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).
+
+## Test the accuracy
+If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
+
+```text Output
+# Launch a server
+python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
+
+# Evaluate
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+
+Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
+This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
+Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.
+
+GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
+You can find additional accuracy eval examples in:
+- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_eval_accuracy_large.py)
+- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/registered/core/test_gpt_oss_1gpu.py)
+
+## Benchmark the speed
+Refer to [Benchmark and Profiling](./benchmark_and_profiling).
+
+## Requesting a review for merge
+You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
+You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
+Then your PR can be merged.
+
+## How to Trigger CI Tests
+
+We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
+Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
+
+**PR authors** can always use `/rerun-failed-ci` on their own PRs, even if they are not listed in `CI_PERMISSIONS.json`.
+
+For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
+
+- `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
+- `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**.
+- `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
+- `/rerun-stage <stage-name>`: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages.
+- `/rerun-test <test-spec> [<test-spec> ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `<test-spec>` is pytest-style `<file>::<TestClass>[.<test_method>]` (the `::TestClass` and `.<test_method>` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once).
+
+If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
+
+To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
+
+Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
+
+If you don’t have permission and you’re not the PR author, please ask maintainers to trigger CI for you.
+
+### CI rate limits
+
+Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
+We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
+
+Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
+
+```yaml Config
+cool-down-minutes:
+  description: "Default cooldown period in minutes; 0 disables rate limiting"
+  type: number
+  default: 120
+```
+
+Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.
+
+## Code style guidance
+- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
+- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
+- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
+  - A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible.
+- Make functions as pure as possible. Avoid in-place modification of arguments.
+- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
+- In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file.
+- Keep tests run fast.
+  - If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
+  - If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
+  - Reuse server launches in your unit tests to make tests run faster.
+- Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python's [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead.
+- When supporting new hardware or features, follow these guidelines:
+  - Do not drastically change existing code.
+  - Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`).
+  - If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
+
+## How to update sgl-kernel
+Since sglang and the `sglang-kernel` (prior `sgl-kernel`) distribution are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR).
+To add a new kernel or modify an existing one in the `sgl-kernel/` source tree, you must use multiple PRs.
+
+Follow these steps:
+
+1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)).
+2. Bump the version of the kernel package (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)).
+   - Once merged, this will trigger an automatic release of the `sglang-kernel` wheel to PyPI.
+   - If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week.
+3. Apply the changes:
+   - Update the `sglang-kernel` version in `sglang/python/pyproject.toml` to use the modified kernels.
+   - Update the related caller code in the sglang to use the new kernel.
+
+## Tips for newcomers
+
+If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase.
+
+Also check out the following materials as startup guide:
+- [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang.
+- [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
+- [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance.
+
+If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
+
+Thank you for your interest in SGLang. Happy coding!

docs_new/docs/developer_guide/development_guide_using_docker.mdx

@@ -0,0 +1,119 @@
+---
+title: "Development Guide Using Docker"
+sidebarTitle: "Using Docker"
+metatags:
+    description: "SGLang Docker development: VSCode dev container, remote tunnels, debugger setup, nsys profiling."
+---
+## Setup VSCode on a Remote Host
+(Optional - you can skip this step if you plan to run sglang dev container locally)
+
+1. In the remote host, download `code` from [VSCode](https://code.visualstudio.com/download) and run `code tunnel` in a shell.
+
+Example
+```bash Command
+wget https://vscode.download.prss.microsoft.com/dbazure/download/stable/fabdb6a30b49f79a7aba0f2ad9df9b399473380f/vscode_cli_alpine_x64_cli.tar.gz
+tar xf vscode_cli_alpine_x64_cli.tar.gz
+
+# https://code.visualstudio.com/docs/remote/tunnels
+./code tunnel
+```
+
+2. In your local machine, press F1 in VSCode and choose "Remote Tunnels: Connect to Tunnel".
+
+## Setup Docker Container
+
+### Option 1. Use the default dev container automatically from VSCode
+There is a `.devcontainer` folder in the sglang repository root folder to allow VSCode to automatically start up within dev container. You can read more about this VSCode extension in VSCode official document [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).
+<Frame>
+  <img src="https://github.com/user-attachments/assets/6a245da8-2d4d-4ea8-8db1-5a05b3a66f6d" alt="VSCode Dev Container Architecture" />
+</Frame>
+
+*Figure 1: Diagram from VSCode official documentation [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).*
+
+To enable this, you only need to:
+1. Start Visual Studio Code and install [VSCode dev container extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
+2. Press F1, type and choose "Dev Container: Open Folder in Container.
+3. Input the `sglang` local repo path in your machine and press enter.
+
+The first time you open it in dev container might take longer due to docker pull and build. Once it's successful, you should set on your status bar at the bottom left displaying that you are in a dev container:
+
+<Frame>
+  <img src="https://github.com/user-attachments/assets/650bba0b-c023-455f-91f9-ab357340106b" alt="VSCode Dev Container Status Bar" />
+</Frame>
+
+Now when you run `sglang.launch_server` in the VSCode terminal or start debugging using F5, sglang server will be started in the dev container with all your local changes applied automatically:
+
+<Frame>
+  <img src="https://github.com/user-attachments/assets/748c85ba-7f8c-465e-8599-2bf7a8dde895" alt="SGLang Server Running in Dev Container" />
+</Frame>
+
+
+### Option 2. Start up containers manually (advanced)
+
+The following startup command is an example for internal development by the SGLang team. You can **modify or add directory mappings as needed**, especially for model weight downloads, to prevent repeated downloads by different Docker containers.
+
+❗️ **Note on RDMA**
+
+    1. `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them but keeping them there does not harm. Thus, we enable these two flags by default in the commands below.
+    2. You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
+
+```bash Command
+# Change the name to yours
+docker run -itd --shm-size 32g --gpus all -v <volumes-to-mount> --ipc=host --network=host --privileged --name sglang_dev lmsysorg/sglang:dev /bin/zsh
+docker exec -it sglang_dev /bin/zsh
+```
+Some useful volumes to mount are:
+1. **Huggingface model cache**: mounting model cache can avoid re-download every time docker restarts. Default location on Linux is `~/.cache/huggingface/`.
+2. **SGLang repository**: code changes in the SGLang local repository will be automatically synced to the .devcontainer.
+
+Example 1: Monting local cache folder `/opt/dlami/nvme/.cache` but not the SGLang repo. Use this when you prefer to manually transfer local code changes to the devcontainer.
+```bash Command
+docker run -itd --shm-size 32g --gpus all -v /opt/dlami/nvme/.cache:/root/.cache --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
+docker exec -it sglang_zhyncs /bin/zsh
+```
+Example 2: Mounting both HuggingFace cache and local SGLang repo. Local code changes are automatically synced to the devcontainer as the SGLang is installed in editable mode in the dev image.
+```bash Command
+docker run -itd --shm-size 32g --gpus all -v $HOME/.cache/huggingface/:/root/.cache/huggingface -v $HOME/src/sglang:/sgl-workspace/sglang --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
+docker exec -it sglang_zhyncs /bin/zsh
+```
+## Debug SGLang with VSCode Debugger
+1. (Create if not exist) open `launch.json` in VSCode.
+2. Add the following config and save. Please note that you can edit the script as needed to apply different parameters or debug a different program (e.g. benchmark script).
+     ```JSON Config
+       {
+          "version": "0.2.0",
+          "configurations": [
+              {
+                  "name": "Python Debugger: launch_server",
+                  "type": "debugpy",
+                  "request": "launch",
+                  "module": "sglang.launch_server",
+                  "console": "integratedTerminal",
+                  "args": [
+                      "--model-path", "meta-llama/Llama-3.2-1B",
+                      "--host", "0.0.0.0",
+                      "--port", "30000",
+                      "--trust-remote-code",
+                  ],
+                  "justMyCode": false
+              }
+          ]
+      }
+    ```
+
+3. Press "F5" to start. VSCode debugger will ensure that the program will pause at the breakpoints even if the program is running at remote SSH/Tunnel host + dev container.
+
+## Profile
+
+```bash Command
+# Change batch size, input, output and add `disable-cuda-graph` (for easier analysis)
+# e.g. DeepSeek V3
+nsys profile -o deepseek_v3 python3 -m sglang.bench_one_batch --batch-size 1 --input 128 --output 256 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --tp 8 --disable-cuda-graph
+```
+
+## Evaluation
+
+```bash Command
+# e.g. gsm8k 8 shot
+python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 2000 --num-shots 8
+```

docs_new/docs/developer_guide/development_jit_kernel_guide.mdx

@@ -0,0 +1,425 @@
+---
+title: "Development Guide for JIT Kernels"
+sidebarTitle: "JIT Kernels"
+metatags:
+    description: "SGLang JIT kernel development: clangd setup, TensorMatcher, LaunchKernel, add_constant example walkthrough."
+---
+## Environment Setup
+
+We strongly recommend using `clangd` as the language server for JIT kernel development.
+For Ubuntu/Debian, you can download clangd from [apt.llvm.org](https://apt.llvm.org/).
+If you are using VS Code, we recommend installing the `clangd` extension for better IDE integration.
+
+All JIT-related files are located in `python/sglang/jit_kernel`.
+Unlike `sgl-kernel`, which compiles CUDA/C++ binaries ahead of time (AOT), just-in-time (JIT) kernels are compiled at runtime.
+Consequently, a static `compile_commands.json` cannot be generated.
+To enable code completion with `clangd`, run `python -m sglang.jit_kernel` to generate a `.clangd` configuration file in your current directory.
+After generating the file, restart the clangd language server. It should now recognize all JIT kernel files.
+
+## Code Structure
+
+### C++ Implementation
+
+C++ source code is located in `python/sglang/jit_kernel/csrc`.
+Reusable functions should be placed in `python/sglang/jit_kernel/include`.
+
+We use [tvm-ffi](https://github.com/apache/tvm-ffi) for efficient foreign language bindings.
+Refer to the [documentation](https://tvm.apache.org/ffi/) for advanced usage, such as exporting C++ objects.
+Typically, `tvm::ffi::TensorView` is sufficient for passing PyTorch Tensors from Python.
+
+### Python Interface
+
+Python interfaces are defined in `python/sglang/jit_kernel`.
+The `load_jit` utility function in `python/sglang/jit_kernel/utils.py` loads and returns the compiled module.
+To export a C++ function (e.g., `cpp_func`), pass `cuda_wrappers=[("func", "cpp_func")]` to `load_jit`.
+The function can then be called in Python as `module.func`.
+
+For caching compiled modules, prefer `sglang.jit_kernel.utils.cache_once` over `functools.lru_cache`.
+`functools.lru_cache` is not compatible with `torch.compile`.
+
+### C++ Utilities
+
+The following C++ utilities are available:
+
+#### Integer Range
+
+Similar to PyTorch, we provide an `irange` function to represent an integer range.
+
+```C++ Example
+#include <sgl_kernel/utils.h>
+
+void test() {
+  for (auto i : host::irange(100)) { // [0, 100)
+    // do something
+  }
+  for (auto i : host::irange(0, 100)) { // [0, 100)
+    // do something
+  }
+}
+
+```
+
+#### Runtime Checking
+
+`RuntimeCheck` validates conditions at runtime. It accepts optional arguments for error reporting.
+If the check fails, these arguments are output to aid debugging.
+`RuntimeDeviceCheck` verifies the status of the last kernel launch.
+
+```C++ Example
+#include <sgl_kernel/utils.h>
+#include <sgl_kernel/utils.cuh>
+
+void test() {
+  host::RuntimeCheck(1 + 1 == 2, 1 + 1, " != ", 2);
+  host::RuntimeDeviceCheck();
+  // check the provided `cudaError_t`
+  host::RuntimeDeviceCheck(cudaGetLastError());
+}
+
+```
+
+#### Tensor Checking
+
+`TensorMatcher` provides a readable way to validate and extract tensor shape information.
+
+```cpp Example
+#include <sgl_kernel/tensor.h>
+
+void test(const tvm::ffi::TensorView k_cache, const tvm::ffi::TensorView v_cache) {
+  using namespace host;
+
+  auto D = SymbolicSize{"D"};  // cache dimension
+  auto N = SymbolicSize{"N"};  // kvcache stride
+  auto dtype = SymbolicDType{};
+  auto device = SymbolicDevice{};
+
+  TensorMatcher({-1, D})  //
+      .with_strides({N, 1})
+      .with_dtype<int32_t, int64_t>(dtype)
+      .with_device<kDLCUDA, kDLCPU>(device)
+      .verify(k_cache)
+      .verify(v_cache);
+}
+```
+
+Configure the `TensorMatcher` with expected stride, dtype, and device properties before verification.
+- If `with_strides` is omitted, the tensor is expected to be contiguous.
+- Template arguments in `with_dtype` restrict the allowed data types.
+- Template arguments in `with_device` restrict the allowed devices.
+- Values passed to `with_xxx` methods enforce equality checks.
+- Passing `-1` for size or stride allows matching any value.
+
+A `Symbolic` variable must resolve to the same value across all verifications.
+Use `.unwrap()` to retrieve the matched value after verification.
+
+> Note: `TensorMatcher` is a temporary expression and should not be stored in a variable.
+
+> Tip: Add `//` at the end of the `TensorMatcher` chain to enforce proper indentation.
+
+#### Kernel Launching
+
+`LaunchKernel::resolve_device` retrieves the current `cudaStream` from PyTorch.
+Kernels can also be launched directly using `LaunchKernel`.
+
+```cpp Example
+#include <sgl_kernel/utils.cuh>
+
+#include <dlpack/dlpack.h>
+
+__global__ void kernel() {}
+
+void test() {
+  const auto num_blocks = 1;
+  const auto num_threads = 32;
+  const auto dynamic_smem = 0;
+
+  DLDevice dev;  // suppose this is initialized properly
+  host::LaunchKernel(num_blocks, num_threads, dev)(kernel);
+
+  cudaStream_t stream = host::LaunchKernel::resolve_device(dev);
+  host::LaunchKernel(num_blocks, num_threads, stream, dynamic_smem)(kernel);
+}
+
+```
+
+## Add new kernels
+
+This section walks through a complete, end-to-end example of adding a new JIT kernel to the system.
+We use a simple add_constant kernel as a running example, which adds a constant integer value to every element of an input tensor.
+
+Conceptually, the Python interface looks like this:
+
+```python Example
+def add_constant(src: torch.Tensor, c: int):
+    return src + c
+```
+
+### STEP 1: Write the C++ kernel
+
+Write your CUDA kernel in [jit_kernel/csrc/add_constant.cuh](https://github.com/sgl-project/sglang/blob/main/python/sglang/jit_kernel/csrc/add_constant.cuh). For demonstration purposes, we pass the constant value as a template parameter.
+
+```cpp Example
+#include <sgl_kernel/tensor.h>   // For TensorMatcher, SymbolicSize, SymbolicDevice
+#include <sgl_kernel/utils.cuh>  // For LaunchKernel
+#include <sgl_kernel/utils.h>    // For div_ceil, RuntimeCheck
+
+#include <dlpack/dlpack.h>
+#include <tvm/ffi/container/tensor.h>
+
+#include <cstddef>
+#include <cstdint>
+
+namespace {
+
+template <int32_t kConstant>
+__global__ void add_constant_kernel(int32_t* dst, const int32_t* src, size_t length) {
+  size_t idx = blockIdx.x * blockDim.x + threadIdx.x;
+  if (idx < length) {
+    dst[idx] = src[idx] + kConstant;
+  }
+}
+
+constexpr size_t kBlockSize = 256;
+
+// You can also use struct with static method as an alternative
+template <int32_t kConstant>
+void add_constant(tvm::ffi::TensorView dst, tvm::ffi::TensorView src) {
+  using namespace host;
+
+  // 1. Validate input tensors
+  SymbolicSize N = {"num_elements"};
+  SymbolicDevice device_;
+  TensorMatcher({N})                  // 1D tensor, must be contiguous
+      .with_dtype<int32_t>()          // must be int32
+      .with_device<kDLCUDA>(device_)  // must be on CUDA device
+      .verify(dst)                    // check tensor dst
+      .verify(src);                   // check tensor src
+
+  // 2. Extract required parameters, prepare for kernel launch
+  const size_t num_elements = N.unwrap();
+  const size_t grid_size = div_ceil(num_elements, kBlockSize);
+  const DLDevice device = device_.unwrap();
+  // some extra runtime checks using host::RuntimeCheck
+  RuntimeCheck(num_elements > 0, "We only support non-empty tensors, got num_elements = ", num_elements);
+
+  // 3. Launch the kernel. Error code will be automatically checked.
+  LaunchKernel(grid_size, kBlockSize, device /*, dynamic_smem*/)(
+      // kernel function
+      add_constant_kernel<kConstant>,
+      // kernel arguments
+      static_cast<int32_t*>(dst.data_ptr()),
+      static_cast<int32_t*>(src.data_ptr()),
+      num_elements);
+}
+
+}  // namespace
+
+```
+
+### STEP 2: Create Python Interfaces
+
+Next, expose the kernel through a Python wrapper.
+Create a new file at [jit_kernel/add_constant.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/jit_kernel/add_constant.py) and expose the needed interfaces.
+
+```python Example
+from __future__ import annotations
+from typing import TYPE_CHECKING
+
+import torch
+
+from sglang.jit_kernel.utils import cache_once, load_jit, make_cpp_args
+
+if TYPE_CHECKING:
+    from tvm_ffi.module import Module
+
+
+@cache_once
+def _jit_add_constant_module(constant: int) -> Module:
+    args = make_cpp_args(constant)  # pass all the template argument
+    return load_jit(
+        "add_constant",
+        *args,
+        cuda_files=["add_constant.cuh"],
+        cuda_wrappers=[("add_constant", f"add_constant<{args}>")],
+    )
+
+
+def add_constant(src: torch.Tensor, constant: int) -> torch.Tensor:
+    if not src.is_cuda:
+        raise RuntimeError("src must be a CUDA tensor")
+    if src.dtype != torch.int32:
+        raise RuntimeError(f"Unsupported dtype {src.dtype}. Supported: int32")
+    dst = torch.empty_like(src)
+    module = _jit_add_constant_module(constant)
+    module.add_constant(dst, src)
+    return dst
+
+```
+
+Keep the Python wrapper thin, but still validate the basic invariants such as device and dtype before dispatch. In the current JIT/FFI path, invalid tensors are not always rejected safely before launch.
+
+### STEP 3: Use your kernel
+
+Finally, import and use the kernel like a regular Python function:
+
+```python Example
+from sglang.jit_kernel.add_constant import add_constant
+```
+
+For a complete, runnable example, refer to [test_add_constant.py](https://github.com/sgl-project/sglang/blob/main/test/registered/jit/test_add_constant.py).
+
+## C++ Include Library Reference
+
+The JIT kernel framework provides a set of reusable C++ headers in
+`python/sglang/jit_kernel/include/sgl_kernel/`. Each header is designed
+to be lightweight and self-contained. Below is a summary of each header
+and its key APIs.
+
+### Core Utilities
+
+<table>
+  <thead>
+    <tr>
+      <th>Header</th>
+      <th>Namespace</th>
+      <th>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>utils.h</code></td>
+      <td><code>host</code></td>
+      <td>Host-side essentials: <code>RuntimeCheck</code>, <code>Panic</code>, <code>div_ceil</code>, <code>irange</code></td>
+    </tr>
+    <tr>
+      <td><code>utils.cuh</code></td>
+      <td><code>device</code> / <code>host</code></td>
+      <td>Type aliases (<code>fp16_t</code>, <code>bf16_t</code>, ...), <code>SGL_DEVICE</code> macro, PDL helpers, <code>LaunchKernel</code>, <code>RuntimeDeviceCheck</code></td>
+    </tr>
+    <tr>
+      <td><code>source_location.h</code></td>
+      <td>(global)</td>
+      <td>Portable <code>std::source_location</code> wrapper for error reporting</td>
+    </tr>
+    <tr>
+      <td><code>runtime.cuh</code></td>
+      <td><code>host::runtime</code></td>
+      <td>CUDA runtime queries: <code>get_blocks_per_sm</code>, <code>get_sm_count</code>, <code>get_cc_major</code>, <code>get_runtime_version</code>, <code>get_available_dynamic_smem_per_block</code></td>
+    </tr>
+  </tbody>
+</table>
+
+### Tensor Validation
+
+<table>
+  <thead>
+    <tr>
+      <th>Header</th>
+      <th>Namespace</th>
+      <th>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>tensor.h</code></td>
+      <td><code>host</code></td>
+      <td><code>TensorMatcher</code>, <code>SymbolicSize</code>, <code>SymbolicDType</code>, <code>SymbolicDevice</code></td>
+    </tr>
+  </tbody>
+</table>
+
+### Math & Type System
+
+<table>
+  <thead>
+    <tr>
+      <th>Header</th>
+      <th>Namespace</th>
+      <th>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>math.cuh</code></td>
+      <td><code>device::math</code></td>
+      <td><code>max</code>, <code>min</code>, <code>abs</code>, <code>sqrt</code>, <code>rsqrt</code>, <code>exp</code>, <code>sin</code>, <code>cos</code>, constants</td>
+    </tr>
+    <tr>
+      <td><code>type.cuh</code></td>
+      <td>(global) / <code>device</code></td>
+      <td><code>dtype_trait&lt;T&gt;</code>, <code>packed_t&lt;T&gt;</code>, <code>device::cast&lt;To&gt;(from)</code></td>
+    </tr>
+  </tbody>
+</table>
+
+### Memory Access
+
+<table>
+  <thead>
+    <tr>
+      <th>Header</th>
+      <th>Namespace</th>
+      <th>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>vec.cuh</code></td>
+      <td><code>device</code></td>
+      <td><code>AlignedVector&lt;T, N&gt;</code> - vectorized load/store (up to 128-bit; 256-bit requires Blackwell GPUs)</td>
+    </tr>
+    <tr>
+      <td><code>tile.cuh</code></td>
+      <td><code>device::tile</code></td>
+      <td><code>Memory&lt;T&gt;</code> - cooperative tiled memory I/O (thread/warp/CTA)</td>
+    </tr>
+  </tbody>
+</table>
+
+### Parallel Primitives
+
+<table>
+  <thead>
+    <tr>
+      <th>Header</th>
+      <th>Namespace</th>
+      <th>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>warp.cuh</code></td>
+      <td><code>device::warp</code></td>
+      <td><code>reduce_sum</code>, <code>reduce_max</code> via <code>__shfl_xor_sync</code></td>
+    </tr>
+    <tr>
+      <td><code>cta.cuh</code></td>
+      <td><code>device::cta</code></td>
+      <td><code>reduce_max</code> across warps via shared memory</td>
+    </tr>
+    <tr>
+      <td><code>atomic.cuh</code></td>
+      <td><code>device::atomic</code></td>
+      <td><code>max</code> - atomic float max (CUDA + ROCm fallback)</td>
+    </tr>
+  </tbody>
+</table>
+
+### Reusable Kernel Templates
+
+<table>
+  <thead>
+    <tr>
+      <th>Header</th>
+      <th>Namespace</th>
+      <th>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>impl/norm.cuh</code></td>
+      <td><code>host::norm</code> / <code>device::norm</code></td>
+      <td>RMSNorm building blocks (warp &amp; CTA paths, <code>StorageType</code>)</td>
+    </tr>
+  </tbody>
+</table>

docs_new/docs/developer_guide/evaluating_new_models.mdx

@@ -0,0 +1,149 @@
+---
+title: "Evaluating New Models with SGLang"
+metatags:
+    description: "SGLang model evaluation: MMLU, GSM8K, GPQA, HumanEval, MMMU benchmarks. Latency and throughput testing commands."
+---
+This document provides commands for evaluating models' accuracy and performance. Before open-sourcing new models, we strongly suggest running these commands to verify whether the score matches your internal benchmark results.
+
+**For cross verification, please submit commands for installation, server launching, and benchmark running with all the scores and hardware requirements when open-sourcing your models.**
+
+[Reference: MiniMax M2](https://github.com/sgl-project/sglang/pull/12129)
+
+## Accuracy
+
+### LLMs
+
+SGLang provides built-in scripts to evaluate common benchmarks.
+
+**MMLU**
+
+```bash Command
+python -m sglang.test.run_eval \
+  --eval-name mmlu \
+  --port 30000 \
+  --num-examples 1000 \
+  --max-tokens 8192
+```
+
+**GSM8K**
+
+```bash Command
+python -m sglang.test.few_shot_gsm8k \
+  --host http://127.0.0.1 \
+  --port 30000 \
+  --num-questions 200 \
+  --num-shots 5
+```
+
+**HellaSwag**
+
+```bash Command
+python benchmark/hellaswag/bench_sglang.py \
+  --host http://127.0.0.1 \
+  --port 30000 \
+  --num-questions 200 \
+  --num-shots 20
+```
+
+**GPQA**
+
+```bash Command
+python -m sglang.test.run_eval \
+  --eval-name gpqa \
+  --port 30000 \
+  --num-examples 198 \
+  --max-tokens 120000 \
+  --repeat 8
+```
+
+<Tip>
+For reasoning models, add `--thinking-mode <mode>` (e.g., `qwen3`, `deepseek-r1`, `deepseek-v3`). You may skip it if the model has forced thinking enabled.
+</Tip>
+
+**HumanEval**
+
+```bash Command
+pip install human_eval
+
+python -m sglang.test.run_eval \
+  --eval-name humaneval \
+  --num-examples 10 \
+  --port 30000
+```
+
+### VLMs
+
+**MMMU**
+
+```bash Command
+python benchmark/mmmu/bench_sglang.py \
+  --port 30000 \
+  --concurrency 64
+```
+
+<Tip>
+You can set max tokens by passing `--extra-request-body '{"max_tokens": 4096}'`.
+</Tip>
+
+For models capable of processing video, we recommend extending the evaluation to include `VideoMME`, `MVBench`, and other relevant benchmarks.
+
+## Performance
+
+Performance benchmarks measure **Latency** (Time To First Token - TTFT) and **Throughput** (tokens/second).
+
+### LLMs
+
+**Latency-Sensitive Benchmark**
+
+This simulates a scenario with low concurrency (e.g., single user) to measure latency.
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --dataset-name random \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+**Throughput-Sensitive Benchmark**
+
+This simulates a high-traffic scenario to measure maximum system throughput.
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --dataset-name random \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+**Single Batch Performance**
+
+You can also benchmark the performance of processing a single batch offline.
+
+```bash Command
+python -m sglang.bench_one_batch_server \
+  --model <model-path> \
+  --batch-size 8 \
+  --input-len 1024 \
+  --output-len 1024
+```
+
+You can run more granular benchmarks:
+
+- **Low Concurrency**: `--num-prompts 10 --max-concurrency 1`
+- **Medium Concurrency**: `--num-prompts 80 --max-concurrency 16`
+- **High Concurrency**: `--num-prompts 500 --max-concurrency 100`
+
+## Reporting Results
+
+For each evaluation, please report:
+
+1.  **Metric Score**: Accuracy % (LLMs and VLMs); Latency (ms) and Throughput (tok/s) (LLMs only).
+2.  **Environment settings**: GPU type/count, SGLang commit hash.
+3.  **Launch configuration**: Model path, TP size, and any special flags.
+4.  **Evaluation parameters**: Number of shots, examples, max tokens.

docs_new/docs/developer_guide/msprobe_debugging_guide.mdx

@@ -0,0 +1,601 @@
+---
+title: "MSProbe Debugging Guide"
+metatags:
+    description: "Debugging AI model accuracy anomalies and numerical errors during inference using MSProbe in SGLang."
+---
+MSProbe is a debugging tool for AI models that diagnoses accuracy anomalies and
+numerical errors during model training and inference. It captures and monitors intermediate data (feature maps, weights,
+activations, layer outputs) and contextual metadata (prompts, tensor dtypes, hardware configuration), and supports
+visual analysis to systematically trace the root cause of accuracy degradation or numerical errors (e.g., NaN/Inf,
+output drift, mismatched predictions).
+
+## Basic Details
+
+### Background Concepts: MSProbe Dumping Levels
+
+MSProbe supports three accuracy levels for data dumping, each for different debugging needs:
+
+- **L0**: Dumps tensors/statistics at the **module level** and generates `construct.json` (for network structure
+  reconstruction in visualization). Requires passing a model/submodule handle.
+- **L1**: Dumps tensors/statistics at the **torch API level**, suitable for fine-grained API-level numerical checking.
+- **mix**: Combines L0 + L1, ideal for scenarios that require both **graph reconstruction** and **numerical comparison**.
+
+### Prerequisites: Install MSProbe
+
+Install MSProbe with pip:
+
+```shell
+pip install mindstudio-probe --pre
+```
+
+### Key Configuration Parameters
+
+MSProbe uses a JSON configuration file for customized data dumping. All core parameters are listed in the table below,
+with the default JSON configuration provided for reference.
+
+#### Configuration Parameter Table
+
+|    Field     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Required |
+|:------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
+|    `task`    | Type of dump task. Common PyTorch values include `"statistics"` and `"tensor"`. A statistics task collects tensor statistics (mean, variance, max, min, etc.) while a tensor task captures arbitrary tensors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |   Yes    |
+| `dump_path`  | Directory where dump results are stored. When omitted, `MSProbe` uses its default path.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |    No    |
+|    `rank`    | Ranks to sample. An empty list collects every rank. For single-card tasks you must set this field to `[]`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |    No    |
+|    `step`    | Token iteration(s) to sample. An empty list means every iteration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |    No    |
+|   `level`    | Dump level string (`"L0"`, `"L1"`, or `"mix"`). `L0` targets `nn.Module`, `L1` targets `torch.api`, and `mix` collects both.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |   Yes    |
+| `async_dump` | Whether to enable asynchronous dump (supported for PyTorch `statistics`/`tensor` tasks). Defaults to `false`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |    No    |
+|   `scope`    | Customize the scope of dump. Provide two module or API names that follow the tool's naming convention to lock a range, only data between the two names will be dumped. An empty list dumps every module or torch API.<br/><br/>Examples:<br/>`"scope": ["Module.conv1.Conv2d.forward.0", "Module.fc2.Linear.forward.0"]`<br/>`"scope": ["Tensor.add.0.forward", "Functional.square.2.forward"]`<br/><br/>The `level` setting determines what can be provided—modules when `level=L0`, APIs when `level=L1`, and either modules or APIs when `level=mix`.                                                                                                                                                                                                                                 |    No    |
+|    `list`    | Customize dump list, only dumps elements from the list. An empty list dumps every module or torch API. Options include:<br/><br/>&#8226;Supply the full names of specific APIs in PyTorch eager mode to only dump those APIs. Example: `"list": ["Tensor.permute.1.forward", "Tensor.transpose.2.forward", "Torch.relu.3.backward"]`.<br/>&#8226;When `level=mix`, you can provide module names so that the dump expands to everything produced while the module is running. Example: `"list": ["Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0"]`.<br/>&#8226;Provide a substring such as `"list": ["relu"]` to dump every API whose name contains the substring. When `level=mix`, modules whose names contain the substring are also expanded. |    No    |
+
+#### Default configuration
+
+```json
+{
+  "task": "statistics",
+  "dump_path": "./dump_path",
+  "rank": [],
+  "step": [],
+  "level": "L1",
+  "async_dump": false,
+  "statistics": {
+    "scope": [],
+    "list": [],
+    "data_mode": [
+      "all"
+    ],
+    "summary_mode": "statistics"
+  },
+  "tensor": {
+    "scope": [],
+    "list": [],
+    "data_mode": [
+      "all"
+    ],
+    "file_format": "npy"
+  },
+  "acc_check": {
+    "white_list": [],
+    "black_list": [],
+    "error_data_path": "./"
+  }
+}
+```
+
+#### Outputs
+
+Dump files are written into `dump_path` you defined. They usually contain:
+
+- `dump.json`, which records metadata such as dtype, shape, min, max, mean, L2 norm, and `requires_grad`.
+- `construct.json`, hierarchical structure description, when `level` is `L0` or `mix` (required for visualization), its
+  content is not empty.
+- `stack.json`, record the call stack information of API/Module.
+- `dump_tensor_data`, generated when `task` is `tensor` and save the collected tensor data.
+
+See [dump directory description](#dump-directory-description) for details.
+
+> **Note**: When MSProbe is enabled, cuda graph is disabled (disable_cuda_graph=True) because MSProbe only supports dump
+> in eager mode, warmup is disabled (skip_server_warmup=True) because there is no need to dump data for this stage.
+
+## End-to-End Examples
+
+MSProbe’s full debugging workflow follows **Enable → Collect Data → Visualize → Analyze Root Cause**. Below is a common
+E2E example for SGLang-based model inference debugging.
+
+### Example : Advanced Debugging with Custom Configuration
+
+Suitable for targeted debugging (e.g., only collect statistics data for specific ranks/steps, enable mix level for graph
+reconstruction + numerical comparison) and root cause analysis via **problem vs. benchmark comparison**.
+
+#### Step 1: Enable
+##### Prepare Custom Configuration JSON
+
+Create `msprobe-config.json` (dump statistics data for rank0/1, step0/1, mix level):
+
+```json
+{
+  "task": "statistics",
+  "dump_path": "./problem_dump",
+  "rank": [
+    0,
+    1
+  ],
+  "step": [
+    0,
+    1
+  ],
+  "level": "mix",
+  "async_dump": false,
+  "statistics": {
+    "scope": [],
+    "list": [],
+    "data_mode": [
+      "all"
+    ],
+    "summary_mode": "statistics"
+  }
+}
+```
+
+##### Enable MSProbe with Custom Configuration in SGLang
+
+Launch the SGLang server and specify the configuration file path with `--msprobe-dump-config`:
+
+```bash
+python3 -m sglang.launch_server \
+ --model-path Qwen/Qwen2.5-0.5B-Instruct \
+ --host 127.0.0.1 \
+ --port 1027 \
+ --msprobe-dump-config /home/msprobe-config.json
+```
+#### Step 2: Collect Data
+##### Collect Dump Data for Problem & Benchmark Sides
+
+Send normal inference requests to trigger model running (MSProbe automatically collects data during request processing):
+
+```bash
+curl -H "Content-type: application/json" \
+ -X POST \
+ -d '{
+     "model": "Qwen/Qwen2.5-0.5B-Instruct",
+     "messages": [
+         {
+             "role": "user",
+             "content": "Hello, my name is"
+         }
+     ],
+     "max_tokens": 10
+ }' \
+ http://127.0.0.1:1027/v1/chat/completions
+```
+
+- **Problem side**: Run the above SGLang server (with the accuracy/numerical issue) and send inference request; dump
+  data is saved to `./problem_dump`.
+- **Benchmark side**: Launch a normal SGLang server (without the issue, e.g., stable framework version/operator) with
+  the **same custom configuration** and send the **same inference request**; rename the dump directory
+  to `./bench_dump`.
+
+> **Key Requirement**: Problem and benchmark dumps must use the same inputs and sampling points (rank/step)
+> for valid comparison.
+
+##### Check Generated Dump Files
+
+Dump files are saved to `./problem_dump` and `./bench_dump` you defined and include core files for subsequent analysis:
+
+- `dump.json`: Records tensor metadata of APIs and modules (dtype, shape, min/max/mean, L2 norm, `requires_grad`, etc.).
+- `stack.json`: Logs call stack information of APIs and modules.
+- `construct.json`: hierarchical structure description, required for visualization, its content is not empty.
+
+#### Step 3: Visualize
+##### Visualize Problem vs. Benchmark Comparison (Multi-Rank)
+
+Generate a multi-rank comparison visualization file (mix level generates `construct.json` for graph reconstruction):
+
+```shell
+msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output
+```
+
+- `-tp`: Path to problem-side dump data
+- `-gp`: Path to benchmark-side dump data
+- `-o`: Output directory for visualization files
+
+If you want overflow check (for NaN/Inf detection), please specify the parameter `-oc`
+
+```shell
+msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output -oc
+```
+
+After the comparison or build task finishes, a `compare_{timestamp}.vis.db` file is created under `graph_output`.
+
+The dumped data can be used to visualize and analyze differences using tables or charts generated with visualization tools such as Matplotlib and Excel.
+
+##### Launch TensorBoard
+
+Start TensorBoard:
+```bash
+tensorboard --logdir ./graph_output --bind_all --port 6006
+```
+#### Step 4: Analyze Root Cause
+##### Locate Root Cause
+
+Root Cause Analysis in TensorBoard:
+- Divergent nodes (with accuracy/numerical differences) are highlighted in **red** (darker red = larger difference).
+- Click on divergent nodes to view detailed tensor data (inputs/outputs, parameters) and API/module call stacks.
+- Use the **search/filter** function to quickly locate key layers/APIs (e.g., "relu", "conv").
+- Switch between ranks/steps via the UI to check cross-rank/cross-step divergence.
+- Check the **overflow check** tab for NaN/Inf values in specific nodes (the direct cause of numerical instability).
+
+##### Verify the Root Cause
+
+After locating the divergent node (e.g., a specific Conv layer or torch API with abnormal tensor values), verify by:
+
+- Narrowing the dump scope to this node (via `scope`/`list` in the configuration file) for fine-grained data collection.
+- Modifying the problematic layer/API (e.g., replacing the operator, adjusting the dtype) and re-running the debugging
+  workflow to confirm the issue is resolved.
+
+## Troubleshooting
+
+### No Dump Files Generated
+
+1. To confirm if MSProbe is installed, use `pip show mindstudio_probe` to troubleshoot. If it is installed, the MSProbe
+   version information will be printed. If it is confirmed that it has not been installed, please
+   use `pip install mindstudio-probe --pre` for installation;
+2. Confirm the `--msprobe-dump-config` parameter points to the **correct JSON file path**.
+
+### Dump Files Are Too Large (Excessive Data)
+
+1. Start with `task: "statistics"` instead of `"tensor"` to collect only tensor statistics (avoids raw tensor dump);
+2. Narrow the dump range with the `scope` field (specify start/end module/API);
+3. Filter dump targets with the `list` field (only dump specific modules/APIs or substrings);
+4. Sample specific `rank` and `step` (avoid dumping all ranks/iterations).
+
+### TensorBoard Visualization Fails
+
+1. Confirm `construct.json` is not empty (requires `level: L0` or `mix` – L1 does not generate graph files);
+2. Check that the `-tp` (problem dump) and `-gp` (benchmark dump) paths point to **valid rank/step subdirectories** (
+   e.g., `step0/rank0`);
+3. Ensure the MSProbe version is up-to-date (reinstall with `pip install mindstudio-probe --pre --upgrade`);
+4. Verify TensorBoard is installed and the `--logdir` parameter points to the directory containing `.vis.db` files (not
+   the file itself).
+
+### Numerical Comparison Shows No Divergence But Model Accuracy Is Low
+
+1. Expand the dump `step` range (check more token iterations for late-stage divergence);
+2. Switch to `task: "tensor"` (statistics may mask subtle numerical differences in raw tensor data);
+3. Ensure the problem and benchmark dumps use **the same input data/hardware configuration** (different inputs lead to
+   invalid comparisons);
+4. Use the `manual mapping` feature in TensorBoard (automatic mapping may miss some nodes for custom models).
+
+---
+
+## Appendix
+
+### Dump directory description
+
+```text
+├── problem_dump or bench_dump
+│   ├── step0
+│   │   ├── rank0
+│   │   │   ├── dump_tensor_data
+│   │   │   │    ├── Tensor.permute.1.forward.pt
+│   │   │   │    ├── Functional.linear.5.backward.output.pt    # Format: {api_type}.{api_name}.{call_count}.{forward/backward}.{input/output}.{arg_index}.
+│   │   │   │    │                                              # arg_index is the nth input or output of the API. If an input is a list, keep numbering with decimals (e.g., 1.1 is the first element of the first argument).
+│   │   │   │    ├── Module.conv1.Conv2d.forward.0.input.0.pt          # Format: {Module}.{module_name}.{class_name}.{forward/backward}.{call_count}.{input/output}.{arg_index}.
+│   │   │   │    ├── Module.conv1.Conv2d.forward.0.parameters.bias.pt  # Module parameter data: {Module}.{module_name}.{class_name}.forward.{call_count}.parameters.{parameter_name}.
+│   │   │   │    └── Module.conv1.Conv2d.parameters_grad.weight.pt     # Module parameter gradients: {Module}.{module_name}.{class_name}.parameters_grad.{parameter_name}. Gradients do not include call_count because the same gradient updates all invocations.
+│   │   │   │                                                          # When the `model` argument passed to dump is a List[torch.nn.Module] or Tuple[torch.nn.Module], module-level data names also include the index inside the list ({Module}.{index}.*), e.g., Module.0.conv1.Conv2d.forward.0.input.0.pt.
+│   │   │   ├── dump.json
+│   │   │   ├── stack.json
+│   │   │   ├── dump_error_info.log
+│   │   │   └── construct.json
+│   │   ├── rank1
+│   │   │   ├── dump_tensor_data
+│   │   │   │   └── ...
+│   │   │   ├── dump.json
+│   │   │   ├── stack.json
+│   │   │   ├── dump_error_info.log
+│   │   │   └── construct.json
+│   │   ├── ...
+│   │   │
+│   │   └── rank7
+│   ├── step1
+│   │   ├── ...
+│   ├── step2
+```
+
+- `rank`: Device ID. Each card writes its data to the corresponding `rank{ID}` directory. In non-distributed scenarios
+  the directory is simply named `rank`.
+- `dump_tensor_data`: Save the collected tensor data.
+- `dump.json`: Statistics for the forward data of each API or module, including names, dtype, shape, max, min, mean, L2
+  norm (square root of the L2 variance), and CRC-32 when `summary_mode="md5"`.
+  See [dump.json file description](#dumpjson-file-description) for details.
+- `dump_error_info.log`: Present only when the dump tool encountered an error and records the failure log.
+- `stack.json`: Call stacks for APIs/modules.
+- `construct.json`: Hierarchical structure description. Empty when `level=L1`.
+
+### dump.json file description
+
+#### L0 level
+
+An L0 `dump.json` contains forward/backward I/O for modules together with parameters and parameter gradients. Using
+PyTorch's `Conv2d` as an example, the network code looks like:
+
+`output = self.conv2(input)  # self.conv2 = torch.nn.Conv2d(64, 128, 5, padding=2, bias=True)`
+
+`dump.json` contains the following entries:
+
+- `Module.conv2.Conv2d.forward.0`: Forward data of the module. `input_args` represents positional inputs, `input_kwargs`
+  represents keyword inputs, `output` stores forward outputs, and `parameters` stores weights/biases.
+- `Module.conv2.Conv2d.parameters_grad`: Parameter gradients (weight and bias).
+- `Module.conv2.Conv2d.backward.0`: Backward data of the module. `input` represents gradients that flow into the
+  module (gradients of the forward outputs) and `output` represents gradients that flow out (gradients of the module
+  inputs).
+
+**Note**: When the `model` parameter passed to the dump API is `List[torch.nn.Module]` or `Tuple[torch.nn.Module]`,
+module-level names include the index inside the list (`{Module}.{index}.*`). Example: `Module.0.conv1.Conv2d.forward.0`.
+
+<details>
+
+<summary>L0 dump.json</summary>
+
+```json
+{
+  "task": "tensor",
+  "level": "L0",
+  "framework": "pytorch",
+  "dump_data_dir": "/dump/path",
+  "data": {
+    "Module.conv2.Conv2d.forward.0": {
+      "input_args": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            8,
+            16,
+            14,
+            14
+          ],
+          "Max": 1.638758659362793,
+          "Min": 0.0,
+          "Mean": 0.2544615864753723,
+          "Norm": 70.50277709960938,
+          "requires_grad": true,
+          "data_name": "Module.conv2.Conv2d.forward.0.input.0.pt"
+        }
+      ],
+      "input_kwargs": {},
+      "output": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            8,
+            32,
+            10,
+            10
+          ],
+          "Max": 1.6815717220306396,
+          "Min": -1.5120246410369873,
+          "Mean": -0.025344856083393097,
+          "Norm": 149.65576171875,
+          "requires_grad": true,
+          "data_name": "Module.conv2.Conv2d.forward.0.output.0.pt"
+        }
+      ],
+      "parameters": {
+        "weight": {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32,
+            16,
+            5,
+            5
+          ],
+          "Max": 0.05992485210299492,
+          "Min": -0.05999220535159111,
+          "Mean": -0.0006165213999338448,
+          "Norm": 3.421217441558838,
+          "requires_grad": true,
+          "data_name": "Module.conv2.Conv2d.forward.0.parameters.weight.pt"
+        },
+        "bias": {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32
+          ],
+          "Max": 0.05744686722755432,
+          "Min": -0.04894155263900757,
+          "Mean": 0.006410328671336174,
+          "Norm": 0.17263513803482056,
+          "requires_grad": true,
+          "data_name": "Module.conv2.Conv2d.forward.0.parameters.bias.pt"
+        }
+      }
+    },
+    "Module.conv2.Conv2d.parameters_grad": {
+      "weight": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32,
+            16,
+            5,
+            5
+          ],
+          "Max": 0.018550323322415352,
+          "Min": -0.008627401664853096,
+          "Mean": 0.0006675920449197292,
+          "Norm": 0.26084786653518677,
+          "requires_grad": false,
+          "data_name": "Module.conv2.Conv2d.parameters_grad.weight.pt"
+        }
+      ],
+      "bias": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32
+          ],
+          "Max": 0.014914230443537235,
+          "Min": -0.006656786892563105,
+          "Mean": 0.002657240955159068,
+          "Norm": 0.029451673850417137,
+          "requires_grad": false,
+          "data_name": "Module.conv2.Conv2d.parameters_grad.bias.pt"
+        }
+      ]
+    },
+    "Module.conv2.Conv2d.backward.0": {
+      "input": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            8,
+            32,
+            10,
+            10
+          ],
+          "Max": 0.0015069986693561077,
+          "Min": -0.001139344065450132,
+          "Mean": 3.3215508210560074e-06,
+          "Norm": 0.020567523315548897,
+          "requires_grad": false,
+          "data_name": "Module.conv2.Conv2d.backward.0.input.0.pt"
+        }
+      ],
+      "output": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            8,
+            16,
+            14,
+            14
+          ],
+          "Max": 0.0007466732058674097,
+          "Min": -0.00044813455315306783,
+          "Mean": 6.814070275140693e-06,
+          "Norm": 0.01474067009985447,
+          "requires_grad": false,
+          "data_name": "Module.conv2.Conv2d.backward.0.output.0.pt"
+        }
+      ]
+    }
+  }
+}
+```
+
+</details>
+
+#### L1 level
+
+An L1 `dump.json` records forward/backward I/O for APIs. Using PyTorch's `relu` function as an
+example (`output = torch.nn.functional.relu(input)`), the file contains:
+
+- `Functional.relu.0.forward`: Forward data of the API. `input_args` are positional inputs, `input_kwargs` are keyword
+  inputs, and `output` stores the forward outputs.
+- `Functional.relu.0.backward`: Backward data of the API. `input` represents the gradients of the forward outputs,
+  and `output` represents the gradients that flow back to the forward inputs.
+
+<details>
+
+<summary>L1 dump.json</summary>
+
+```json
+{
+  "task": "tensor",
+  "level": "L1",
+  "framework": "pytorch",
+  "dump_data_dir": "/dump/path",
+  "data": {
+    "Functional.relu.0.forward": {
+      "input_args": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32,
+            16,
+            28,
+            28
+          ],
+          "Max": 1.3864083290100098,
+          "Min": -1.3364859819412231,
+          "Mean": 0.03711778670549393,
+          "Norm": 236.20692443847656,
+          "requires_grad": true,
+          "data_name": "Functional.relu.0.forward.input.0.pt"
+        }
+      ],
+      "input_kwargs": {},
+      "output": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32,
+            16,
+            28,
+            28
+          ],
+          "Max": 1.3864083290100098,
+          "Min": 0.0,
+          "Mean": 0.16849493980407715,
+          "Norm": 175.23345947265625,
+          "requires_grad": true,
+          "data_name": "Functional.relu.0.forward.output.0.pt"
+        }
+      ]
+    },
+    "Functional.relu.0.backward": {
+      "input": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32,
+            16,
+            28,
+            28
+          ],
+          "Max": 0.0001815402356442064,
+          "Min": -0.00013352684618439525,
+          "Mean": 0.00011915402356442064,
+          "Norm": 0.007598237134516239,
+          "requires_grad": false,
+          "data_name": "Functional.relu.0.backward.input.0.pt"
+        }
+      ],
+      "output": [
+        {
+          "type": "torch.Tensor",
+          "dtype": "torch.float32",
+          "shape": [
+            32,
+            16,
+            28,
+            28
+          ],
+          "Max": 0.0001815402356442064,
+          "Min": -0.00012117840378778055,
+          "Mean": 2.0098118724831693e-08,
+          "Norm": 0.006532244384288788,
+          "requires_grad": false,
+          "data_name": "Functional.relu.0.backward.output.0.pt"
+        }
+      ]
+    }
+  }
+}
+```
+
+</details>
+
+#### mix level
+
+A `mix` dump.json contains both L0 and L1 level data; the file format is the same as the examples above.

docs_new/docs/developer_guide/overview.mdx

@@ -0,0 +1,13 @@
+---
+title: Developer Guide
+description: Contributing to SGLang — development setup, benchmarking, and evaluation.
+---
+
+- [Contribution Guide](./contribution_guide)
+- [Development Guide (Docker)](./development_guide_using_docker)
+- [JIT Kernels](./development_jit_kernel_guide)
+- [Quantization Contribution Guide](./quantization_contribution_guide)
+- [Benchmark and Profiling](./benchmark_and_profiling)
+- [Bench Serving](./bench_serving)
+- [Evaluating New Models](./evaluating_new_models)
+- [MSProbe Debugging Guide](./msprobe_debugging_guide)

docs_new/docs/developer_guide/quantization_contribution_guide.mdx

@@ -0,0 +1,96 @@
+---
+title: "Quantization Contribution Guide"
+metatags:
+    description: "Guidance for contributing SGLang quantization methods, schemes, backend kernels, tests, and validation results."
+---
+
+This guide describes how to add or refactor quantization support in SGLang. It focuses on the common structure used by weight-only and weight-activation quantization methods such as AWQ, GPTQ, compressed-tensors, ModelSlim, Quark, and related backend kernels.
+
+## Design Goals
+
+Quantization code should keep the quantization format semantics separate from hardware-specific execution. This makes it easier to add new formats, reuse kernels across formats, and review platform-specific changes independently.
+
+Follow the architecture proposed in [Quantization Modifications](https://github.com/sgl-project/sglang/issues/15194):
+
+- **Config**: parses model and runtime quantization parameters, validates supported options, and selects the proper scheme.
+- **Scheme**: owns quantized weight creation, weight loading, post-processing, and quantized layer wiring for Linear, MoE, embedding, or other module types.
+- **Backend kernel**: wraps hardware-specific execution, layout conversion, kernel selection, and kernel calls for GPU (CUDA/HIP/XPU), NPU, or other backends.
+
+Avoid putting config parsing, weight loading, and backend kernel calls in a single monolithic file. If a method needs multiple formats or backends, add a package under `python/sglang/srt/layers/quantization/<method>/` and split schemes into `schemes/`.
+
+## Recommended File Layout
+
+Use this layout for a quantization method that has multiple schemes or backend-specific execution paths:
+
+```text
+python/sglang/srt/layers/quantization/<method>/
+  __init__.py
+  <method>.py
+  schemes/
+    __init__.py
+    <method>_scheme.py
+    <method>_linear.py
+    <method>_moe.py
+    <method>_<variant>.py
+```
+
+Backend kernels should live under the hardware backend they target:
+
+```text
+python/sglang/srt/hardware_backend/gpu/quantization/<method>_kernels.py
+python/sglang/srt/hardware_backend/npu/quantization/<method>_kernels.py
+```
+
+Keep shared method selection in the quantization package and keep backend imports narrow. This prevents circular imports and keeps non-target platforms from importing unavailable kernel dependencies.
+
+## Adding or Refactoring a Quantization Method
+
+1. Define the config entry point and register it through `python/sglang/srt/layers/quantization/__init__.py` when needed.
+2. Add explicit scheme selection helpers such as `get_linear_scheme` and `get_moe_scheme`.
+3. Move layer-specific weight creation and weight loading into scheme classes.
+4. Move GPU (CUDA/HIP/XPU), NPU, or other hardware kernel calls into backend kernel modules.
+5. Keep Linear, MoE, embedding, and non-linear module handling explicit. Do not assign a Linear quantization method to a module type that needs different semantics.
+6. Preserve compatibility for existing quantized checkpoints and runtime flags.
+7. Add tests that cover both config parsing and execution paths touched by the change.
+
+For examples, see the AWQ and GPTQ refactors:
+
+- [PR #21126](https://github.com/sgl-project/sglang/pull/21126): splits AWQ schemes, weight initialization, and backend kernel calls.
+- [PR #26402](https://github.com/sgl-project/sglang/pull/26402): applies the same scheme/kernel split to GPTQ.
+
+## Tests and Validation
+
+Quantization changes can affect both accuracy and performance. Include validation that matches the blast radius of the change.
+
+For Python-only structure changes:
+
+```bash
+ruff check <changed-python-files>
+git diff --check
+```
+
+For quantized model behavior:
+
+- Launch at least one representative model for each touched quantization method.
+- Send a `/generate` request and confirm the output path succeeds.
+- Run an accuracy sanity test if the change can affect numerics.
+- Include warmup-aware benchmark results when the change affects kernel calls, layout conversion, or dispatch.
+
+For backend-specific changes:
+
+- Validate GPU changes on a supported GPU environment (NVIDIA, AMD, or Intel).
+- Validate NPU changes on a supported Ascend environment.
+- Include the exact model, quantization flag, backend flag, hardware, and command used in the PR description.
+
+## PR Checklist
+
+Before requesting review, make sure the PR description includes:
+
+- The quantization method and backend paths changed.
+- The issue, design proposal, or roadmap item the PR follows.
+- Any compatibility notes for existing checkpoints or flags.
+- Accuracy results when model outputs can change.
+- Benchmark or profiling results when runtime performance can change.
+- The exact local checks and model launch tests that were run.
+
+Use the general [Contribution Guide](./contribution_guide) for source setup, formatting, unit tests, CI triggering, and review process details.

docs_new/docs/developer_guide/release_process.mdx

@@ -0,0 +1,21 @@
+---
+title: "PyPI Package Release Process"
+metatags:
+    description: "SGLang PyPI release: version update, upload_pypi.sh script, GitHub release creation."
+---
+## Update the version in code
+Update the package version in `python/pyproject.toml` and `python/sglang/__init__.py`.
+
+## Upload the PyPI package
+
+```text Output
+pip install build twine
+```
+
+```text Output
+cd python
+bash upload_pypi.sh
+```
+
+## Make a release in GitHub
+Make a new release https://github.com/sgl-project/sglang/releases/new.

docs_new/docs/developer_guide/setup_github_runner.mdx

@@ -0,0 +1,54 @@
+---
+title: "Set Up Self-Hosted Runners for GitHub Action"
+metatags:
+    description: "SGLang GitHub Actions self-hosted runner: Docker setup for NVIDIA/AMD GPUs, config.sh and run.sh."
+---
+## Add a Runner
+
+### Step 1: Start a docker container.
+
+**You can mount a folder for the shared huggingface model weights cache. **
+The command below uses `/tmp/huggingface` as an example.
+
+```
+docker pull nvidia/cuda:12.9.1-devel-ubuntu22.04
+# Nvidia
+docker run --shm-size 128g -it -v /tmp/huggingface:/hf_home --gpus all nvidia/cuda:12.9.1-devel-ubuntu22.04 /bin/bash
+# AMD
+docker run --rm --device=/dev/kfd --device=/dev/dri --group-add video --shm-size 128g -it -v /tmp/huggingface:/hf_home lmsysorg/sglang:v0.5.8-rocm700-mi30x /bin/bash
+# AMD just the last 2 GPUs
+docker run --rm --device=/dev/kfd --device=/dev/dri/renderD176 --device=/dev/dri/renderD184 --group-add video --shm-size 128g -it -v /tmp/huggingface:/hf_home lmsysorg/sglang:v0.5.8-rocm700-mi30x /bin/bash
+```
+
+### Step 2: Configure the runner by `config.sh`
+
+Run these commands inside the container.
+
+```text Output
+apt update && apt install -y curl python3-pip git
+pip install --upgrade pip
+export RUNNER_ALLOW_RUNASROOT=1
+```
+
+Then follow https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners to run `config.sh`
+
+**Notes**
+- Do not need to specify the runner group
+- Give it a name (e.g., `test-sgl-gpu-0`) and some labels (e.g., `1-gpu-h100`). The labels can be edited later in Github Settings.
+- Do not need to change the work folder.
+
+### Step 3: Run the runner by `run.sh`
+
+- Set up environment variables
+```text Output
+export HF_HOME=/hf_home
+export SGLANG_IS_IN_CI=true
+export HF_TOKEN=hf_xxx
+export OPENAI_API_KEY=sk-xxx
+export CUDA_VISIBLE_DEVICES=0
+```
+
+- Run it forever
+```text Output
+while true; do ./run.sh; echo "Restarting..."; sleep 2; done
+```

docs_new/docs/get-started/install.mdx

@@ -0,0 +1,241 @@
+---
+title: Installation
+description: Install SGLang with pip/uv, source, Docker, Kubernetes, and cloud deployment options.
+keywords:
+  - installation
+  - sglang
+  - pip
+  - docker
+---
+You can install SGLang using one of the methods below.
+This page primarily applies to common NVIDIA GPU platforms.
+For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../hardware-platforms/amd_gpu), [Apple Metal](../hardware-platforms/apple_metal), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPU](../hardware-platforms/tpu), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/ascend_npu), and [Intel XPU](../hardware-platforms/xpu).
+
+<Note>
+Prerequisites: Python 3.10 or higher.
+</Note>
+
+## Method 1: With pip or uv
+
+It is recommended to use uv for faster installation:
+
+```bash Command
+pip install --upgrade pip
+pip install uv
+uv pip install --prerelease=allow sglang
+```
+
+The major version of Cuda is 13 by default. To install sglang under Cuda 12 with pip or uv, please try the following commands:
+```bash Command
+pip install --upgrade pip
+pip install uv
+uv pip install --prerelease=allow sglang
+uv pip install --force-reinstall  torch==2.11.0 torchaudio==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu129
+uv pip install --force-reinstall sglang-kernel --index-url https://docs.sglang.ai/whl/cu129/
+uv pip install --force-reinstall sgl-deep-gemm --index-url https://docs.sglang.ai/whl/cu129/ --no-deps
+```
+
+### Nightly builds
+
+To pick up the latest features and fixes before the next stable release, install a nightly build. Nightly wheels are built from the latest `main` and published to the SGLang wheel index. Add that index with `--extra-index-url`, and combine `--prerelease=allow` with `--index-strategy unsafe-best-match` so uv considers the nightly (pre-release) version alongside PyPI:
+
+```bash Command
+pip install --upgrade pip
+pip install uv
+uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
+```
+
+To install a nightly build under Cuda 12, swap the index to `cu129`:
+```bash Command
+pip install --upgrade pip
+pip install uv
+uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu129/ sglang
+```
+
+### Quick fixes to common problems
+- If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions:
+  1. Use `export CUDA_HOME=/usr/local/cuda-<your-cuda-version>` to set the `CUDA_HOME` environment variable.
+  2. Install FlashInfer first following [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above.
+
+## Method 2: From source
+
+```bash Command
+# Use the last release branch
+git clone -b v0.5.12 https://github.com/sgl-project/sglang.git
+cd sglang
+
+# Install the python packages
+pip install --upgrade pip
+pip install -e "python"
+```
+
+**Quick fixes to common problems**
+
+- If you want to develop SGLang, you can try the dev docker image. Please refer to [setup docker container](../developer_guide/development_guide_using_docker#setup-docker-container). The docker image is `lmsysorg/sglang:dev`.
+
+## Method 3: Using docker
+
+The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
+Replace `<secret>` below with your huggingface hub [token](https://huggingface.co/docs/hub/en/security-tokens).
+
+<Note>
+`latest` and `dev` are **mutable** tags: `latest` always points at the newest stable release, while `dev` is rebuilt daily from the latest `main` and includes build/development tools. Because they are overwritten over time, pin an immutable version tag for reproducible deployments — e.g. `lmsysorg/sglang:v0.5.12`. Browse all released versions on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags).
+</Note>
+
+```bash Command
+docker run --gpus all \
+    --shm-size 32g \
+    -p 30000:30000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    --env "HF_TOKEN=<secret>" \
+    --ipc=host \
+    lmsysorg/sglang:latest \
+    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
+```
+
+For production deployments, use the `runtime` variant which is significantly smaller (~40% reduction) by excluding build tools and development dependencies:
+
+```bash Command
+docker run --gpus all \
+    --shm-size 32g \
+    -p 30000:30000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    --env "HF_TOKEN=<secret>" \
+    --ipc=host \
+    lmsysorg/sglang:latest-runtime \
+    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
+```
+
+You can also find the nightly docker images [here](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
+
+Notes:
+- SGLang is shipped with CUDA 13 environment by default. To run SGLang on CUDA 12 environment, please use images with `-cu12` or `-cu129` suffix, such as `lmsysorg/sglang:latest-cu129` or `lmsysorg/sglang:dev-cu12`.
+
+## Method 4: Using Kubernetes
+
+Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs).
+
+<Accordion title="More">
+
+1. Option 1: For single node serving (typically when the model size fits into GPUs on one node)
+
+   Execute command `kubectl apply -f docker/k8s-sglang-service.yaml`, to create k8s deployment and service, with llama-31-8b as example.
+
+2. Option 2: For multi-node serving (usually when a large model requires more than one GPU node, such as `DeepSeek-R1`)
+
+   Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service.
+
+</Accordion>
+
+## Method 5: Using docker compose
+
+<Accordion title="More">
+
+> This method is recommended if you plan to serve it as a service.
+> A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
+
+1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine
+2. Execute the command `docker compose up -d` in your terminal.
+
+</Accordion>
+
+## Method 6: Run on Kubernetes or Clouds with SkyPilot
+
+<Accordion title="More">
+
+To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).
+
+1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
+2. Deploy on your own infra with a single command and get the HTTP API endpoint:
+
+<Accordion title={<>SkyPilot YAML: <code>sglang.yaml</code></>}>
+
+```yaml Config
+# sglang.yaml
+envs:
+  HF_TOKEN: null
+
+resources:
+  image_id: docker:lmsysorg/sglang:latest
+  accelerators: A100
+  ports: 30000
+
+run: |
+  conda deactivate
+  python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+</Accordion>
+
+```bash Command
+# Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
+HF_TOKEN=<secret> sky launch -c sglang --env HF_TOKEN sglang.yaml
+
+# Get the HTTP API endpoint
+sky status --endpoint 30000 sglang
+```
+
+3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).
+
+</Accordion>
+
+## Method 7: Run on AWS SageMaker
+
+<Accordion title="More">
+
+To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy)
+
+Amazon Web Services provide supports for SGLang containers along with routine security patching. For available SGLang containers, check out [AWS SGLang DLCs](https://aws.github.io/deep-learning-containers/reference/available_images/#sglang).
+
+To deploy a pre-built SGLang Deep Learning Container without building your own image, see [Amazon SageMaker AI](/docs/basic_usage/aws_sagemaker).
+
+To host a model with your own container, follow the following steps:
+
+1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script.
+2. Push your container onto AWS ECR.
+
+<Accordion title={<>Dockerfile Build Script: <code>build-and-push.sh</code></>}>
+
+```bash Command
+#!/bin/bash
+AWS_ACCOUNT="<YOUR_AWS_ACCOUNT>"
+AWS_REGION="<YOUR_AWS_REGION>"
+REPOSITORY_NAME="<YOUR_REPOSITORY_NAME>"
+IMAGE_TAG="<YOUR_IMAGE_TAG>"
+
+ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
+IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}"
+
+echo "Starting build and push process..."
+
+# Login to ECR
+echo "Logging into ECR..."
+aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY}
+
+# Build the image
+echo "Building Docker image..."
+docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile .
+
+echo "Pushing ${IMAGE_URI}"
+docker push ${IMAGE_URI}
+
+echo "Build and push completed successfully!"
+```
+
+</Accordion>
+
+3. Deploy a model for serving on AWS Sagemaker, refer to [deploy_and_serve_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk).
+   1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker.
+   2. To modify your model serving parameters, the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script allows for all available options within `python3 -m sglang.launch_server --help` cli by specifying environment variables with prefix `SM_SGLANG_`.
+   3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli.
+   4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`.
+
+</Accordion>
+
+## Common Notes
+
+- [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub.
+- To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`.

docs_new/docs/get-started/quickstart.mdx

@@ -0,0 +1,332 @@
+---
+title: "Quickstart"
+description: "Get up and running with SGLang in minutes: install, launch a server, and send your first request."
+---
+
+## Overview
+
+This guide walks you through the entire flow of getting started with SGLang:
+
+1. **Install** SGLang
+2. **Launch** an inference server
+3. **Send requests** using cURL, OpenAI Python client, Python `requests`, or the native SGLang API
+
+By the end, you'll have a working SGLang server responding to your prompts.
+
+---
+
+## Prerequisites
+
+- **Python**: 3.10 or higher
+- **GPU**: NVIDIA GPU with CUDA support (sm80 and above, e.g., A10, A100, L4, L40S, H100)
+- **OS**: Linux (recommended)
+
+<Note>
+For other platforms, see the dedicated guides for [AMD GPUs](../hardware-platforms/amd_gpu), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPUs](../hardware-platforms/tpu), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/ascend_npu), and [Intel XPU](../hardware-platforms/xpu).
+</Note>
+
+---
+
+## Installation
+
+<Tabs>
+  <Tab title="Pip / uv (Recommended)">
+    We recommend using **uv** for faster installation:
+
+```bash
+pip install --upgrade pip
+pip install uv
+uv pip install --prerelease=allow sglang
+```
+  </Tab>
+  <Tab title="From Source">
+```bash
+# Clone and install from source
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+pip install --upgrade pip
+pip install -e "python"
+```
+  </Tab>
+  <Tab title="Docker">
+    The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags).
+
+    Replace `<secret>` with your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens):
+
+    ```bash
+    docker run --gpus all \
+        --shm-size 32g \
+        -p 30000:30000 \
+        -v ~/.cache/huggingface:/root/.cache/huggingface \
+        --env "HF_TOKEN=<secret>" \
+        --ipc=host \
+        lmsysorg/sglang:latest \
+        python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
+    ```
+
+    For production deployments, use the smaller **runtime** variant (~40% size reduction):
+
+    ```bash
+    docker run --gpus all \
+        --shm-size 32g \
+        -p 30000:30000 \
+        -v ~/.cache/huggingface:/root/.cache/huggingface \
+        --env "HF_TOKEN=<secret>" \
+        --ipc=host \
+        lmsysorg/sglang:latest-runtime \
+        python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
+    ```
+  </Tab>
+</Tabs>
+
+<Tip>
+If you encounter `OSError: CUDA_HOME environment variable is not set`, set it with:
+```bash
+export CUDA_HOME=/usr/local/cuda-<your-cuda-version>
+```
+</Tip>
+
+---
+
+## Launch a Server
+
+Start the SGLang server with a model. Here we use `qwen/qwen2.5-0.5b-instruct` as a lightweight example:
+
+```bash
+python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000
+```
+
+Wait until you see `The server is fired up and ready to roll!` in the terminal output.
+
+<Note>
+Once the server is running, API documentation is available at:
+- **Swagger UI**: `http://localhost:30000/docs`
+- **ReDoc**: `http://localhost:30000/redoc`
+- **OpenAPI Spec**: `http://localhost:30000/openapi.json`
+</Note>
+
+<Info>
+The server automatically applies the chat template from the Hugging Face tokenizer. You can override it with `--chat-template` when launching.
+</Info>
+
+---
+
+## Send Requests
+
+SGLang is fully **OpenAI API-compatible**, so you can use the same tools and libraries you already know.
+
+### Using cURL
+
+```bash
+curl http://localhost:30000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "qwen/qwen2.5-0.5b-instruct",
+    "messages": [
+      {"role": "user", "content": "What is the capital of France?"}
+    ]
+  }'
+```
+
+### Using OpenAI Python Client
+
+Install the OpenAI Python library if you haven't:
+
+```bash
+pip install openai
+```
+
+Then send a request:
+
+```python Example
+import openai
+
+client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+)
+
+print(response.choices[0].message.content)
+```
+
+#### Streaming
+
+```python Example
+import openai
+
+client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
+
+response = client.chat.completions.create(
+    model="qwen/qwen2.5-0.5b-instruct",
+    messages=[
+        {"role": "user", "content": "List 3 countries and their capitals."},
+    ],
+    temperature=0,
+    max_tokens=64,
+    stream=True,
+)
+
+for chunk in response:
+    if chunk.choices[0].delta.content:
+        print(chunk.choices[0].delta.content, end="", flush=True)
+```
+
+### Using Python Requests
+
+```python Example
+import requests
+
+url = "http://localhost:30000/v1/chat/completions"
+
+data = {
+    "model": "qwen/qwen2.5-0.5b-instruct",
+    "messages": [{"role": "user", "content": "What is the capital of France?"}],
+}
+
+response = requests.post(url, json=data)
+print(response.json())
+```
+
+### Using the Native `/generate` API
+
+SGLang also provides a native `/generate` endpoint for more flexibility.
+
+```python Example
+import requests
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+    },
+)
+
+print(response.json())
+```
+
+#### Streaming with `/generate`
+
+```python Example
+import requests
+import json
+
+response = requests.post(
+    "http://localhost:30000/generate",
+    json={
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 32,
+        },
+        "stream": True,
+    },
+    stream=True,
+)
+
+prev = 0
+for chunk in response.iter_lines(decode_unicode=False):
+    chunk = chunk.decode("utf-8")
+    if chunk and chunk.startswith("data:"):
+        if chunk == "data: [DONE]":
+            break
+        data = json.loads(chunk[5:].strip("\n"))
+        output = data["text"]
+        print(output[prev:], end="", flush=True)
+        prev = len(output)
+```
+
+---
+
+## Offline Batch Inference (No Server)
+
+SGLang also supports offline batch inference using the `Engine` class directly -- no HTTP server required.
+
+```python Example
+import sglang as sgl
+
+llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
+
+prompts = [
+    "Hello, my name is",
+    "The president of the United States is",
+    "The capital of France is",
+    "The future of AI is",
+]
+
+sampling_params = {"temperature": 0.8, "top_p": 0.95}
+
+outputs = llm.generate(prompts, sampling_params)
+
+for prompt, output in zip(prompts, outputs):
+    print(f"Prompt: {prompt}\nGenerated text: {output['text']}\n")
+
+llm.shutdown()
+```
+
+---
+
+## Common Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="CUDA_HOME not set">
+    Set the `CUDA_HOME` environment variable to your CUDA install root:
+    ```bash
+    export CUDA_HOME=/usr/local/cuda-<your-cuda-version>
+    ```
+  </Accordion>
+  <Accordion title="FlashInfer issues on sm75+ devices">
+    Switch to alternative backends by adding these flags when launching the server:
+    ```bash
+    --attention-backend triton --sampling-backend pytorch
+    ```
+  </Accordion>
+  <Accordion title="Reinstalling FlashInfer">
+    ```bash
+    pip3 install --upgrade flashinfer-python --force-reinstall --no-deps
+    rm -rf ~/.cache/flashinfer
+    ```
+  </Accordion>
+  <Accordion title="ptxas error on B300/GB300 (sm_103a)">
+    ```bash
+    export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
+    ```
+  </Accordion>
+</AccordionGroup>
+
+---
+
+{/*
+WIP, TBD linked later
+## What's Next?
+
+<CardGroup cols={2}>
+  <Card title="OpenAI-Compatible APIs" href="../basic_usage/openai_api_completions">
+    Explore the full Chat Completions and Completions APIs, including multi-turn conversations.
+  </Card>
+  <Card title="Vision Language Models" href="../basic_usage/openai_api_vision">
+    Send image inputs alongside text using OpenAI-compatible vision APIs.
+  </Card>
+  <Card title="Sampling Parameters" href="../basic_usage/sampling_params">
+    Fine-tune generation with temperature, top-p, frequency penalty, and more.
+  </Card>
+  <Card title="Server Arguments" href="../advanced_features/server_arguments">
+    Customize server behavior with advanced launch arguments like tensor parallelism.
+  </Card>
+  <Card title="Structured Outputs" href="../advanced_features/structured_outputs">
+    Constrain model output to JSON, regex, or EBNF grammars.
+  </Card>
+  <Card title="Ollama-Compatible API" href="../basic_usage/ollama_api">
+    Use the familiar Ollama CLI and Python library with SGLang as the backend.
+  </Card>
+</CardGroup>
+*/}

docs_new/docs/hardware-platforms/amd_gpu.mdx

@@ -0,0 +1,196 @@
+---
+title: "AMD GPUs"
+---
+This document describes how to run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
+
+## System Configuration
+
+When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:
+
+- [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
+- [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
+- [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
+- [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
+- [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)
+
+**NOTE:** We strongly recommend reading these docs and guides entirely to fully utilize your system.
+
+Below are a few key settings to confirm or enable for SGLang:
+
+### Update GRUB Settings
+
+In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:
+
+```text GRUB Configuration
+pci=realloc=off iommu=pt
+```
+
+Afterward, run `sudo update-grub` (or your distro’s equivalent) and reboot.
+
+### Disable NUMA Auto-Balancing
+
+```bash Disable NUMA
+sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
+```
+
+You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).
+
+Again, please go through the entire documentation to confirm your system is using the recommended configuration.
+
+## Install SGLang
+
+You can install SGLang using one of the methods below.
+
+### Install from Source
+
+```bash Command
+# Use the last release branch
+git clone -b v0.5.12 https://github.com/sgl-project/sglang.git
+cd sglang
+
+# Compile sgl-kernel
+pip install --upgrade pip
+cd sgl-kernel
+python setup_rocm.py install
+
+# Install sglang python package along with diffusion support
+cd ..
+rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
+pip install -e "python[all_hip]"
+```
+
+### Install Using Docker (Recommended)
+
+The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
+
+The steps below show how to build and use an image.
+
+1. Build the docker image.
+   If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.
+
+   ```bash Command
+   docker build -t sglang_image -f rocm.Dockerfile .
+   ```
+
+2. Create a convenient alias.
+
+   ```bash Command
+   alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
+       --ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
+       --security-opt seccomp=unconfined \
+       -v $HOME/dockerx:/dockerx \
+       -v /data:/data'
+   ```
+
+   If you are using RDMA, please note that:
+     - `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
+     - You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
+
+3. Launch the server.
+
+   **NOTE:** Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
+
+   ```bash Command
+   drun -p 30000:30000 \
+       -v ~/.cache/huggingface:/root/.cache/huggingface \
+       --env "HF_TOKEN=<secret>" \
+       sglang_image \
+       python3 -m sglang.launch_server \
+       --model-path NousResearch/Meta-Llama-3.1-8B \
+       --host 0.0.0.0 \
+       --port 30000
+   ```
+
+4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.
+
+   ```bash Command
+   drun sglang_image \
+       python3 -m sglang.bench_serving \
+       --backend sglang \
+       --dataset-name random \
+       --num-prompts 4000 \
+       --random-input 128 \
+       --random-output 128
+   ```
+
+With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities.
+
+## Quantization on AMD GPUs
+
+The [Quantization documentation](../advanced_features/quantization#platform-compatibility) has a full compatibility matrix. The short version: FP8, AWQ, MXFP4, W8A8, GPTQ, compressed-tensors, Quark, and **petit_nvfp4** (NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel)) all work on AMD. Methods that depend on Marlin or NVIDIA-specific kernels (`awq_marlin`, `gptq_marlin`, `gguf`, `modelopt_fp8`, `modelopt_fp4`) do not.
+
+A few things to keep in mind:
+
+- FP8 works via Aiter or Triton. Pre-quantized FP8 models like DeepSeek-V3/R1 work out of the box.
+- AWQ uses Triton dequantization kernels on AMD. The faster Marlin path is not available.
+- MXFP4 requires CDNA3/CDNA4 and `SGLANG_USE_AITER=1`.
+- `petit_nvfp4` enables NVFP4 models (e.g., [Llama 3.3 70B FP4](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4)) on MI250/MI300X via [Petit](https://github.com/causalflow-ai/petit-kernel). Install with `pip install petit-kernel`; no `--quantization` flag needed when loading pre-quantized NVFP4 models.
+- `quark_int4fp8_moe` is an AMD-only online quantization method for MoE models on CDNA3/CDNA4.
+
+Several of these backends are accelerated by [Aiter](https://github.com/ROCm/aiter). Enable it with:
+
+```bash Command
+export SGLANG_USE_AITER=1
+```
+
+Example -- serving an AWQ model:
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4 \
+    --trust-remote-code \
+    --port 30000 --host 0.0.0.0
+```
+
+Example -- FP8 online quantization:
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
+    --quantization fp8 \
+    --port 30000 --host 0.0.0.0
+```
+
+## Examples
+
+### Running DeepSeek-V3
+
+The only difference when running DeepSeek-V3 is in how you start the server. Here's an example command:
+
+```bash Command
+drun -p 30000:30000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    --ipc=host \
+    --env "HF_TOKEN=<secret>" \
+    sglang_image \
+    python3 -m sglang.launch_server \
+    --model-path deepseek-ai/DeepSeek-V3 \ # <- here
+    --tp 8 \
+    --trust-remote-code \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.
+
+### Running Llama3.1
+
+Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server, shown by the following example command:
+
+```bash Command
+drun -p 30000:30000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    --ipc=host \
+    --env "HF_TOKEN=<secret>" \
+    sglang_image \
+    python3 -m sglang.launch_server \
+    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here
+    --tp 8 \
+    --trust-remote-code \
+    --host 0.0.0.0 \
+    --port 30000
+```
+
+### Warmup Step
+
+When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.

docs_new/docs/hardware-platforms/apple_metal.mdx

@@ -0,0 +1,124 @@
+---
+title: "Apple Silicon with Metal"
+metatags:
+    description: "Run SGLang on Apple Silicon using the Metal backend."
+---
+
+This document describes how run SGLang on Apple Silicon using [Metal (MLX)](https://opensource.apple.com/projects/mlx/). If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
+
+## Prerequisites
+
+Building the native Metal kernels in `sgl-kernel` requires the Apple
+toolchain (`clang++`, the Metal framework headers, and `xcrun`). These ship
+with the **Xcode Command Line Tools**, which cannot be installed via `pip`:
+
+```bash
+xcode-select --install
+```
+
+If you have the full Xcode app installed, the Command Line Tools are already
+available. You can verify with `xcode-select -p && xcrun --find metal`.
+
+## Install SGLang
+
+You can install SGLang using one of the methods below.
+
+### Install from Source
+
+```bash
+# Use the default branch
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+
+# Create and activate a virtual environment
+uv venv -p 3.12 sglang-metal
+source sglang-metal/bin/activate
+
+# (Optional) Compile sgl-kernel
+uv pip install --upgrade pip
+uv run sgl-kernel/setup_metal.py install
+
+# Install sglang python package along with diffusion support
+rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
+uv pip install -e "python[all_mps]"
+```
+
+## Launch of the Serving Engine
+
+Launch the server with:
+
+```bash
+SGLANG_USE_MLX=1 python -m sglang.launch_server \
+  --model <MODEL_ID_OR_PATH> \
+  --disable-cuda-graph \
+  --host 0.0.0.0
+```
+
+**Key Parameters Explained:**
+
+1. `SGLANG_USE_MLX=1` - Enables the use of MLX as the SGLang runtime backend (if disabled, SGLang will fall back to `torch.mps`, which has less support)
+2. `--disable-cuda-graph` - Disables usage of CUDA graph, which is not relevant for Apple Metal.
+3. `--disable-overlap-schedule` - Disables overlap scheduling (enabled/not present by default) achieved using MLX's `async_eval()`
+4. `SGLANG_MLX_USE_CUSTOM_ROPE=1` - Enables the optional custom Metal RoPE kernel. It is disabled by default, so the MLX backend uses the standard RoPE path unless you opt in for A/B testing.
+5. `SGLANG_MLX_FUSE_SWIGLU=1` - Enables the use of fused Swish-Gated Linear Unit kernel (disabled by default)
+6. `SGLANG_MLX_CLEAR_CACHE_STEPS=256` - Sets the number of decode steps before clearing the MLX cache (256 by default)
+
+## Quantization
+
+The MLX backend supports two quantization paths on Apple Silicon:
+
+1. **Pre-quantized HF repos.** Any `mlx-community/<model>-4bit` (or `-8bit`) repo loads directly through `mlx_lm.load(...)` — no extra flag needed.
+   ```bash
+   SGLANG_USE_MLX=1 python -m sglang.launch_server \
+     --model-path mlx-community/Qwen3-0.6B-4bit \
+     --disable-cuda-graph
+   ```
+2. **On-the-fly quantization.** For any fp16 model, pass `--quantization mlx_q4` or `--quantization mlx_q8` to have sglang quantize the weights at load time via `mlx_lm.utils.quantize_model` (group size 64, the mlx-community default). The quantized weights stay in process memory; the on-disk model is untouched.
+   ```bash
+   SGLANG_USE_MLX=1 python -m sglang.launch_server \
+     --model-path Qwen/Qwen3-0.6B \
+     --quantization mlx_q4 \
+     --disable-cuda-graph
+   ```
+   Expected log line:
+   ```
+   Quantizing MLX model on-the-fly: bits=4 group_size=64 (preset=mlx_q4)
+   Quantization complete in 0.13s — active mem: 1.11 GB -> 0.31 GB (71.9% reduction)
+   ```
+   The MLX backend silently ignores `--quantization mlx_q4` when the model is already quantized in its HF config (path 1), so the same flag is safe to pass either way.
+
+## Benchmarking with Requests
+
+`sglang.benchmark_one_batch` calls the synchronous prefill/decode methods directly without going through the scheduler and the overlap code path.
+
+`sglang.benchmark_offline_throughput` can toggle overlap scheduling as it uses the scheduler and the overlap code path by using the flag `--disable-overlap-schedule`.
+
+### Throughput Testing
+
+Basic synchronous one batch throughput:
+```bash
+SGLANG_USE_MLX=1 python -m sglang.bench_one_batch \
+  --model-path <MODEL_ID_OR_PATH> \
+  --disable-cuda-graph \
+  --tp-size 1 \
+  --batch-size 1 \
+  --input-len 60 \
+  --output-len 10
+```
+
+Synchronous offline throughput:
+```bash
+SGLANG_USE_MLX=1 python -m sglang.bench_offline_throughput \
+  --model-path <MODEL_ID_OR_PATH> \
+  --disable-cuda-graph \
+  --num-prompts 1 \
+  --disable-overlap-schedule
+```
+
+Asynchronous offline throughput:
+```bash
+SGLANG_USE_MLX=1 python -m sglang.bench_offline_throughput \
+  --model-path <MODEL_ID_OR_PATH> \
+  --disable-cuda-graph \
+  --num-prompts 1
+```

docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx

@@ -0,0 +1,222 @@
+---
+title: "Contribution Guide"
+mode: wide
+metatags:
+    description: "Set up the Ascend NPU development environment, run tests, build documentation, and open SGLang pull requests."
+---
+
+Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.
+
+## Install SGLang from Source
+
+### Prepare Environment
+
+Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](./ascend_npu) to install the necessary dependencies. We recommend [using docker](./ascend_npu#method-2-using-docker-image) to build the environment.
+
+### Fork and clone the repository
+
+**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.
+
+```bash
+git clone https://github.com/<your_user_name>/sglang.git
+# if you are using docker, the environment is already set up.
+cd sglang
+export PYTHONPATH=$PWD/python:$PYTHONPATH
+```
+
+## Format code with pre-commit
+
+We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:
+
+```bash
+pip3 install pre-commit
+pre-commit install
+pre-commit run --all-files
+```
+
+- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
+- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
+- Link checking with lychee is **enforced in CI**. By default, it is not blocking local commits.
+- To run local link checks manually, use: `pre-commit run --hook-stage manual lychee --all-files`.
+
+## Run and add tests
+
+All NPU tests are end-to-end (E2E) and require launching a server with real model weights.
+Tests live under [`test/registered/ascend/`](https://github.com/sgl-project/sglang/tree/main/test/registered/ascend), organized by model type and functionality:
+
+```
+ascend/
+├── llm_models/          # Per-model inference accuracy
+├── vlm_models/          # Vision-language models
+├── embedding_models/    # Embedding model tests
+├── rerank_models/       # Reranker model tests
+├── reward_models/       # Reward model tests
+├── interface/           # API correctness, function calling
+├── basic_function/      # Cache, sampling, quantization, etc.
+└── test_npu_memory_consumption.py
+```
+
+### Adding a test
+
+See [`test_npu_sampling_backend.py`](https://github.com/sgl-project/sglang/tree/main/test/registered/ascend/basic_function/backends/test_npu_sampling_backend.py) for a complete example. Key steps:
+
+1. Place your test file in the appropriate directory under `test/registered/ascend/`.
+2. Extend `CustomTestCase` (from `sglang.test.test_utils`) for CI retry support.
+3. Launch server with `popen_launch_server()` in `setUpClass` and clean up with `kill_process_tree()` in `tearDownClass`.
+4. Register your test with `register_npu_ci()`:
+   ```python
+   from sglang.test.ci.ci_register import register_npu_ci
+   register_npu_ci(est_time=400, suite="stage-b-test-1-npu-a2", nightly=False)
+   register_npu_ci(est_time=400, suite="nightly-1-npu-a3", nightly=True)
+   ```
+
+### Running tests locally
+
+```bash
+pytest test/registered/ascend/llm_models/test_npu_qwen3_0_6b.py -v
+```
+
+For detailed instructions, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
+
+### Register models for CI
+
+If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps:
+
+1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
+2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}".
+  If this is not the case, use following command on CI server:
+
+    ```bash
+    modelscope download \
+      --model {your_model_repo}/{your_model} \
+      --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
+    ```
+
+  > Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model.
+
+3. Add model to `python/sglang/test/ascend/test_ascend_utils.py` (use docker `"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"` path).
+
+## Write documentation
+
+We recommend new contributors start by writing documentation, which helps you quickly understand SGLang codebase.
+For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).
+
+## Test the accuracy
+
+If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
+
+```bash
+# Launch a server
+python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
+
+# Evaluate. --base-url must point at the server launched above.
+# The default SGLang server port is 30000; change it if you launched
+# the server with a different --port.
+python3 -m sglang.test.run_eval --base-url http://localhost:30000 --eval-name gsm8k --num-examples 200
+```
+
+Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
+This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
+Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.
+
+GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
+You can find additional accuracy eval examples in:
+- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/manual/eval/test_eval_accuracy_large.py)
+- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/manual/core/test_gpt_oss_1gpu.py)
+
+## Benchmark the speed
+
+Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling).
+
+## Requesting a review for merge
+
+You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
+You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
+Then your PR can be merged.
+
+## How to Trigger CI Tests
+
+We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
+Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
+
+**PR authors** can always use `/rerun-failed-ci` on their own PRs, even if they are not listed in `CI_PERMISSIONS.json`.
+
+For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
+
+- `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
+- `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**.
+- `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
+- `/rerun-stage <stage-name>`: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages.
+- `/rerun-test <test-spec> [<test-spec> ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `<test-spec>` is pytest-style `<file>::<TestClass>[.<test_method>]` (the `::TestClass` and `.<test_method>` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once).
+
+If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
+
+To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
+
+Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
+
+If you don’t have permission, please ask maintainers to trigger CI for you.
+
+### CI rate limits
+
+Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
+We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
+
+Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
+
+```yaml
+cool-down-minutes:
+  description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting"
+  type: number
+  default: 120
+```
+
+Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.
+
+## Code style guidance
+
+- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
+- Minimize device synchronization. Reduce expensive CPU-NPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
+- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
+  - A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible.
+- Make functions as pure as possible. Avoid in-place modification of arguments.
+- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_pp_mixin.py`)
+- In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file.
+- Keep tests run fast.
+  - If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
+  - If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
+  - Reuse server launches in your unit tests to make tests run faster.
+- Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python’s [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead.
+- When supporting new hardware or features, follow these guidelines:
+  - Do not drastically change existing code.
+  - Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_npu.py`).
+  - If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
+
+## How to update sgl-kernel-npu
+
+Sgl-kernel-npu is the separate kernel package for Ascend NPU, containing both Ascend C and Triton operators. It is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository.
+
+For detailed guidance on developing and integrating operators (Ascend C directory structure, PyTorch op registration, build, test, and code style), see the [Ascend NPU Operator Development Guide](./ascend_npu_operator_development).
+
+### Multi-PR workflow
+
+Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow:
+
+1. **Submit sgl-kernel-npu PR**: Add or modify operators in the sgl-kernel-npu repository following the operator development guide. Ensure all tests pass.
+2. **Bump sgl-kernel-npu version**: Update the version number. Merging triggers an automatic PyPI release. If not urgent, wait for a regular release (typically within one week).
+3. **Reference the new version in SGLang**:
+   - Update the `SGLANG_KERNEL_NPU_TAG` argument in [`docker/npu.Dockerfile`](https://github.com/sgl-project/sglang/blob/main/docker/npu.Dockerfile) to the new sgl-kernel-npu release tag.
+   - Use the new operator in SGLang code.
+
+## Tips for newcomers
+
+If you want to contribute but don’t have a specific idea in mind, pick issues labeled ["good first issue" or "help wanted"](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase.
+
+Also check out the following materials as startup guide:
+- [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang.
+- [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
+- [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance.
+
+If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
+
+Thank you for your interest in SGLang. Happy coding!

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx

@@ -0,0 +1,572 @@
+---
+title: SGLang installation with NPUs support
+description: "Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services."
+---
+You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
+
+## Component Version Mapping For SGLang
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Obtain Way</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HDK</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>25.5.2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://www.hiascend.com/hardware/firmware-drivers/community?product=1&model=30&cann=9.0.0&driver=Ascend+HDK+25.5.2">link</a></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CANN</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>9.0.0</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="#obtain-cann-image">Obtain Images</a></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Pytorch Adapter</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>26.0.0</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://gitcode.com/Ascend/pytorch/releases">link</a></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MemFabric</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0.8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install memfabric-hybrid==1.0.8`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Triton</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3.2.1.dev20260530</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install triton-ascend==3.2.1.dev20260530 \`<br/>`--extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \`<br/>`--trusted-host mirrors.huaweicloud.com`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang NPU Kernel</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2026.05.01.post3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://github.com/sgl-project/sgl-kernel-npu/releases/tag/2026.05.01.post3">link</a></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MemFabric-zbal</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.1.1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install memfabric-zbal==1.1.1`</td>
+    </tr>
+  </tbody>
+</table>
+
+### Obtain CANN Image
+
+<Warning>
+Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space.
+</Warning>
+
+You can obtain the dependency of a specified version of CANN through an image.
+
+<Tabs>
+  <Tab title="Atlas 800I A3">
+
+```bash Command
+docker pull quay.io/ascend/cann:9.0.0-a3-ubuntu22.04-py3.11
+```
+
+  </Tab>
+  <Tab title="Atlas 800I A2">
+
+```bash Command
+docker pull quay.io/ascend/cann:9.0.0-910b-ubuntu22.04-py3.11
+```
+
+  </Tab>
+</Tabs>
+
+## Preparing the Running Environment
+
+### Method 1: Installing from source with prerequisites
+
+#### Python Version
+
+**Only `python==3.11` is supported currently**. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda).
+
+```bash Command
+conda create --name sglang_npu python=3.11
+conda activate sglang_npu
+```
+
+Note on Anaconda repository restrictions
+If you encounter an error like “Terms of Service have not been accepted” during the conda create step, the default Anaconda repository is blocking package downloads. To resolve this, configure a mirror (e.g., Tsinghua Open Source Mirror):
+
+```bash Command
+# Add Tsinghua mirrors
+conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
+conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
+conda config --set show_channel_urls yes
+conda config --remove channels defaults
+```
+
+Edit the system-level conda config to remove any hardcoded defaults, e.g. vi ~/miniconda3/.condarc
+Then remove the failed environment and recreate it:
+
+```bash Command
+conda clean -i
+conda env remove -n sglang_npu
+conda create --name sglang_npu python=3.11
+conda activate sglang_npu
+```
+
+#### CANN
+
+Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 9.0.0, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0008.html?OS=openEuler&InstallType=local)
+
+#### MemFabric-Hybrid
+
+If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
+
+```bash Command
+pip install memfabric-hybrid==1.0.8
+```
+
+#### MemFabric-zbal
+
+MemFabric-zbal is a Zero Buffer Acceleration Library of high-performance operators for LLM inference and training on Ascend,
+accelerating computation by eliminating intermediate memory buffers; it is required only on **aarch64** clusters and is
+installed in addition to MemFabric-Hybrid.
+
+```bash Command
+# Only needed on aarch64 (arm64) hosts
+pip install memfabric-zbal==1.1.1
+```
+
+#### Pytorch and Pytorch Framework Adaptor on Ascend
+
+```bash Command
+PYTORCH_VERSION=2.10.0
+TORCHVISION_VERSION=0.25.0
+TORCH_NPU_VERSION=2.10.0
+pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
+pip install torch_npu==$TORCH_NPU_VERSION
+```
+
+If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md)
+
+#### Triton on Ascend
+
+We provide our own implementation of Triton for Ascend.
+
+```bash Command
+pip install triton-ascend==3.2.1.dev20260530 \
+  --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \
+  --trusted-host mirrors.huaweicloud.com
+```
+
+For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
+
+#### SGLang Kernels NPU
+
+We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md).
+
+#### DeepEP-compatible Library
+
+We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md).
+
+#### Some other dependencies
+
+```bash Command
+# libGL
+apt update
+apt install libgl1 libglib2.0-0
+
+# ensure setuptools contains pkg_resources module
+pip install "setuptools<80"
+```
+
+#### Installing SGLang from source
+
+```bash Command
+# Use the last release branch
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+mv python/pyproject_npu.toml python/pyproject.toml
+pip install -e python[all_npu]
+```
+
+### Method 2: Using Docker Image
+
+#### Obtain Image
+
+You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
+
+<Warning>
+Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
+</Warning>
+
+1. Download SGLang image
+
+<Tip>
+We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.13.post1-cann9.0.0-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes.
+</Tip>
+
+<Tabs>
+  <Tab title="Atlas 800I A3">
+
+```bash Command
+# Stable release
+docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3
+
+# Daily build
+docker pull quay.io/ascend/sglang:main-cann9.0.0-a3
+```
+
+  </Tab>
+  <Tab title="Atlas 800I A2">
+
+```bash Command
+# Stable release
+docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b
+
+# Daily build
+docker pull quay.io/ascend/sglang:main-cann9.0.0-910b
+```
+
+  </Tab>
+</Tabs>
+
+2. Build an image based on Dockerfile
+
+```bash Command
+# Clone the SGLang repository
+git clone https://github.com/sgl-project/sglang.git
+cd sglang/docker
+
+# Build the docker image
+# Replace <arch_tag> with the target architecture, e.g. amd64, arm64.
+# Optional build arguments:
+#   --build-arg DEVICE_TYPE=910b          # Required for Atlas 800I A2
+#   --build-arg APTMIRROR=<mirror_url>    # Use a custom APT mirror to improve download speed
+# If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV.
+docker build --build-arg TARGETARCH=<arch_tag> -t <image_name> -f npu.Dockerfile .
+```
+
+#### Create Docker
+
+__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
+
+<Tabs>
+  <Tab title="Atlas 800I A3">
+
+```bash Command
+# Create a shortcut 'drun' to launch a privileged Docker container
+alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
+    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
+    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
+    --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
+    --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
+    --device=/dev/davinci_manager --device=/dev/hisi_hdc \
+    --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
+    --volume /etc/ascend_install.info:/etc/ascend_install.info \
+    --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
+
+# Add HF_TOKEN env for download model by SGLang.
+# The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C)
+drun --env "HF_TOKEN=<secret>" \
+    <image_name> \
+    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
+```
+
+  </Tab>
+  <Tab title="Atlas 800I A2">
+
+```bash Command
+# Create a shortcut 'drun' to launch a privileged Docker container
+alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
+    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
+    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
+    --device=/dev/davinci_manager --device=/dev/hisi_hdc \
+    --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
+    --volume /etc/ascend_install.info:/etc/ascend_install.info \
+    --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
+
+# Add HF_TOKEN env for download model by SGLang.
+# The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C)
+drun --env "HF_TOKEN=<secret>" \
+    <image_name> \
+    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
+```
+
+  </Tab>
+</Tabs>
+
+<Tip>
+SGLang will serve on `http://127.0.0.1:30000` by default. You can change the host and port by `--host` and `--port` parameters.
+</Tip>
+
+## System Settings
+
+### CPU performance power scheme
+
+The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
+
+```bash Command
+echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+
+# Make sure changes are applied successfully
+cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
+```
+
+### Disable NUMA balancing
+
+```bash Command
+sudo sysctl -w kernel.numa_balancing=0
+# Check
+cat /proc/sys/kernel/numa_balancing # shows 0
+```
+
+### Prevent swapping out system memory
+
+```bash Command
+sudo sysctl -w vm.swappiness=10
+
+# Check
+cat /proc/sys/vm/swappiness # shows 10
+```
+
+## Running SGLang Service
+
+### Running Service For Large Language Models
+
+#### PD Mixed Scene
+
+```bash Command
+# Enabling CPU Affinity
+export SGLANG_SET_CPU_AFFINITY=1
+python3 -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.1-8B-Instruct \
+  --attention-backend ascend \
+  --host 127.0.0.1 \
+  --port 8000
+```
+
+#### PD Disaggregation Scene
+
+1. Launch Prefill Server
+
+<Tabs>
+  <Tab title="Atlas 800I A3">
+
+```bash Command
+# Enabling CPU Affinity
+export SGLANG_SET_CPU_AFFINITY=1
+
+# PREFILL_IP: IP address of the first Prefill Server
+# FREE_PORT: any available port
+# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
+export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --disaggregation-mode prefill \
+    --disaggregation-transfer-backend ascend \
+    --disaggregation-bootstrap-port 8995 \
+    --attention-backend ascend \
+    --device npu \
+    --base-gpu-id 0 \
+    --tp-size 1 \
+    --host 127.0.0.1 \
+    --port 8000
+```
+
+  </Tab>
+  <Tab title="Atlas 800I A2">
+
+```bash Command
+# Enabling CPU Affinity
+export SGLANG_SET_CPU_AFFINITY=1
+
+# PREFILL_IP: IP address of the first Prefill Server
+# FREE_PORT: any available port
+# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
+export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
+export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --disaggregation-mode prefill \
+    --disaggregation-transfer-backend ascend \
+    --disaggregation-bootstrap-port 8995 \
+    --attention-backend ascend \
+    --device npu \
+    --base-gpu-id 0 \
+    --tp-size 1 \
+    --host 127.0.0.1 \
+    --port 8000
+```
+
+  </Tab>
+</Tabs>
+
+2. Launch Decode Server
+
+<Tabs>
+  <Tab title="Atlas 800I A3">
+
+```bash Command
+# PREFILL_IP: IP address of the first Prefill Server
+# FREE_PORT: any available port
+# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
+export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --disaggregation-mode decode \
+    --disaggregation-transfer-backend ascend \
+    --attention-backend ascend \
+    --device npu \
+    --base-gpu-id 1 \
+    --tp-size 1 \
+    --host 127.0.0.1 \
+    --port 8001
+```
+
+  </Tab>
+  <Tab title="Atlas 800I A2">
+
+```bash Command
+# PREFILL_IP: IP address of the first Prefill Server
+# FREE_PORT: any available port
+# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
+export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
+export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
+python3 -m sglang.launch_server \
+    --model-path meta-llama/Llama-3.1-8B-Instruct \
+    --disaggregation-mode decode \
+    --disaggregation-transfer-backend ascend \
+    --attention-backend ascend \
+    --device npu \
+    --base-gpu-id 1 \
+    --tp-size 1 \
+    --host 127.0.0.1 \
+    --port 8001
+```
+
+  </Tab>
+</Tabs>
+
+3. Launch Router
+
+```bash Command
+python3 -m sglang_router.launch_router \
+    --pd-disaggregation \
+    --policy cache_aware \
+    --prefill http://127.0.0.1:8000 8995 \
+    --decode http://127.0.0.1:8001 \
+    --host 127.0.0.1 \
+    --port 6688
+```
+
+### Running Service For Multimodal Language Models
+
+#### PD Mixed Scene
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
+    --host 127.0.0.1 \
+    --port 8000 \
+    --tp 4 \
+    --device npu \
+    --attention-backend ascend \
+    --mm-attention-backend ascend_attn \
+    --disable-radix-cache \
+    --trust-remote-code \
+    --enable-multimodal \
+    --sampling-backend ascend
+```
+
+## Testing the Service
+
+Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests.
+
+### Which port to send requests to
+
+The port you use depends on your deployment mode:
+
+| Scenario | Where to send requests |
+|---|---|
+| Non-PD (single server) | The server's `--port` (e.g., `8000` in the examples above) |
+| Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes |
+| PD disaggregation | The router's `--port` (e.g., `6688` in the examples above); do **not** send requests directly to prefill or decode servers |
+
+<Tip>
+SGLang serves on port `30000` by default if `--port` is not specified. The examples in this guide use explicit ports for clarity.
+
+If you are using PD disaggregation, replace `8000` with your router's port (e.g., `6688`) in the following examples.
+</Tip>
+
+### Health Check
+
+```bash Command
+curl http://127.0.0.1:8000/health
+```
+
+A successful response returns HTTP 200 with an empty body.
+
+### Generate (Native Endpoint)
+
+```bash Command
+curl http://127.0.0.1:8000/generate \
+  -H "Content-Type: application/json" \
+  -d '{
+    "text": "What is the capital of France?",
+    "sampling_params": {"temperature": 0, "max_new_tokens": 128}
+  }'
+```
+
+The expected output should contain "Paris".
+
+### Chat Completions (OpenAI-Compatible)
+
+```bash Command
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "meta-llama/Llama-3.1-8B-Instruct",
+    "messages": [{"role": "user", "content": "What is the capital of France?"}]
+  }'
+```
+
+Some models return responses accompanied with thinking process content. To disable this output, configure parameters as follows:
+
+```bash Command
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Eco-Tech/Qwen3.5-27B-w8a8-mtp",
+    "messages": [{"role": "user", "content": "What is the capital of France?"}],
+    "chat_template_kwargs": {"enable_thinking": false}
+  }'
+```
+
+The expected output should contain "Paris".
+
+### Multimodal Chat Completions
+
+<Note>
+The image URL in the example below references an external resource (`raw.githubusercontent.com`). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data.
+</Note>
+
+```bash Command
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen3-VL-30B-A3B-Instruct",
+    "messages": [{
+      "role": "user",
+      "content": [
+        {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
+        {"type": "text", "text": "Describe this image."}
+      ]
+    }]
+  }'
+```

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx

@@ -0,0 +1,304 @@
+---
+title: "Ascend NPU Accuracy Evaluation"
+metatags:
+  description: "Complete guide for SGLang model accuracy evaluation on Ascend NPU using EvalScope, covering text/multimodal scenarios."
+---
+
+# Ascend NPU Accuracy Evaluation
+
+This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using a tool: **EvalScope**. The following scenarios are covered:
+
+- **Online Testing**: Evaluate via API interface after starting SGLang server
+- **Text Models**: Using Qwen2.5-7B-Instruct as example
+- **Multimodal Models**: Using Qwen2.5-VL-7B-Instruct as example
+
+---
+
+## Environment Setup
+
+<Warning>
+Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
+</Warning>
+
+First, launch the SGLang environment using the provided container image:
+
+<Tabs>
+  <Tab title="Atlas 800I A3">
+
+```shell Command
+export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3
+
+docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
+    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
+    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
+    --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
+    --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
+    --device=/dev/davinci_manager \
+    --device=/dev/hisi_hdc \
+    --volume /usr/local/sbin:/usr/local/sbin \
+    --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
+    --volume /etc/ascend_install.info:/etc/ascend_install.info \
+    --volume /var/queue_schedule:/var/queue_schedule \
+    --volume ~/.cache/:/root/.cache/ \
+    --entrypoint=bash \
+    $IMAGE
+```
+
+  </Tab>
+  <Tab title="Atlas 800I A2">
+
+```shell Command
+export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b
+
+docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
+    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
+    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
+    --device=/dev/davinci_manager \
+    --device=/dev/hisi_hdc \
+    --volume /usr/local/sbin:/usr/local/sbin \
+    --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
+    --volume /etc/ascend_install.info:/etc/ascend_install.info \
+    --volume /var/queue_schedule:/var/queue_schedule \
+    --volume ~/.cache/:/root/.cache/ \
+    --entrypoint=bash \
+    $IMAGE
+```
+
+  </Tab>
+</Tabs>
+
+---
+
+## Using EvalScope
+
+[EvalScope](https://github.com/modelscope/evalscope) is a comprehensive model evaluation framework from ModelScope, supporting both accuracy evaluation and performance stress testing.
+
+### Install EvalScope
+
+```shell Command
+# Method 1: Installing via pip
+pip install evalscope
+
+# Method 2: Installing from source
+git clone https://github.com/modelscope/evalscope.git
+cd evalscope/
+pip install -e .
+```
+
+### Online Text Model Testing
+
+This section covers online evaluation scenarios where the SGLang server is already running.
+
+#### Start SGLang Server
+
+```shell Command
+# Set HuggingFace mirror (if network access is restricted)
+export HF_ENDPOINT=https://hf-mirror.com
+
+# Start text model server
+sglang serve --model-path /home/weights/Qwen2.5-7B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000 &
+```
+
+For more details of SGLang server, refer to the [Ascend NPU Quick Start](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start)
+
+#### Execute Accuracy Evaluation
+
+EvalScope connects to the SGLang server via OpenAI-compatible API. The following example uses the GSM8K dataset:
+
+```shell Command
+evalscope eval \
+ --model /home/weights/Qwen2.5-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets gsm8k \
+ --limit 10
+```
+
+Upon completion, results similar to the following will be displayed:
+
+```text
++---------------------+-----------+----------+----------+-------+---------+---------+
+| Model               | Dataset   | Metric   | Subset   |   Num |   Score | Cat.0   |
++=====================+===========+==========+==========+=======+=========+=========+
+| Qwen2.5-7B-Instruct | gsm8k     | mean_acc | main     |     5 |     1.0 | default |
++---------------------+-----------+----------+----------+-------+---------+---------+
+```
+
+> **Note**: Output format may vary slightly across different EvalScope versions. The above example is from EvalScope 1.6.x. Ensure the `--model` parameter matches the model name returned by the SGLang server's `/v1/models` endpoint. When starting the server with an HF path (e.g., `Qwen/Qwen2.5-7B-Instruct`), use that path directly. For local paths, pass the full path or the model name returned by `/v1/models`.
+
+#### Common Datasets for Online Evaluation
+
+```shell Command
+# MMLU
+evalscope eval \
+ --model /home/weights/Qwen2.5-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets mmlu
+
+# CEval (Chinese evaluation)
+evalscope eval \
+ --model /home/weights/Qwen2.5-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets ceval
+
+# MATH-500
+evalscope eval \
+ --model /home/weights/Qwen2.5-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets math_500
+
+# HumanEval (code generation)
+evalscope eval \
+ --model /home/weights/Qwen2.5-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets humaneval
+```
+
+### Online Multimodal Model Testing
+
+#### Start Multimodal Model Server
+
+```shell Command
+# Start multimodal model server (Qwen2.5-VL-7B-Instruct)
+# Multimodal models require both --attention-backend and --mm-attention-backend
+sglang serve --model-path /home/weights/Qwen2.5-VL-7B-Instruct \
+    --attention-backend ascend \
+    --mm-attention-backend ascend_attn \
+    --host 0.0.0.0 --port 30000 &
+```
+
+#### Execute Multimodal Accuracy Evaluation
+
+```shell Command
+# MMBench (multimodal evaluation)
+evalscope eval \
+ --model /home/weights/Qwen2.5-VL-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets mm_bench
+
+# MMMU (multimodal comprehensive understanding)
+evalscope eval \
+ --model /home/weights/Qwen2.5-VL-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets mmmu
+
+# HallusionBench (hallucination evaluation)
+evalscope eval \
+ --model /home/weights/Qwen2.5-VL-7B-Instruct \
+ --api-url http://localhost:30000/v1 \
+ --api-key EMPTY \
+ --eval-type openai_api \
+ --datasets hallusion_bench
+```
+
+For more details, refer to the [EvalScope documentation](https://evalscope.readthedocs.io/).
+
+---
+
+## Troubleshooting
+
+### SGLang Server Startup Failure
+
+1. Verify device mapping: A2 uses `davinci[0-7]`, A3 uses `davinci[0-15]`
+2. Confirm image tag matches device type: A2 uses `...-910b`, A3 uses `...-a3`
+3. Check NPU status with `npu-smi info`
+4. First run requires model download; set `HF_ENDPOINT=https://hf-mirror.com` if network access is restricted
+
+### EvalScope Connection Failure to Server
+
+1. Confirm SGLang server started successfully (look for `Application startup complete` in logs)
+2. Verify `--api-url` points to the correct port (SGLang defaults to `30000`)
+3. Ensure URL ends with `/v1`, e.g., `http://localhost:30000/v1`
+
+### EvalScope SSL certificate verification failed
+
+When using EvalScope commands without specifying a dataset or model path, it will attempt to download automatically, which may encounter an SSL certificate verification error:
+
+```text
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 605, in get
+    return self.request("GET", url, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 592, in request
+    resp = self.send(prep, **send_kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 706, in send
+    r = adapter.send(request, **kwargs)
+        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/adapters.py", line 676, in send
+    raise SSLError(e, request=request)
+requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)')))
+[ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
+```
+**Temporary workaround (test only):**
+Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`.
+
+<Warning>
+This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production.
+</Warning>
+
+**Stable solution:**
+The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle:
+
+```shell
+# Obtain the CA certificate from your network administrator
+# Then set the environment variable:
+export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt
+```
+
+<Note>
+This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations.
+</Note>
+
+If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below.
+
+### EvalScope Request Retry Timeout
+If EvalScope keeps retrying requests with errors like:
+```
+2026-06-22 03:09:03 - evalscope - WARNING: Attempt 4 / 5 failed: ....... Retrying...
+2026-06-22 03:09:14 - evalscope - INFO: Evaluating[ceval]   0%| 0/520 [Elapsed: 02:00 < Remaining: ?, ?it/s]
+2026-06-22 03:09:19,557 - openai._base_client - INFO: Retrying request to /chat/completions in 0.447260 seconds
+2026-06-22 03:09:26,088 - openai._base_client - INFO: Retrying request to /chat/completions in 0.992551 seconds
+```
+This is usually caused by the HTTP proxy intercepting requests to the local SGLang server. Disable the proxy with:
+```shell Command
+unset http_proxy
+unset https_proxy
+unset HTTP_PROXY
+unset HTTPS_PROXY
+```
+
+### Download Dataset Error
+
+For this error
+
+```text
+root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
+--2026-05-12 12:08:01--  https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
+Connecting to 141.5.152.215:6688... connected.
+ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Secure Internet Gateway CA V2,OU=IT,O=Huawei,L=Shenzhen,ST=GuangDong,C=CN’:
+  Self-signed certificate encountered.
+To connect to www.modelscope.cn insecurely, use `--no-check-certificate`.
+```
+
+You can add `--no-check-certificate'
+
+```bash
+wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate
+```
+
+For additional assistance, refer to [SGLang GitHub Issues](https://github.com/sgl-project/sglang/issues).

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx

@@ -0,0 +1,156 @@
+---
+title: "Environment Variables"
+metatags:
+    description: "Reference commonly used Ascend NPU environment variables for configuring SGLang runtime behavior."
+---
+SGLang supports various environment variables related to Ascend NPU that can be used to configure its runtime behavior.
+This document provides a list of commonly used environment variables and aims to stay updated over time.
+
+## Directly Used in SGLang
+
+<table>
+  <thead>
+    <tr>
+      <th>Environment Variable</th>
+      <th>Description</th>
+      <th>Default Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>SGLANG_NPU_USE_MLAPO</code></td>
+      <td>Adopts the <code>MLAPO</code> fusion operator in attention <br/> preprocessing stage of the MLA model.</td>
+      <td><code>false</code></td>
+    </tr>
+    <tr>
+      <td><code>SGLANG_USE_FIA_NZ</code></td>
+      <td>Reshapes KV Cache for FIA NZ format.<br/> <code>SGLANG_USE_FIA_NZ</code> must be enabled with <code>SGLANG_NPU_USE_MLAPO</code></td>
+      <td><code>false</code></td>
+    </tr>
+    <tr>
+      <td><code>SGLANG_NPU_USE_MULTI_STREAM</code></td>
+      <td>Enable dual-stream computation of shared experts <br/> and routing experts in DeepSeek models.<br/> Enable dual-stream computation in DeepSeek DSA Indexer.</td>
+      <td><code>false</code></td>
+    </tr>
+    <tr>
+      <td><code>SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT</code></td>
+      <td>Disable cast model weight tensor to a specific NPU <br/> ACL format.</td>
+      <td><code>false</code></td>
+    </tr>
+    <tr>
+      <td><code>SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK</code></td>
+      <td>The maximum number of dispatched tokens on each rank.</td>
+      <td><code>128</code></td>
+    </tr>
+  </tbody>
+</table>
+
+## Used in DeepEP Ascend
+
+<table>
+  <thead>
+    <tr>
+      <th>Environment Variable</th>
+      <th>Description</th>
+      <th>Default Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS</code></td>
+      <td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank.</td>
+      <td><code>8192</code></td>
+    </tr>
+    <tr>
+      <td><code>DEEPEP_NORMAL_LONG_SEQ_ROUND</code></td>
+      <td>Enable long-sequence token pipelining in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank.</td>
+      <td><code>1</code></td>
+    </tr>
+    <tr>
+      <td><code>DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ</code></td>
+      <td>Enable long-sequence token pipelining in combine stage. <br/> The value <code>0</code> means disabled.</td>
+      <td><code>0</code></td>
+    </tr>
+    <tr>
+      <td><code>MOE_ENABLE_TOPK_NEG_ONE</code></td>
+      <td>Needs to be enabled when the expert ID to be processed by <br/> DEEPEP contains -1.</td>
+      <td><code>0</code></td>
+    </tr>
+    <tr>
+      <td><code>DEEP_NORMAL_MODE_USE_INT8_QUANT</code></td>
+      <td>
+        <strong>Deprecated — will be removed in a future release.</strong>
+        <br/>
+        When set to <code>1</code>, quantizes intermediate activations to INT8 in
+        <br/>the DeepEP dispatch operator during normal mode, reducing communication
+        <br/>volume for W8A8-quantized MoE models. This variable will become a no-op;
+        <br/>the quantization behavior will be inferred automatically.
+      </td>
+      <td><code>0</code></td>
+    </tr>
+  </tbody>
+</table>
+
+## Others
+
+<table>
+  <thead>
+    <tr>
+      <th>Environment Variable</th>
+      <th>Description</th>
+      <th>Default Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>TASK_QUEUE_ENABLE</code></td>
+      <td>Used to control the optimization level of the dispatch queue<br/> about the task_queue operator. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/730/comref/Envvariables/docs/zh/environment_variable_reference/TASK_QUEUE_ENABLE.md">Detail</a></td>
+      <td><code>1</code></td>
+    </tr>
+    <tr>
+      <td><code>INF_NAN_MODE_ENABLE</code></td>
+      <td>Controls whether the chip uses saturation mode or INF_NAN mode. <a href="https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0056.html">Detail</a></td>
+      <td><code>1</code></td>
+    </tr>
+    <tr>
+      <td><code>STREAMS_PER_DEVICE</code></td>
+      <td>Configures the maximum number of streams for the stream pool. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/720/comref/Envvariables/Envir_041.html">Detail</a></td>
+      <td><code>32</code></td>
+    </tr>
+    <tr>
+      <td><code>PYTORCH_NPU_ALLOC_CONF</code></td>
+      <td>Controls the behavior of the cache allocator. <br/>This variable changes memory usage and may cause performance fluctuations. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/700/comref/Envvariables/Envir_012.html">Detail</a></td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>ASCEND_MF_STORE_URL</code></td>
+      <td>The address of config store in MemFabric during PD separation, <br/>which is generally set to the IP address of the P primary node<br/> with an arbitrary port number.</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>ASCEND_LAUNCH_BLOCKING</code></td>
+      <td>Controls whether synchronous mode is enabled during operator execution. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/710/comref/Envvariables/Envir_006.html">Detail</a></td>
+      <td><code>0</code></td>
+    </tr>
+    <tr>
+      <td><code>HCCL_OP_EXPANSION_MODE</code></td>
+      <td>Configures the expansion position for communication algorithm scheduling. <a href="https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0094.html">Detail</a></td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>HCCL_BUFFSIZE</code></td>
+      <td>Controls the size of the buffer area for shared data between two NPUs. <br/>The unit is MB, and the value must be greater than or equal to 1. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/60RC3/ptmoddevg/trainingmigrguide/performance_tuning_0047.html">Detail</a></td>
+      <td><code>200</code></td>
+    </tr>
+    <tr>
+      <td><code>HCCL_SOCKET_IFNAME</code></td>
+      <td>Configures the name of the network card used by the Host <br/>during HCCL initialization. <a href="https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/apiref/envvar/envref_07_0075.html">Detail</a></td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>GLOO_SOCKET_IFNAME</code></td>
+      <td>Configures the network interface name for GLOO communication.</td>
+      <td></td>
+    </tr>
+  </tbody>
+</table>

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx

@@ -0,0 +1,305 @@
+---
+title: "Ascend NPU Troubleshooting and FAQ"
+metatags:
+  description: "Troubleshooting and frequently asked questions for Ascend NPU"
+---
+
+Contributions are welcome. Feel free to add more.
+
+## 1. Context corruption with GLOO op.preamble.length <= op.nbytes in PD disaggregation
+
+### Error message
+
+```text highlight=2,5-7
+[2026-04-07 13:24:13 TP0] Decode batch, #running-req: 10, #token: 485248, token usage: 0.94, pre-allocated usage: 0.51, #prealloc-req: 1, #transfer-req: 12, #retracted-req: 0, npu graph: True, gen throughput (token/s): 259.82, #queue-req: 0
+[2026-04-07 13:24:13 TP0] Context corruption detected: Request 3b5dcfe1575d4e1f9b18c953de878a93 (bootstrap_room=7451500070298748792) received metadata from bootstrap_room=4125156593077881415. Metadata buffer index: 1. This indicates metadata buffer index collision.
+[2026-04-07 13:24:13] INFO:      127.0.0.1:59272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
+[2026-04-07 13:24:13] INFO:      127.0.0.1:34000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
+terminate called after throwing an instance of 'gloo::EnforceNotMet'
+ what():  [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:456] op.preamble.length <= op.nbytes. 4 vs 3
+Fatal Python error: Aborted
+
+Thread 0x0000fff873f6f120 (most recent call first):
+ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 1499 in heartbeat_checker
+```
+
+### Cause
+
+(Possibly, not precisely located) High-concurrency long sequences fill up the transfer buffer, causing buffer index collision and data corruption.
+
+### Solution
+
+1. Disable overlap on the Prefill node with `--disable-overlap-schedule`. The Prefill node in PD disaggregation must not enable overlap, otherwise it causes timing issues that lead to out-of-order reception on the Decode node.
+2. Even with overlap disabled, multi-Prefill node high-concurrency long-sequence scenarios may still encounter this issue with low probability. This is a known issue pending resolution.
+
+## 2. Graph mode aclnnInplaceFillScalar error
+
+### Error message
+
+```text highlight=2
+(SGLangEngine pid=3872176) [rank0]:[E414 12:14:41.204711510 compiler_depend.ts:444] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:26 NPU function error:
+ call aclnnInplaceFillScalar failed, error code is 507000
+(SGLangEngine pid=3872176) [ERROR] 2026-04-14-12:14:41 (PID:3874122, Device:0, RankID:-1) ERR00100 PTA call acl api failed
+(SGLangEngine pid=3872176) [Error]: An internal error occurs in the runtime module on the host.
+(SGLangEngine pid=3872176) Rectify the fault based on the error information in the ascend log.
+(SGLangEngine pid=3872176) [PID: 3874122] 2026-04-14-12:14:41.897.548 AclNN_Runtime_Error(EZ9903): aclrtLaunchKerneWithHostArgs failed, return: 507000
+(SGLangEngine pid=3872176)  Solution: In this scenario, collect the plog when the fault occurs and locate the fault based on the plog.
+(SGLangEngine pid=3872176)  TraceBack (most recent call last):
+(SGLangEngine pid=3872176)  Check kernel task failed, stream_id=2028, task_id=48, retCode=0x7080005.[FUNC:LaunchKernel][FILE:context.cc][LINE:1585]
+(SGLangEngine pid=3872176)  rtsLaunchKernelWithHostArgs execution failed, reason=kernel type error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:61]
+(SGLangEngine pid=3872176)  rtsLaunchKernelWithHostArgs failed, runtime result = 507000.[FUNC:ReportCallError][FILE:Log_inner.cpp][LINE:148]
+(SGLangEngine pid=3872176)  aclrtLaunchKerneWWithHostArgs failed, return: 507000
+(SGLangEngine pid=3872176)  Launch kernel failed.
+(SGLangEngine pid=3872176)  #### KernelLaunch failed: /home/850b160/cann-8.5.8/opp/built-in/op_impl/ai_core/tbe//kernel/ascend910_93/ops_legacy/fill/Fill_41dadce325bOf810d03359af2a38990b_high_performance.o
+(SGLangEngine pid=3872176)  Kernel Run failed. opType: 18, Fill
+(SGLangEngine pid=3872176)  launch failed for Fill, errno:361001.
+(SGLangEngine pid=3872176)
+(SGLangEngine pid=3872176) Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler depend.ts:26 (most recent call first):
+(SGLangEngine pid=3872176) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+ 0xb0 (0xffff806848c0 in /root/anaconda3/envs/slime_re/Lib/python3.11/site-packages/torch/lib/libc10.so)
+(SGLangEngine pid=3872176) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x68(0xffff8062c140 in /root/anaconda3/envs/slime_re/Tib/puthon3.11/site-packages/torch/lib/libc10.so)
+(SGLangEngine pid=3872176) frame #2: <unknown function> + 0x110e2b4 (0xffff6d44e2b4 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
+(SGLangEngine pid=3872176) frame #3: <unknown function> + 0x29f0894(0xffff6ed30894 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
+(SGLangEngine pid=3872176) frame #4: <unknown function> + 0x9cc708(0xffff6cd0c700 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
+(SGLangEngine pid=3872176) frame #5: <unknown function> + 0x9cd2dc (0xffff6cd0d2dc in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
+```
+
+### Cause
+
+Too many captured graphs cause a conflict in the graph mode update stream. Each graph is placed on a separate stream, but the number of streams is limited. If too many graphs are captured, conflicts occur.
+
+### Solution
+
+- CANN 8.5 + PTA 2.8 should have resolved this issue.
+- If your versions do not match, reduce the number of captured graphs to 10 or fewer.
+
+## 3. alloc_extend_kernel error
+
+### Error message
+
+```text
+[root@os-node-created-6z9tp pd_7p1d_tp2_20260414_030537]# grep -nR -E "aivec error|ACL" *,log
+prefill 1.log:5739:EZ9999[PID: 164243] 2026-04-14-06:40:35.033.745 (EZ9999): The error from device(chipId:0, dieId:1), serial number is 1, there is an exception of aivec error, core id is 23, error code = 0, dump info: pc start: 0x12420156a000, current: 0x12420156ad74, vec error info: 0xdb1751f50e, mte error info: 0x98f6388707, ifu error info: 0x212c93fc00000, ccu error info: 0x998e981c00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc00.[FUNC:PrintCoreInfo][FILE:device error core proc.cc][LINE:347]
+prefill_1.log:5747:EZ9999[PID: 164242] 2026-04-14-06:40:35.034.456 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 1, there is an exception of aivec error, core id is 45, error code = O, dump info: pc start: 0x12400156a000, current: 0x12400156ad74, vec error info: 0x7304583407, mte error info: 0x27e2b1c765, ifu error info: 0x212c93fc00000, ccu error info: 0x6e0836f300000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc0O.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
+prefill_1.Log:5784:RuntimeError: ACL stream synchronize failed, error code:507035
+prefill_1.Log:5816:RuntimeError: ACL stream synchronize failed, error code:507035
+prefill 2.log:2566:EZ9999[PID: 163475] 2026-04-14-05:28:43.282.016 (EZ9999):  The error from device(chipId:1, dieId:1), serial number is 1, there is an exception of aivec error, core id is 20, error code = 0, dump info: pc start: 0x124601550000, current: 0x124601550d74, vec error info: 0x9b07e496e3, mte error info: 0xc2a6806828, ifu error info: 0x212c93f200000, ccu error info: 0xaa64401100000000, cube error info: O, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core proc.cc][LINE:347]
+prefill 2.log:2574:EZ9999[PID: 163474] 2026-04-14-05:28:43.282.809 (EZ9999):  The error from device(chipId:1, dieId:0), serial number is 1, there is an exception of aivec error, core id is 22, error code = 0, dump info: pc start: 0x124401550000, current: 0x124401550d74, vec error info: 0xa80496049a, mte error info: 0xa0770e2bf2, ifu error info: 0x212c93f200000, ccu error info: 0x1083000000000000, cube error info: O, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
+prefill 2.log:2611:RuntimeError: ACL stream synchronize failed, error code:507035
+prefill 2.log:2644:RuntimeError: ACL stream synchronize failed, error code:507035
+prefill 4.log:1323:EZ9999[PID: 163478] 2026-04-14-05:08:15.363.509 (EZ9999):  The error from device(chipId:3, dieId:0), serial number is 1, there is an exception of aivec error, core id is 5, error code = 0, dump info: pc start: 0x124c00dff000, current: 0x124c00dff860, vec error info: 0xf311fb8727, mte error info: 0x45418486a, ifu error info: 0x212c93fa00000, ccu error info: 0x17a9996f00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
+prefill 4.log:1331:EZ9999[PID: 163479] 2026-04-14-05:08:15.364.016 (EZ9999):  The error from device(chipId:3, dieId:1), serial number is 1, there is an exception of aivec error, core id is 40, error code = 0, dump info: pc start: 0x124e00dff000, current: 0x124e00dff860, vec error info: 0xf11a9cf188, mte error info: 0xfb7710514a, ifu error info: 0x212c93fa00000, ccu error info: 0x1f5828a900000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
+prefill 4.Log:1368:RuntimeError: ACL stream synchronize failed, error code:507035
+prefill 4.log:1400:RuntimeError: ACL stream synchronize failed, error code:507035
+prefill 7.log:3017:EZ9999[PID: 164628] 2026-04-14-05:34:12.369.755 (EZ9999): The error from device(chipId:6, dieId:0), serial number is 1, there is an exception of aivec error, core id is 33, error code = 0, dump info: pc start: 0x1258015ae000, current: 0x1258015aed74, vec error info: 0x740053222c, mte error info: 0x97103df5a0, ifu error info: 0x212c93f400000, ccu error info: 0x391c89ab00000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
+prefill 7.log:3025:EZ9999[PID: 164629] 2026-04-14-05:34:12.369.742 (EZ9999):  The error from device(chipId:6, dieId:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = 0, dump info: pc start: 0x125a015ae000, current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
+prefill 7.log:3062:RuntimeError: ACL stream synchronize failed, error code:507035
+prefill 7.log:3094:RuntimeError: ACL stream synchronize failed, error code:507035
+```
+
+### Error plog
+
+```text highlight=21
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.353.461 [stars engine.cc:1534]170327 ProcLogicCaReport:Task run failed, device id=13, stream id=43, task id=13145, sqe type=0(ffts), errType=0x1(task exception), sqSwStatus=0
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.720 device error core proc.cc:3211170327 AddExceptionReqInfo:add error register: core id=42, stream id=43, task id=13145
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.730 device error core proc.cc:3471170327 PrintCorelnfo:The error from device(chipld:6, dield:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = O, dump info: pc start: 0x125a015ae000,current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100cd6400.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.774 device error core proc.cc:3601170327 PrintCorelnto:The extend into: errcode:(0, 0x8000, 0) errorStr: When the D-cache reads and writes data to the UB, the response value returned bv the bus is a non-zero value. fixp_error0 info: 0x670ef, fixp error1 info: 0xfc, fsmId:0, tslot:2, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.787 device error core proc.cc:4341170327 ProcessStarsCoreErrorInfo:devId=13, streamId=43, taskId=13145, MTE errorCode=0.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.795 davinci task.cc:2011170327 SetStarsResultForDavinciTask:AIV Kernel happen error, retCode=0x31.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.644 davinci kernel task.cc:15821170327 PreCheckTaskErr:Kernel task happen error retCode=0x31, vector core exception1.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.705 davinci kernel task.cc:14241170327 GetArasInfo:[AIC INFO] aras(0 to 9) after execute:0x3fffffb9000, 0, 0, 0x12c9323ff600, 0x12c93231ee00, 0x12c93f1d8800,0x12c93f3ff600,0x12c958200000,0x100000003, 0xaaaa00000001.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.710 [davinci kernel task.cc:1427]170327 GetArgsInfo:tilingKey = 0, print 1 Times totalLen=(10*8), argsSize=80, blockDim=3
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.717 [davinci kernel task.cc:1468]170327 PrintErrorInfoForDavinciTask:[AIC INFO] after execute:arqs print end
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.751 davinci kernel task.cc:14981170327 PrintErrorInfoForDavinciTask:[DFX INFO]Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, flip num=56, fault kernel_name=alloc_extend _kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248.
+[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.823 [dump manager.cpp:41][tid:170327] An exception callback message is received.
+[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.971 [kernel info collector.cpp:384][tid:170327] Get error register information. coreNum=0
+[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.981 kernel info collector.cpp:4771tid:1703271 It is Non-SuperKernel. functionCount=1, qlobalCount=1
+[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.987 [dump args.cpp:668][tid:170327] In arqAddr[0x12c100cd6400]|arqSize[80]dfxAddr[(nil)]|dfxSize[0] has invalid attribute.
+[ERROR] IDEDD(164629,):2026-04-14-05:34:12.383.894 [dump_printf.cpp:1118][tid:170327] infoAddr is null
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.908 [stream.cc:1332]170327 GetError:Stream Synchronize failed, stream id=43, retCode=0x31, [vector core exception].
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.911 [stream.cc:1335]170327 GetError:AIV Kernel happen error, retCode=0x31.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.929 [stream.cc:1335]170327 GetError:[AIC_INFO] after execute:args print end
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.936 stream.cc:13351170327 GetError: DFX INFO1Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, tlip num=56,
+ fault kernel_name=alloc_extend_kernel_18, ault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.943 [stream.cc:3549]170327 EnterFailureAbort:stream id=43 enter failure abort.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.973 [stars_engine.cc:1427]170327 StarsResumeRtsa:stop scheduling in abort failure mode: stream id=43, sq id=6,sq head=801, task id=13145, taskType=66.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.972 [stream.cc:1463]164629 SynchronizeExecutedTask:context is abort, status=0x715005e.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.978 [stream.cc:1516]164629 Synchronizelmpl:failed, stream_id=43, error=0x715005e
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.982 [api error.cc:1015]164629 StreamSynchronize:Stream synchronize failed, stream_id=43, timeout=-1ms.
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.991 [apic stream.cc:154]164629 rtStreamSynchronize:ErrCode=507035, desc=[vector core exception], InnerCode=0x715005e
+[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.997 [error_message_manage.cc:61]164629 FuncErrorReason:rtStreamSynchronize execution failed, reason=vector core exception
+[ERROR] ASCENDCL(164629,):2026-04-14-05:34:12.384.269 [stream.cpp:140]164629 acIrtSynchronizeStreamImpl:synchronize stream failed, runtime result = 507035
+```
+
+### Cause
+
+The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pending resolution.
+
+### Solution
+
+Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead.
+
+```python highlight=15-36
+    def alloc_extend(
+        self,
+        prefix_lens: torch.Tensor,
+        prefix_lens_cpu: torch.Tensor,
+        seq_lens: torch.Tensor,
+        seq_lens_cpu: torch.Tensor,
+        last_loc: torch.Tensor,
+        extend_num_tokens: int,
+        num_new_pages: int = None,
+    ):
+        ...
+        if num_new_pages_item > len(self.free_pages):
+            return None
+
+        # if num_new_pages_item < 200:
+        #     from sgl_kernel_npu.mem_cache.allocator import alloc_extend_kernel
+
+        #     out_indices = torch.empty(
+        #         (extend_num_tokens,),
+        #         dtype=torch.int64,
+        #         device=self.device,
+        #     )
+        #     max_num_extend_tokens = next_power_of_2(extend_num_tokens)
+        #     bs = prefix_lens.shape[0]
+        #     alloc_extend_kernel[(bs,)](
+        #         prefix_lens,
+        #         seq_lens,
+        #         last_loc,
+        #         self.free_pages,
+        #         out_indices,
+        #         next_power_of_2(bs),
+        #         self.page_size,
+        #         max_num_extend_tokens,
+        #     )
+
+        # else:
+        out_indices = torch.empty(
+            (extend_num_tokens,),
+            dtype=torch.int32,
+            device=self.device,
+        )
+        ...
+```
+
+## 4. Out of NPU memory
+
+```text highlight=7
+  File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 175, in calculate_pool_sizes
+    return MemoryPoolConfig(max_total_num_tokens=max_total_num_tokens)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "<string>", line 8, in __init__
+  File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 44, in __post_init__
+    raise RuntimeError(msg)
+RuntimeError: Not enough memory. Please try to increase --mem-fraction-static.
+```
+
+### Solution
+
+First, use the `npu-smi info` command to check the NPU memory usage.
+
+If the NPUs are occupied by other processes, use `--base-gpu-id` to specify the starting device index.
+
+If the NPUs are not occupied, you can use `--tp` to deploy across multiple devices, or reduce the KV cache memory usage by decreasing the `--mem-fraction-static` value. For detailed tuning guidance, see [Hyperparameter Tuning](/docs/advanced_features/hyperparameter_tuning).
+
+## 5. How to update sgl-kernel-npu
+
+### Solution
+
+```bash
+git clone https://github.com/sgl-project/sgl-kernel-npu.git
+
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+cd sgl-kernel-npu
+# Building Project
+bash build.sh
+
+pip install output/sgl_kernel_npu*.whl --force-reinstall
+
+# (Optional) Confirm whether the import can be successfully
+python -c "import sgl_kernel_npu; print(sgl_kernel_npu.__path__)"
+
+rm -rf sgl-kernel-npu
+```
+
+## 6. `[Errno 101] Network is unreachable` when downloading HuggingFace datasets
+
+### Error message
+
+```text highlight=1-2
+'[Errno 101] Network is unreachable' thrown while requesting HEAD https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
+Retrying in 1s [Retry 1/5].
+Traceback (most recent call last):
+  File "<frozen runpy>", line 198, in _run_module_as_main
+  File "<frozen runpy>", line 88, in _run_code
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 2353, in <module>
+    run_benchmark(args)
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 1848, in run_benchmark
+    input_requests = get_dataset(args, tokenizer, model_id)
+                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/__init__.py", line 44, in get_dataset
+    return dataset.load(tokenizer=tokenizer, model_id=model_id)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 45, in load
+    return sample_random_requests(
+           ^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 89, in sample_random_requests
+    dataset_path = download_and_cache_hf_file(
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/utils.py", line 98, in download_and_cache_hf_file
+    return hf_hub_download(repo_id=repo_id, filename=filename, repo_type=repo_type)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  ...
+RuntimeError: Cannot send a request, as the client has been closed.
+[ERROR] 2026-05-18-11:58:31 (PID:215, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
+```
+
+### Cause
+
+The machine cannot directly access the HuggingFace server due to network restrictions (firewall, proxy, or regional access limitations).
+
+### Solution
+
+- **Use an HF mirror site** — set the `HF_ENDPOINT` environment variable to a mirror (e.g., hf-mirror.com):
+
+  ```bash
+  export HF_ENDPOINT=https://hf-mirror.com
+  ```
+
+- **Use a proxy** — if you have an HTTP proxy available:
+
+  ```bash
+  export http_proxy=http://your-proxy:port
+  export https_proxy=http://your-proxy:port
+  ```
+
+- **Download the dataset manually** — use a machine with network access to download the file, then transfer it to the target machine. Use `--dataset-path` to specify the local file path:
+
+  ```bash
+  python -m sglang.bench_serving \
+    --dataset-name random \
+    --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
+    ...
+  ```
+
+## 7. Unexpected type fp8
+
+### Cause
+
+FP8 model is not supported
+
+### Solution
+
+Replace model weights, e.g., switch Qwen/Qwen3.5-27B-FP8 to Eco-Tech/Qwen3.5-27B-w8a8-mtp.
+
+## 8. Docker image versions: stable release vs. daily build
+
+Docker images for Ascend NPU are available in two types:
+
+- **Stable release** — validated version with a specific tag, e.g., `quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3`.
+  Recommended for production deployments.
+- **Daily build** — includes the latest development changes, e.g., `quay.io/ascend/sglang:main-cann8.5.0-a3`. Use this
+  if you need the latest features or bug fixes that have not yet been included in a stable release.
+
+If you encounter issues with a stable release, try switching to a daily build to see if the issue has been resolved in the latest development version.

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5.2_examples.mdx

@@ -0,0 +1,379 @@
+---
+title: "GLM-5.2 examples"
+description: "Deploy GLM-5.2 with SGLang on Ascend NPUs, including single-node, multi-node, and Prefill-Decode disaggregation modes."
+---
+
+# GLM-5.2 examples
+
+## Introduction
+
+The GLM (General Language Model) series is an open-source bilingual large language model family jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. This series of models has performed outstandingly in the field of Chinese NLP with its unique unified pre-training framework and bilingual capabilities. GLM-5.2 adopts the DeepSeek-V3/V3.2 architecture, including the sparse attention (DSA) and multi-token prediction (MTP). Ascend supports GLM-5.2 with 0Day based on the SGLang inference framework, achieving low-code seamless enablement and compatibility with the mainstream distributed parallel capabilities within the current SGLang framework. We welcome developers to download and experience it.
+
+## Environment Preparation
+
+### Model Weight
+
+- `GLM-5.2`(BF16 version): [Download model weight](https://huggingface.co/collections/zai-org/glm-52).
+- `GLM-5.2-w8a8`(Quantized version without mtp): [Download model weight](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/).
+- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.
+
+
+### Installation
+
+The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
+
+```bash
+#Atlas 800 A3
+docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-a3-glm5.2-20260615
+#Atlas 800 A2
+docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.0.0-910b-glm5.2-20260615
+
+#start container
+docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
+--privileged=true --net=host \
+-v /var/queue_schedule:/var/queue_schedule \
+-v /etc/ascend_install.info:/etc/ascend_install.info \
+-v /usr/local/sbin:/usr/local/sbin \
+-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
+--device=/dev/davinci0:/dev/davinci0  \
+--device=/dev/davinci1:/dev/davinci1  \
+--device=/dev/davinci2:/dev/davinci2  \
+--device=/dev/davinci3:/dev/davinci3  \
+--device=/dev/davinci4:/dev/davinci4  \
+--device=/dev/davinci5:/dev/davinci5  \
+--device=/dev/davinci6:/dev/davinci6  \
+--device=/dev/davinci7:/dev/davinci7  \
+--device=/dev/davinci8:/dev/davinci8  \
+--device=/dev/davinci9:/dev/davinci9  \
+--device=/dev/davinci10:/dev/davinci10  \
+--device=/dev/davinci11:/dev/davinci11  \
+--device=/dev/davinci12:/dev/davinci12  \
+--device=/dev/davinci13:/dev/davinci13  \
+--device=/dev/davinci14:/dev/davinci14  \
+--device=/dev/davinci15:/dev/davinci15  \
+--device=/dev/davinci_manager:/dev/davinci_manager \
+--device=/dev/hisi_hdc:/dev/hisi_hdc \
+--entrypoint=bash \
+swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
+```
+
+## Deployment
+
+### Single-node Deployment
+
+- Quantized model `glm5.2_w8a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
+
+**A3 series**
+
+Run the following script to execute online inference.
+
+```shell
+# high performance cpu
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+# bind cpu
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+# cann
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export STREAMS_PER_DEVICE=32
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+export SGLANG_ENABLE_SPEC_V2=1
+# MTP OVERLAP
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_NPU_USE_MULTI_STREAM=1
+
+export HCCL_BUFFSIZE=1000
+export HCCL_OP_EXPANSION_MODE=AIV
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+# DEEPEP
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
+export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
+
+MODEL_PATH={"weights path"}
+
+python3 -m sglang.launch_server \
+        --model-path $MODEL_PATH \
+        --attention-backend ascend \
+        --device npu \
+        --tp-size 16 --nnodes 1 --node-rank 0 \
+        --chunked-prefill-size 16384 --max-prefill-tokens 280000 \
+        --trust-remote-code \
+        --host 127.0.0.1 \
+        --mem-fraction-static 0.7 \
+        --port 8000 \
+        --served-model-name glm-5 \
+        --cuda-graph-bs 16 \
+        --quantization modelslim \
+        --speculative-draft-model-quantization unquant \
+        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4  \
+        --moe-a2a-backend deepep --deepep-mode auto
+```
+
+**A2 series**
+
+Run the following script to execute online inference.
+```shell
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+# cann
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export STREAMS_PER_DEVICE=32
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+export HCCL_BUFFSIZE=1000
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export TRANSFORMERS_VERBOSITY=error
+
+#DEEPEP
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
+export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+
+MODEL_PATH={"weights path"}
+
+python3 -m sglang.launch_server \
+        --model-path $MODEL_PATH \
+        --attention-backend ascend \
+        --device npu \
+        --tp-size 8 \
+        --nnodes 1 \
+        --dp-size 1 \
+        --enable-dp-attention \
+        --chunked-prefill-size -1 \
+        --max-prefill-tokens 65536 \
+        --trust-remote-code \
+        --mem-fraction-static 0.9 \
+        --served-model-name glm-5 \
+        --cuda-graph-bs 8 \
+        --max-running-requests 102 \
+        --quantization modelslim \
+        --speculative-draft-model-quantization unquant \
+        --moe-a2a-backend deepep --deepep-mode auto \
+        --load-balance-method round_robin
+```
+
+### Multi-node Deployment
+
+- Quantized model `glm5.2_w8a8` can be deployed on 2 Atlas 800 A3 (64G × 16) .
+
+**A3 series**
+
+Modify the IP of 2 nodes, then run the same scripts on two nodes.
+
+**node 0/1**
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+# bind cpu
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+# cann
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export STREAMS_PER_DEVICE=32
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+# MTP OVERLAP
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+
+export SGLANG_NPU_USE_MULTI_STREAM=1
+export HCCL_BUFFSIZE=1000
+export HCCL_OP_EXPANSION_MODE=AIV
+
+# Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+
+# DEEPEP
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
+export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+
+
+IPS=('your ip1' 'your ip2')
+IP_MASTER="${IPS[0]}:your port"
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+MODEL_PATH={"weights path"}
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+for i in "${!IPS[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${IPS[$i]}" || "$LOCAL_HOST2" == "${IPS[$i]}" ]];
+    then
+        echo "${IPS[$i]}"
+        python3 -m sglang.launch_server \
+        --model-path $MODEL_PATH \
+        --attention-backend ascend \
+        --device npu \
+        --tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $IP_MASTER \
+        --chunked-prefill-size 16384 --max-prefill-tokens 131072 \
+        --trust-remote-code \
+        --host 127.0.0.1 \
+        --mem-fraction-static 0.8 \
+        --port 8000 \
+        --served-model-name glm-5 \
+        --cuda-graph-max-bs-decode 32 \
+        --moe-a2a-backend deepep \
+        --deepep-mode auto \
+        --speculative-draft-model-quantization unquant \
+        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4  \
+        --disable-radix-cache
+        NODE_RANK=$i
+        break
+    fi
+done
+```
+
+### Prefill-Decode Disaggregation
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+# pd transfer, prefill master IP
+export ASCEND_MF_STORE_URL="tcp://x.x.x.x:24707"
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+P_IP=('your ip1')
+D_IP=('your ip2')
+
+MODEL_PATH={"weights path"}
+
+export TRANSFORMERS_VERBOSITY=error
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+# prefill
+for i in "${!P_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+    then
+        echo "${P_IP[$i]}"
+        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
+        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
+        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
+        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+        export TASK_QUEUE_ENABLE=2
+        export HCCL_SOCKET_IFNAME=lo
+        export GLOO_SOCKET_IFNAME=lo
+
+        # P节点
+        python -m sglang.launch_server --model-path ${MODEL_PATH}  --disaggregation-mode prefill --host ${P_IP[$i]} \
+        --port 8000 --disaggregation-bootstrap-port 8998 --trust-remote-code --nnodes 1 --node-rank $i \
+        --tp-size 16 --mem-fraction-static 0.8 --attention-backend ascend --device npu --quantization modelslim \
+        --disaggregation-transfer-backend ascend --max-running-requests 64 \
+        --served-model-name glm-5 --chunked-prefill-size 524288 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \
+        --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \
+        --dp-size 4 --enable-dp-attention \
+        --load-balance-method round_robin \
+        --enable-dp-lm-head --moe-dense-tp 1 \
+        --speculative-draft-model-quantization unquant \
+        --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
+
+        # cp
+        #--enable-nsa-prefill-context-parallel \
+        #--nsa-prefill-cp-mode in-seq-split \
+        #--attn-cp-size 4 \
+        NODE_RANK=$i
+        break
+    fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+    then
+        echo "${D_IP[$i]}"
+
+        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+        export SGLANG_ENABLE_SPEC_V2=1
+        export HCCL_BUFFSIZE=650
+
+        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
+        export TASK_QUEUE_ENABLE=0
+
+        export HCCL_SOCKET_IFNAME=lo
+        export GLOO_SOCKET_IFNAME=lo
+
+        export SGLANG_NPU_USE_MULTI_STREAM=1
+
+        python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
+        --port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \
+        --mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \
+        --served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \
+        --cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
+        --tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16  --load-balance-method round_robin \
+        --speculative-draft-model-quantization unquant \
+        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
+        NODE_RANK=$i
+        break
+    fi
+done
+```
+
+```shell
+python3 -m sglang_router.launch_router \
+--pd-disaggregation \
+--policy round_robin \
+--prefill http://{P_MASTER_IP}:8000 8998 \
+--decode http://{D_MASTER_IP}:8003 \
+--host {P_MASTER_IP} \
+--port 6688
+```
+
+### Using Benchmark
+
+Refer to [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling) for details.

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx

@@ -0,0 +1,520 @@
+---
+title: "Ascend NPU Operator Development Guide"
+description: "How to develop custom operators (Ascend C / Triton) for Ascend NPU and integrate them into the SGLang inference engine."
+---
+
+## Overview
+
+[SGL-Kernel-NPU](https://github.com/sgl-project/sgl-kernel-npu) is the official
+operator library provided by the SGLang framework for Ascend NPU. It includes
+two types of operator implementations:
+
+1. **Ascend C operators**: High-performance C++ kernels written in Ascend C,
+   compiled into `libsgl_kernel_npu.so`, and registered through PyTorch's custom
+   operator mechanism (`TORCH_LIBRARY_FRAGMENT`). Called in SGLang via
+   `torch.ops.npu.<op_name>()`.
+2. **Triton operators**: Python kernels written in Triton, adapted for Ascend
+   NPU. Called directly via `from sgl_kernel_npu.xxx import ...`.
+
+When SGLang detects an NPU device, it automatically loads `sgl_kernel_npu` and
+uses its operators in place of GPU counterparts, providing optimized inference
+on Ascend hardware.
+
+## Directory Structure
+
+```text
+sgl-kernel-npu/
+├── csrc/                          # Ascend C operator C++ sources
+│   ├── CMakeLists.txt             # Build configuration
+│   ├── pytorch_extensions.cpp     # PyTorch op registration (core integration file)
+│   └── <op_name>/                 # One directory per operator
+│       ├── op_host/               # Host-side code (validation, tiling, launch)
+│       │   ├── <op_name>.cpp
+│       │   └── tiling/            # Optional: tiling data
+│       └── op_kernel/             # Device-side code (Ascend C kernel on AICore)
+│           └── <op_name>_kernel.cpp
+├── include/
+│   └── sgl_kenel_npu_ops.h        # C++ interface declarations
+├── python/
+│   └── sgl_kernel_npu/
+│       └── sgl_kernel_npu/
+│           ├── __init__.py        # Loads libsgl_kernel_npu.so
+│           ├── attention/         # Triton attention kernels
+│           ├── norm/              # Triton normalization kernels
+│           ├── activation/        # Triton activation kernels
+│           ├── fla/               # Triton linear attention kernels
+│           ├── mamba/             # Triton Mamba kernels
+│           ├── moe/               # Triton MoE kernels
+│           └── sample/            # Triton speculative decoding kernels
+├── tests/
+│   └── python/sgl_kernel_npu/     # One test file per operator
+├── build.sh                       # Build script
+└── CMakeLists.txt                 # Root CMake configuration
+```
+
+## Developing Ascend C Operators
+
+A complete Ascend C operator consists of two parts:
+
+- **Device part**: Kernel code running on the NPU AICore, responsible for actual
+  computation. Written using the Ascend C API.
+- **Host part**: Code running on the CPU, responsible for parameter validation,
+  data pre-processing, tiling, and kernel launch.
+
+We recommend starting with the
+[helloworld](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld)
+example, a simple operator that performs element-wise addition on two tensors.
+
+### Step 1: Create the operator directory and files
+
+Create a new operator directory under `csrc/`, following the `op_host/` +
+`op_kernel/` structure:
+
+```text
+csrc/<op_name>/
+├── op_host/
+│   └── <op_name>.cpp
+└── op_kernel/
+    └── <op_name>_kernel.cpp
+```
+
+### Step 2: Write the Device-side Kernel (op_kernel)
+
+Device-side code runs on AICore and follows the Ascend C programming model. The
+core structure is a class with `Init()` and `Process()` methods, plus an
+`extern "C"` entry function.
+
+Using helloworld as an example:
+
+```cpp
+// csrc/helloworld/op_kernel/kernel_helloworld.cpp
+#include "kernel_operator.h"
+
+constexpr int32_t BUFFER_NUM = 2;
+
+class KernalHelloworld {
+public:
+    __aicore__ inline KernalHelloworld() {}
+
+    __aicore__ inline void Init(GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength)
+    {
+        // Compute workload for current block
+        this->blockLength = totalLength / AscendC::GetBlockNum();
+        this->tileNum = 8;
+        this->tileLength = this->blockLength / this->tileNum / BUFFER_NUM;
+
+        // Set global memory buffers
+        xGm.SetGlobalBuffer((__gm__ half *)x + this->blockLength * AscendC::GetBlockIdx(), this->blockLength);
+        yGm.SetGlobalBuffer((__gm__ half *)y + this->blockLength * AscendC::GetBlockIdx(), this->blockLength);
+        zGm.SetGlobalBuffer((__gm__ half *)z + this->blockLength * AscendC::GetBlockIdx(), this->blockLength);
+
+        // Initialize pipeline queues
+        pipe.InitBuffer(inQueueX, BUFFER_NUM, this->tileLength * sizeof(half));
+        pipe.InitBuffer(inQueueY, BUFFER_NUM, this->tileLength * sizeof(half));
+        pipe.InitBuffer(outQueueZ, BUFFER_NUM, this->tileLength * sizeof(half));
+    }
+
+    __aicore__ inline void Process()
+    {
+        int32_t loopCount = this->tileNum * BUFFER_NUM;
+        for (int32_t i = 0; i < loopCount; i++) {
+            CopyIn(i);    // Move data from Global Memory to Local Memory
+            Compute(i);   // Compute on Local Memory
+            CopyOut(i);   // Move results back to Global Memory
+        }
+    }
+
+private:
+    __aicore__ inline void CopyIn(int32_t progress) { /* data copy-in... */ }
+    __aicore__ inline void Compute(int32_t progress) { /* core computation... */ }
+    __aicore__ inline void CopyOut(int32_t progress) { /* data copy-out... */ }
+
+private:
+    AscendC::TPipe pipe;
+    AscendC::TQue<AscendC::TPosition::VECIN, BUFFER_NUM> inQueueX, inQueueY;
+    AscendC::TQue<AscendC::TPosition::VECOUT, BUFFER_NUM> outQueueZ;
+    AscendC::GlobalTensor<half> xGm, yGm, zGm;
+    uint32_t blockLength, tileNum, tileLength;
+};
+
+// Entry function: the compile tool auto-generates aclrtlaunch_<op_name>.h from this name
+extern "C" __global__ __aicore__ void helloworld(
+    GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength)
+{
+    KernalHelloworld op;
+    op.Init(x, y, z, totalLength);
+    op.Process();
+}
+```
+
+**Key points:**
+
+- Class methods must be marked with `__aicore__`, indicating they run on AICore.
+- Use `AscendC::TPipe` + `AscendC::TQue` to build a pipeline that overlaps data
+  movement and computation.
+- The entry function must be declared `extern "C" __global__ __aicore__`. The
+  compile tool generates a host-callable launch header
+  `aclrtlaunch_<func_name>.h` from the function name.
+- Simple operators (e.g., helloworld, cache_assign, lora) do not need extra
+  workspace memory. Complex operators (e.g., mla_preprocess, alloc_extend,
+  build_tree) require temporary workspace memory and are compiled separately in
+  `CMakeLists.txt`.
+
+For more in-depth Ascend C programming knowledge, refer to the
+[Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html).
+
+### Step 3: Write the Host-side Code (op_host)
+
+Host-side code is responsible for passing PyTorch Tensors to the kernel and
+launching it. The key macro is `EXEC_KERNEL_CMD` (located in
+`csrc/utils/torch_helper.h`).
+
+```cpp
+// csrc/helloworld/op_host/helloworld.cpp
+#include "defines.h"                   // Provides HOST_API macro
+#include "torch_helper.h"              // Provides EXEC_KERNEL_CMD macro
+#include "aclrtlaunch_helloworld.h"    // Auto-generated by compile tool
+
+namespace sglang {
+namespace npu_kernel {
+
+HOST_API at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y)
+{
+    // Create output tensor
+    at::Tensor z = at::empty_like(x);
+
+    // Define block count
+    uint32_t blockDim = 8;
+
+    // Compute total element count
+    uint32_t totalLength = 1;
+    for (uint32_t size : x.sizes()) {
+        totalLength *= size;
+    }
+
+    // Launch kernel via EXEC_KERNEL_CMD macro
+    EXEC_KERNEL_CMD(helloworld, blockDim, x, y, z, totalLength);
+    return z;
+}
+
+} // namespace npu_kernel
+} // namespace sglang
+```
+
+**Key points:**
+
+- The namespace must be `sglang::npu_kernel`.
+- Function signatures follow the pattern
+  `at::Tensor <op_name>(const at::Tensor &input, ...)`.
+- For operators with multiple outputs, use
+  `std::tuple<at::Tensor, at::Tensor, ...>` or non-const reference parameters.
+
+### Step 4: Declare the C++ Interface (include/sgl_kenel_npu_ops.h)
+
+Add the operator function declaration in `include/sgl_kenel_npu_ops.h`:
+
+```cpp
+// include/sgl_kenel_npu_ops.h
+namespace sglang {
+namespace npu_kernel {
+
+at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y);
+
+} // namespace npu_kernel
+} // namespace sglang
+```
+
+### Step 5: Register the PyTorch Custom Operator (pytorch_extensions.cpp)
+
+Register the operator in `csrc/pytorch_extensions.cpp` in two steps: define the
+schema and bind the implementation.
+
+```cpp
+// csrc/pytorch_extensions.cpp
+namespace {
+
+// 1. Define operator schema (used by torch.compile, etc.)
+TORCH_LIBRARY_FRAGMENT(npu, m)
+{
+    m.def("helloworld(Tensor x, Tensor y) -> Tensor");
+    // ... other operator schemas ...
+}
+
+// 2. Bind implementation for the PrivateUse1 device (i.e., NPU)
+TORCH_LIBRARY_IMPL(npu, PrivateUse1, m)
+{
+    m.impl("helloworld", TORCH_FN(sglang::npu_kernel::helloworld));
+    // ... other operator implementations ...
+}
+
+} // namespace
+```
+
+**Schema conventions:**
+
+- The namespace is fixed to `npu`. In SGLang, operators are called via
+  `torch.ops.npu.<op_name>()`.
+- Output tensor parameters use the `Tensor(a!)` mutating annotation.
+- Optional parameters use the `Tensor?` annotation, with `c10::optional<T>`
+  handling in the impl.
+- For detailed schema syntax, see the
+  [PyTorch Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func).
+
+**Implementation binding rules:**
+
+- The device name is fixed to `PrivateUse1` (PyTorch NPU backend identifier).
+- Use the `TORCH_FN` macro to bind to the implementation function.
+- For complex operators with optional parameters, use lambda expressions to
+  unpack the arguments.
+
+### Step 6: Update the Build Configuration (csrc/CMakeLists.txt)
+
+Add the new operator's source files to `csrc/CMakeLists.txt`:
+
+**For operators not requiring workspace** (simple operators), add kernel source
+to `no_workspace_kernel`:
+
+```cmake
+ascendc_library(no_workspace_kernel STATIC
+    # ... existing kernel files ...
+    ${PROJECT_OP_SRC_BASE}/<op_name>/op_kernel/<op_name>_kernel.cpp
+)
+```
+
+**For operators requiring workspace** (complex operators), add kernel source to
+`workspace_kernel` with the `-DHAVE_WORKSPACE -DHAVE_TILING` compile flags:
+
+```cmake
+ascendc_library(workspace_kernel STATIC
+    # ... existing kernel files ...
+    ${PROJECT_OP_SRC_BASE}/<op_name>/op_kernel/<op_name>_kernel.cpp
+)
+```
+
+**Add host source files to `OP_SRCS`:**
+
+```cmake
+FILE(GLOB OP_SRCS
+    # ... existing host files ...
+    ${PROJECT_OP_SRC_BASE}/<op_name>/op_host/<op_name>.cpp
+)
+```
+
+### Step 7: Build
+
+Build following the steps in the
+[python/sgl_kernel_npu/README.md](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md):
+
+```bash
+cd sgl-kernel-npu
+
+# Build all modules
+bash build.sh
+
+# Install the sgl_kernel_npu wheel
+pip install output/sgl_kernel_npu*.whl
+```
+
+The compiled `libsgl_kernel_npu.so` is copied into
+`python/sgl_kernel_npu/sgl_kernel_npu/lib/` and loaded by the Python package.
+
+## Developing Triton Operators
+
+Triton operators are located under `python/sgl_kernel_npu/sgl_kernel_npu/`,
+organized by function category:
+
+```text
+python/sgl_kernel_npu/sgl_kernel_npu/
+├── attention/     # Attention (decode_attention, sinks_attention)
+├── norm/          # Normalization (rmsnorm, fused_qk_norm, l1_norm)
+├── activation/    # Activation (swiglu_oai, swiglu_quant)
+├── fla/           # Linear attention (chunk, cumsum, wy_fast)
+├── mamba/         # Mamba-related (causal_conv1d, state_update)
+├── moe/           # MoE-related (mul_add, zero_experts)
+└── sample/        # Speculative decoding (verify_tree_greedy)
+```
+
+**Development steps:**
+
+1. Create a new `.py` file in the appropriate category subdirectory.
+2. Write the kernel using the Triton language, using existing operators in the
+   same directory as templates.
+3. Export functions in the corresponding `__init__.py` if needed.
+4. Write tests under `tests/python/sgl_kernel_npu/`.
+
+**Note:** Many Triton operators are adapted from SGLang's GPU Triton kernels
+(e.g., comments in `fla/utils.py` note the original source). Pay special
+attention to differences between NPU and GPU when adapting.
+
+## Integrating Operators into SGLang
+
+### Ascend C Operator Integration
+
+After completing the [Steps 1-7](#developing-ascend-c-operators) above (writing
+the kernel, registering the torch op, building), and installing the
+`sgl-kernel-npu` wheel, call the operator in SGLang as follows:
+
+```python
+import sgl_kernel_npu  # Loading the library auto-triggers libsgl_kernel_npu.so loading
+
+# Call the operator
+result = torch.ops.npu.helloworld(x, y)
+```
+
+Real-world usage in SGLang (from `sglang/srt/speculative/eagle_utils.py`):
+
+```python
+torch.ops.npu.build_tree_kernel_efficient(
+    parent_list, selected_index, verified_seq_len, tree_mask,
+    positions, retrive_index, retrive_next_token,
+    retrive_next_sibling, topk, depth, draft_token_num, tree_mask_mode
+)
+```
+
+### Triton Operator Integration
+
+Import and call directly via Python:
+
+```python
+from sgl_kernel_npu.attention.decode_attention import decode_attention_fwd
+from sgl_kernel_npu.norm.rmsnorm_bias import rmsnorm_bias
+from sgl_kernel_npu.mamba.causal_conv1d import causal_conv1d_fwd
+
+# Direct function call
+output = decode_attention_fwd(q, k, v, ...)
+```
+
+Real-world usage in SGLang (from `sglang/srt/models/llama.py`):
+
+```python
+from sgl_kernel_npu.norm.split_qkv_rmsnorm_rope import split_qkv_rmsnorm_rope
+```
+
+### sgl-kernel-npu Wheel Update Process
+
+Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates
+require a multi-PR workflow:
+
+1. **Submit sgl-kernel-npu PR**: Add/modify operators in the sgl-kernel-npu
+   repository, ensuring all tests pass.
+2. **Bump sgl-kernel-npu version**: Update the version number in sgl-kernel-npu.
+   Merging triggers an automatic PyPI release.
+3. **Reference the new version in SGLang**:
+   - Update the `sgl-kernel-npu` version requirement in SGLang's
+     `python/pyproject.toml`.
+   - Use the new operator in SGLang code.
+
+If not urgent, you can wait for a regular release (typically within one week).
+
+## Writing Unit Tests
+
+Each operator needs a corresponding unit test under
+`tests/python/sgl_kernel_npu/`, using Python's `unittest` framework.
+
+Test file naming convention: `test_<op_name>.py`
+
+```python
+# tests/python/sgl_kernel_npu/test_helloworld.py
+import unittest
+import torch
+import sgl_kernel_npu
+
+class TestHelloworld(unittest.TestCase):
+    def test_helloworld_basic(self):
+        x = torch.randn(2048, dtype=torch.float16, device="npu")
+        y = torch.randn(2048, dtype=torch.float16, device="npu")
+
+        z = torch.ops.npu.helloworld(x, y)
+        expected = x + y
+
+        torch.testing.assert_close(z, expected)
+
+if __name__ == "__main__":
+    unittest.main()
+```
+
+Run tests:
+
+```bash
+python tests/python/sgl_kernel_npu/test_helloworld.py
+```
+
+**Testing checklist:**
+
+- Cover typical input shapes (power-of-2 sizes and non-standard sizes).
+- Cover different data types (bf16 / fp16, etc.).
+- For operators with in-place behavior, verify correctness of output tensors.
+- Compare against PyTorch native computation to verify accuracy.
+
+## Code Style
+
+### Pre-commit Checks
+
+sgl-kernel-npu uses pre-commit for consistent code style:
+
+```bash
+pip3 install pre-commit
+cd sgl-kernel-npu
+pre-commit install
+pre-commit run --all-files
+```
+
+**Note:** If `pre-commit run --all-files` fails the first time, run it again to
+ensure all lint errors are auto-fixed. All code must pass checks before
+submitting a PR.
+
+### C++ Code Style
+
+- Use the C++17 standard.
+- Place all operator implementations under the `sglang::npu_kernel` namespace.
+- Follow existing code style; format using `.clang-format`.
+- Use `TORCH_CHECK` for error checking (not the standard GE `OP_ADD` macros).
+- Do not include unnecessary GE registration code (e.g., `OP_ADD()` macros).
+
+### Python Code Style
+
+- Follow PEP 8.
+- Use snake_case for file and function names.
+- When adapting from SGLang GPU code, note the original source in the file
+  header.
+
+### General Principles
+
+- Avoid code duplication: extract shared functions for any repeated code blocks
+  over 5 lines.
+- Minimize device synchronization: reduce CPU-NPU sync operations like
+  `tensor.item()` or `tensor.cpu()`.
+- Keep functions pure: avoid in-place argument modification.
+- Keep files concise: split files exceeding 2,000 lines.
+
+## Submitting a PR
+
+1. **Fork the repo**: Fork
+   [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) on GitHub,
+   then clone locally.
+2. **Create a branch**: Create a new branch from `main`, e.g.,
+   `feature/add-my-op`.
+3. **Develop and test**: Develop the operator and write tests following the
+   steps above. Ensure all tests pass.
+4. **Run pre-commit**: Ensure code formatting compliance.
+5. **Commit and push**:
+
+   ```bash
+   git add .
+   git commit -m "feat: add <op_name> operator"
+   git push origin feature/add-my-op
+   ```
+
+6. **Create a PR**: Open a Pull Request on GitHub from your branch to
+   `sgl-project/sgl-kernel-npu:main`.
+7. **Wait for CI and review**: CI checks include linting, compilation, and
+   operator tests. After passing, wait for maintainer review and merge.
+
+## References
+
+- [SGL-Kernel-NPU Official Repository](https://github.com/sgl-project/sgl-kernel-npu)
+- [SGL-Kernel-NPU Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md)
+- [Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html)
+- [PyTorch Custom Ops Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func)
+- [helloworld Example Operator](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld)
+- [SGLang Contribution Guide](/docs/hardware-platforms/ascend-npus/ascend_contribution_guide)

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx

@@ -0,0 +1,173 @@
+---
+title: "Operator Performance Optimizing Guidance"
+metatags:
+    description: "Operator Performance Optimizing Guidance for NPU"
+---
+
+## Performance_benchmark
+
+### Obtaining Performance Data
+
+Before optimizing the performance, you need to obtain accurate performance data, understand the current performance status, and analyze the next optimization direction based on the performance status. MindStudio provides realistic methods for testing the performance of Triton operators.
+
+#### Device-end
+
+The msProf tool is used to collect and analyze key performance indicators of operators running on the Ascend AI Processor. You can use the output performance data to quickly locate the software and hardware performance bottlenecks of operators and improve the efficiency of operator performance analysis.
+
+```bash
+msprof op python3 test_xxxxx.py
+```
+
+The following is a case of using msprof for data collection.
+
+| Attribute | Value |
+|------|------|
+| Name | DequantSwigluQuant_int32_high_performance_100000000 |
+| Type | DequantSwigluQuant |
+| OP State | static |
+| Accelerator Core | AI_VECTOR_CORE |
+| Start Time(us) | 1774489226717521.715 |
+| Duration(us) | 102.824 |
+| Wait Time(us) | 0 |
+| Block Dim | 36 |
+| Mix Block Dim | 0 |
+| HF32 Eligible | NO |
+| Input Shapes | 163840,1024;128,1024;163840;;;;128 |
+| Input Data Types | INT32;FLOAT;FLOAT;DT_UNDEFINED;DT_UNDEFINED;DT_UNDEFINED;INT64 |
+| Input Formats | ND;ND;ND;NULL;NULL;NULL;ND |
+| Output Shapes | 163840,512;163840 |
+| Output Data Types | INT8;FLOAT |
+| Output Formats | ND;ND |
+| Context ID | N/A |
+| aicore_time(us) | 0 |
+| aic_total_cycles | 0 |
+| aic_mac_time(us) | 0 |
+| aic_mac_ratio | 0 |
+| aic_scalar_time(us) | 0 |
+| aic_scalar_ratio | 0 |
+| aic_mte1_time(us) | 0 |
+| aic_mte1_ratio | 0 |
+| aic_mte2_time(us) | 0 |
+| aic_mte2_ratio | 0 |
+| aic_fixpipe_time(us) | 0 |
+| aic_fixpipe_ratio | 0 |
+| aic_icache_miss_rate | 0 |
+| aiv_time(us) | 59.128 |
+| aiv_total_cycles | 3512188 |
+| aiv_vec_time(us) | 36.708 |
+| aiv_vec_ratio | 0.621 |
+| aiv_scalar_time(us) | 41.403 |
+| aiv_scalar_ratio | 0.7 |
+| aiv_mte2_time(us) | 11.975 |
+| aiv_mte2_ratio | 0.203 |
+| aiv_mte3_time(us) | 9.738 |
+| aiv_mte3_ratio | 0.165 |
+| aiv_icache_miss_rate | 0.005 |
+| cube_utilization(%) | 0 |
+
+Below is the field-by-field breakdown of the operator performance record, aligned with the official specification.
+
+### 1. Basic Identification Fields
+
+| Field | Value | Definition (per official docs) |
+|-------|-------|--------------------------------|
+| Name | DequantSwigluQuant_int32_high_performance_100000000 | **Op Name**: Name of the fused operator (dequantization + SwiGLU activation + quantization), with an int32 high-performance implementation suffix. |
+| Type | DequantSwigluQuant | **OP Type**: Functional category of the operator. |
+| OP State | static | **OP State**: Indicates a static operator whose shape and scheduling logic are determined at compile time. |
+| Accelerator Core | AI_VECTOR_CORE | **Task Type**: The operator runs on the AI Vector Core; other common types include AI_CORE (matrix computation core) and AI_CPU. |
+
+### 2. Timing & Scheduling Fields
+
+| Field | Value | Definition (per official docs) |
+|-------|-------|--------------------------------|
+| Start Time(us) | 1774489226717521.715 | **Task Start Time**: Absolute start timestamp of the operator task on the device side, in microseconds. |
+| Duration(us) | 102.824 | **Task Duration**: End-to-end total latency of the operator, including dispatch time, accelerator execution time, and completion response time, in microseconds. |
+| Wait Time(us) | 0 | **Task Wait Time**: Time interval between the end of the previous task and the start of the current task. A value of 0 means no idle wait between task dispatches. |
+
+### 3. Core Configuration & Precision Fields
+
+| Field | Value | Definition (per official docs) |
+|-------|-------|--------------------------------|
+| Block Dim | 36 | **Block Num**: Number of parallel thread blocks for the operator task, corresponding to Block Dim in the SIMT programming model. One AI Vector Core executes only one thread block at a time, so this value reflects the scale of occupied parallel compute resources. |
+| Mix Block Dim | 0 | **Mix Block Num**: Number of blocks on the secondary accelerator if the operator runs on both AI Core and Vector Core. A value of 0 means the operator runs exclusively on AI_VECTOR_CORE with no hybrid core scheduling. |
+| HF32 Eligible | NO | **HF32 Eligible**: Indicates whether the HF32 high-precision floating-point format is enabled; `NO` means it is not used. This field is reported only at the `--task-time=l1` collection level. |
+
+### 4. Input & Output Information
+
+| Field | Value | Definition (per official docs) |
+|-------|-------|--------------------------------|
+| Input Shapes | 163840,1024;128,1024;16384;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. |
+| Input Data Types | INT32;FLOAT;FLOAT;DT_UNDEFINED;DT_UNDEFINED;DT_UNDEFINED;INT64 | **Input Data Types**: Data types of inputs, in the same order as input shapes. |
+| Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | **Input Formats**: Memory layout of inputs; `ND` stands for N-dimensional tensor format, `NULL` corresponds to scalar/undefined inputs. |
+| Output Shapes | 163840,512;163840 | **Output Shapes**: Dimensions of two output tensors, separated by semicolons: `[163840,512]` and `[163840]`. |
+| Output Data Types | INT8;FLOAT | **Output Data Types**: Output 1 is INT8 (quantized result), output 2 is FLOAT. |
+| Output Formats | ND;ND | **Output Formats**: Both outputs use standard ND layout. |
+| Context ID | N/A | **Context ID**: Identifier for sub-tasks at Sub Task granularity; N/A means no sub-task splitting for this operator. |
+
+### 5. AI Core Performance Metrics (aic_* series)
+
+| Field | Value | Definition (per official docs) |
+|-------|-------|--------------------------------|
+| aicore_time(us) | 0 | Theoretical execution time on AI Core, in microseconds. |
+| aic_total_cycles | 0 | Total execution cycles on AI Core. |
+| aic_mac_time(us) / aic_mac_ratio | 0 / 0 | Latency and cycle ratio of cube (matrix multiplication) instructions. |
+| aic_scalar_time(us) / aic_scalar_ratio | 0 / 0 | Latency and cycle ratio of scalar instructions. |
+| aic_mte1_time(us) / aic_mte1_ratio | 0 / 0 | Latency and cycle ratio of L1→L0A/L0B data move instructions. |
+| aic_mte2_time(us) / aic_mte2_ratio | 0 / 0 | Latency and cycle ratio of DDR→AICORE read move instructions. |
+| aic_fixpipe_time(us) / aic_fixpipe_ratio | 0 / 0 | Latency and cycle ratio of L0C→OUT/L1 move instructions. |
+| aic_icache_miss_rate | 0 | Instruction cache miss rate of AI Core. |
+
+### 6. AI Vector Core Performance Metrics (aiv_* series)
+
+| Field | Value | Definition & Interpretation |
+|-------|-------|------------------------------|
+| aiv_time(us) | 59.128 | **aiv_time**: Theoretical execution time on Vector Core under ideal conditions (all blocks scheduled simultaneously with equal duration). In practice, this value is slightly smaller than real execution time due to staggered block startup. |
+| aiv_total_cycles | 3512188 | **aiv_total_cycles**: Total cycles executed on Vector Core, summed across all blocks. |
+| aiv_vec_time(us) / aiv_vec_ratio | 36.708 / 0.621 | Latency (us) and cycle ratio (62.1%) of vector computation instructions. Vector operations are the core compute workload of this operator. |
+| aiv_scalar_time(us) / aiv_scalar_ratio | 41.403 / 0.7 | Latency (us) and cycle ratio (70%) of scalar instructions. The sum exceeds 100% because scalar and vector pipelines run in parallel with independent cycle counters. |
+| aiv_mte2_time(us) / aiv_mte2_ratio | 11.975 / 0.203 | Latency and cycle ratio (20.3%) of read memory move instructions (DDR/on-chip memory → Vector Core). |
+| aiv_mte3_time(us) / aiv_mte3_ratio | 9.738 / 0.165 | Latency and cycle ratio (16.5%) of write memory move instructions (Vector Core → DDR/on-chip memory). |
+| aiv_icache_miss_rate | 0.005 | Vector Core instruction cache miss rate of 0.5%, extremely low and indicates efficient instruction fetch. |
+
+### 7. Utilization Metrics
+
+| Field | Value | Definition (per official docs) |
+|-------|-------|--------------------------------|
+| cube_utilization(%) | 0 | **cube_utilization**: Utilization rate of the matrix multiplication unit. The value is 0 because the operator is purely vector-based. |
+
+## Optimization
+
+### Specification
+
+#### 1. Ascend core compute units
+
+- AI Core: the core that actually performs matrix/vector computation
+- Vector Unit: responsible for SIMD computation (similar to CUDA Core)
+- Scalar Unit: responsible for control/loop
+- L0/L1/L2 cache: The smaller the size, the faster the speed. L0 is only 64 KB, L1 is 256 KB, and L2 is shared.
+
+#### 2. Ascend memory hierarchy (from fastest to slowest)
+
+- Register → Fastest
+- L0/L1 cache → Very fast
+- On-chip cache (L2) → Fast
+- DDR (host memory) → Slowest
+
+### 3. Characteristics of Ascend instructions
+
+Good at accessing large contiguous memory blocks
+
+Dislikes discrete access, stride access, and random access
+
+Must be 128-bit/256-bit aligned
+
+Must be vectorized.
+
+### Tips
+
+1. Ascend 910 series usually has only 40 or 48 vector cores. If the number of grids exceeds 40 or 48 vector cores, the grids will be delivered in a queue, resulting in a long waiting time. Therefore, the number of cores for high-performance implementation does not exceed the number of vector cores.
+2. Try to use up all the UB as much as possible. Move a large block size at a time to ensure that the bound is in the MTE. No Redundant Copy.
+3. If the offset is a negative number, the current triton-ascend considers it as a discrete memory access scenario. As a result, the performance severely deteriorates, and the data is read from the entire DMA block instead of being read in scalar mode.
+4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32 bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation.
+5. Use Double Buffer, parallelizes computation and data transfer. While computing one block of data, another block of data is being transferred to L1.
+6. If hostbound behavior is severe, core binding can be used to address it.

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx

@@ -0,0 +1,804 @@
+---
+title: "Ascend NPU Optimization"
+metatags:
+  description: "Learn how to optimize SGLang for Ascend NPU using the best practice configurations for mainstream LLM models"
+---
+
+This guide explains the role of each parameter used in SGLang deployments on Ascend NPU. It uses
+the [DeepSeek-V3.2 best practice configuration](/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2#pd-disaggregation)
+as the reference example. For a complete list of tested deployment configurations, see the
+[Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/best_practice) page.
+
+<Note>
+  Parameters in this guide fall into two categories:
+
+  - **Required configurations** (marked with `[Required]`): These must be set correctly for the target deployment scenario (e.g., multi-node communication, PD disaggregation). Incorrect values will cause deployment failures or incorrect behavior.
+  - **Performance optimizations**: These improve throughput, latency, or memory efficiency. The optimal values depend on your specific model, hardware, and workload and may require tuning. Where the optimal value is not obvious, tuning guidance is provided.
+</Note>
+
+## System-Level Optimizations
+
+The following system-level tuning steps reduce OS interference and improve CPU scheduling determinism:
+
+<table>
+  <thead>
+    <tr>
+      <th>Command / Variable</th>
+      <th>Purpose</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`</td>
+      <td>Locks all CPU cores to the maximum frequency, eliminating DVFS-induced latency jitter during inference-critical paths.</td>
+    </tr>
+    <tr>
+      <td>`sysctl -w vm.swappiness=0`</td>
+      <td>Minimizes kernel swapping of anonymous pages. Reduces the risk of page faults on NPU memory buffers pinned to host RAM.</td>
+    </tr>
+    <tr>
+      <td>`sysctl -w kernel.numa_balancing=0`</td>
+      <td>Disables automatic NUMA page migration. Prevents the kernel from moving memory pages between NUMA nodes while inference is running, which would cause latency spikes.</td>
+    </tr>
+    <tr>
+      <td>`sysctl -w kernel.sched_migration_cost_ns=50000`</td>
+      <td>Sets a minimum task migration cost, discouraging the scheduler from moving inference threads between CPU cores unnecessarily.</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_SET_CPU_AFFINITY=1`</td>
+      <td>Binds SGLang worker processes to specific CPU cores, avoiding cross-core migration overhead for high-frequency scheduling loops.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Memory & Device Configuration
+
+<table>
+  <thead>
+    <tr>
+      <th>Variable / Argument</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`PYTORCH_NPU_ALLOC_CONF=expandable_segments:True`</td>
+      <td>Enables the expandable NPU memory allocator, allowing the memory pool to grow dynamically. This avoids out-of-memory errors when workloads have variable memory requirements and is essential for large models such as MoE architectures.</td>
+      <td>`expandable_segments:True`</td>
+    </tr>
+    <tr>
+      <td>`STREAMS_PER_DEVICE=32`</td>
+      <td>Sets the maximum number of parallel streams per NPU device. More streams allow better overlap between compute and communication operations. The default of `32` is sufficient for most deployments; increase only if profiling reveals stream contention in complex pipeline parallelism setups.</td>
+      <td>`32`</td>
+    </tr>
+    <tr>
+      <td>`--mem-fraction-static`</td>
+      <td>Controls the fraction of NPU memory allocated to model weights and the KV cache pool. Lower values leave headroom for intermediate activations; higher values maximize KV cache capacity for serving more concurrent requests. The optimal value depends on your model size, sequence length, and available NPU memory. Start conservatively and increase gradually while monitoring for out-of-memory errors.</td>
+      <td>Prefill: `0.73`, Decode: `0.79`</td>
+    </tr>
+  </tbody>
+</table>
+
+## Communication Configuration
+
+<table>
+  <thead>
+    <tr>
+      <th>Variable</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`HCCL_BUFFSIZE`</td>
+      <td>Sets the HCCL communication buffer size in MB. Larger buffers increase throughput for bulk transfers but consume more host memory. The optimal value depends on your communication pattern — larger values benefit prefill (bulk token transfers), while smaller values are sufficient for decode (small batches). Tune based on your expected token dispatch volume.</td>
+      <td>Prefill: `1200`, Decode: `400`</td>
+    </tr>
+    <tr>
+      <td>`HCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME`</td>
+      <td><strong>[Required for multi-node]</strong> Specifies the network interface used for HCCL and GLOO distributed communication. Must be set to the high-bandwidth inter-node network interface (e.g., RDMA-capable NIC) for multi-node deployments. Without this, the framework may default to a low-bandwidth interface, severely degrading distributed communication performance. Not needed for single-node deployments.</td>
+      <td>Set per-cluster</td>
+    </tr>
+  </tbody>
+</table>
+
+## MoE & Expert Parallelism
+
+<table>
+  <thead>
+    <tr>
+      <th>Variable / Argument</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`--moe-a2a-backend`</td>
+      <td>Selects the all-to-all communication backend for MoE expert dispatch and combine. On Ascend NPU, the primary options are `deepep` (DeepEP) and `ascend_fuseep` (Ascend Fused EP). DeepEP is optimized for large-scale models with flexible prefill/decode dispatch paths; `ascend_fuseep` provides a general fused MoE dispatch path.</td>
+      <td>`deepep`</td>
+    </tr>
+    <tr>
+      <td>`--deepep-mode`</td>
+      <td>Selects the DeepEP operating mode. Available options: `normal` (optimized for high throughput, long sequences, and large token counts — suitable for prefill), `low_latency` (optimized for low latency, CUDA Graph compatible, small batches — suitable for decode), and `auto` (switches automatically based on the operation type). Use `auto` if unsure; use explicit modes when managing prefill/decode independently in PD disaggregation.</td>
+      <td>Prefill: `normal`, Decode: `low_latency`</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
+      <td>Sets the maximum number of tokens that a single rank can dispatch in one DeepEP operation (hard upper limit: `1024`). Larger values accommodate more tokens per dispatch but increase buffer allocation overhead. For prefill, use a large value or unbounded (`0`) since many tokens are processed. For decode, set to match your expected tokens per iteration. Must satisfy: `max-running-requests * (1 + draft_tokens) <= this value`.</td>
+      <td>Prefill: `0` (unbounded), Decode: `8`</td>
+    </tr>
+    <tr>
+      <td>`DEEP_NORMAL_MODE_USE_INT8_QUANT`</td>
+      <td>When set to `1`, quantizes intermediate activations to INT8 in the DeepEP dispatch operator, reducing communication volume during MoE dispatch. This is most beneficial for large-scale multi-node prefill with many tokens. The trade-off is a small accuracy impact from quantization and additional compute for the quantize/dequantize operations.</td>
+      <td>Prefill: `1`</td>
+    </tr>
+    <tr>
+      <td>`TASK_QUEUE_ENABLE`</td>
+      <td>Controls the ASCEND Runtime task queue optimization level: `0` = disabled, `1` = default optimization, `2` = aggressive optimization with greater task fusion and overlap. Higher levels improve throughput but may interfere with CUDA Graph-launched tasks. Start with `1` for general use. Use `2` for throughput-critical prefill workloads; use `0` for decode where CUDA Graph compatibility is needed.</td>
+      <td>Prefill: `2`, Decode: `0`</td>
+    </tr>
+    <tr>
+      <td>`--moe-dense-tp-size`</td>
+      <td>Sets the tensor parallelism size for MoE dense (shared) MLP layers. When using DP attention, setting this to `1` avoids an unnecessary all-reduce across the DP group for the dense MLP layers, since each DP shard already has the full weight. In deployments without DP attention, set this to match your TP size.</td>
+      <td>`1`</td>
+    </tr>
+  </tbody>
+</table>
+
+## Prefill Optimizations
+
+These arguments and environment variables are critical for tuning prefill performance:
+
+<table>
+  <thead>
+    <tr>
+      <th>Argument / Variable</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`--chunked-prefill-size`</td>
+      <td>Sets the maximum number of tokens per prefill chunk. A positive value enables chunked prefill, which interleaves prefill and decode for better concurrency in mixed workloads. Set to `-1` to disable chunking and process each request in a single forward pass, which is preferred for dedicated prefill servers with long-context sequences.</td>
+      <td>`-1`</td>
+    </tr>
+    <tr>
+      <td>`--max-prefill-tokens`</td>
+      <td>Limits the total number of tokens the prefill server can process in one batch. The effective bound is `max(this value, model_max_context_length)`. Set this based on your target sequence length and available NPU memory to bound memory usage while maximizing throughput. Tune by increasing until you encounter out-of-memory errors.</td>
+      <td>`68000`</td>
+    </tr>
+    <tr>
+      <td>`--max-running-requests`</td>
+      <td>Limits the number of concurrent requests being processed. For prefill, a low value (e.g., `1`) dedicates more compute and memory to each request, achieving higher per-request throughput — ideal for dedicated prefill nodes processing long sequences. For general-purpose serving, use a higher value to support multi-request concurrency.</td>
+      <td>`1`</td>
+    </tr>
+    <tr>
+      <td>`--disable-radix-cache`</td>
+      <td>Disables prefix caching via RadixAttention. Set this flag when processing non-overlapping long sequences where prefix caching provides no benefit and only consumes memory. Leave unset (radix cache enabled) for chat/conversation workloads with shared system prompts.</td>
+      <td>true</td>
+    </tr>
+    <tr>
+      <td>`--disable-cuda-graph`</td>
+      <td>Disables CUDA Graph capture. CUDA Graphs reduce kernel launch overhead for small, predictable batch sizes, making them ideal for decode. For prefill with large and variable batch sizes, CUDA Graphs provide minimal benefit and can cause issues with dynamic shapes. Set this flag on prefill nodes; leave unset on decode nodes.</td>
+      <td>true</td>
+    </tr>
+    <tr>
+      <td>`--enable-dsa-prefill-context-parallel`</td>
+      <td><strong>(DeepSeek V3.2 DSA-specific)</strong> Enables context parallelism for the long-sequence prefill phase of DeepSeek V3.2 with DSA (DeepSeek Sparse Attention). Distributes the sequence across CP ranks to parallelize the computationally expensive DSA prefill for ultra-long contexts.</td>
+      <td>Enabled</td>
+    </tr>
+    <tr>
+      <td>`--dsa-prefill-cp-mode`</td>
+      <td><strong>(DeepSeek V3.2 DSA-specific)</strong> Controls how the long sequence is split across context parallel ranks: `in-seq-split` divides each sequence uniformly across CP ranks, optimal for single-request prefill. `round-robin-split` (code default) distributes tokens by index mod CP size, supporting multi-batch prefill. Only effective when `--enable-dsa-prefill-context-parallel` is enabled.</td>
+      <td>`in-seq-split`</td>
+    </tr>
+    <tr>
+      <td>`--attn-cp-size`</td>
+      <td>Specifies the context parallelism group size for attention computation. Larger values distribute the sequence across more ranks, reducing per-rank memory and compute at the cost of increased communication. For models with DSA, this controls the CP size for sparse attention prefill. Set to the number of available devices for maximum parallelization.</td>
+      <td>`32`</td>
+    </tr>
+  </tbody>
+</table>
+
+## Decode Optimizations
+
+These arguments and environment variables are critical for tuning decode performance:
+
+<table>
+  <thead>
+    <tr>
+      <th>Argument / Variable</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`--dp-size` / `--data-parallel-size`</td>
+      <td>Sets the data parallelism degree for the decode server. With DP attention enabled, attention layers are sharded across DP ranks while FFN/MoE layers use tensor parallelism. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides evenly into your total card count, with remaining cards used for TP/EP.</td>
+      <td>`8`</td>
+    </tr>
+    <tr>
+      <td>`--ep` / `--ep-size` / `--expert-parallel-size`</td>
+      <td>Sets the expert parallelism degree. For MoE models, this distributes experts across cards, reducing per-card expert loading overhead and enabling all-to-all dispatch. The code default is `1`; set explicitly for MoE models. The optimal value depends on your model's expert count and architecture. DeepSeek V3.2 with 256 routed experts uses `ep=32`. For models with fewer experts, use a proportionally smaller value.</td>
+      <td>`32`</td>
+    </tr>
+    <tr>
+      <td>`--enable-dp-attention`</td>
+      <td>Enables data parallelism for attention layers while keeping tensor parallelism for FFN/MoE layers. This is a key optimization for decode throughput — attention is DP-sharded to reduce KV cache duplication, while MoE layers remain TP-sharded to leverage expert parallelism. Best suited for MoE models where attention is not the compute bottleneck.</td>
+      <td>Enabled</td>
+    </tr>
+    <tr>
+      <td>`--enable-dp-lm-head`</td>
+      <td>Enables vocabulary parallelism across the DP attention group, sharding the LM head weight across ranks. Each rank only computes logits for its vocabulary shard, avoiding a costly all-gather of logits across the DP group. This is essential when DP attention is enabled to maintain throughput.</td>
+      <td>Enabled</td>
+    </tr>
+    <tr>
+      <td>`--cuda-graph-max-bs-decode`</td>
+      <td>Caps the maximum batch size for which CUDA Graphs are captured. Larger values cover more batch sizes but increase graph capture time and memory overhead. If your `max-running-requests` is high but typical batch sizes are lower, use a smaller value to reduce capture overhead. Tune based on your observed batch size distribution during serving.</td>
+      <td>`4`</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_SCHEDULER_SKIP_ALL_GATHER=1`</td>
+      <td>When DP attention is enabled, the scheduler normally performs an all-gather across DP ranks to determine the full set of ready requests. Setting this to `1` skips that operation, reducing decode scheduling latency. Only safe when load is balanced across DP ranks (e.g., via a round-robin load balancing policy). Disable if you observe uneven load distribution across DP ranks.</td>
+      <td>`1`</td>
+    </tr>
+  </tbody>
+</table>
+
+## Speculative Decoding
+
+Speculative decoding reduces per-token latency by generating draft tokens that are then verified by the target model:
+
+<table>
+  <thead>
+    <tr>
+      <th>Argument / Variable</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`--speculative-algorithm`</td>
+      <td>Selects the speculative decoding algorithm. `NEXTN` (aliased to `EAGLE`) uses the model's built-in MTP (Multi-Token Prediction) heads, requiring no separate draft model. `EAGLE3` uses an external draft model, which can achieve higher acceptance rates at the cost of additional memory. Other built-in options include `STANDALONE`, `NGRAM`, and `DFLASH`, plus any plugin-registered name via `SpeculativeAlgorithm.register`. Choose `NEXTN` for models with native MTP support (e.g., DeepSeek V3.2/R1); choose `EAGLE3` for models without MTP (e.g., Qwen MoE).</td>
+      <td>`NEXTN`</td>
+    </tr>
+    <tr>
+      <td>`--speculative-num-steps`</td>
+      <td>Number of speculative forward passes per iteration. More steps can increase the acceptance length and throughput but add latency. For prefill, use a small value (`1`) to minimize prefill latency impact. For decode, use a larger value (`2`–`4`) to maximize throughput. Tune based on your latency vs throughput requirements.</td>
+      <td>Prefill: `1`, Decode: `3`</td>
+    </tr>
+    <tr>
+      <td>`--speculative-eagle-topk`</td>
+      <td>Limits the number of draft tokens considered per position. Lower values reduce compute on unlikely tokens and are required for the overlap scheduler (enabled by default). Higher values may increase acceptance rates but add overhead. Start with `1` for the overlap scheduler; otherwise, `4`–`8` is typical.</td>
+      <td>`1`</td>
+    </tr>
+    <tr>
+      <td>`--speculative-num-draft-tokens`</td>
+      <td>Number of draft tokens generated per speculative step. Higher values increase potential acceptance length and throughput but add per-step computation. Balance against your latency budget — prefill typically uses fewer draft tokens (`2`) to minimize overhead; decode can use more (`4`) to maximize throughput.</td>
+      <td>Prefill: `2`, Decode: `4`</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1`</td>
+      <td>Enables the overlap plan stream feature for EAGLE v2/v3 speculative decoding workers. This overlaps draft model computation with target model verification, effectively hiding draft latency. Enable when using EAGLE-based speculative decoding; not applicable for NEXTN.</td>
+      <td>`1`</td>
+    </tr>
+  </tbody>
+</table>
+
+## Quantization
+
+<table>
+  <thead>
+    <tr>
+      <th>Argument</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`--quantization modelslim`</td>
+      <td>Uses the Ascend ModelSlim quantization tool to load W8A8 pre-quantized model weights. This reduces model weight footprint by approximately 50% compared to BF16, allowing larger models to fit in NPU memory with minimal accuracy degradation. The quantization method is auto-detected from the model's `quant_model_description.json` file.</td>
+      <td>`modelslim`</td>
+    </tr>
+  </tbody>
+</table>
+
+## Throughput Configuration
+
+<table>
+  <thead>
+    <tr>
+      <th>Argument / Variable</th>
+      <th>Purpose</th>
+      <th>Reference Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`--tokenizer-worker-num`</td>
+      <td>Sets the number of parallel tokenizer worker processes. Increasing this allows concurrent tokenization of multiple input/output streams, preventing the tokenizer from becoming a bottleneck under high request concurrency. Set based on your CPU core count and expected request rate.</td>
+      <td>`4`</td>
+    </tr>
+    <tr>
+      <td>`--load-balance-method`</td>
+      <td>Selects the DP load balancing strategy. Available options: `auto` (default, automatically selects the best strategy), `round_robin` (assigns requests to DP ranks in rotation for even distribution), `total_tokens` (balances by token load), `total_requests` (balances by request count), `follow_bootstrap_room` (follows the bootstrap room assignment). Start with `round_robin` for simple even distribution; use `total_tokens` if your requests have highly variable lengths.</td>
+      <td>`round_robin`</td>
+    </tr>
+    <tr>
+      <td>`ASCEND_MF_STORE_URL`</td>
+      <td><strong>[Required for PD disaggregation]</strong> Sets the MemFabric config store address for PD disaggregation. This is the prefill primary node's IP with an arbitrary port, used by the decode nodes to discover and connect to the MemFabric-based KV cache transfer service. Omit this for non-disaggregated deployments.</td>
+      <td>Prefill IP with port</td>
+    </tr>
+  </tbody>
+</table>
+
+## Additional Ascend NPU-Specific Parameters
+
+The following environment variables are used in other best practice configurations and may be applicable depending on your model and deployment:
+
+<table>
+  <thead>
+    <tr>
+      <th>Variable</th>
+      <th>Purpose</th>
+      <th>Typical Usage</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>`HCCL_OP_EXPANSION_MODE=AIV`</td>
+      <td>Configures the HCCL communication algorithm scheduling to use AIV (Ascend Intelligent Vision) expansion mode, which can improve communication efficiency for certain collective operations.</td>
+      <td>Used in Qwen MoE and R1 non-DSA configurations</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_NPU_FUSED_MOE_MODE`</td>
+      <td>Controls the fused MoE optimization mode on Ascend NPU. `1` is default; `2` enables a more aggressive fusion strategy (`DISPATCH_FFN_COMBINE`) that can improve MoE dispatch throughput. Mode `2` requires `--quantization modelslim`. Used primarily with DeepSeek R1 models.</td>
+      <td>`1` or `2`</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_NPU_USE_MLAPO=1`</td>
+      <td><strong>(DeepSeek MLA-specific)</strong> Adopts the `MLAPO` fusion operator in the MLA (Multi-Head Latent Attention) preprocessing stage for DeepSeek models with MLA architecture.</td>
+      <td>Used with DeepSeek R1</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_USE_FIA_NZ=1`</td>
+      <td><strong>(DeepSeek MLA-specific)</strong> Reshapes the KV Cache into FIA NZ format for improved memory access efficiency. Must be used together with `SGLANG_NPU_USE_MLAPO=1`.</td>
+      <td>Used with DeepSeek R1</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_NPU_USE_MULTI_STREAM=1`</td>
+      <td><strong>(DeepSeek MoE-specific)</strong> Enables dual-stream computation for shared experts and routing experts in DeepSeek MoE models, allowing the two expert types to execute concurrently on separate streams.</td>
+      <td>Used with DeepSeek R1</td>
+    </tr>
+    <tr>
+      <td>`SGLANG_USE_AG_AFTER_QLORA=1`</td>
+      <td>Delays the all-gather operation until after Q-LoRA processing. This reduces communication overhead by performing Q-LoRA projection before the all-gather, requiring fewer bytes to be transferred.</td>
+      <td>Used with DeepSeek V3.2/R1 prefill</td>
+    </tr>
+  </tbody>
+</table>
+
+## Feature Compatibility
+
+The table shows feature-by-feature compatibility of SGLang capabilities on Ascend NPU. Features that have performance impact on Ascend are included. Each cell at the intersection of two features indicates whether they can be used together.
+
+The symbols used are defined as follows:
+
+- 🟢 = Full compatibility
+- 🟠 = Partial compatibility
+- ❌ = No compatibility
+- ❔ = Unknown or TBD
+
+<table>
+  <thead>
+    <tr>
+      <th>Feature</th>
+      <th>Tensor Parallelism</th>
+      <th>Data Parallelism</th>
+      <th>Expert Parallelism</th>
+      <th>Context Parallelism</th>
+      <th>PD Disaggregation</th>
+      <th>Quantization</th>
+      <th>Chunked Prefill</th>
+      <th>NPU Graph</th>
+      <th>Speculative Decoding</th>
+      <th>PrefixCache</th>
+      <th>Overlap Schedule</th>
+      <th>DP LM Head</th>
+      <th>MLAPO</th>
+      <th>Multistream MoE</th>
+      <th>EPLB</th>
+      <th>NZ Weight Format</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>Tensor Parallelism</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>Data Parallelism</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>Expert Parallelism</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>Context Parallelism</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>❔</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>PD Disaggregation</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>Quantization</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>Chunked Prefill</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td></td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>NPU Graph</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td></td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+    </tr>
+    <tr>
+      <td>Speculative Decoding</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>PrefixCache</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>Overlap Schedule</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>DP LM Head</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>❔</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>MLAPO</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>🟢</td>
+      <td>❔</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>Multistream MoE</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td></td>
+      <td>❔</td>
+      <td>🟢</td>
+    </tr>
+    <tr>
+      <td>EPLB</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>❔</td>
+      <td>❔</td>
+      <td></td>
+      <td>🟠</td>
+    </tr>
+    <tr>
+      <td>NZ Weight Format</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟢</td>
+      <td>🟠</td>
+      <td></td>
+    </tr>
+  </tbody>
+</table>
+
+## Feature descriptions
+
+### Tensor Parallelism (`--tp-size`)
+
+Splits model weights across multiple NPU devices so that large models can be loaded and run cooperatively. Each device holds a shard of every weight tensor and communicates via HCCL all-reduce. This is the primary mechanism for deploying models that exceed single-device memory.
+
+### Data Parallelism (`--dp-size`, `--enable-dp-attention`)
+
+Replicates the model across independent device groups to increase throughput by processing multiple requests in parallel. With --enable-dp-attention, the attention layers are replicated across DP ranks while the FFN/MoE layers remain tensor-parallel/expert-parallel. This reduces the communication overhead of attention layers in TP (which is significant during small-batch decode) while keeping the heavy compute and memory demands of MoE layers distributed.
+
+### Expert Parallelism (`--ep-size`)
+
+Each device holds a subset of experts and routes tokens via all-to-all communication based on the model's gating network. Supports load balancing methods including default round-robin expert placement and dynamic Expert Parallelism Load Balancing (EPLB).
+
+### Context Parallelism (`--attn-cp-size`)
+
+Splits long input sequences across devices so that the KV cache and attention computation for a single request are distributed. This enables serving very long context lengths (e.g., 128K tokens) that exceed single-device memory. On Ascend, `--attn-cp-size` must equal `--tp-size`.
+
+### PD Disaggregation (`--disaggregation-mode`)
+
+Separates the prefill (P) and decode (D) phases onto different device groups, allowing each phase to be independently optimized for its compute and memory characteristics. Prefill nodes handle long input processing at high throughput, while decode nodes focus on low-latency token generation. Uses the Ascend MemFabric transfer backend (`--disaggregation-transfer-backend ascend`).
+
+### Quantization (`--quantization`)
+
+Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput. On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static). Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors) depends on the availability of specific Ascend custom kernels in your environment; see quantization for details and compatibility matrices.
+
+### Chunked Prefill (`--chunked-prefill-size`)
+
+Breaks large prefill computations into smaller, fixed-size chunks for better scheduling interleaving with decode batches. This prevents long prefill requests from blocking decode latency. Use `-1` to disable chunked prefill on dedicated prefill nodes. Has partial compatibility with most features because chunk boundaries introduce scheduling complexity.
+
+### NPU Graph (`--cuda-graph-bs`)
+
+Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes.
+
+<Note>
+  `--enable-torch-compile` is incompatible with NPU Graph. When torch.compile is enabled, NPU Graph must be disabled via `--disable-cuda-graph`.
+</Note>
+
+### Speculative Decoding (`--speculative-algorithm`)
+
+Reduces per-token latency by predicting multiple future tokens in a single forward pass, then verifying them against the model. Ascend supports `NEXTN` (for DeepSeek models, using the model's own hidden states) and `EAGLE3` (for Qwen MoE models, using a separate draft model). Controlled by `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. On Ascend, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` enables the optimized overlap scheduler for speculative decoding.
+
+### PrefixCache (`--disable-radix-cache`)
+
+Reuses KV cache across requests that share common prompt prefixes (Radix Cache), reducing repeated prefill computation and lowering time-to-first-token. Enabled by default; disable with `--disable-radix-cache` when prefix reuse is not expected (e.g., dedicated prefill nodes in PD disaggregation, or random-input benchmarks).
+
+### Overlap Schedule
+
+On Ascend, this is primarily controlled via environment variable SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1, which enables the optimized overlap scheduler for speculative decoding scenarios. Most effective during decode with speculative decoding and DP attention, where expert dispatch/reduce communication can be overlapped with the next batch's computation.
+
+### DP LM Head (`--enable-dp-lm-head`)
+
+Shards the language model head (the final vocabulary projection layer) across DP ranks to reduce memory consumption and improve decode throughput. Without this, each DP rank holds a full copy of the LM head, which is wasteful for large vocabularies. Typically used together with `--enable-dp-attention`.
+
+### MLAPO (`SGLANG_NPU_USE_MLAPO=1`)
+
+Enables the MLAPO (MLA Pre-Processing Optimization) fusion operator for DeepSeek-style Multi-head Latent Attention models, replacing multiple separate kernels with a single fused kernel for Q/K/V projection and absorption. Reduces prefill latency and memory bandwidth usage. Used in most DeepSeek best-practice configurations.
+
+### Multistream MoE (`SGLANG_NPU_USE_MULTI_STREAM=1`)
+
+Enables dual-stream parallel execution for MoE layers, where shared experts and routing experts run concurrently on separate NPU streams. This overlaps shared-expert computation with routed-expert dispatch, improving MoE layer throughput.
+
+### EPLB (`--enable-eplb`)
+
+Enables Expert Parallelism Load Balancing to dynamically redistribute experts across devices based on workload, ensuring even expert utilization and preventing hotspots. Uses the `deepseek` balancing algorithm. Compatible with the `deepep` MoE A2A backend, but not with `ascend_fuseep`.
+
+### NZ Weight Format (`SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT`)
+
+Casts model weight tensors to the Ascend NPU ACL FRACTAL_NZ format (format 29) for improved memory access efficiency on Da Vinci AI cores. Enabled by default globally; disable with `SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT=1` when the format conflicts with other features (e.g., EPLB). Applied to linear weights, MoE weights, quantized weights, MLA QKV projections, and KV Cache (via `PA_NZ` / `nzcache` modes for FIA). Related env var: `SGLANG_USE_FIA_NZ=1` (requires `SGLANG_NPU_USE_MLAPO=1`).
+
+## See Also
+
+- [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/best_practice) — Complete deployment configurations and benchmark results for all supported models
+- [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables) — Reference for all Ascend NPU-related environment variables
+- [DeepSeek V3.2 Guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) — Detailed usage guide for DeepSeek V3.2 deployment
+- [Expert Parallelism](/docs/advanced_features/expert_parallelism) — DeepEP configuration and tuning guide

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx

@@ -0,0 +1,770 @@
+---
+title: "Ascend NPU Performance Testing"
+metatags:
+  description: "Performance testing for SGLang on Ascend NPUs for text generation, multimodal, and embedding models in both online and offline serving modes."
+---
+
+This page walks through performance testing your SGLang deployment on Ascend NPUs. We cover three model types — text generation (`Qwen/Qwen2.5-7B-Instruct`), multimodal vision (`Qwen/Qwen2.5-VL-7B-Instruct`), and embedding (`Qwen/Qwen3-Embedding-8B`) — in both online and offline serving modes. You can use [Evalscope](https://evalscope.readthedocs.io/en/latest/), [AISBench](https://ais-bench-benchmark.readthedocs.io/en/latest/), or SGLang's built-in benchmarking tools.
+
+<Note>The benchmark output examples in this guide are for illustration only. Actual performance depends on your hardware (e.g., Atlas 800I A2 vs A3), model version, SGLang version, and deployment configuration. Always run benchmarks on your own hardware to obtain accurate performance data.</Note>
+
+## 1. Prepare
+
+### 1.1 Start SGLang server
+
+Launch the server with the appropriate flags for each model type. Make sure SGLang is installed first — see [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) for environment setup.
+
+<Tabs>
+  <Tab title="Text Generation">
+
+```shell Command
+# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
+sglang serve --model-path Qwen/Qwen2.5-7B-Instruct
+```
+
+  </Tab>
+  <Tab title="Multimodal">
+
+```shell Command
+# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
+sglang serve --model-path Qwen/Qwen2.5-VL-7B-Instruct --mm-attention-backend ascend_attn
+```
+
+  </Tab>
+  <Tab title="Embedding">
+
+```shell Command
+# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
+sglang serve --model-path Qwen/Qwen3-Embedding-8B --is-embedding
+```
+
+  </Tab>
+</Tabs>
+
+<Tip>Add `&` at the end of the command to run the server in the background, or open a new terminal to run the benchmark commands in the following sections.</Tip>
+
+<Note>The server binds to `http://127.0.0.1:30000` by default. All online benchmarks below assume the server is running at that address. The `--is-embedding` flag is required for embedding models.</Note>
+
+### 1.2 Install benchmarking tools
+
+`bench_serving` and `bench_offline_throughput` are built into SGLang and require no extra installation. For Evalscope and AISBench, set up each in its own virtual environment:
+
+<Tabs>
+  <Tab title="Evalscope">
+
+```shell Command
+python3 -m venv .evalscope_venv
+source .evalscope_venv/bin/activate
+pip install evalscope[perf] -U
+```
+
+  </Tab>
+  <Tab title="AISBench">
+
+```shell Command
+python3 -m venv .aisbench_venv
+source .aisbench_venv/bin/activate
+
+git clone https://github.com/AISBench/benchmark.git
+cd benchmark/
+pip3 install -e ./ --use-pep517
+
+pip3 install -r requirements/api.txt
+pip3 install -r requirements/extra.txt
+```
+
+Run `ais_bench -h` to verify.
+
+<Note>AISBench requires Python 3.10-3.12. After installation, all AISBench commands must be run from the `benchmark/` directory (the cloned repo root). Set `stream=True` and `ignore_eos=True` in the model config for accurate results.</Note>
+
+  </Tab>
+</Tabs>
+
+## 2. Online Service: Text Generation Model
+
+Test `Qwen/Qwen2.5-7B-Instruct` via the online serving endpoint.
+
+<Note>Before running any benchmark in this section, make sure the SGLang text-generation server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) for the launch command.</Note>
+
+<Tip>For performance testing, prefer random datasets (`--dataset random`, `--dataset-name random`) over real datasets. Random datasets let you pin `--min-prompt-length` / `--max-prompt-length` and `--min-tokens` / `--max-tokens` to fixed values, producing consistent, repeatable results. Real datasets (ShareGPT, openqa, etc.) have variable input lengths that add noise and make cross-run comparisons unreliable.</Tip>
+
+### 2.1 Using Evalscope
+
+<Note>Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang server running at `http://127.0.0.1:30000`.</Note>
+
+Run the following command to run a performance test against the server:
+
+```shell Command
+evalscope perf \
+  --parallel 10 \
+  --number 20 \
+  --model Qwen/Qwen2.5-7B-Instruct \
+  --url http://127.0.0.1:30000/v1/chat/completions \
+  --api openai \
+  --dataset random \
+  --max-tokens 1024 \
+  --min-tokens 1024 \
+  --prefix-length 0 \
+  --min-prompt-length 1024 \
+  --max-prompt-length 1024 \
+  --tokenizer-path Qwen/Qwen2.5-7B-Instruct \
+  --extra-args '{"ignore_eos": true}'
+```
+
+<Tip>If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.</Tip>
+
+Example output (for illustration only — actual results depend on your hardware and configuration):
+
+```text
+Benchmarking summary:
+┌────────────────────────────┬─────────────┐
+│ Metric                     │       Value │
+├────────────────────────────┼─────────────┤
+│ ── General ──              │             │
+│ Test Duration (s)          │       89.34 │
+│ Concurrency                │          10 │
+│ Request Rate (req/s)       │       -1.00 │
+│ Total / Success / Failed   │ 20 / 20 / 0 │
+│ Req Throughput (req/s)     │        0.22 │
+│ ── Latency ──              │             │
+│ Avg Latency (s)            │       44.67 │
+│ TTFT (ms)                  │      578.51 │
+│ TPOT (ms)                  │       43.10 │
+│ ITL (ms)                   │       43.12 │
+│ ── Tokens ──               │             │
+│ Avg Input Tokens           │     1024.00 │
+│ Avg Output Tokens          │     1024.00 │
+│ Output Throughput (tok/s)  │      229.24 │
+│ Total Throughput (tok/s)   │      458.49 │
+│ ── Speculative Decoding ── │             │
+│ Decoded Tok/Iter           │        1.00 │
+│ Spec. Accept Rate          │        0.00 │
+└────────────────────────────┴─────────────┘
+
+Percentile results:
+┌────────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
+│ Metric         │      1% │      5% │     10% │     25% │     50% │     75% │     90% │     95% │     99% │
+├────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
+│ Latency (s)    │   44.47 │   44.47 │   44.47 │   44.47 │   44.86 │   44.86 │   44.86 │   44.86 │   44.86 │
+│ TTFT (ms)      │  138.12 │  142.07 │  426.17 │  426.87 │  783.67 │  785.26 │  786.85 │  787.97 │  787.97 │
+│ ITL (ms)       │   41.84 │   42.14 │   42.22 │   42.36 │   42.57 │   42.80 │   42.99 │   49.24 │   49.84 │
+│ TPOT (ms)      │   42.71 │   42.71 │   42.71 │   43.05 │   43.08 │   43.43 │   43.43 │   43.71 │   43.71 │
+│ Input tokens   │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │
+│ Output tokens  │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │
+│ Output (tok/s) │   22.83 │   22.83 │   22.83 │   22.83 │   23.02 │   23.03 │   23.03 │   23.03 │   23.03 │
+│ Total (tok/s)  │   45.65 │   45.65 │   45.65 │   45.65 │   46.05 │   46.05 │   46.05 │   46.05 │   46.05 │
+│ Decode (tok/s) │   22.88 │   23.03 │   23.03 │   23.07 │   23.21 │   23.42 │   23.42 │   23.42 │   23.42 │
+└────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
+...
+```
+
+<Note>See the [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) for full details.</Note>
+
+### 2.2 Using AISBench
+
+<Note>Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands must be run from the `benchmark/` directory. SGLang server running at `http://127.0.0.1:30000`. Set `stream=True` and `ignore_eos=True` in the model config for accurate results.</Note>
+
+Two files need to be configured for performance testing.
+
+First, describe the model and server settings in `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py`:
+
+```python vllm_api_stream_chat.py
+# more details: https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/scenes_intro/performance_benchmark.html
+from ais_bench.benchmark.models import VLLMCustomAPIChat
+from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content
+
+models = [
+    dict(
+        attr="service",
+        type=VLLMCustomAPIChat,
+        abbr="vllm-api-stream-chat",
+        path="Qwen/Qwen2.5-7B-Instruct",
+        model="Qwen/Qwen2.5-7B-Instruct",
+        stream=True,
+        request_rate=0,
+        use_timestamp=False,
+        retry=2,
+        api_key="",
+        host_ip="127.0.0.1",
+        host_port=30000,
+        url="",
+        max_out_len=512,
+        batch_size=32,
+        trust_remote_code=False,
+        generation_kwargs=dict(
+            temperature=0.01,
+            ignore_eos=True,
+        ),
+        pred_postprocessor=dict(type=extract_non_reasoning_content),
+    )
+]
+```
+
+<Tip>If the model has already been downloaded, point `path` to the local model path instead of the model id.</Tip>
+
+Second, configure random prompt lengths in `ais_bench/datasets/synthetic/synthetic_config.py`:
+
+```python synthetic_config.py
+# more details: https://ais-bench-benchmark.readthedocs.io/en/latest/advanced_tutorials/synthetic_dataset.html
+synthetic_config = {
+    "Type":"tokenid",
+    "RequestCount": 10,
+    "TrustRemoteCode": False,
+    "StringConfig" : {
+        "Input" : {
+            "Method": "uniform",
+            "Params": {"MinValue": 1, "MaxValue": 200}
+        },
+        "Output" : {
+            "Method": "gaussian",
+            "Params": {"Mean": 100, "Var": 200, "MinValue": 1, "MaxValue": 100}
+        }
+    },
+    "TokenIdConfig" : {
+        "RequestSize": 10,
+        "PrefixLen": 0
+    }
+}
+```
+
+Run with a synthetic dataset:
+
+```shell Command
+ais_bench --models vllm_api_stream_chat --datasets synthetic_gen_string -m perf
+```
+
+Example output (for illustration only — actual results depend on your hardware and configuration):
+
+```text
+╒══════════════════════════╤═════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════╕
+│ Performance Parameters   │ Stage   │ Average         │ Min             │ Max             │ Median          │ P75             │ P90             │ P99             │  N  │
+╞══════════════════════════╪═════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════╡
+│ E2EL                     │ total   │ 3896.4 ms       │ 3081.6 ms       │ 4175.3 ms       │ 4013.8 ms       │ 4123.4 ms       │ 4137.1 ms       │ 4171.5 ms       │ 10  │
+├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
+│ TTFT                     │ total   │ 411.6 ms        │ 346.7 ms        │ 439.7 ms        │ 416.3 ms        │ 426.6 ms        │ 434.4 ms        │ 439.2 ms        │ 10  │
+├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
+│ TPOT                     │ total   │ 38.3 ms         │ 37.4 ms         │ 39.0 ms         │ 38.3 ms         │ 38.7 ms         │ 38.9 ms         │ 39.0 ms         │ 10  │
+├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
+│ ITL                      │ total   │ 38.7 ms         │ 0.0 ms          │ 156.5 ms        │ 38.9 ms         │ 39.0 ms         │ 39.2 ms         │ 117.1 ms        │ 10  │
+├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
+│ InputTokens              │ total   │ 123.4           │ 34.0            │ 228.0           │ 130.5           │ 170.5           │ 217.2           │ 226.92          │ 10  │
+├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
+│ OutputTokens             │ total   │ 92.1            │ 69.0            │ 100.0           │ 95.0            │ 99.75           │ 100.0           │ 100.0           │ 10  │
+├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
+│ OutputTokenThroughput    │ total   │ 23.5937 token/s │ 22.3912 token/s │ 24.2616 token/s │ 23.7399 token/s │ 23.9919 token/s │ 24.2027 token/s │ 24.2557 token/s │ 10  │
+╘══════════════════════════╧═════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════╛
+╒══════════════════════════╤═════════╤══════════════════╕
+│ Common Metric            │ Stage   │ Value            │
+╞══════════════════════════╪═════════╪══════════════════╡
+│ Benchmark Duration       │ total   │ 4175.4485 ms     │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Total Requests           │ total   │ 10               │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Failed Requests          │ total   │ 0                │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Success Requests         │ total   │ 10               │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Concurrency              │ total   │ 9.3317           │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Max Concurrency          │ total   │ 32               │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Request Throughput       │ total   │ 2.395 req/s      │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Total Input Tokens       │ total   │ 1234             │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Prefill Token Throughput │ total   │ 299.8329 token/s │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Total Generated Tokens   │ total   │ 921              │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Input Token Throughput   │ total   │ 295.5371 token/s │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Output Token Throughput  │ total   │ 220.5751 token/s │
+├──────────────────────────┼─────────┼──────────────────┤
+│ Total Token Throughput   │ total   │ 516.1122 token/s │
+╘══════════════════════════╧═════════╧══════════════════╛
+```
+
+<Note>See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details.</Note>
+
+### 2.3 Using bench_serving
+
+SGLang's built-in `bench_serving` requires no extra installation. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark.
+
+<Note>See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for all backends, datasets, and advanced options.</Note>
+
+```shell Command
+python -m sglang.bench_serving \
+  --backend sglang-oai \
+  --base-url http://127.0.0.1:30000 \
+  --model Qwen/Qwen2.5-7B-Instruct \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 512 \
+  --random-range-ratio 1 \
+  --num-prompts 100 \
+  --max-concurrency 32
+```
+
+<Note>
+`--dataset-name random` samples token IDs from the ShareGPT dataset to generate realistic input; the first run downloads ShareGPT from Hugging Face automatically.
+
+1. If you have network issues, set `export HF_ENDPOINT=https://hf-mirror.com` to use domestic mirror.
+2. If downloading still fails, manually download the dataset file `ShareGPT_V3_unfiltered_cleaned_split.json` locally, upload it to your server, then specify the file directory via `--dataset-path` to run offline.
+</Note>
+
+<Tip>Set `--random-range-ratio 1` for fixed input/output lengths (recommended for consistent comparisons) or `0` (default) for uniform distribution. Add `--request-rate` to control the request rate. For all backends, datasets, and advanced options, see the full [Bench Serving Guide](/docs/developer_guide/bench_serving).</Tip>
+
+Example output (for illustration only — actual results depend on your hardware and configuration):
+
+```text
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai
+Traffic request rate:                    inf
+Max request concurrency:                 32
+Successful requests:                     100
+Benchmark duration (s):                  47.51
+Total input tokens:                      102400
+Total input text tokens:                 102400
+Total generated tokens:                  51200
+Total generated tokens (retokenized):    51195
+Request throughput (req/s):              2.10
+Input token throughput (tok/s):          2155.35
+Output token throughput (tok/s):         1077.68
+Peak output token throughput (tok/s):    1587.00
+Peak concurrent requests:                64
+Total token throughput (tok/s):          3233.03
+Concurrency:                             26.93
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   12793.49
+Median E2E Latency (ms):                 12940.17
+P90 E2E Latency (ms):                    13049.86
+P99 E2E Latency (ms):                    13051.61
+---------------Time to First Token----------------
+Mean TTFT (ms):                          1423.99
+Median TTFT (ms):                        1489.29
+P99 TTFT (ms):                           2325.56
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          22.25
+Median TPOT (ms):                        22.22
+P99 TPOT (ms):                           25.08
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           22.26
+Median ITL (ms):                         20.74
+P95 ITL (ms):                            21.40
+P99 ITL (ms):                            23.62
+Max ITL (ms):                            2229.30
+==================================================
+```
+
+#### SGLang Serving Benchmark Result — Complete Reference
+
+The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments.
+
+##### Test Configuration
+
+<table>
+  <thead>
+    <tr>
+      <th>Parameter</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>Backend</code></td>
+      <td>The serving backend under test (e.g., <code>sglang</code>, <code>vllm</code>).</td>
+    </tr>
+    <tr>
+      <td><code>Traffic request rate</code></td>
+      <td>Request generation rate in req/s. <code>inf</code> means maximum rate (concurrency-bounded). <code>trace</code> indicates trace timestamp mode. A fixed value enforces constant inter-arrival time.</td>
+    </tr>
+    <tr>
+      <td><code>Max request concurrency</code></td>
+      <td>Maximum number of concurrent requests from the client side. Displays <code>not set</code> when unspecified.</td>
+    </tr>
+  </tbody>
+</table>
+
+##### Core Statistics & Throughput Metrics
+
+<table>
+  <thead>
+    <tr>
+      <th>Parameter</th>
+      <th>Description</th>
+      <th>Format Specification</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr><td><code>Successful requests</code></td><td>Total number of successfully completed requests (HTTP 200, no generation errors).</td><td>Integer, no decimal places</td></tr>
+    <tr><td><code>Benchmark duration (s)</code></td><td>Total elapsed time from first request sent to last response fully received (seconds).</td><td>2 decimal places</td></tr>
+    <tr><td><code>Total input tokens</code></td><td>Total number of input (prompt) tokens across all requests, counted by server-side tokenizer.</td><td>Integer, no decimal places</td></tr>
+    <tr><td><code>Total input text tokens</code></td><td>Same as <code>Total input tokens</code>. For multimodal inputs, this may differ.</td><td>Integer, no decimal places</td></tr>
+    <tr><td><code>Total generated tokens</code></td><td>Total number of output tokens actually generated by the server (server-side tokenizer count).</td><td>Integer, no decimal places</td></tr>
+    <tr><td><code>Total generated tokens (retokenized)</code></td><td>Output text re-tokenized by the client using its own tokenizer. A large discrepancy indicates tokenizer mismatch or special tokens in output.</td><td>Integer, no decimal places</td></tr>
+    <tr><td><code>Request throughput (req/s)</code></td><td>Number of successful requests processed per second. Formula: <code>Successful requests / Benchmark duration (s)</code>.</td><td>2 decimal places</td></tr>
+    <tr><td><code>Input token throughput (tok/s)</code></td><td>Number of input tokens processed per second. Formula: <code>Total input tokens / Benchmark duration (s)</code>.</td><td>2 decimal places</td></tr>
+    <tr><td><code>Output token throughput (tok/s)</code></td><td>Number of output tokens generated per second. Formula: <code>Total generated tokens / Benchmark duration (s)</code>.</td><td>2 decimal places</td></tr>
+    <tr><td><code>Peak output token throughput (tok/s)</code></td><td>Observed instantaneous peak output token generation rate during the test (computed over a sliding window).</td><td>2 decimal places</td></tr>
+    <tr><td><code>Peak concurrent requests</code></td><td>Maximum number of requests being processed simultaneously on the server side. May exceed client-side <code>Max request concurrency</code> due to queueing.</td><td>Integer, no decimal places</td></tr>
+    <tr><td><code>Total token throughput (tok/s)</code></td><td>Sum of input and output token throughputs. Formula: <code>Input token throughput + Output token throughput</code>.</td><td>2 decimal places</td></tr>
+    <tr><td><code>Concurrency</code></td><td>Average number of concurrent requests during the test (Little's Law). Formula: <code>Sum of all E2E latencies / Benchmark duration</code>.</td><td>2 decimal places</td></tr>
+  </tbody>
+</table>
+
+##### End-to-End Latency (E2E Latency)
+
+<table>
+  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>
+  <tbody>
+    <tr><td><code>Mean E2E Latency (ms)</code></td><td>Arithmetic mean</td><td>2 decimal places</td></tr>
+    <tr><td><code>Median E2E Latency (ms)</code></td><td>50th percentile</td><td>2 decimal places</td></tr>
+    <tr><td><code>P90 E2E Latency (ms)</code></td><td>90th percentile (90% of requests have latency ≤ this value)</td><td>2 decimal places</td></tr>
+    <tr><td><code>P99 E2E Latency (ms)</code></td><td>99th percentile</td><td>2 decimal places</td></tr>
+  </tbody>
+</table>
+
+##### Time to First Token (TTFT)
+
+<table>
+  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>
+  <tbody>
+    <tr><td><code>Mean TTFT (ms)</code></td><td>Arithmetic mean</td><td>2 decimal places</td></tr>
+    <tr><td><code>Median TTFT (ms)</code></td><td>50th percentile</td><td>2 decimal places</td></tr>
+    <tr><td><code>P99 TTFT (ms)</code></td><td>99th percentile</td><td>2 decimal places</td></tr>
+  </tbody>
+</table>
+
+##### Time per Output Token (TPOT) – Excluding First Token
+
+Formula: <code>(E2E Latency - TTFT) / (Number of output tokens - 1)</code>
+
+<table>
+  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>
+  <tbody>
+    <tr><td><code>Mean TPOT (ms)</code></td><td>Arithmetic mean</td><td>2 decimal places</td></tr>
+    <tr><td><code>Median TPOT (ms)</code></td><td>50th percentile</td><td>2 decimal places</td></tr>
+    <tr><td><code>P99 TPOT (ms)</code></td><td>99th percentile</td><td>2 decimal places</td></tr>
+  </tbody>
+</table>
+
+##### Inter-Token Latency (ITL)
+
+<table>
+  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>
+  <tbody>
+    <tr><td><code>Mean ITL (ms)</code></td><td>Average inter-token interval</td><td>2 decimal places</td></tr>
+    <tr><td><code>Median ITL (ms)</code></td><td>50th percentile inter-token interval</td><td>2 decimal places</td></tr>
+    <tr><td><code>P95 ITL (ms)</code></td><td>95th percentile (used to detect stalls)</td><td>2 decimal places</td></tr>
+    <tr><td><code>P99 ITL (ms)</code></td><td>99th percentile</td><td>2 decimal places</td></tr>
+    <tr><td><code>Max ITL (ms)</code></td><td>Maximum observed inter-token interval; useful for identifying severe blocking events</td><td>2 decimal places</td></tr>
+  </tbody>
+</table>
+
+## 3. Online Service: Multimodal Model
+
+Test `Qwen/Qwen2.5-VL-7B-Instruct` for vision-language tasks.
+
+<Note>Before running any benchmark in this section, make sure the SGLang multimodal server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Multimodal tab for the launch command.</Note>
+
+<Tip>For consistent, repeatable results, set `--random-range-ratio 1` to fix input/output lengths, or `0` (default) for uniform distribution.</Tip>
+
+### 3.1 Using Evalscope
+
+<Note>Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang multimodal server running at `http://127.0.0.1:30000`.</Note>
+
+Evalscope's `perf` tool uses the OpenAI-compatible `/v1/chat/completions` endpoint. Use `--dataset random_vl` for randomized multimodal data with image generation:
+
+```shell Command
+evalscope perf \
+  --parallel 10 \
+  --number 20 \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --url http://127.0.0.1:30000/v1/chat/completions \
+  --api openai \
+  --dataset random_vl \
+  --min-tokens 1024 \
+  --max-tokens 1024 \
+  --prefix-length 0 \
+  --min-prompt-length 1024 \
+  --max-prompt-length 1024 \
+  --image-width 512 \
+  --image-height 512 \
+  --image-format RGB \
+  --image-num 1 \
+  --tokenizer-path Qwen/Qwen2.5-VL-7B-Instruct \
+  --extra-args '{"ignore_eos": true}'
+```
+
+<Tip>If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.</Tip>
+
+### 3.2 Using AISBench
+
+<Note>Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands run from the `benchmark/` directory. SGLang multimodal server running at `http://127.0.0.1:30000`. AISBench does not include a built-in multimodal dataset — you must provide your own.</Note>
+
+First, edit `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py` to configure the vision model:
+
+```python vllm_api_stream_chat.py
+from ais_bench.benchmark.models import VLLMCustomAPIChat
+from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content
+
+models = [
+    dict(
+        attr="service",
+        type=VLLMCustomAPIChat,
+        abbr="vllm-api-stream-chat",
+        path="Qwen/Qwen2.5-VL-7B-Instruct",
+        model="Qwen/Qwen2.5-VL-7B-Instruct",
+        stream=True,
+        request_rate=0,
+        use_timestamp=False,
+        retry=2,
+        api_key="",
+        host_ip="127.0.0.1",
+        host_port=30000,
+        url="",
+        max_out_len=256,
+        batch_size=16,
+        trust_remote_code=False,
+        generation_kwargs=dict(
+            temperature=0.01,
+            ignore_eos=True,
+        ),
+        pred_postprocessor=dict(type=extract_non_reasoning_content),
+    )
+]
+```
+
+<Tip>If the model has already been downloaded, point `path` to the local model path instead of the model id.</Tip>
+
+Next, download a multimodal dataset such as mmstar:
+
+```shell Command
+# Download the mmstar dataset (from within the benchmark/ directory)
+cd ais_bench/datasets
+mkdir mmstar
+cd mmstar
+wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
+```
+
+Run the performance test:
+
+```shell Command
+ais_bench --models vllm_api_stream_chat --datasets mmstar_gen -m perf
+```
+
+Example output (for illustration only — actual results depend on your hardware and configuration):
+
+```text
+╒══════════════════════════╤═════════╤═════════════════╤════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤══════╕
+│ Performance Parameters   │ Stage   │ Average         │ Min            │ Max             │ Median          │ P75             │ P90             │ P99             │  N   │
+╞══════════════════════════╪═════════╪═════════════════╪════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪══════╡
+│ E2EL                     │ total   │ 6190.9 ms       │ 5071.4 ms      │ 8464.8 ms       │ 6126.6 ms       │ 6475.2 ms       │ 6833.5 ms       │ 7897.9 ms       │ 1500 │
+├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
+│ TTFT                     │ total   │ 693.3 ms        │ 96.0 ms        │ 2161.5 ms       │ 747.4 ms        │ 870.9 ms        │ 1032.3 ms       │ 1620.8 ms       │ 1500 │
+├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
+│ TPOT                     │ total   │ 21.6 ms         │ 17.8 ms        │ 32.1 ms         │ 21.3 ms         │ 23.1 ms         │ 24.5 ms         │ 29.1 ms         │ 1500 │
+├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
+│ ITL                      │ total   │ 25.5 ms         │ 0.0 ms         │ 1951.1 ms       │ 18.8 ms         │ 19.7 ms         │ 37.3 ms         │ 121.8 ms        │ 1500 │
+├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
+│ InputTokens              │ total   │ 0.0             │ 0.0            │ 0.0             │ 0.0             │ 0.0             │ 0.0             │ 0.0             │ 1500 │
+├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
+│ OutputTokens             │ total   │ 256.0           │ 256.0          │ 256.0           │ 256.0           │ 256.0           │ 256.0           │ 256.0           │ 1500 │
+├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
+│ OutputTokenThroughput    │ total   │ 41.6779 token/s │ 30.243 token/s │ 50.4791 token/s │ 41.7847 token/s │ 44.6424 token/s │ 45.6484 token/s │ 46.0932 token/s │ 1500 │
+╘══════════════════════════╧═════════╧═════════════════╧════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧══════╛
+╒═════════════════════════╤═════════╤══════════════════╕
+│ Common Metric           │ Stage   │ Value            │
+╞═════════════════════════╪═════════╪══════════════════╡
+│ Benchmark Duration      │ total   │ 582099.6816 ms   │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Total Requests          │ total   │ 1500             │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Failed Requests         │ total   │ 0                │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Success Requests        │ total   │ 1500             │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Concurrency             │ total   │ 15.9532          │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Max Concurrency         │ total   │ 16               │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Request Throughput      │ total   │ 2.5769 req/s     │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Total Input Tokens      │ total   │ 0                │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Total Generated Tokens  │ total   │ 384000           │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Input Token Throughput  │ total   │ 0.0 token/s      │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Output Token Throughput │ total   │ 659.6808 token/s │
+├─────────────────────────┼─────────┼──────────────────┤
+│ Total Token Throughput  │ total   │ 659.6808 token/s │
+╘═════════════════════════╧═════════╧══════════════════╛
+```
+
+<Note>See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details.</Note>
+
+### 3.3 Using bench_serving (image dataset)
+
+Set `--dataset-name image` for image datasets. `bench_serving` will generate random prompts with image inputs. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark.
+
+<Note>See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for the full list of image-related flags.</Note>
+
+```shell Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --base-url http://127.0.0.1:30000 \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --dataset-name image \
+  --random-input-len 1024 \
+  --random-output-len 512 \
+  --random-range-ratio 1 \
+  --num-prompts 32 \
+  --max-concurrency 16 \
+  --image-count 1 \
+  --image-resolution 720p
+```
+
+Example output (for illustration only — actual results depend on your hardware and configuration):
+
+```text
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     32
+Benchmark duration (s):                  51.74
+Total input tokens:                      73464
+Total input text tokens:                 35128
+Total input vision tokens:               38336
+Total generated tokens:                  16384
+Total generated tokens (retokenized):    9300
+Request throughput (req/s):              0.62
+Input token throughput (tok/s):          1419.96
+Output token throughput (tok/s):         316.68
+Peak output token throughput (tok/s):    800.00
+Peak concurrent requests:                32
+Total token throughput (tok/s):          1736.64
+Concurrency:                             15.98
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   25841.84
+Median E2E Latency (ms):                 25842.85
+P90 E2E Latency (ms):                    26296.42
+P99 E2E Latency (ms):                    26303.13
+---------------Time to First Token----------------
+Mean TTFT (ms):                          12211.59
+Median TTFT (ms):                        14405.77
+P99 TTFT (ms):                           15837.60
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          26.67
+Median TPOT (ms):                        21.75
+P99 TPOT (ms):                           41.89
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           26.67
+Median ITL (ms):                         20.34
+P95 ITL (ms):                            20.85
+P99 ITL (ms):                            21.70
+Max ITL (ms):                            11309.91
+==================================================
+```
+
+## 4. Online Service: Embedding Model
+
+Test `Qwen/Qwen3-Embedding-8B` on the embedding API endpoint.
+
+<Note>Before running any benchmark in this section, make sure the SGLang embedding server is running with `--is-embedding` at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Embedding tab for the launch command. AISBench does not support embedding endpoints — use `bench_serving` or Evalscope instead.</Note>
+
+### 4.1 Using Evalscope
+
+<Note>Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang embedding server running with `--is-embedding` at `http://127.0.0.1:30000`.</Note>
+
+Evalscope supports embedding evaluation. For performance testing the embedding API directly:
+
+```shell Command
+evalscope perf \
+  --parallel 10 \
+  --number 20 \
+  --model Qwen/Qwen3-Embedding-8B \
+  --url http://127.0.0.1:30000/v1/embeddings \
+  --api openai_embedding \
+  --dataset random_embedding \
+  --min-prompt-length 1024 \
+  --max-prompt-length 1024 \
+  --tokenizer-path Qwen/Qwen3-Embedding-8B
+```
+
+<Tip>If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.</Tip>
+
+<Note>Evalscope's embedding performance testing support may vary by version. If the `perf` command does not accept the embeddings endpoint, use [`bench_serving` with `--backend sglang-embedding`](#4-2-using-bench_serving-embedding-backend) as the primary option.</Note>
+
+### 4.2 Using bench_serving (embedding backend)
+
+`bench_serving` is built into SGLang. Use `--backend sglang-embedding` to target the `/v1/embeddings` endpoint. Make sure the server is running with `--is-embedding` at `http://127.0.0.1:30000`.
+
+```shell Command
+python -m sglang.bench_serving \
+  --backend sglang-embedding \
+  --base-url http://127.0.0.1:30000 \
+  --model Qwen/Qwen3-Embedding-8B \
+  --dataset-name random \
+  --random-input-len 512 \
+  --random-output-len 0 \
+  --num-prompts 1000 \
+  --max-concurrency 64 \
+  --request-rate 32
+```
+
+<Note>`--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available. Set `--random-output-len 0` for embedding benchmarks — no output tokens are generated.</Note>
+
+Example output (for illustration only — actual results depend on your hardware and configuration):
+
+```text
+============ Serving Benchmark Result ============
+Backend:                                 sglang-embedding
+Traffic request rate:                    32.0
+Max request concurrency:                 64
+Successful requests:                     1000
+Benchmark duration (s):                  31.86
+Total input tokens:                      257891
+Total input text tokens:                 257891
+Request throughput (req/s):              31.39
+Input token throughput (tok/s):          8094.67
+Peak concurrent requests:                62
+Concurrency:                             6.67
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   212.34
+Median E2E Latency (ms):                 160.97
+P90 E2E Latency (ms):                    267.31
+P99 E2E Latency (ms):                    1445.94
+==================================================
+```
+
+## 5. Offline Performance Testing
+
+SGLang's `Engine` API runs inference in-process, without an HTTP server, letting you measure maximum throughput. `bench_offline_throughput` is built into SGLang and requires no extra installation or running server.
+
+<Note>`bench_offline_throughput` currently only supports text-generation (LLM) benchmarks. Multimodal and embedding models are not supported.</Note>
+
+### 5.1 Using bench_offline_throughput
+
+`bench_offline_throughput` uses the `Engine` API internally and measures pure inference throughput without HTTP overhead:
+
+```shell Command
+python -m sglang.bench_offline_throughput \
+  --model-path Qwen/Qwen2.5-7B-Instruct \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 512 \
+  --num-prompts 500
+```
+
+<Note>`--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available.</Note>
+
+<Tip>`--dataset-name random` with `--random-input-len` and `--random-output-len` gives you full control over input/output token counts. Fixed-length random data eliminates variance from real datasets, making throughput comparisons across runs deterministic and reliable.</Tip>
+
+## See also
+
+- [Bench Serving Guide](/docs/developer_guide/bench_serving) — all backends, datasets, and advanced options for `bench_serving`
+- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — environment setup for Ascend NPUs
+- [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) — full Evalscope documentation
+- [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) — full AISBench documentation

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx

@@ -0,0 +1,545 @@
+---
+title: "Ascend NPU Performance Profiling Guide"
+metatags:
+  description: "Ascend NPU performance profiling guide: use SGLang's built-in PyTorch Profiler for operator-level performance analysis of Ascend NPU inference services."
+---
+
+During inference serving, it is sometimes necessary to monitor the internal
+execution flow of the serving framework to identify performance issues. By
+collecting start/end timestamps of key flows, identifying critical functions or
+iterations, recording key events, and gathering relevant information, you can
+quickly locate performance bottlenecks.
+
+This guide walks you through the complete workflow of collecting performance
+data in an SGLang Ascend NPU inference service — from preparation, collection,
+and analysis to visualization — helping you get started with performance
+profiling quickly.
+
+For more profiling scenarios (e.g., Nsight Systems, PD disaggregation, etc.),
+see [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling).
+
+## Ascend PyTorch Profiler
+
+SGLang has built-in PyTorch Profiler support. Through the Ascend `torch_npu`
+backend, you can directly collect NPU operator-level performance data. No
+additional packages are required — profiling start/stop is controlled via API
+requests.
+
+### 1. Environment Setup
+
+Launch an SGLang online service and set the `SGLANG_TORCH_PROFILER_DIR`
+environment variable to control where performance files are saved. Once the
+service starts, profiling is ready on standby.
+
+```shell Command
+# Set the performance data output directory
+export SGLANG_TORCH_PROFILER_DIR=./sglang_profile
+
+# Start SGLang server (use local model path or HuggingFace model id)
+sglang serve \
+  --model-path /path/to/your/model \
+  --attention-backend ascend \
+  --host 0.0.0.0 --port 30000 \
+  --tp-size 1 \
+  --max-running-requests 128
+```
+
+**Profiling-related environment variables:**
+
+<table>
+  <thead>
+    <tr>
+      <th>Variable</th>
+      <th>Description</th>
+      <th>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>SGLANG_TORCH_PROFILER_DIR</code></td>
+      <td>Trace file output directory</td>
+      <td><code>/tmp</code></td>
+    </tr>
+    <tr>
+      <td><code>SGLANG_PROFILE_WITH_STACK</code></td>
+      <td>Record Python call stack (True / False)</td>
+      <td><code>True</code></td>
+    </tr>
+    <tr>
+      <td><code>SGLANG_PROFILE_RECORD_SHAPES</code></td>
+      <td>Record operator input shapes (True / False)</td>
+      <td><code>True</code></td>
+    </tr>
+  </tbody>
+</table>
+
+<Note>
+Priority order: **API parameter > Environment variable > Default value**.
+</Note>
+
+### 2. Collection Methods
+
+SGLang provides four collection methods. The core differences are **whether you
+need to manually send `/start_profile` and `/stop_profile`**. All four methods
+produce identical results — choose the most convenient one.
+
+**Method comparison:**
+
+<table>
+  <thead>
+    <tr>
+      <th>Method</th>
+      <th>Manual start_profile</th>
+      <th>Manual stop_profile</th>
+      <th>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>A: API manual start/stop</td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Maximum flexibility for precise control</td>
+    </tr>
+    <tr>
+      <td>B: API auto-stop</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>Set <code>num_steps</code>, auto-stops and generates output</td>
+    </tr>
+    <tr>
+      <td>C: bench_serving --profile</td>
+      <td>No</td>
+      <td>No</td>
+      <td>Benchmark + profiling in one command</td>
+    </tr>
+    <tr>
+      <td>D: sglang.profiler CLI</td>
+      <td>No</td>
+      <td>No</td>
+      <td>Standalone profiling CLI tool</td>
+    </tr>
+  </tbody>
+</table>
+
+#### Method A: API Manual Start/Stop
+
+Send `/start_profile` to start → send workload requests → send `/stop_profile`
+to stop. After stopping, the server automatically parses the data — **no need to
+manually call `analyse()`**.
+
+```bash Command
+# Step 1: Start profiling (no num_steps, requires manual stop)
+curl -X POST http://127.0.0.1:30000/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{
+    "output_dir": "./sglang_profile",
+    "start_step": 1,
+    "activities": ["CPU", "GPU"]
+  }'
+
+# Step 2: Send workload requests (using curl as example)
+curl http://127.0.0.1:30000/generate \
+  -H "Content-Type: application/json" \
+  -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 10}}'
+
+# Step 3: Stop profiling
+curl -X POST http://127.0.0.1:30000/stop_profile
+```
+
+<Note>
+`/stop_profile` returns `"Stop profiling. This will take some time."` — the
+server needs time to flush trace data to disk and parse it. Wait for the
+response to complete.
+This method takes a significant amount of time to parse
+profiling data; consider using **Method B** instead to avoid lengthy waits.
+</Note>
+
+#### Method B: API Auto-Stop
+
+Specify `num_steps` in the `/start_profile` request. Profiling stops
+automatically after N steps and generates output — **no need to manually send
+`/stop_profile`**.
+
+```bash Command
+# num_steps=10, wait 3 warmup steps, auto-stop after 10 steps
+curl -X POST http://127.0.0.1:30000/start_profile \
+  -H "Content-Type: application/json" \
+  -d '{
+    "output_dir": "./sglang_profile",
+    "start_step": 3,
+    "num_steps": 10,
+    "activities": ["CPU", "GPU"]
+  }'
+
+# Just send workload — no /stop_profile needed
+curl http://127.0.0.1:30000/generate \
+  -H "Content-Type: application/json" \
+  -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}'
+```
+
+#### Method C: bench_serving --profile
+
+Use SGLang's built-in `bench_serving` with the `--profile` flag.
+**Automatically handles `/start_profile` and `/stop_profile`** — no manual API
+calls needed.
+
+```bash Command
+# With --profile-steps: auto-stops after N steps and generates output
+python -m sglang.bench_serving \
+  --backend sglang \
+  --base-url http://127.0.0.1:30000 \
+  --model /path/to/your/model \
+  --tokenizer /path/to/your/model \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 100 \
+  --num-prompts 10 \
+  --profile \
+  --profile-steps 10 \
+  --profile-output-dir ./sglang_profile
+
+# Without --profile-steps: /stop_profile sent automatically after benchmark
+python -m sglang.bench_serving \
+  --backend sglang \
+  --base-url http://127.0.0.1:30000 \
+  --model /path/to/your/model \
+  --tokenizer /path/to/your/model \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 100 \
+  --num-prompts 10 \
+  --profile \
+  --profile-output-dir ./sglang_profile
+```
+
+<Note>
+`--profile-steps N` sends `"num_steps": N` to the server's `/start_profile`, so
+the server auto-stops and parses data after N steps — bench_serving skips
+sending `/stop_profile`.
+</Note>
+
+<Note>
+`bench_serving --profile` creates a timestamp subdirectory inside
+`--profile-output-dir` (e.g. `<output_dir>/<timestamp>/`). The output path is
+shown in the server log as `Profiling done. Traces are saved to: <path>`.
+</Note>
+
+<Note>
+On Ascend NPU, SGLang uses `torch_npu._apply_patches()` to automatically
+redirect PyTorch Profiler's CUDA activity to NPU, so
+`activities: ["CPU", "GPU"]` actually captures NPU operator events.
+</Note>
+
+**`bench_serving --profile` parameters:**
+
+<table>
+  <thead>
+    <tr><th>Parameter</th><th>Description</th></tr>
+  </thead>
+  <tbody>
+    <tr><td><code>--profile</code></td><td>Enable auto profiling start/stop</td></tr>
+    <tr><td><code>--profile-steps N</code></td><td>Auto-stop after N steps (skips /stop_profile)</td></tr>
+    <tr><td><code>--profile-output-dir</code></td><td>Trace output directory</td></tr>
+  </tbody>
+</table>
+
+#### Method D: sglang.profiler CLI
+
+Use the `sglang.profiler` CLI module, which automatically sends
+`/start_profile` and waits for completion. **Start `sglang.profiler` first,
+then send inference requests** (otherwise there are no steps to capture and the
+profiler will wait indefinitely).
+
+```bash Command
+# Terminal 1: Start sglang.profiler first (sends /start_profile, then waits for completion)
+python3 -m sglang.profiler \
+  --url http://127.0.0.1:30000 \
+  --output-dir ./my_profiles \
+  --num-steps 3 \
+  --cpu --gpu &
+```
+
+```bash Command
+# Terminal 2: Wait for "Waiting for N steps" output from Terminal 1, then send requests.
+# The profiler starts recording once /start_profile is received by the server.
+# Requests sent before the server receives /start_profile will not be captured.
+curl http://127.0.0.1:30000/generate \
+  -H "Content-Type: application/json" \
+  -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}'
+```
+
+A simpler and more reliable approach is to use `bench_serving --profile`, which
+handles both steps automatically:
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --base-url http://127.0.0.1:30000 \
+  --model /path/to/your/model \
+  --tokenizer /path/to/your/model \
+  --dataset-name random \
+  --random-input-len 128 \
+  --random-output-len 32 \
+  --num-prompts 10 \
+  --profile \
+  --profile-steps 3 \
+  --profile-output-dir ./my_profiles
+```
+
+<Note>
+`sglang.profiler` is essentially a CLI wrapper around the `/start_profile` API.
+Advanced options like `--profile-by-stage` are also supported. On Ascend NPU,
+trace flushing is asynchronous and may take a while — the CLI may occasionally
+block waiting for flush. If it times out, use Method B (API auto-stop) or
+Method C (bench_serving --profile) instead.
+</Note>
+
+**`sglang.profiler` CLI parameters:**
+
+<table>
+  <thead>
+    <tr><th>Parameter</th><th>Description</th></tr>
+  </thead>
+  <tbody>
+    <tr><td><code>--url</code></td><td>SGLang server address</td></tr>
+    <tr>
+      <td><code>--output-dir</code></td>
+      <td>Output directory (defaults to <code>SGLANG_TORCH_PROFILER_DIR</code>)</td>
+    </tr>
+    <tr><td><code>--num-steps</code></td><td>Number of steps to profile</td></tr>
+    <tr>
+      <td><code>--profile-by-stage</code></td>
+      <td>Profile prefill / decode stages separately</td>
+    </tr>
+    <tr><td><code>--profile-prefix</code></td><td>Trace filename prefix</td></tr>
+    <tr>
+      <td><code>--cpu</code> / <code>--gpu</code> / <code>--mem</code> / <code>--rpd</code></td>
+      <td>Activity types to collect</td>
+    </tr>
+  </tbody>
+</table>
+
+### 3. Full Parameter Reference
+
+All methods ultimately send a `/start_profile` request to the server. The full
+set of supported parameters:
+
+<table>
+  <thead>
+    <tr>
+      <th>Parameter</th>
+      <th>Description</th>
+      <th>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>output_dir</code></td>
+      <td>
+        Output directory. Falls back to
+        <code>SGLANG_TORCH_PROFILER_DIR</code> or <code>/tmp</code>
+      </td>
+      <td><code>/tmp</code></td>
+    </tr>
+    <tr>
+      <td><code>num_steps</code></td>
+      <td>
+        Number of steps. If set, profiling auto-stops — no /stop_profile needed
+      </td>
+      <td>None</td>
+    </tr>
+    <tr>
+      <td><code>start_step</code></td>
+      <td>
+        Step index to start profiling (inclusive), for skipping warmup
+      </td>
+      <td>0</td>
+    </tr>
+    <tr>
+      <td><code>activities</code></td>
+      <td>
+        Activity types: CPU, GPU, MEM, RPD. On Ascend NPU, only CPU and GPU are supported.
+        MEM depends on CUDA memory APIs and will be silently ignored. RPD requires ROCm and will cause an error.
+      </td>
+      <td><code>["CPU", "GPU"]</code></td>
+    </tr>
+    <tr>
+      <td><code>profile_by_stage</code></td>
+      <td>Profile prefill and decode stages separately</td>
+      <td><code>false</code></td>
+    </tr>
+    <tr>
+      <td><code>with_stack</code></td>
+      <td>
+        Record Python call stack. Also controllable via
+        <code>SGLANG_PROFILE_WITH_STACK</code>
+      </td>
+      <td><code>true</code></td>
+    </tr>
+    <tr>
+      <td><code>record_shapes</code></td>
+      <td>
+        Record operator input shapes. Also controllable via
+        <code>SGLANG_PROFILE_RECORD_SHAPES</code>
+      </td>
+      <td><code>true</code></td>
+    </tr>
+    <tr>
+      <td><code>profile_prefix</code></td>
+      <td>Prefix for trace filenames</td>
+      <td>None</td>
+    </tr>
+    <tr>
+      <td><code>profile_stages</code></td>
+      <td>
+        Stages to profile, e.g. <code>["prefill", "decode"]</code>.
+        Requires <code>profile_by_stage</code>
+      </td>
+      <td>None</td>
+    </tr>
+  </tbody>
+</table>
+
+### 4. Finding Output Files
+
+**The server log explicitly indicates where traces are saved.** You can find
+them via:
+
+- **When profiling starts**: server log outputs
+  `Profiling starts. Traces will be saved to: <path> (with profile id: <id>)`
+
+```text
+[2026-05-19 13:23:15] Profiling starts. Traces will be saved to: /tmp/1779196995.6948605 (with profile id: 1779196995.6979997)
+[2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text.
+[2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text.
+[2026-05-19 13:23:15] INFO:     127.0.0.1:40714 - "POST /start_profile HTTP/1.1" 200 OK
+```
+
+- **When profiling stops**: server log outputs
+  `Profiling done. Traces are saved to: <path>`
+
+```text
+[2026-05-19 13:23:17] Stop profiling...
+[2026-05-19 13:23:17] [WARNING] [350443] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
+[rank0]:[W519 13:23:17.084812760 compiler_depend.ts:3136] Warning: The indexFromRank 0is not equal indexFromCurDevice 4 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator())
+[2026-05-19 13:23:17] [INFO] [352725] profiler.py: Start parsing profiling data: /tmp/1779196995.6948605/localhost.localdomain_350443_20260519132315700_ascend_pt
+[2026-05-19 13:23:22] [INFO] [352734] profiler.py: CANN profiling data parsed in a total time of 0:00:04.022310
+[2026-05-19 13:23:32] [INFO] [352725] profiler.py: All profiling data parsed in a total time of 0:00:14.305669
+[2026-05-19 13:23:32] Profiling done. Traces are saved to: /tmp/1779196995.6948605
+```
+
+- **CLI output**: `sglang.profiler` outputs `Dump profiling traces to <path>`
+
+```text
+Dump profiling traces to /tmp/1779243331.3219
+Waiting for 10 steps and the trace to be flushed.... (profile_by_stage=False)
+```
+
+The directory structure is
+`<output_dir>/<hostname>_<pid>_<timestamp>_ascend_pt/`. When using Method C
+(`bench_serving --profile`), a timestamp subdirectory is added:
+`<output_dir>/<timestamp>/<hostname>_<pid>_<timestamp>_ascend_pt/`. Always check the server log for the exact path:
+`Profiling done. Traces are saved to: <path>`.
+
+### 5. Viewing Results
+
+After profiling stops (either `/stop_profile` returns or `num_steps`
+auto-triggers), the server **automatically parses the raw data**. The
+`ASCEND_PROFILER_OUTPUT` directory directly contains the following visualization
+files — **no need to manually call `analyse()`**:
+
+<table>
+  <thead>
+    <tr><th>File</th><th>Description</th></tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>trace_view.json</code></td>
+      <td>
+        Chrome Tracing format. Open in
+        <a href="https://www.hiascend.com/document/detail/zh/mindstudio/81RC1/GUI_baseddevelopmenttool/msascendinsightug/Insight_userguide_0002.html">MindStudio Insight</a>
+      </td>
+    </tr>
+    <tr><td><code>analysis.db</code></td><td>Database-format performance data</td></tr>
+    <tr>
+      <td><code>ascend_pytorch_profiler_0.db</code></td>
+      <td>Database-format performance data</td>
+    </tr>
+    <tr><td><code>kernel_details.csv</code></td><td>Kernel-level data</td></tr>
+    <tr><td><code>operator_details.csv</code></td><td>Operator-level data</td></tr>
+    <tr><td><code>step_trace_time.csv</code></td><td>Step trace timing data</td></tr>
+  </tbody>
+</table>
+
+<Note>
+`trace_view.json` can also be opened using Chrome's built-in
+`chrome://tracing` or [Perfetto UI](https://ui.perfetto.dev/).
+</Note>
+
+<Note>
+In a multi-node deployment, each node generates its own trace files.
+To merge them into a single unified trace, set `"merge_profiles": true` in the `/start_profile` request.
+However, on Ascend NPU, the merge feature does **not** fully support the `*_ascend_pt` format — merged results may be incomplete or incorrect.
+It is recommended to view `trace_view.json` on each node individually instead.
+For more details, see [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling#profiler-trace-merger-for-distributed-traces).
+</Note>
+
+### 6. Re-parsing Raw Data (Optional)
+
+If you need to **re-parse existing data** with different parameters, or if
+profiling was interrupted and `ASCEND_PROFILER_OUTPUT` was not auto-generated,
+use `torch_npu`'s `analyse()` tool:
+
+```python
+from torch_npu.profiler.profiler import analyse
+analyse("./sglang_profile/<hostname>_*_ascend_pt/")
+```
+
+<Note>
+Normally **no need** to manually run `analyse()` — the server already parses
+data automatically. Only use this for re-parsing or handling interrupted data.
+Running `analyse()` when ASCEND_PROFILER_OUTPUT already exists will overwrite the original directory.
+If the original data is still needed, back it up before running `analyse()`.
+</Note>
+
+## Best Practices
+
+### Common Notes
+
+- **Finding output**: Check the server log for
+  `Profiling starts. Traces will be saved to: <path>` and
+  `Profiling done. Traces are saved to: <path>`, or `sglang.profiler` output for
+  `Dump profiling traces to <path>`.
+- **Control trace file size**: Reduce the number of requests and output length
+  using `--num-prompts` and `--random-output-len` to avoid trace files too large
+  for browsers.
+- **Warmup iterations**: Set `start_step` to skip the first few warmup steps and
+  capture performance data under steady state.
+- **Profile step count**: Large values for `num_steps` or `--profile-steps` can
+  lead to lengthy profiling data parsing times. Reduce these values
+  appropriately when you only need a quick overview.
+- **CUDA Graph impact**: To see the full Python call stack → operator mapping in
+  traces, add `--disable-cuda-graph` when starting the server. Note that this
+  reduces decode performance — only use during profiling. To analyze CUDA Graph
+  capture specifically, use `--enable-profile-cuda-graph` — traces are saved to
+  `SGLANG_TORCH_PROFILER_DIR/graph_capture_profile/`.
+- **Multi-node deployment**: In multi-node environments, performance data is
+  distributed across nodes. On Ascend NPU, the `merge_profiles` feature has
+  limited support — check `*_ascend_pt/ASCEND_PROFILER_OUTPUT/trace_view.json`
+  on each node individually. In PD disaggregation mode, prefill and decode
+  workers must be profiled separately — see
+  [Profile In PD Disaggregation Mode](/docs/developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode).
+
+## See Also
+
+- [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling)
+  — General SGLang profiling guide
+- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start)
+  — Ascend NPU environment setup
+- [Ascend NPU Optimization](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization)
+  — Ascend NPU optimization parameters
+- [Ascend NPU Performance Testing](/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing)
+  — Ascend NPU performance benchmarking
+- [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables)
+  — Environment variable reference

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx

@@ -0,0 +1,507 @@
+---
+title: "Quantization on Ascend"
+metatags:
+    description: "Load, export, and serve quantized models on Ascend NPUs with SGLang."
+---
+To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
+
+SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
+
+[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
+<table>
+  <thead>
+    <tr>
+      <th>Quantization scheme</th>
+      <th>Layer type</th>
+      <th>A2 Supported</th>
+      <th>A3 Supported</th>
+      <th>A5 Supported</th>
+      <th>Diffusion models</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>W4A4 dynamic</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+    </tr>
+    <tr>
+      <td>W8A8 static</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+    </tr>
+    <tr>
+      <td>W8A8 dynamic</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+    </tr>
+    <tr>
+      <td>MXFP8 (<a href="https://github.com/sgl-project/sglang/pull/20922">Diffusion</a>, <a href="https://github.com/sgl-project/sglang/pull/22352">LLM dense</a>)</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/22338">MXFP4</a></td>
+      <td>Linear</td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/23650">MXFP4 W4A8</a></td>
+      <td>Linear</td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+    </tr>
+    <tr>
+      <td>W4A4 dynamic</td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+    </tr>
+    <tr>
+      <td>W4A8 dynamic</td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+    </tr>
+    <tr>
+      <td>W8A8 dynamic</td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/20922">MXFP8</a></td>
+      <td>MoE</td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'blue'}}>WIP</strong></td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
+<table>
+  <thead>
+    <tr>
+      <th>Quantization scheme</th>
+      <th>Layer type</th>
+      <th>A2 Supported</th>
+      <th>A3 Supported</th>
+      <th>A5 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>W4A16</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td>W8A16</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td>W4A16</td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+GPTQ on Ascend support
+<table>
+  <thead>
+    <tr>
+      <th>Quantization scheme</th>
+      <th>Layer type</th>
+      <th>A2 Supported</th>
+      <th>A3 Supported</th>
+      <th>A5 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/15203">W4A16</a></td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/15203">W8A16</a></td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/16364">W4A16 MOE</a></td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/16364">W8A16 MOE</a></td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
+<table>
+  <thead>
+    <tr>
+      <th>Quantization scheme</th>
+      <th>Layer type</th>
+      <th>A2 Supported</th>
+      <th>A3 Supported</th>
+      <th>A5 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>W4A16</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td>W8A16</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td>W4A16</td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td>W8A16</td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+Compressed-tensors (LLM Compressor) on Ascend support:
+<table>
+  <thead>
+    <tr>
+      <th>Quantization scheme</th>
+      <th>Layer type</th>
+      <th>A2 Supported</th>
+      <th>A3 Supported</th>
+      <th>A5 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/14504">W8A8 dynamic</a></td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/14736">W4A8 dynamic with/without activation clip</a></td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/12759">W4A16 MOE</a></td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td><a href="https://github.com/sgl-project/sglang/pull/14504">W8A8 dynamic</a></td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
+<table>
+  <thead>
+    <tr>
+      <th>Quantization type</th>
+      <th>Layer type</th>
+      <th>A2 Supported</th>
+      <th>A3 Supported</th>
+      <th>A5 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>All GGUF types (standard, K-quant)</td>
+      <td>Linear</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+    <tr>
+      <td>All GGUF types (standard, K-quant)</td>
+      <td>MoE</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+**Usage Examples:**
+
+- Dense model (e.g. Qwen3-14B-Q4_K_M.gguf):
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen3-14B-Q4_K_M.gguf \
+    --device npu --attention-backend ascend \
+    --host 0.0.0.0 --port 30000 \
+    --mem-fraction-static 0.7 --tp-size 2
+```
+
+- MoE model (e.g. Qwen3-30B-A3B-Q4_K_M.gguf):
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen3-30B-A3B-Q4_K_M.gguf \
+    --device npu --attention-backend ascend \
+    --host 0.0.0.0 --port 30000 \
+    --mem-fraction-static 0.8 --tp-size 2
+```
+
+> **Implementation Notes:**
+>
+> - GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead).
+> - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation.
+> - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.
+
+**MXFP8 for LLM dense models (e.g. Qwen3 / Qwen3.5):**
+
+LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend A5 series or newer (`npu_dynamic_mx_quant` is not available on A2 / A3).
+
+- Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time):
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --quantization mxfp8 \
+    --device npu --attention-backend ascend \
+    --host 0.0.0.0 --port 30000 \
+    --mem-fraction-static 0.8 --tp-size 1
+```
+
+- Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`):
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path /path/to/Qwen3-8B-W8A8-MXFP8 \
+    --device npu --attention-backend ascend \
+    --host 0.0.0.0 --port 30000 \
+    --mem-fraction-static 0.8 --tp-size 1
+```
+
+> **Implementation Notes:**
+> - Online path: `Fp8Config.get_quant_method()` dispatches to `NPUMXFP8LinearMethod`. Weights are quantized once at load via `npu_dynamic_mx_quant(weight, dst_type=torch_npu.float8_e4m3fn)` and pre-transposed to `[in, out]`; activations are per-token quantized at inference and matmul runs via `npu_quant_matmul(..., group_sizes=[1, 1, 32])` (block_size = 32).
+> - Offline path: `ModelSlimMXFP8Scheme` loads `float8_e4m3fn` weights + `float8_e8m0fnu` block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (`.data` assignment) — calling `.contiguous()` would physically reorder the pre-quantized layout and break the block-scale mapping.
+> - MoE MXFP8 (FusedMoE/TP) for LLMs is tracked separately and not part of this PR.
+
+**MXFP4 W4A8 for LLM dense models (e.g. Qwen3 / Qwen3.5):**
+
+LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend A5 series or newer.
+
+- Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time):
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path Qwen/Qwen3-8B \
+    --quantization mxfp_w4a8 \
+    --device npu --attention-backend ascend \
+    --host 0.0.0.0 --port 30000 \
+    --mem-fraction-static 0.8 --tp-size 1
+```
+
+- Offline W4A8 quantization (msmodelslim pre-quantized weights, `W4A8_MXFP` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`).
+
+> **Implementation Notes:**
+> - Weights are packed FP4 (`float4_e2m1fn_x2`, two nibbles per byte) with a UE8M0 per-block shared exponent (block_size = 32); activations are per-token MXFP8. Matmul runs via `npu_quant_matmul(..., x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[0, 0, 32])`.
+> - The packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype` / `input_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject.
+> - Online and offline share the same kernel path and layout; they differ only in the weight source (RTN at load vs msmodelslim calibration).
+
+## Diffusion Model Quantization on Ascend NPU
+
+SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires A5; the ModelSlim W8A8/W4A4 schemes work on A2/A3.
+
+**Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend A5
+
+<table>
+  <thead>
+    <tr>
+      <th>Quantization method</th>
+      <th><code>quant_type</code> in JSON</th>
+      <th>Scheme class</th>
+      <th>Mode</th>
+      <th>A2/A3 Supported</th>
+      <th>A5 Supported</th>
+      <th>Trigger</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>MXFP8 (W8A8)</td>
+      <td>—</td>
+      <td><code>MXFP8Config</code></td>
+      <td>Online</td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><code>--quantization mxfp8</code></td>
+    </tr>
+    <tr>
+      <td>MXFP8 (W8A8)</td>
+      <td><code>W8A8_MXFP8</code></td>
+      <td><code>ModelSlimMXFP8Scheme</code></td>
+      <td>Offline</td>
+      <td><strong style={{color: 'red'}}>x</strong></td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td>auto-detected from <code>quant_model_description.json</code></td>
+    </tr>
+    <tr>
+      <td>W8A8 static</td>
+      <td><code>W8A8</code></td>
+      <td><code>ModelSlimW8A8Int8</code></td>
+      <td>Offline</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td>auto-detected from <code>quant_model_description.json</code></td>
+    </tr>
+    <tr>
+      <td>W8A8 dynamic</td>
+      <td><code>W8A8_DYNAMIC</code></td>
+      <td><code>ModelSlimW8A8Int8</code></td>
+      <td>Offline</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td>auto-detected from <code>quant_model_description.json</code></td>
+    </tr>
+    <tr>
+      <td>W4A4 dynamic</td>
+      <td><code>W4A4_DYNAMIC</code></td>
+      <td><code>ModelSlimW4A4Int4</code></td>
+      <td>Offline</td>
+      <td><strong style={{color: 'green'}}>√</strong></td>
+      <td><strong style={{color: 'orange'}}>TBD</strong></td>
+      <td>auto-detected from <code>quant_model_description.json</code></td>
+    </tr>
+  </tbody>
+</table>
+
+### Online MXFP8 Quantization
+
+Online quantization dynamically quantizes FP16/BF16 weights to MXFP8 at load time using `npu_dynamic_mx_quant` + `npu_quant_matmul` CANN kernels. Pass `--quantization mxfp8` to override auto-detection.
+
+```bash Command
+# Start the diffusion server with online MXFP8 quantization
+sglang serve \
+  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
+  --quantization mxfp8 \
+  --num-gpus 4
+```
+
+```bash Command
+# One-shot generation
+sglang generate \
+  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
+  --quantization mxfp8 \
+  --prompt "a beautiful sunset over the mountains" \
+  --save-output
+```
+
+### Offline MXFP8 Quantization (ModelSlim)
+
+For offline quantization, pre-quantize the model with msModelSlim and load the resulting checkpoint. The quantization scheme is auto-detected from `quant_model_description.json`, so no extra `--quantization` flag is needed.
+
+**Step 1: Quantize with msModelSlim**
+
+```bash Command
+msmodelslim quant \
+  --model_path /path/to/wan2_2_float_weights \
+  --save_path /path/to/wan2_2_mxfp8_weights \
+  --device npu \
+  --model_type Wan2_2 \
+  --quant_type mxfp8 \
+  --trust_remote_code True
+```
+
+> Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim.
+
+**Step 2: Convert to Diffusers format**
+
+msModelSlim saves quantized Wan2.2 weights in the original Wan format. Convert to Diffusers format using the provided repack script:
+
+```bash Command
+python python/sglang/multimodal_gen/tools/wan_repack.py \
+  --input-path /path/to/wan2_2_mxfp8_weights \
+  --output-path /path/to/wan2_2_mxfp8_diffusers
+```
+
+Then copy all files from the original Diffusers checkpoint (except the `transformer`/`transformer_2` folders) into the output directory.
+
+**Step 3: Run inference**
+
+```bash Command
+sglang generate \
+  --model-path /path/to/wan2_2_mxfp8_diffusers \
+  --prompt "a beautiful sunset over the mountains" \
+  --save-output
+```
+
+For pre-quantized checkpoints available on ModelScope, see [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx

@@ -0,0 +1,150 @@
+---
+title: "Ascend NPU Quickstart"
+metatags:
+  description: "Quickstart for running SGLang on Ascend NPUs with the official container image, including server launch and test request examples."
+---
+
+<Note>
+This page covers only the simplest deployment flow using the official container image. For the complete installation guide
+across all scenarios (source install, Docker build, system settings, version mapping, etc.),
+see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus/ascend_npu).
+</Note>
+
+## Prerequisites
+
+### Supported Devices
+
+- Atlas 800I A2 inference series (Atlas 800I A2)
+- Atlas 800I A3 inference series (Atlas 800I A3)
+
+## Setup environment using container
+
+<Warning>
+Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.
+</Warning>
+
+<Tip>
+We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.13.post1-cann9.0.0-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes.
+</Tip>
+
+<Tabs>
+  <Tab title="Atlas 800I A3">
+
+```shell Command
+# Stable release
+export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3
+
+# Daily build
+export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3
+
+docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
+    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
+    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
+    --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
+    --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
+    --device=/dev/davinci_manager \
+    --device=/dev/hisi_hdc \
+    --volume /usr/local/sbin:/usr/local/sbin \
+    --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
+    --volume /etc/ascend_install.info:/etc/ascend_install.info \
+    --volume /var/queue_schedule:/var/queue_schedule \
+    --volume ~/.cache/:/root/.cache/ \
+    --entrypoint=bash \
+    $IMAGE
+```
+
+  </Tab>
+  <Tab title="Atlas 800I A2">
+
+```shell Command
+# Stable release
+export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b
+
+# Daily build
+export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b
+
+docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
+    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
+    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
+    --device=/dev/davinci_manager \
+    --device=/dev/hisi_hdc \
+    --volume /usr/local/sbin:/usr/local/sbin \
+    --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
+    --volume /etc/ascend_install.info:/etc/ascend_install.info \
+    --volume /var/queue_schedule:/var/queue_schedule \
+    --volume ~/.cache/:/root/.cache/ \
+    --entrypoint=bash \
+    $IMAGE
+```
+
+  </Tab>
+</Tabs>
+
+## Usage
+
+The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version.
+
+### Start SGLang server
+
+SGLang will automatically download the model from Hugging Face.
+
+```shell Command
+# Set HF_ENDPOINT to a mirror site if network is not available
+export HF_ENDPOINT=https://hf-mirror.com
+
+# Set your own HF_TOKEN to download restricted models
+export HF_TOKEN=<secret>
+
+# Start SGLang server
+# It may take several minutes to download the model on the first run
+sglang serve --model-path Qwen/Qwen2.5-7B-Instruct --attention-backend ascend &
+```
+
+If you see output like the following, the server is running.
+
+```log Output
+INFO:     Waiting for application startup.
+INFO:     Application startup complete.
+INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
+The server is fired up and ready to roll!
+```
+
+### Send a test request
+
+You can do inference using the server:
+
+```shell Command
+curl -X POST http://localhost:30000/generate \
+    -H "Content-Type: application/json" \
+    -d '{
+        "text": "The capital of France is",
+        "sampling_params": {
+            "temperature": 0,
+            "max_new_tokens": 16
+        }
+    }'
+```
+
+If the "text" field in the response contains "Paris", the server is working as expected.
+
+### Stop server and exit container
+
+The SGLang server is running as a background process. You can send a `SIGINT` signal to stop it.
+
+```shell Command
+SGLANG_PID=$(pgrep -f "sglang serve")
+kill -SIGINT $SGLANG_PID
+```
+
+The output should be like the following:
+
+```log Output
+INFO:     Shutting down
+INFO:     Waiting for application shutdown.
+INFO:     Application shutdown complete.
+INFO:     Finished server process [25310]
+```
+
+The server has now stopped. You can verify it with `ps -ef | grep sglang`, then exit the container by pressing `Ctrl+D`.

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_ring_sp_performance.mdx

@@ -0,0 +1,110 @@
+---
+title: "Ascend NPU Ring-SP Performance (Wan2.1-T2V-1.3B)"
+metatags:
+    description: "This page reports Ring-SP performance on Ascend NPU with torchnpu==2.10.0."
+---
+
+This page reports Ring-SP performance on Ascend NPU with `torch_npu==2.10.0`.
+
+- Baseline config: `ulysses=1, ring=1` (short: `u1r1`)
+- Ring-SP config: `ulysses=1, ring=2` (short: `u1r2`)
+
+## Benchmark Setup
+
+- Model: `Wan2.1-T2V-1.3B-Diffusers`
+- Prompt: `"a cat is playing piano"`
+- Framework command: `sglang generate`
+- Runtime: `torch_npu==2.10.0`
+
+## Generate Commands
+
+### Baseline (`u1r1`)
+
+```bash
+sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
+    --prompt "a cat is playing piano" --num-gpus 1 --ring-degree 1 \
+    --save-output
+```
+
+### Ring-SP (`u1r2`)
+
+```bash
+sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
+    --prompt "a cat is playing piano" --num-gpus 2 --ring-degree 2 \
+    --save-output
+```
+
+## Benchmarks
+
+Benchmark Disclaimer
+
+These numbers are from one fixed setup and one prompt case. Actual performance may vary by model settings, environment, and workload.
+
+### Stage Time Breakdown
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Stage / Metric</th>
+      <th><code>u1r2</code> (s)</th>
+      <th><code>u1r1</code> baseline (s)</th>
+      <th>Speedup</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>InputValidation</td>
+      <td>0.0003</td>
+      <td>0.0002</td>
+      <td>0.67x</td>
+    </tr>
+    <tr>
+      <td>TextEncoding</td>
+      <td>3.5936</td>
+      <td>3.5820</td>
+      <td>1.00x</td>
+    </tr>
+    <tr>
+      <td>LatentPreparation</td>
+      <td>0.0007</td>
+      <td>0.0055</td>
+      <td>7.86x</td>
+    </tr>
+    <tr>
+      <td>TimestepPreparation</td>
+      <td>0.0008</td>
+      <td>0.0007</td>
+      <td>0.88x</td>
+    </tr>
+    <tr>
+      <td>Denoising</td>
+      <td>121.2788</td>
+      <td>239.2580</td>
+      <td>1.97x</td>
+    </tr>
+    <tr>
+      <td>Decoding</td>
+      <td>13.8685</td>
+      <td>16.4969</td>
+      <td>1.19x</td>
+    </tr>
+    <tr>
+      <td><strong>Total (Pixel data generated)</strong></td>
+      <td><strong>141.86</strong></td>
+      <td><strong>266.50</strong></td>
+      <td><strong>1.88x</strong></td>
+    </tr>
+  </tbody>
+</table>
+
+## Summary
+
+- With `torch_npu==2.10.0`, Ring-SP (`u1r2`) runs successfully on NPU for this case.
+- End-to-end generation time improves from `266.50s` to `141.86s` (`1.88x`).
+- The main gain comes from `DenoisingStage` (`1.97x`), while decoding also improves (`1.19x`).

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx

@@ -0,0 +1,713 @@
+---
+title: "Support Models on Ascend NPU"
+metatags:
+  description: "Documentation for Support Models on Ascend NPU"
+---
+This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
+Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included.
+You are welcome to enable various models based on your business requirements.
+
+## Large Language Models
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/DeepSeek-V4-Pro-w4a8-mtp</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek V3/V3.1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.2-W8A8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528-W8A8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V2-Lite-W8A8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3.6-35B-A3B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/Qwen3.6-27B-w8a8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/Qwen3.5-27B-w8a8-mtp</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3.5-9B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3.5-4B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3.5-0.8B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-30B-A3B-Instruct-2507</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-32B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-0.6B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B-W8A8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Next-80B-A3B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-7B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QWQ-32B-W8A8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>QWQ</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>meta-llama/Llama-4-Scout-17B-16E-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/Llama-3.1-8B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/llama-2-7b</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/Llama-3.2-1B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mistralai/Mistral-7B-Instruct-v0.2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>google/gemma-3-4b-it</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>microsoft/Phi-4-multimodal-instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Phi</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>allenai/OLMoE-1B-7B-0924</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>OLMoE</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stabilityai/stablelm-2-1_6b</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>StableLM</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CohereForAI/c4ai-command-r-v01</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Command-R</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>huihui-ai/grok-2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Grok</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/chatglm2-6b</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ChatGLM</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ExaONE 3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>xverse/XVERSE-MoE-A36B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>XVERSE</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFaceTB/SmolLM-1.7B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>SmolLM</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/GLM-5.2-w8a8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/GLM-5.1-w4a8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/GLM-5-w4a8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/glm-4-9b-chat</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-4</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>XiaomiMiMo/MiMo-7B-RL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiMo</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>arcee-ai/AFM-4.5B-Base</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Arcee AFM-4.5B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Howeee/persimmon-8b-chat</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Persimmon</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>inclusionAI/Ling-lite</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Ling</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ibm-granite/granite-3.1-8b-instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Granite</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/dbrx-instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DBRX (Databricks)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>baichuan-inc/Baichuan2-13B-Chat</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Baichuan 2 (7B, 13B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>baidu/ERNIE-4.5-21B-A3B-PT</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ERNIE-4.5 (4.5, 4.5MoE series)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>OpenBMB/MiniCPM3-4B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM (v3, 4B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/Kimi-K2.6-w4a8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/Kimi-K2.5-w4a8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>moonshotai/Kimi-K2-Thinking</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>eigen-ai-labs/gpt-oss-120b-bf16</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GPTOSS</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>allenai/OLMo-2-1124-7B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>OLMo</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Eco-Tech/MiniMax-M2.5-w8a8-QuaRot</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniMax-M2.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>cyankiwi/MiniMax-M2-BF16</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniMax-M2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>upstage/SOLAR-10.7B-Instruct-v1.0</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Solar</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FLM/Tele-FLM</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tele FLM (52B-1T)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>bigcode/starcoder2-7b</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>StarCoder2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>arcee-ai/Trinity-Mini</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Trinity (Nano, Mini)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>OrionStarAI/Orion-14B-Base</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Orion (14B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>EleutherAI/gpt-j-6b</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GPT-J (6B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+  </tbody>
+</table>
+
+## Multimodal Language Models
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family (Variants)</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-VL-3B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5-VL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-VL-72B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5-VL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-30B-A3B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-VL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-8B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-VL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-4B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-VL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-235B-A22B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-VL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/deepseek-vl2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-VL2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/Janus-Pro-1B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Janus-Pro (1B, 7B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/Janus-Pro-7B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Janus-Pro (1B, 7B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openbmb/MiniCPM-V-2_6</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM-V / MiniCPM-o</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openbmb/MiniCPM-o-2_6</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM-V / MiniCPM-o</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>google/gemma-3-4b-it</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma 3 (Multimodal)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mistralai/Mistral-Small-3.1-24B-Instruct-2503</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral-Small-3.1-24B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>microsoft/Phi-4-multimodal-instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Phi-4-multimodal-instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>XiaomiMiMo/MiMo-VL-7B-RL</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiMo-VL (7B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/llava-v1.6-34b</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA (v1.5 & v1.6)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lmms-lab/llava-next-72b</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA-NeXT (8B, 72B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>moonshotai/Kimi-VL-A3B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi-VL (A3B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/GLM-4.5V</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-4.5V (106B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/Llama-3.2-11B-Vision-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama 3.2 Vision (11B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>rednote-hilab/dots.ocr</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DotsVLM-OCR</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Omni-30B-A3B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-Omni</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stepfun-ai/Step3-VL-10B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Step3-VL (10B)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+  </tbody>
+</table>
+
+## Embedding Models
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>intfloat/e5-mistral-7b-instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>E5 (Llama/Mistral based)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>iic/gte_Qwen2-1.5B-instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GTE-Qwen2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Embedding-8B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-Embedding</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Alibaba-NLP/gme-Qwen2-VL-2B-Instruct</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GME (Multimodal)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/clip-vit-large-patch14-336</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CLIP</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>BAAI/bge-large-en-v1.5</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BGE</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+  </tbody>
+</table>
+
+## Reward Models
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Skywork/Skywork-Reward-Llama-3.1-8B-v0.2</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama3.1 Reward</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Shanghai_AI_Laboratory/internlm2-7b-reward</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>InternLM 2 Reward</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-Math-RM-72B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5 Reward - Math</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Howeee/Qwen2.5-1.5B-apeach</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5 Reward - Sequence</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+  </tbody>
+</table>
+
+## Rerank Models
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+    <col style={{width: "25.0%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>BAAI/bge-reranker-v2-m3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BGE-Reranker</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Reranker-8B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-Reranker (decoder-only yes/no)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-Reranker-2B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-VL-Reranker (multimodal yes/no)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+  </tbody>
+</table>

docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx

@@ -0,0 +1,526 @@
+---
+title: "How to Support New Models"
+metatags:
+  description: "This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations."
+---
+
+This document explains how to add support for new language models and multimodal large language models (MLLMs) in
+SGLang. It also covers how to test new models and register external implementations.
+
+## How to Support a New Language Model
+
+To support a new model in SGLang, you only need to add a single file under
+the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn
+from existing model implementations and create a new file for your model. For most models, you should be able to find a
+similar model to start with (e.g., starting from Llama). Also refer how
+to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang)
+
+NPU adaptations are embedded in existing model files (e.g., `llama.py`, `qwen3_vl.py`) through `_is_npu` conditional
+branches. The NPU hardware backend lives at `sglang/srt/hardware_backend/npu/`. Some ops may need to use `torch_npu`
+APIs in place of CUDA equivalents.
+
+## How to Support a New Multimodal Large Language Model
+
+To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the
+standard LLM support:
+
+1. **Register your new model as multimodal**:
+   Extend `is_multimodal_model`
+   in [model_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561)
+   to return `True` for your model.
+
+2. **Register a new chat-template**:
+   Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py) and the corresponding matching function.
+
+3. **Multimodal Data Processor**:
+   Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your
+   model’s dedicated processor.
+   See [Multimodal Processors](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors)
+   for more details.
+
+4. **Handle Multimodal Tokens**:
+   Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be
+   expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data
+   with `RadixAttention`.
+
+5. **Handle Image Feature Extraction**:
+   Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model.
+
+6. **Adapt to Vision Attention**:
+   Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.
+
+You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or
+other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs.
+
+<Tip>
+On Ascend NPU, ensure vision processors and image feature extraction are compatible with the `torch_npu` backend.
+Refer to `vit_npu_graph_runner.py` under `hardware_backend/npu/graph_runner/` and `qwen_vl_processor.py` under
+`hardware_backend/npu/modules/` for NPU vision adaptation patterns.
+</Tip>
+
+## Testing and Debugging
+
+Please note all your testing and benchmarking results in PR description.
+
+### Benchmark
+
+- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow [https://docs.sglang.io/developer_guide/benchmark_and_profiling.html](https://docs.sglang.io/developer_guide/benchmark_and_profiling.html) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer).
+- **(Optional) Other evals**: If you ran other evals, please note the results in PR description.
+
+<Tip>
+For NPU-adapted models: add the corresponding test under `test/registered/ascend/` and verify correctness on Ascend NPU
+hardware; run benchmarks on the NPU device and report performance metrics (TTFT, throughput), comparing against SGLang
+GPU results as the primary baseline. Fall back to HF Transformer comparison when no GPU adaptation is available.
+</Tip>
+
+## Port a Model from vLLM to SGLang
+
+The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable
+resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models
+from vLLM to SGLang.
+
+To port a model from vLLM to SGLang:
+
+- Compare these two files for guidance:
+  - [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py)
+  - [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py)
+- The major differences include:
+  - **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`).
+  - **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.**
+  - **(For multimodal models) Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.**
+  - **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers.
+  - **Change the `forward()` functions** to accept a `forward_batch: ForwardBatch` argument carrying the full batch
+    state (positions, `input_ids`, KV cache indices, sampling info, etc.). SGLang’s top-level model `forward()` is the
+    entry point and returns the final `LogitsProcessorOutput` directly — it does both the backbone forward and the
+    logits computation, unlike vLLM which splits these into `forward()` + `compute_logits()`.
+  - **Add `EntryClass`** at the end.
+  - **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components.
+  - **For Ascend NPU**: Reference existing NPU-adapted models (e.g., `llama.py`, `deepseek_v2.py`) for NPU-specific
+    patterns, such as replacing CUDA kernels with `torch_npu` equivalents. The NPU backend is at
+    `sglang/srt/hardware_backend/npu/`.
+
+Note: make sure you add your new model to the supported models list in the supported models documentation.
+
+## Registering an External Model Implementation
+
+In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server.
+This allows you to integrate your model without modifying the source code.
+
+For example:
+
+```python Register Model
+from sglang.srt.models.registry import ModelRegistry
+from sglang.srt.entrypoints.http_server import launch_server
+
+# For a single model, add it to the registry:
+ModelRegistry.models[model_name] = model_class
+
+# For multiple models, you can imitate the import_model_classes() function:
+from functools import lru_cache
+
+@lru_cache()
+def import_new_model_classes():
+    model_arch_name_to_cls = {}
+    # Populate model_arch_name_to_cls with your new model classes.
+    ...
+    return model_arch_name_to_cls
+
+ModelRegistry.models.update(import_new_model_classes())
+
+# Launch the server with your server arguments:
+launch_server(server_args)
+```
+
+## Example: Implementing and Serving a Llama Wrapper Model
+
+Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](/docs/basic_usage/offline_engine_api).
+
+### Implementing Our Model
+
+To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit.
+
+Let's start by defining our model in a file called `llama_wrapper.py`.
+The first step is to import the necessary libraries from SRT, which is SGLang's internal backend.
+
+```python Example
+# In the file `llama_wrapper.py`
+
+import torch
+from transformers import LlamaConfig
+from typing import Optional
+from sglang.srt.layers.logits_processor import LogitsProcessorOutput
+from sglang.srt.layers.quantization.base_config import QuantizationConfig
+from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors
+
+from sglang.srt.models.llama import LlamaForCausalLM
+```
+
+Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`.
+Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219).
+Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us.
+
+```python Class Definition
+class LlamaWrapper(LlamaForCausalLM):
+    def __init__(
+        self,
+        config: LlamaConfig,
+        quant_config: Optional[QuantizationConfig] = None,
+        prefix: str = "",
+    ) -> None:
+        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
+```
+
+Now, we want to define the `forward` method, which is what will be called at inference time.
+Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references.
+To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py).
+
+```python Forward Method Signature
+    @torch.no_grad()
+    def forward(
+        self,
+        input_ids: torch.Tensor,
+        positions: torch.Tensor,
+        forward_batch: ForwardBatch,
+        pp_proxy_tensors: Optional[PPProxyTensors] = None,
+        input_embeds: Optional[torch.Tensor] = None,
+        get_embedding: bool = False,
+    ) -> LogitsProcessorOutput:
+```
+
+We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method.
+After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`).
+
+```python Call Model and LogitsProcessor
+        hidden_states = self.model(
+            input_ids,
+            positions,
+            forward_batch,
+            input_embeds,
+            pp_proxy_tensors=pp_proxy_tensors,
+        )
+
+        res: LogitsProcessorOutput = self.logits_processor(
+            input_ids,
+            hidden_states,
+            self.lm_head,
+            forward_batch,
+        )
+```
+
+After receiving the logits for the next token, we can finally perform our biasing step.
+
+```python Logit Biasing
+        orig_logits = res.next_token_logits
+        res.next_token_logits = torch.where(
+            orig_logits > 0,
+            orig_logits.sqrt(),
+            orig_logits
+        )
+
+        return res
+```
+
+Now, our `LlamaWrapper` model is created and ready to be served!
+
+### Serving Our Model Via SGLang's Offline Engine
+
+The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server.
+
+First, create a new file called `run.py`.
+Now, we must ensure that SGLang's `ModelRegistry` can find our model.
+To do this, we first download the model's configuration and weights from Huggingface.
+
+```python Example
+# In the file `run.py`
+
+import asyncio
+from functools import lru_cache
+from huggingface_hub import snapshot_download
+from llama_wrapper import LlamaWrapper # Make sure to import our new model!
+import sglang as sgl
+from sglang.srt.models.registry import ModelRegistry
+
+# Make sure to request access to this model on Huggingface, then export your
+# `HF_TOKEN` to download the model snapshot
+llama_dir = snapshot_download(
+    repo_id="meta-llama/Llama-3.1-8B-Instruct",
+    local_dir="./llama_ckpt",
+)
+```
+
+Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`.
+That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model.
+
+```json Example
+{
+  "architectures": [
+    "LlamaWrapper"
+  ],
+  ...
+}
+```
+
+However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model.
+Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation".
+
+```python Register LlamaWrapper
+@lru_cache()
+def import_new_model_classes():
+    model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper}
+    return model_arch_name_to_cls
+
+ModelRegistry.models.update(import_new_model_classes())
+```
+
+Lastly, when we create our `Engine`, we just pass in the path to the local model directory.
+Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint.
+
+```python Example
+def main():
+    llm = sgl.Engine(model_path="./llama_ckpt")
+    sampling_params = {"temperature": 0.2, "top_k": 5}
+    prompts = [
+        "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
+        "Provide a concise factual statement about France’s capital city. The capital of France is",
+        "Explain possible future trends in artificial intelligence. The future of AI is",
+    ]
+
+    asyncio.run(run_llm(llm, sampling_params, prompts))
+
+    llm.shutdown()
+
+async def run_llm(
+    llm,
+    sampling_params,
+    prompts,
+) -> None:
+    outputs = await llm.async_generate(prompts, sampling_params)
+
+    for prompt, output in zip(prompts, outputs):
+        print(f"\nPrompt: {prompt}")
+        print(f"Generated text: {output['text']}")
+
+if __name__ == "__main__":
+    main()
+```
+
+Now, when we call `python run.py`, we will get the outputs of our newly created model!
+
+## Serving External Models via the Standard CLI
+
+The previous sections show how to register a model programmatically via `ModelRegistry` and serve it through the Offline Engine. Similar to vLLM model plugin, there is an alternative that lets you keep using the standard `python -m sglang.launch_server` CLI without modifying any SGLang source code: you can register your model using the `SGLANG_EXTERNAL_MODEL_PACKAGE` environment variable.
+
+<Tip>
+On Ascend NPU, `--device` and `--attention-backend` are auto-detected and can be omitted from the launch command.
+SGLang sets the device to `npu` and attention backend to `ascend` automatically when `torch.npu.is_available()`.
+</Tip>
+
+### The `EntryClass` Variable
+
+When SGLang scans a model package, it looks for the variable `EntryClass` at the module level of your Python file. The [model registry](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/registry.py) imports your file, checks for `EntryClass`, and registers the class assigned to it. If you are using a model based on HuggingFace, the name of this class needs to match the `"architectures"` field in your model's `config.json`.
+
+For example, if you are implementing a Llama wrapper, add this line at the end of your model file:
+
+```python Example
+# This is what "Add EntryClass at the end" means
+EntryClass = LlamaWrapper
+```
+
+### Example: Text-Only Model
+
+Using the same Llama wrapper from the previous section, here is how to package and serve it via the CLI.
+
+1. Create your project
+
+```text
+sglang_custom_project/
+|----setup.py
+|----custom_llm/
+     |----__init__.py
+     |----llama_wrapper.py
+```
+
+Write the `setup.py`:
+
+```python Example
+# sglang_custom_project/setup.py
+
+from setuptools import setup, find_packages
+setup(
+    name="sglang-custom-plugins",
+    version="0.1",
+    packages=find_packages(),
+)
+```
+
+2. Write your model code
+
+Inside `llama_wrapper.py`, write your model and include `EntryClass`:
+
+```python Example
+# sglang_custom_project/custom_llm/llama_wrapper.py
+
+import torch
+from typing import Optional
+from sglang.srt.layers.logits_processor import LogitsProcessorOutput
+from sglang.srt.layers.quantization.base_config import QuantizationConfig
+from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors
+from sglang.srt.models.llama import LlamaForCausalLM
+
+class LlamaWrapper(LlamaForCausalLM):
+    def __init__(self, config, quant_config: Optional[QuantizationConfig] = None,
+                 prefix: str = "") -> None:
+        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
+    @torch.no_grad()
+    def forward(self, input_ids, positions, forward_batch,
+                pp_proxy_tensors=None, input_embeds=None, get_embedding=False):
+        hidden_states = self.model(
+            input_ids, positions, forward_batch, input_embeds,
+            pp_proxy_tensors=pp_proxy_tensors,
+        )
+        res: LogitsProcessorOutput = self.logits_processor(
+            input_ids, hidden_states, self.lm_head, forward_batch,
+        )
+
+        orig = res.next_token_logits
+        res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig)
+        return res
+
+# Don't forget to add EntryClass
+EntryClass = LlamaWrapper
+```
+
+3. Install your package
+
+Run this inside your `sglang_custom_project` directory to install your code into the active Python environment:
+
+```bash Command
+pip install -e .
+```
+
+4. Update your `config.json`
+
+Update the `config.json` under your HuggingFace model checkpoint directory so the `architectures` field matches your class name:
+
+```json Config
+{
+    "architectures": ["LlamaWrapper"],
+    ...
+}
+```
+
+5. Launch the server
+
+Set the environment variable before running the CLI:
+
+```bash Command
+export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_llm
+python -m sglang.launch_server \
+    --model-path /path/to/Llama-3.1-8B-Instruct \
+    --port 8000
+```
+
+The `SGLANG_EXTERNAL_MODEL_PACKAGE` should be the parent folder name containing your model-related code. In this example, it should be `custom_llm`.
+
+### Example: Multimodal Model
+
+If you are working with multimodal models, setting `SGLANG_EXTERNAL_MODEL_PACKAGE` alone is not enough. SGLang also needs to recognize your architecture as multimodal to enable the image/video processing pipelines, and it needs a custom processor.
+
+You can handle this by setting two additional environment variables:
+
+- `SGLANG_EXTERNAL_MM_MODEL_ARCH`: Adds your architecture name to SGLang's internal list of multimodal models.
+- `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`: Tells SGLang where to find your custom processor class.
+
+For example, let's build a custom model based on Qwen2-VL-Instruct that takes the square root of the logits.
+
+Create the project:
+
+```text
+sglang_custom_project_vl/
+|----setup.py
+|----custom_vlm/
+     |----__init__.py
+     |----qwenvl_wrapper.py
+```
+
+Write `setup.py`:
+
+```python Example
+# sglang_custom_project_vl/setup.py
+
+from setuptools import setup, find_packages
+setup(
+    name="sglang-custom-plugins-vl",
+    version="0.1",
+    packages=find_packages(),
+)
+```
+
+Write the model in `qwenvl_wrapper.py`:
+
+```python Example
+# sglang_custom_project_vl/custom_vlm/qwenvl_wrapper.py
+import torch
+from sglang.srt.models.qwen2_vl import Qwen2VLForConditionalGeneration
+from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor
+
+class CustomQwen2VL(Qwen2VLForConditionalGeneration):
+    def forward(self, input_ids, positions, forward_batch,
+                input_embeds=None, get_embedding=False):
+        res = super().forward(
+            input_ids, positions, forward_batch,
+            input_embeds=input_embeds, get_embedding=get_embedding
+        )
+        if not get_embedding:
+            orig = res.next_token_logits
+            res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig)
+        return res
+
+class CustomQwen2VLProcessor(QwenVLImageProcessor):
+    models = [CustomQwen2VL]
+
+    def __init__(self, hf_config, server_args, _processor, *args, **kwargs):
+        super().__init__(hf_config, server_args, _processor, *args, **kwargs)
+
+EntryClass = CustomQwen2VL
+```
+
+**Note:** you don't need a separate `EntryClass` for the custom processor as long as you associate the processor with the specific model class.
+
+Install the package, update `config.json`, and launch:
+
+```bash Command
+pip install -e .
+```
+
+```json Config
+{
+  "architectures": ["CustomQwen2VL"],
+  ...
+}
+```
+
+```bash Command
+export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_vlm
+export SGLANG_EXTERNAL_MM_MODEL_ARCH=CustomQwen2VL
+export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=custom_vlm
+
+python -m sglang.launch_server \
+    --model-path /path/to/Qwen2-VL-2B-Instruct \
+    --port 8000 \
+    --enable-multimodal
+```
+
+## Documentation
+
+Add to table of supported models in [generative_models.mdx](/docs/supported-models/generative_models) or [multimodal_language_models.mdx](/docs/supported-models/multimodal_language_models)
+
+<Tip>
+For NPU-adapted models, also add entries to the NPU support models table in
+[ascend_npu_support_models.mdx](./ascend_npu_support_models).
+</Tip>
+
+---
+
+By following these guidelines, you can add support for new language models and multimodal large language models in
+SGLang and ensure they are thoroughly tested and easily integrated into the system.

docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2.mdx

@@ -0,0 +1,623 @@
+---
+title: "DeepSeek-V3.2"
+metatags:
+  description: "Best Practice for DeepSeek-V3.2 on Ascend NPU"
+---
+
+<Note>
+This page focuses on optimal configuration and benchmark results for DeepSeek-V3.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-V3.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2).
+</Note>
+
+### Low Latency
+
+| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128K+1K | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) |
+| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128K+1K | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) |
+
+### High Throughput
+
+| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128K+1K | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) |
+
+## Optimal Configuration
+
+<a id="pd-disaggregation" title="Referenced by external docs. Verify before removing."></a>
+
+### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K 26ms
+
+**Model**: DeepSeek-V3.2
+
+**Hardware**: Atlas 800I A3
+
+**Cards**: 32
+
+**Deploy Mode**: PD Disaggregation
+
+**Quantization**: W8A8 INT8
+
+**Dataset**: 128K+1K
+
+**TPOT**: 26ms
+
+#### Model Deployment
+
+```bash Command
+# ============================================================
+# Before running, update the following variables:
+#   P_IP: prefill node IP address
+#   D_IP: decode node IP address
+#   ASCEND_MF_STORE_URL: prefill node IP with port
+#   MODEL_PATH: path to the model weights directory
+#   HCCL_SOCKET_IFNAME: network interface name for HCCL
+#   GLOO_SOCKET_IFNAME: network interface name for Gloo
+# ============================================================
+
+
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export SGLANG_SET_CPU_AFFINITY=1
+export STREAMS_PER_DEVICE=32
+
+P_IP=('<your prefill ip1>' '<your prefill ip2>')
+D_IP=('<your decode ip1>' '<your decode ip2>')
+
+export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"
+
+MODEL_PATH=/path/to/model-weights
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+# prefill
+for i in "${!P_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+    then
+        echo "${P_IP[$i]}"
+        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+        export GLOO_SOCKET_IFNAME=<network-interface>
+        export HCCL_BUFFSIZE=1200
+        export HCCL_SOCKET_IFNAME=<network-interface>
+        export TASK_QUEUE_ENABLE=2
+
+        python3 -m sglang.launch_server \
+        --model-path ${MODEL_PATH} \
+        --disaggregation-mode prefill \
+        --host ${P_IP[$i]} \
+        --port 8000 \
+        --dist-init-addr ${P_IP[0]}:5000 \
+        --disaggregation-bootstrap-port 8998 \
+        --node-rank $i \
+        --nnodes 2 \
+        --tp 32 \
+        --watchdog-timeout 9000 \
+        --mem-fraction-static 0.73 \
+        --disable-radix-cache \
+        --chunked-prefill-size -1 \
+        --max-prefill-tokens 68000 \
+        --max-running-requests 1 \
+        --moe-a2a-backend deepep \
+        --deepep-mode normal \
+        --quantization modelslim \
+        --disaggregation-transfer-backend ascend \
+        --disable-cuda-graph \
+        --moe-dense-tp-size 1 \
+        --enable-nsa-prefill-context-parallel \
+        --nsa-prefill-cp-mode in-seq-split \
+        --attn-cp-size 32 \
+        --speculative-algorithm NEXTN \
+        --speculative-num-steps 1 \
+        --speculative-eagle-topk 1 \
+        --speculative-num-draft-tokens 2 \
+        --reasoning-parser deepseek-v3 \
+        --tool-call-parser deepseekv32 \
+        --trust-remote-code \
+        --attention-backend ascend \
+        --device npu
+        NODE_RANK=$i
+        break
+    fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+    then
+        echo "${D_IP[$i]}"
+        export GLOO_SOCKET_IFNAME=<network-interface>
+        export HCCL_BUFFSIZE=400
+        export HCCL_SOCKET_IFNAME=<network-interface>
+        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8
+        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+        export SGLANG_ENABLE_SPEC_V2=1
+        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
+        export TASK_QUEUE_ENABLE=0
+
+        python3 -m sglang.launch_server \
+        --model-path ${MODEL_PATH} \
+        --disaggregation-mode decode \
+        --host ${D_IP[$i]} \
+        --port 8001 \
+        --dist-init-addr ${D_IP[0]}:5000 \
+        --node-rank $i \
+        --nnodes 2 \
+        --tp 32 \
+        --dp 8 \
+        --ep 32 \
+        --moe-dense-tp-size 1 \
+        --enable-dp-attention \
+        --enable-dp-lm-head \
+        --watchdog-timeout 9000 \
+        --mem-fraction-static 0.79 \
+        --disable-radix-cache \
+        --chunked-prefill-size -1 \
+        --max-prefill-tokens 68000 \
+        --max-running-requests 32 \
+        --cuda-graph-max-bs-decode 4 \
+        --moe-a2a-backend deepep \
+        --deepep-mode low_latency \
+        --quantization modelslim \
+        --speculative-algorithm NEXTN \
+        --speculative-num-steps 3 \
+        --speculative-eagle-topk 1 \
+        --speculative-num-draft-tokens 4 \
+        --disaggregation-transfer-backend ascend \
+        --reasoning-parser deepseek-v3 \
+        --tool-call-parser deepseekv32 \
+        --trust-remote-code \
+        --attention-backend ascend \
+        --device npu
+        NODE_RANK=$i
+        break
+    fi
+done
+```
+
+```shell Command
+# ============================================================
+# Before running, replace the following placeholders:
+#   <your prefill ip>: prefill node IP address
+#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
+# ============================================================
+
+python -m sglang_router.launch_router \
+    --pd-disaggregation \
+    --policy cache_aware \
+    --prefill http://<your prefill ip>:8000 8998 \
+    --decode http://<your decode ip1>:8001 \
+    --host 127.0.0.1 \
+    --port 6688 \
+    --mini-lb
+```
+
+#### Benchmark
+
+We tested it based on the `RANDOM` dataset.
+
+```shell Command
+python -m sglang.bench_serving \
+    --dataset-name random \
+    --backend sglang \
+    --host 127.0.0.1 \
+    --port 6688 \
+    --max-concurrency 8 \
+    --random-input-len 131072 \
+    --random-output-len 1024 \
+    --num-prompts 8 \
+    --random-range-ratio 1
+```
+
+### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS16
+
+**Model**: DeepSeek-V3.2
+
+**Hardware**: Atlas 800I A3
+
+**Cards**: 32
+
+**Deploy Mode**: PD Disaggregation
+
+**Quantization**: W8A8 INT8
+
+**Dataset**: 128K+1K
+
+**TPOT**: 107ms
+
+#### Model Deployment
+
+```bash Command
+# ============================================================
+# Before running, update the following variables:
+#   P_IP: prefill node IP address
+#   D_IP: decode node IP address
+#   ASCEND_MF_STORE_URL: prefill node IP with port
+#   MODEL_PATH: path to the model weights directory
+#   HCCL_SOCKET_IFNAME: network interface name for HCCL
+#   GLOO_SOCKET_IFNAME: network interface name for Gloo
+# ============================================================
+
+
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export SGLANG_SET_CPU_AFFINITY=1
+export STREAMS_PER_DEVICE=32
+
+P_IP=('<your prefill ip1>' '<your prefill ip2>')
+D_IP=('<your decode ip1>' '<your decode ip2>')
+
+export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"
+
+MODEL_PATH=/path/to/model-weights
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+# prefill
+for i in "${!P_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+    then
+        echo "${P_IP[$i]}"
+        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+        export GLOO_SOCKET_IFNAME=<network-interface>
+        export HCCL_BUFFSIZE=1200
+        export HCCL_SOCKET_IFNAME=<network-interface>
+        export TASK_QUEUE_ENABLE=2
+
+        python3 -m sglang.launch_server \
+        --model-path ${MODEL_PATH} \
+        --disaggregation-mode prefill \
+        --host ${P_IP[$i]} \
+        --port 8000 \
+        --dist-init-addr ${P_IP[0]}:5000 \
+        --disaggregation-bootstrap-port 8998 \
+        --node-rank $i \
+        --nnodes 2 \
+        --tp 32 \
+        --watchdog-timeout 9000 \
+        --mem-fraction-static 0.73 \
+        --disable-radix-cache \
+        --chunked-prefill-size -1 \
+        --max-prefill-tokens 68000 \
+        --max-running-requests 1 \
+        --moe-a2a-backend deepep \
+        --deepep-mode normal \
+        --quantization modelslim \
+        --disaggregation-transfer-backend ascend \
+        --disable-cuda-graph \
+        --moe-dense-tp-size 1 \
+        --enable-nsa-prefill-context-parallel \
+        --nsa-prefill-cp-mode in-seq-split \
+        --attn-cp-size 32 \
+        --speculative-algorithm NEXTN \
+        --speculative-num-steps 1 \
+        --speculative-eagle-topk 1 \
+        --speculative-num-draft-tokens 2 \
+        --reasoning-parser deepseek-v3 \
+        --tool-call-parser deepseekv32 \
+        --trust-remote-code \
+        --attention-backend ascend \
+        --device npu
+        NODE_RANK=$i
+        break
+    fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+    then
+        echo "${D_IP[$i]}"
+        export GLOO_SOCKET_IFNAME=<network-interface>
+        export HCCL_BUFFSIZE=400
+        export HCCL_SOCKET_IFNAME=<network-interface>
+        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8
+        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+        export SGLANG_ENABLE_SPEC_V2=1
+        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
+        export TASK_QUEUE_ENABLE=0
+
+        python3 -m sglang.launch_server \
+        --model-path ${MODEL_PATH} \
+        --disaggregation-mode decode \
+        --host ${D_IP[$i]} \
+        --port 8001 \
+        --dist-init-addr ${D_IP[0]}:5000 \
+        --node-rank $i \
+        --nnodes 2 \
+        --tp 32 \
+        --dp 8 \
+        --ep 32 \
+        --moe-dense-tp-size 1 \
+        --enable-dp-attention \
+        --enable-dp-lm-head \
+        --watchdog-timeout 9000 \
+        --mem-fraction-static 0.79 \
+        --disable-radix-cache \
+        --chunked-prefill-size -1 \
+        --max-prefill-tokens 68000 \
+        --max-running-requests 32 \
+        --cuda-graph-max-bs-decode 4 \
+        --moe-a2a-backend deepep \
+        --deepep-mode low_latency \
+        --quantization modelslim \
+        --speculative-algorithm NEXTN \
+        --speculative-num-steps 3 \
+        --speculative-eagle-topk 1 \
+        --speculative-num-draft-tokens 4 \
+        --disaggregation-transfer-backend ascend \
+        --reasoning-parser deepseek-v3 \
+        --tool-call-parser deepseekv32 \
+        --trust-remote-code \
+        --attention-backend ascend \
+        --device npu
+        NODE_RANK=$i
+        break
+    fi
+done
+```
+
+```shell Command
+# ============================================================
+# Before running, replace the following placeholders:
+#   <your prefill ip>: prefill node IP address
+#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
+# ============================================================
+
+python -m sglang_router.launch_router \
+    --pd-disaggregation \
+    --policy cache_aware \
+    --prefill http://<your prefill ip>:8000 8998 \
+    --decode http://<your decode ip1>:8001 \
+    --host 127.0.0.1 \
+    --port 6688 \
+    --mini-lb
+```
+
+#### Benchmark
+
+We tested it based on the `RANDOM` dataset.
+
+```shell Command
+python -m sglang.bench_serving \
+    --dataset-name random \
+    --backend sglang \
+    --host 127.0.0.1 \
+    --port 6688 \
+    --max-concurrency 16 \
+    --random-input-len 131072 \
+    --random-output-len 1024 \
+    --num-prompts 16 \
+    --random-range-ratio 1
+```
+
+### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS8
+
+**Model**: DeepSeek-V3.2
+
+**Hardware**: Atlas 800I A3
+
+**Cards**: 32
+
+**Deploy Mode**: PD Disaggregation
+
+**Quantization**: W8A8 INT8
+
+**Dataset**: 128K+1K
+
+**TPOT**: 26ms
+
+#### Model Deployment
+
+```bash Command
+# ============================================================
+# Before running, update the following variables:
+#   P_IP: prefill node IP address
+#   D_IP: decode node IP address
+#   ASCEND_MF_STORE_URL: prefill node IP with port
+#   MODEL_PATH: path to the model weights directory
+#   HCCL_SOCKET_IFNAME: network interface name for HCCL
+#   GLOO_SOCKET_IFNAME: network interface name for Gloo
+# ============================================================
+
+
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export SGLANG_SET_CPU_AFFINITY=1
+export STREAMS_PER_DEVICE=32
+
+P_IP=('<your prefill ip1>' '<your prefill ip2>')
+D_IP=('<your decode ip1>' '<your decode ip2>')
+
+export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"
+
+MODEL_PATH=/path/to/model-weights
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+# prefill
+for i in "${!P_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+    then
+        echo "${P_IP[$i]}"
+        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+        export GLOO_SOCKET_IFNAME=<network-interface>
+        export HCCL_BUFFSIZE=1200
+        export HCCL_SOCKET_IFNAME=<network-interface>
+        export TASK_QUEUE_ENABLE=2
+
+        python3 -m sglang.launch_server \
+        --model-path ${MODEL_PATH} \
+        --disaggregation-mode prefill \
+        --host ${P_IP[$i]} \
+        --port 8000 \
+        --dist-init-addr ${P_IP[0]}:5000 \
+        --disaggregation-bootstrap-port 8998 \
+        --node-rank $i \
+        --nnodes 2 \
+        --tp 32 \
+        --watchdog-timeout 9000 \
+        --mem-fraction-static 0.73 \
+        --disable-radix-cache \
+        --chunked-prefill-size -1 \
+        --max-prefill-tokens 68000 \
+        --max-running-requests 1 \
+        --moe-a2a-backend deepep \
+        --deepep-mode normal \
+        --quantization modelslim \
+        --disaggregation-transfer-backend ascend \
+        --disable-cuda-graph \
+        --moe-dense-tp-size 1 \
+        --enable-nsa-prefill-context-parallel \
+        --nsa-prefill-cp-mode in-seq-split \
+        --attn-cp-size 32 \
+        --speculative-algorithm NEXTN \
+        --speculative-num-steps 1 \
+        --speculative-eagle-topk 1 \
+        --speculative-num-draft-tokens 2 \
+        --reasoning-parser deepseek-v3 \
+        --tool-call-parser deepseekv32 \
+        --trust-remote-code \
+        --attention-backend ascend \
+        --device npu
+        NODE_RANK=$i
+        break
+    fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+    then
+        echo "${D_IP[$i]}"
+        export GLOO_SOCKET_IFNAME=<network-interface>
+        export HCCL_BUFFSIZE=400
+        export HCCL_SOCKET_IFNAME=<network-interface>
+        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8
+        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+        export SGLANG_ENABLE_SPEC_V2=1
+        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
+        export TASK_QUEUE_ENABLE=0
+
+        python3 -m sglang.launch_server \
+        --model-path ${MODEL_PATH} \
+        --disaggregation-mode decode \
+        --host ${D_IP[$i]} \
+        --port 8001 \
+        --dist-init-addr ${D_IP[0]}:5000 \
+        --node-rank $i \
+        --nnodes 2 \
+        --tp 32 \
+        --dp 8 \
+        --ep 32 \
+        --moe-dense-tp-size 1 \
+        --enable-dp-attention \
+        --enable-dp-lm-head \
+        --watchdog-timeout 9000 \
+        --mem-fraction-static 0.79 \
+        --disable-radix-cache \
+        --chunked-prefill-size -1 \
+        --max-prefill-tokens 68000 \
+        --max-running-requests 32 \
+        --cuda-graph-max-bs-decode 4 \
+        --moe-a2a-backend deepep \
+        --deepep-mode low_latency \
+        --quantization modelslim \
+        --speculative-algorithm NEXTN \
+        --speculative-num-steps 3 \
+        --speculative-eagle-topk 1 \
+        --speculative-num-draft-tokens 4 \
+        --disaggregation-transfer-backend ascend \
+        --reasoning-parser deepseek-v3 \
+        --tool-call-parser deepseekv32 \
+        --trust-remote-code \
+        --attention-backend ascend \
+        --device npu
+        NODE_RANK=$i
+        break
+    fi
+done
+```
+
+```shell Command
+# ============================================================
+# Before running, replace the following placeholders:
+#   <your prefill ip>: prefill node IP address
+#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
+# ============================================================
+
+python -m sglang_router.launch_router \
+    --pd-disaggregation \
+    --policy cache_aware \
+    --prefill http://<your prefill ip>:8000 8998 \
+    --decode http://<your decode ip1>:8001 \
+    --host 127.0.0.1 \
+    --port 6688 \
+    --mini-lb
+```
+
+#### Benchmark
+
+We tested it based on the `RANDOM` dataset.
+
+```shell Command
+python -m sglang.bench_serving \
+    --dataset-name random \
+    --backend sglang \
+    --host 127.0.0.1 \
+    --port 6688 \
+    --max-concurrency 8 \
+    --random-input-len 131072 \
+    --random-output-len 1024 \
+    --num-prompts 8 \
+    --random-range-ratio 1
+```