chore: import upstream snapshot with attribution

docs/diffusion/performance/attention_backends.md

@@ -0,0 +1,166 @@
+# Attention Backends
+
+This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them.
+
+## Overview
+
+Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`.
+
+Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`) and therefore applies to any model component using these layers (e.g. diffusion transformer / DiT and encoders).
+
+When using the diffusers backend, `--attention-backend` is passed through to diffusers'
+`set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`).
+
+- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA.
+- **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
+- **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA.
+- **MUSA**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
+- **MPS**: always uses PyTorch SDPA.
+- **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA.
+
+## Backend options
+
+For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
+
+| CLI value | Enum value | Notes |
+|---|---|---|
+| `fa` / `fa3` / `fa4` | `FA` | FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`). |
+| `torch_sdpa` | `TORCH_SDPA` | PyTorch `scaled_dot_product_attention`. |
+| `sliding_tile_attn` | `SLIDING_TILE_ATTN` | Sliding Tile Attention (STA). Requires `st_attn`. Configure via `--attention-backend-config`. |
+| `sage_attn` | `SAGE_ATTN` | Requires `sageattention`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120 (compute capability 8.0/8.6/8.9/9.0/12.0); see upstream `setup.py`: https://github.com/thu-ml/SageAttention/blob/main/setup.py. |
+| `sage_attn_3` | `SAGE_ATTN_3` | Requires SageAttention3 installed per upstream instructions. |
+| `video_sparse_attn` | `VIDEO_SPARSE_ATTN` | Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
+| `vmoba_attn` | `VMOBA_ATTN` | Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
+| `aiter` | `AITER` | Requires `aiter`. |
+| `aiter_sage` | `AITER_SAGE` | Requires `aiter`. |
+| `sla_attn` | `SLA_ATTN` | Sparse Linear Attention. Requires `SpargeAttn`. Install with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation`. |
+| `sage_sla_attn` | `SAGE_SLA_ATTN` | SageAttention + Sparse Linear Attention. Requires `SpargeAttn` (same install as SLA). |
+| `sparse_video_gen_2_attn` | `SPARSE_VIDEO_GEN_2_ATTN` | Requires `svg`. See installation instructions at https://github.com/svg-project/Sparse-VideoGen. |
+| `laser_attn` | `LASER_ATTN` | Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU. |
+| `block_sparse_attn` | `BLOCK_SPARSE_ATTN` | Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU. |
+| `rain_fusion_attn` | `RAIN_FUSION_ATTN` | Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU. |
+
+## Selection priority
+
+The selection order in `runtime/layers/attention/selector.py` is:
+
+1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)`
+2. Component override from `--component-attention-backends` while that component is being constructed
+3. CLI `--attention-backend` (`ServerArgs.attention_backend`)
+4. Auto selection (platform capability, dtype, and installed packages)
+
+## Configuration
+
+Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts:
+- A path to a JSON or YAML configuration file.
+- A JSON string (e.g., `'{"sparsity": 0.5}'`).
+- Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`).
+
+### Supported Configuration Parameters
+
+**Sliding Tile Attention (`sliding_tile_attn`)**
+
+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `mask_strategy_file_path` | `str` | **Required.** Path to the mask strategy JSON file. | - |
+| `sta_mode` | `str` | Mode of STA. | `STA_inference` |
+| `skip_time_steps` | `int` | Number of steps to use full attention before switching to sparse attention. | `15` |
+
+**Video Sparse Attention (`video_sparse_attn`)**
+
+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `sparsity` | `float` | Validation sparsity (0.0 - 1.0). | `0.0` |
+
+**V-MoBA (`vmoba_attn`)**
+
+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `temporal_chunk_size` | `int` | Chunk size for temporal dimension. | - |
+| `temporal_topk` | `int` | Top-K tokens to select in temporal dimension. | - |
+| `spatial_chunk_size` | `list[int]` | Chunk size for spatial dimension (H, W). | - |
+| `spatial_topk` | `int` | Top-K tokens to select in spatial dimension. | - |
+| `st_chunk_size` | `list[int]` | Chunk size for spatiotemporal dimension (T, H, W). | - |
+| `st_topk` | `int` | Top-K tokens to select in spatiotemporal dimension. | - |
+| `moba_select_mode` | `str` | Selection mode (e.g., `threshold`). | `threshold` |
+| `moba_threshold` | `float` | Threshold value for selection. | `0.25` |
+| `moba_threshold_type` | `str` | Type of thresholding (e.g., `query_head`). | `query_head` |
+| `first_full_step` | `int` | Number of initial steps to use full attention. | `12` |
+| `first_full_layer` | `int` | Number of initial layers to use full attention. | `0` |
+| `temporal_layer` | `int` | Number of temporal layers. | `1` |
+| `spatial_layer` | `int` | Number of spatial layers. | `1` |
+| `st_layer` | `int` | Number of spatiotemporal layers. | `1` |
+
+**Block Sparse attention ( `block_sparse_attn` ) and  Rain Fusion attention ( `rain_fusion_attn` )**
+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `skip_first_steps` | `int` | Number of steps to use laser attention before switching to sparse attention. | `10` |
+| `sparsity` | `float` | The sparsity coefficient must be in the range (0, 1). | `0.2` |
+
+## Platform support matrix
+
+| Backend | CUDA | ROCm | XPU | MUSA | MPS | NPU | Notes |
+|---|---:|---:|---:|---:|---:|---:|---|
+| `fa` | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. No extra installations are required for NPU |
+| `torch_sdpa` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Most compatible option across platforms. |
+| `sliding_tile_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`. |
+| `sage_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). |
+| `sage_attn_3` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). |
+| `video_sparse_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
+| `sla_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `SpargeAttn`. |
+| `sage_sla_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `SpargeAttn`. |
+| `vmoba_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
+| `aiter` | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | Requires `aiter`. |
+| `aiter_sage` | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | Requires `aiter`. |
+| `sparse_video_gen_2_attn` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | CUDA-only. Requires `svg`. |
+| `laser_attn` | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | NPU-only. Requires `attentions` from `sgl_kernel_npu`. Uses SDPA if seqlen is less than 2048. |
+| `block_sparse_attn` | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | NPU-only. Requires `attentions` from `sgl_kernel_npu`.  Configure via `--attention-backend-config`. |
+| `rain_fusion_attn` | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | NPU-only. Requires `attentions` from `sgl_kernel_npu`. Configure via `--attention-backend-config`. |
+
+## Usage
+
+### Select a backend via CLI
+
+```bash
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "..." \
+  --attention-backend fa
+```
+
+```bash
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "..." \
+  --attention-backend torch_sdpa
+```
+
+### Override one component
+
+Use component overrides when a specific module needs different attention semantics from the main transformer:
+
+```bash
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "..." \
+  --attention-backend fa \
+  --component-attention-backends text_encoder=torch_sdpa
+```
+
+Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`.
+
+### Using Sliding Tile Attention (STA)
+
+```bash
+# Pass the mask strategy file path via config
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "..." \
+  --attention-backend sliding_tile_attn \
+  --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"
+```
+
+### Notes for ROCm / MPS
+
+- ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment.
+- MPS: the platform implementation always uses `torch_sdpa`.

docs/diffusion/performance/cache/cache_dit.md

@@ -0,0 +1,418 @@
+# Cache-DiT
+
+SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss.
+
+## Overview
+
+**Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop:
+
+- **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences
+- **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions
+- **SCM (Step Computation Masking)**: Step-level caching control for additional speedup
+
+## Basic Usage
+
+Enable Cache-DiT by exporting the environment variable and using `sglang generate` or `sglang serve` :
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A beautiful sunset over the mountains"
+```
+
+## Diffusers Backend
+
+Cache-DiT supports loading acceleration configs from a custom YAML file. For
+diffusers pipelines (`diffusers` backend), pass the YAML/JSON path via `--cache-dit-config`. This
+flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`).
+
+### Single GPU inference
+
+Define a `cache.yaml` file that contains:
+
+- DBCache + TaylorSeer
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+```
+
+Then apply the config with:
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --model-path Qwen/Qwen-Image \
+  --cache-dit-config cache.yaml \
+  --prompt "A beautiful sunset over the mountains"
+```
+
+- DBCache + TaylorSeer + SCM (Step Computation Mask)
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+  # Must set the num_inference_steps for SCM. The SCM will automatically
+  # generate the steps computation mask based on the num_inference_steps.
+  # Reference: https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#scm-steps-computation-masking
+  num_inference_steps: 28
+  steps_computation_mask: fast
+```
+
+- DBCache + TaylorSeer + SCM (Step Computation Mask) + Cache CFG
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+  num_inference_steps: 28
+  steps_computation_mask: fast
+  enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc.
+```
+
+### Distributed inference
+
+- 1D Parallelism
+
+Define a parallelism only config yaml `parallel.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+```
+
+Then, apply the distributed inference acceleration config from yaml. `ulysses_size: auto` means that cache-dit will auto detect the `world_size` as the ulysses_size. Otherwise, you should manually set it as specific int number, e.g, 4.
+
+Then apply the distributed config with: (Note: please add `--num-gpus N` to specify the number of gpus for distributed inference)
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --num-gpus 4 \
+  --model-path Qwen/Qwen-Image \
+  --cache-dit-config parallel.yaml \
+  --prompt "A futuristic cityscape at sunset"
+```
+
+- 2D Parallelism
+
+You can also define a 2D parallelism config yaml `parallel_2d.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  tp_size: 2
+  attention_backend: native
+```
+Then, apply the 2D parallelism config from yaml. Here `tp_size: 2` means using tensor parallelism with size 2. The `ulysses_size: auto` means that cache-dit will auto detect the `world_size // tp_size` as the ulysses_size.
+
+- 3D Parallelism
+
+You can also define a 3D parallelism config yaml `parallel_3d.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: 2
+  ring_size: 2
+  tp_size: 2
+  attention_backend: native
+```
+Then, apply the 3D parallelism config from yaml. Here `ulysses_size: 2`, `ring_size: 2`, `tp_size: 2` means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2.
+
+- Ulysses Anything Attention
+
+To enable Ulysses Anything Attention, you can define a parallelism config yaml `parallel_uaa.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  ulysses_anything: true
+```
+
+- Ulysses FP8 Communication
+
+For device that don't have NVLink support, you can enable Ulysses FP8 Communication to further reduce the communication overhead. You can define a parallelism config yaml `parallel_fp8.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  ulysses_float8: true
+```
+
+- Async Ulysses CP
+
+You can also enable async ulysses CP to overlap the communication and computation. Define a parallelism config yaml `parallel_async.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  ulysses_async: true # Now, only support for FLUX.1, Qwen-Image, Ovis-Image and Z-Image.
+```
+Then, apply the config from yaml. Here `ulysses_async: true` means enabling async ulysses CP.
+
+- TE-P and VAE-P
+
+You can also specify the extra parallel modules in the yaml config. For example, define a parallelism config yaml `parallel_extra.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  extra_parallel_modules: ["text_encoder", "vae"]
+```
+
+
+### Hybrid Cache and Parallelism
+
+Define a hybrid cache and parallel acceleration config yaml `hybrid.yaml` file that contains:
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  extra_parallel_modules: ["text_encoder", "vae"]
+```
+
+Then, apply the hybrid cache and parallel acceleration config from yaml.
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --num-gpus 4 \
+  --model-path Qwen/Qwen-Image \
+  --cache-dit-config hybrid.yaml \
+  --prompt "A beautiful sunset over the mountains"
+```
+
+### Attention Backend
+
+In some cases, users may want to only specify the attention backend without any other optimization configs. In this case, you can define a yaml file `attention.yaml` that only contains:
+
+```yaml
+attention_backend: "flash" # '_flash_3' for Hopper
+```
+
+### Quantization
+
+You can also specify the quantization config in the yaml file, required `torchao>=0.16.0`. For example, define a yaml file `quantize.yaml` that contains:
+
+```yaml
+quantize_config: # quantization configuration for transformer modules
+  # float8 (DQ), float8_weight_only, float8_blockwise, int8 (DQ), int8_weight_only, etc.
+  quant_type: "float8"
+  # layers to exclude from quantization (transformer). layers that contains any of the
+  # keywords in the exclude_layers list will be excluded from quantization. This is useful
+  # for some sensitive layers that are not robust to quantization, e.g., embedding layers.
+  exclude_layers:
+    - "embedder"
+    - "embed"
+  verbose: false # whether to print verbose logs during quantization
+```
+Then, apply the quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. For example:
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --model-path Qwen/Qwen-Image \
+  --warmup \
+  --cache-dit-config quantize.yaml \
+  --enable-torch-compile \
+  --dit-cpu-offload false \
+  --text-encoder-cpu-offload false \
+  --prompt "A beautiful sunset over the mountains"
+```
+
+### Combined Configs: Cache + Parallelism + Quantization
+
+You can also combine all the above configs together in a single yaml file `combined.yaml` that contains:
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  extra_parallel_modules: ["text_encoder", "vae"]
+quantize_config:
+  quant_type: "float8"
+  exclude_layers:
+    - "embedder"
+    - "embed"
+  verbose: false
+```
+Then, apply the combined cache, parallelism and quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization.
+
+## Advanced Configuration
+
+### DBCache Parameters
+
+DBCache controls block-level caching behavior:
+
+| Parameter | Env Variable              | Default | Description                              |
+|-----------|---------------------------|---------|------------------------------------------|
+| Fn        | `SGLANG_CACHE_DIT_FN`     | 1       | Number of first blocks to always compute |
+| Bn        | `SGLANG_CACHE_DIT_BN`     | 0       | Number of last blocks to always compute  |
+| W         | `SGLANG_CACHE_DIT_WARMUP` | 4       | Warmup steps before caching starts       |
+| R         | `SGLANG_CACHE_DIT_RDT`    | 0.24    | Residual difference threshold            |
+| MC        | `SGLANG_CACHE_DIT_MC`     | 3       | Maximum continuous cached steps          |
+
+### TaylorSeer Configuration
+
+TaylorSeer improves caching accuracy using Taylor expansion:
+
+| Parameter | Env Variable                  | Default | Description                     |
+|-----------|-------------------------------|---------|---------------------------------|
+| Enable    | `SGLANG_CACHE_DIT_TAYLORSEER` | false   | Enable TaylorSeer calibrator    |
+| Order     | `SGLANG_CACHE_DIT_TS_ORDER`   | 1       | Taylor expansion order (1 or 2) |
+
+### Combined Configuration Example
+
+DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters
+simultaneously:
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+SGLANG_CACHE_DIT_FN=2 \
+SGLANG_CACHE_DIT_BN=1 \
+SGLANG_CACHE_DIT_WARMUP=4 \
+SGLANG_CACHE_DIT_RDT=0.4 \
+SGLANG_CACHE_DIT_MC=4 \
+SGLANG_CACHE_DIT_TAYLORSEER=true \
+SGLANG_CACHE_DIT_TS_ORDER=2 \
+sglang generate --model-path black-forest-labs/FLUX.1-dev \
+    --prompt "A curious raccoon in a forest"
+```
+
+### SCM (Step Computation Masking)
+
+SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and
+which to use cached results.
+
+**SCM Presets**
+
+SCM is configured with presets:
+
+| Preset   | Compute Ratio | Speed    | Quality    |
+|----------|---------------|----------|------------|
+| `none`   | 100%          | Baseline | Best       |
+| `slow`   | ~75%          | ~1.3x    | High       |
+| `medium` | ~50%          | ~2x      | Good       |
+| `fast`   | ~35%          | ~3x      | Acceptable |
+| `ultra`  | ~25%          | ~4x      | Lower      |
+
+**Usage**
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+SGLANG_CACHE_DIT_SCM_PRESET=medium \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A futuristic cityscape at sunset"
+```
+
+**Custom SCM Bins**
+
+For fine-grained control over which steps to compute vs cache:
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
+SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A futuristic cityscape at sunset"
+```
+
+**SCM Policy**
+
+| Policy    | Env Variable                          | Description                                 |
+|-----------|---------------------------------------|---------------------------------------------|
+| `dynamic` | `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` | Adaptive caching based on content (default) |
+| `static`  | `SGLANG_CACHE_DIT_SCM_POLICY=static`  | Fixed caching pattern                       |
+
+## Environment Variables
+
+All Cache-DiT parameters can be configured via environment variables.
+See [Environment Variables](../../environment_variables.md) for the complete list.
+
+## Supported Models
+
+SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
+
+| Model Family | Example Models              |
+|--------------|-----------------------------|
+| Wan          | Wan2.1, Wan2.2              |
+| Flux         | FLUX.1-dev, FLUX.2-dev      |
+| Z-Image      | Z-Image-Turbo               |
+| Qwen         | Qwen-Image, Qwen-Image-Edit |
+| Hunyuan      | HunyuanVideo                |
+
+## Performance Tips
+
+1. **Start with defaults**: The default parameters work well for most models
+2. **Use TaylorSeer**: It typically improves both speed and quality
+3. **Tune R threshold**: Lower values = better quality, higher values = faster
+4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance
+5. **Warmup matters**: Higher warmup = more stable caching decisions
+
+## Limitations
+
+- **SGLang-native pipelines**: Distributed support (TP/SP) is not yet validated; Cache-DiT will be automatically
+  disabled when `world_size > 1`.
+- **SCM minimum steps**: SCM requires >= 8 inference steps to be effective
+- **Model support**: Only models registered in Cache-DiT's BlockAdapterRegister are supported
+
+## Troubleshooting
+
+### SCM disabled for low step count
+
+For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache
+acceleration still works.
+
+## References
+
+- [Cache-DiT](https://github.com/vipshop/cache-dit)
+- [SGLang Diffusion](../index.md)

docs/diffusion/performance/cache/index.md

@@ -0,0 +1,65 @@
+# Caching Acceleration
+
+SGLang provides two complementary caching strategies for Diffusion Transformer (DiT) models. Both reduce denoising cost by skipping redundant computation, but they operate at different levels.
+
+## Overview
+
+SGLang supports two complementary caching approaches:
+
+| Strategy | Scope | Mechanism | Best For |
+|----------|-------|-----------|----------|
+| **Cache-DiT** | Block-level | Skip individual transformer blocks dynamically | Advanced, higher speedup |
+| **TeaCache** | Timestep-level | Skip entire denoising steps based on L1 similarity | Simple, built-in |
+
+## Cache-DiT
+
+[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with
+advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**.
+
+See [cache_dit.md](cache_dit.md) for detailed configuration.
+
+### Quick Start
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A beautiful sunset over the mountains"
+```
+
+### Key Features
+
+- **DBCache**: Dynamic block-level caching based on residual differences
+- **TaylorSeer**: Taylor expansion-based calibration for optimized caching
+- **SCM**: Step-level computation masking for additional speedup
+
+## TeaCache
+
+TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
+
+See [teacache.md](teacache.md) for detailed documentation.
+
+### Quick Overview
+
+- Tracks L1 distance between modulated inputs across timesteps
+- When accumulated distance is below threshold, reuses cached residual
+- Supports CFG with separate positive/negative caches
+
+### Supported Models
+
+- Wan (wan2.1, wan2.2)
+- Hunyuan (HunyuanVideo)
+- Z-Image
+
+For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.
+
+```{toctree}
+:maxdepth: 1
+
+cache_dit
+teacache
+```
+
+## References
+
+- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
+- [TeaCache Paper](https://arxiv.org/abs/2411.14324)

docs/diffusion/performance/cache/teacache.md

@@ -0,0 +1,84 @@
+# TeaCache
+
+> **Note**: This is one of two caching strategies available in SGLang.
+> For an overview of all caching options, see [caching](../index.md).
+
+TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
+
+## Overview
+
+TeaCache works by:
+1. Tracking the L1 distance between modulated inputs across consecutive timesteps
+2. Accumulating the rescaled L1 distance over steps
+3. When accumulated distance is below a threshold, reusing the cached residual
+4. Supporting CFG (Classifier-Free Guidance) with separate positive/negative caches
+
+## How It Works
+
+### L1 Distance Tracking
+
+At each denoising step, TeaCache computes the relative L1 distance between the current and previous modulated inputs:
+
+```
+rel_l1 = |current - previous|.mean() / |previous|.mean()
+```
+
+This distance is then rescaled using polynomial coefficients and accumulated:
+
+```
+accumulated += poly(coefficients)(rel_l1)
+```
+
+### Cache Decision
+
+- If `accumulated >= threshold`: Force computation, reset accumulator
+- If `accumulated < threshold`: Skip computation, use cached residual
+
+### CFG Support
+
+For models that support CFG cache separation (Wan, Hunyuan, Z-Image), TeaCache maintains separate caches for positive and negative branches:
+- `previous_modulated_input` / `previous_residual` for positive branch
+- `previous_modulated_input_negative` / `previous_residual_negative` for negative branch
+
+For models that don't support CFG separation (Flux, Qwen), TeaCache is automatically disabled when CFG is enabled.
+
+## Configuration
+
+TeaCache is configured via `TeaCacheParams` in the sampling parameters:
+
+```python
+from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams
+
+params = TeaCacheParams(
+    teacache_thresh=0.1,           # Threshold for accumulated L1 distance
+    coefficients=[1.0, 0.0, 0.0],  # Polynomial coefficients for L1 rescaling
+)
+```
+
+### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `teacache_thresh` | float | Threshold for accumulated L1 distance. Lower = more caching, faster but potentially lower quality |
+| `coefficients` | list[float] | Polynomial coefficients for L1 rescaling. Model-specific tuning |
+
+### Model-Specific Configurations
+
+Different models may have different optimal configurations. The coefficients are typically tuned per-model to balance speed and quality.
+
+## Supported Models
+
+TeaCache is built into the following model families:
+
+| Model Family | CFG Cache Separation | Notes |
+|--------------|---------------------|-------|
+| Wan (wan2.1, wan2.2) | Yes | Full support |
+| Hunyuan (HunyuanVideo) | Yes | To be supported |
+| Z-Image | Yes | To be supported |
+| Flux | No | To be supported |
+| Qwen | No | To be supported |
+
+
+## References
+
+- [TeaCache: Accelerating Diffusion Models with Temporal Similarity](https://arxiv.org/abs/2411.14324)

docs/diffusion/performance/deployment_cookbook.md

@@ -0,0 +1,94 @@
+# Deployment Cookbook
+
+This page gives practical defaults for choosing CPU offload, FSDP, CFG parallelism, SP, and TP.
+
+## Quick Rule
+
+Use the simplest setting that fits your memory target:
+
+| Goal                                       | Recommended setting                                                                |
+|--------------------------------------------|------------------------------------------------------------------------------------|
+| Fastest single-GPU run when the model fits | Disable CPU offload and do not use FSDP.                                           |
+| Lower single-GPU memory usage              | Use component CPU offload, or layerwise DiT offload for supported Wan/MOVA models. |
+| Faster multi-GPU Qwen/Wan CFG generation   | Use FSDP with CFG parallelism and disable CPU offload.                             |
+| Sequence length or video-shape scaling     | Use SP/Ulysses/Ring when the model benefits from sequence parallelism.             |
+| TP compatibility or encoder-heavy paths    | Set TP explicitly; do not treat TP as the default latency optimization.            |
+
+Base the decision on available memory on the selected GPU(s).
+
+- For multi-GPU deployment: the least-free selected GPU is the bottleneck. A busy 80GiB GPU can behave like a much smaller GPU.
+- For single-GPU deployment: FSDP shards DiT weights across multiple GPUs. It is not useful for keeping a single-GPU deployment on one GPU; for that case use CPU offload.
+
+## Performance Modes
+
+`--performance-mode` applies safe presets without overriding explicit offload, FSDP, or parallelism flags. `auto` is the default. Use `manual` when you need to keep performance-related server args under explicit user control. `--mode` is a short alias.
+
+| Mode       | Meaning                                                                                                                   |
+|------------|---------------------------------------------------------------------------------------------------------------------------|
+| `manual`   | Keeps performance-related server args under explicit user control.                                                       |
+| `auto`     | Default. Keeps legacy safe offload defaults and uses FSDP/CFG only on validated multi-GPU deployments where FSDP can replace DiT offload. |
+| `speed`    | Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset; may OOM.           |
+| `memory`   | Favors lower GPU memory. Uses component offload, or Wan/MOVA layerwise DiT offload when supported.                         |
+
+`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. Text encoder, image encoder, and other component residency still follow the offload policy unless the model marks a high-memory resident path as safe. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk.
+
+The modes tune residency for native pipeline components declared to the component residency manager. Today this covers the major DiT, text/image encoder, VAE, vocoder, and upsampler components; DiT can use layerwise offload when supported, while text encoders use either resident execution or component CPU offload. Do not assume text-encoder layerwise offload unless a model implements and validates it.
+
+NOTE:
+The preset is intentionally coarse. A future continuous value such as `0.0` to `1.0` could express the speed-memory tradeoff more precisely, but it would need model-specific memory models and clearer user expectations. Until then, use the preset plus explicit flags for overrides.
+
+Examples:
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --num-gpus 2 \
+  --performance-mode auto
+```
+
+```bash
+sglang generate \
+  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
+  --performance-mode memory
+```
+
+Explicit flags win over the mode:
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --num-gpus 2 \
+  --performance-mode auto \
+  --use-fsdp-inference false
+```
+
+In this example, `auto` will not re-enable FSDP. The same applies to parallelism; for example, `--enable-cfg-parallel false` keeps CFG parallelism disabled.
+
+## Interpreting The Levers
+
+**No offload** keeps model components resident on GPU. It is usually fastest when memory is sufficient.
+
+**Component CPU offload** lowers GPU memory by moving large components to CPU. It is simple and robust, but it usually trades latency for memory.
+
+**Layerwise DiT offload** lowers DiT memory further for supported Wan/MOVA models by moving DiT layers between CPU and GPU. It can be the best single-GPU memory mode, but may increase latency and lower throughput.
+
+**FSDP** shards DiT weights across multiple GPUs and all-gathers weights during forward. It can reduce DiT CPU offload cost on multi-GPU deployments, especially for validated Wan I2V workloads.
+
+FSDP sharding granularity matters. SGLang Diffusion prefers sharding direct repeated transformer block entries such as `transformer_blocks.0` or `blocks.0`. Coarser sharding lowers wrapper count but can increase all-gather peak memory; finer sharding can reduce transient memory but adds communication and scheduling overhead. If a model does not define an explicit sharding rule, the loader falls back to repeated block class names and common direct numbered block paths.
+
+**CFG parallelism** splits positive and negative CFG branches across GPUs. For Qwen/Wan workloads with normal step counts, this is the most reliable multi-GPU speedup observed so far.
+
+**SP/Ulysses/Ring** splits sequence work. It can help video workloads, but validated Qwen/Wan runs showed CFG parallelism outperforming SP for latency.
+
+**TP** is supported for compatibility and some model structures, but current measurements do not make it the default latency path for Qwen/Wan.
+
+## Current Benchmark Takeaways
+
+Observed regular-scale trends:
+
+- Z-Image: single-GPU no-offload was faster than FSDP/SP in the tested setting; keep FSDP off unless memory or parallelism requires it.
+- Qwen-Image: keep the default non-FSDP path unless a specific FSDP/SP/Ring setting has been benchmarked on the target hardware.
+- Wan: FSDP can replace DiT offload on validated multi-GPU workloads, while text/image encoders may still need component offload. Keep model-specific precision checks before making FSDP automatic for a path.
+- Component offload mainly reduced memory; it did not improve latency in the tested no-offload-vs-offload runs.
+
+Always benchmark with your actual resolution, frame count, step count, and GPU type before locking production defaults.

docs/diffusion/performance/index.md

@@ -0,0 +1,44 @@
+# Performance
+
+This section covers the main performance levers for SGLang Diffusion: attention backends, caching acceleration, and profiling.
+
+## Overview
+
+| Optimization | Type | Description |
+|--------------|------|-------------|
+| **Cache-DiT** | Caching | Block-level caching with DBCache, TaylorSeer, and SCM |
+| **TeaCache** | Caching | Timestep-level caching based on temporal similarity |
+| **Attention Backends** | Kernel | Optimized attention implementations (FlashAttention, SageAttention, etc.) |
+| **Profiling** | Diagnostics | PyTorch Profiler and Nsight Systems guidance |
+
+## Start Here
+
+- Use [Attention Backends](attention_backends.md) to choose the best backend for your model and hardware.
+- Use [Deployment Cookbook](deployment_cookbook.md) to choose CPU offload, FSDP, CFG parallelism, SP, and TP.
+- Use [Caching Acceleration](cache/index.md) to reduce denoising cost with Cache-DiT or TeaCache.
+- Use [Profiling](profiling.md) when you need to diagnose a bottleneck rather than guess.
+
+## Caching at a Glance
+
+- [Cache-DiT](cache/cache_dit.md) is block-level caching for diffusers pipelines and higher speedup-oriented tuning.
+- [TeaCache](cache/teacache.md) is timestep-level caching built into SGLang model families.
+
+```{toctree}
+:maxdepth: 1
+
+attention_backends
+deployment_cookbook
+cache/index
+profiling
+```
+
+## Current Baseline Snapshot
+
+For Ring SP benchmark details, see:
+
+- [Ring SP Performance](ring_sp_performance.md)
+
+## References
+
+- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
+- [TeaCache Paper](https://arxiv.org/abs/2411.14324)

docs/diffusion/performance/profiling.md

@@ -0,0 +1,136 @@
+# Profiling Multimodal Generation
+
+This guide covers profiling techniques for multimodal generation pipelines in SGLang.
+
+## PyTorch Profiler
+
+PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics.
+
+### Denoising Stage Profiling
+
+Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step):
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+  --seed 0 \
+  --profile
+```
+
+**Parameters:**
+- `--profile`: Enable profiling for the denoising stage
+- `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
+  - Smaller values reduce trace file size
+  - Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step
+
+### Full Pipeline Profiling
+
+Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.):
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+  --seed 0 \
+  --profile \
+  --profile-all-stages
+```
+
+**Parameters:**
+- `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising
+
+### Output Location
+
+By default, trace files are saved in the ./logs/ directory.
+
+The exact output file path will be shown in the console output, for example:
+
+```bash
+[mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz
+```
+
+### View Traces
+
+Load and visualize trace files at:
+- https://ui.perfetto.dev/ (recommended)
+- chrome://tracing (Chrome only)
+
+For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.
+
+
+### `--perf-dump-path` (Stage/Step Timing Dump)
+
+Besides profiler traces, you can also dump a lightweight JSON report that contains:
+- stage-level timing breakdown for the full pipeline
+- step-level timing breakdown for the denoising stage (per diffusion step)
+
+This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike).
+
+The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key.
+
+Example:
+
+```bash
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "<PROMPT>" \
+  --perf-dump-path perf.json
+```
+
+## Nsight Systems
+
+Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns.
+
+### Installation
+
+See the [SGLang profiling guide](https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/benchmark_and_profiling.md#profile-with-nsight) for installation instructions.
+
+### Basic Profiling
+
+Profile the entire pipeline execution:
+
+```bash
+nsys profile \
+  --trace-fork-before-exec=true \
+  --cuda-graph-trace=node \
+  --force-overwrite=true \
+  -o QwenImage \
+  sglang generate \
+    --model-path Qwen/Qwen-Image \
+    --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+    --seed 0
+```
+
+### Targeted Stage Profiling
+
+Use `--delay` and `--duration` to capture specific stages and reduce file size:
+
+```bash
+nsys profile \
+  --trace-fork-before-exec=true \
+  --cuda-graph-trace=node \
+  --force-overwrite=true \
+  --delay 10 \
+  --duration 30 \
+  -o QwenImage_denoising \
+  sglang generate \
+    --model-path Qwen/Qwen-Image \
+    --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+    --seed 0
+```
+
+**Parameters:**
+- `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
+- `--duration N`: Capture for N seconds (focus on specific stages)
+- `--force-overwrite`: Overwrite existing output files
+
+## Notes
+
+- **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems
+- **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline
+- **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads
+
+## FAQ
+
+- If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time.

docs/diffusion/performance/ring_sp_performance.md

@@ -0,0 +1,67 @@
+# Ring SP Benchmark: Wan2.2-TI2V-5B (u1r2 vs Baseline)
+
+This page reports Ring-SP performance for `Wan2.2-TI2V-5B-Diffusers` using:
+
+- Parallel config: `sp=2, ulysses=1, ring=2` (short: `u1r2`)
+- Baseline config: `sp=1, ulysses=1, ring=1` (short: `u1r1`)
+
+## Benchmark Setup
+
+- Model: `Wan2.2-TI2V-5B-Diffusers`
+- GPU: `48G RTX40 series * 2`
+
+## Online Serving
+
+### Ring SP (`u1r2`)
+
+```bash
+sglang serve \
+  --model-type diffusion \
+  --model-path /model/HuggingFace/Wan-AI/Wan2.2-TI2V-5B-Diffusers \
+  --num-gpus 2 --sp-degree 2 --ulysses-degree 1 --ring-degree 2 \
+  --port 8898
+```
+
+### Baseline (`u1r1`)
+
+```bash
+sglang serve \
+  --model-type diffusion \
+  --model-path /model/HuggingFace/Wan-AI/Wan2.2-TI2V-5B-Diffusers \
+  --num-gpus 1 --sp-degree 1 --ulysses-degree 1 --ring-degree 1 \
+  --port 8898
+```
+
+## Benchmarks
+
+### Benchmark Disclaimer
+
+These benchmarks are provided for reference under one specific setup and command configuration. Actual performance may vary with model settings, runtime environment, and request patterns.
+
+### Stage Time Breakdown
+
+| Stage / Metric | `u1r2` (s) | `u1r1` baseline (s) | Speedup |
+|---|---:|---:|---:|
+| InputValidation | 0.1060 | 0.1029 | 0.97x |
+| TextEncoding | 1.3965 | 2.2261 | 1.59x |
+| LatentPreparation | 0.0002 | 0.0002 | 1.00x |
+| TimestepPreparation | 0.0003 | 0.0004 | 1.33x |
+| Denoising | 52.6358 | 71.6785 | 1.36x |
+| Decoding | 7.6708 | 13.4314 | 1.75x |
+| **Total** | **63.74** | **90.63** | **1.42x** |
+
+### Memory Usage
+
+| Memory Metric | `u1r2` (GB) | `u1r1` baseline (GB) | Delta |
+|---|---:|---:|---:|
+| Peak GPU Memory | 20.07 | 27.40 | -7.33 |
+| Peak Allocated | 13.35 | 20.40 | -7.05 |
+| Memory Overhead | 6.72 | 7.00 | -0.28 |
+| Overhead Ratio | 33.5% | 25.6% | +7.9pp |
+
+## Summary
+
+- End-to-end latency improves from `90.63s` to `63.74s` (`1.42x`).
+- Main gains come from `Denoising` (`1.36x`) and `Decoding` (`1.75x`).
+- Absolute memory usage drops noticeably on Ring-SP (`Peak GPU Memory -7.33GB`, `Peak Allocated -7.05GB`).
+- Overhead ratio rises (`+7.9pp`), so future tuning can focus on reducing communication/runtime overhead while preserving the latency gain.