wehub-resource-sync
150 lines
4.1 KiB
Markdown
150 lines
4.1 KiB
Markdown
# HSDP Guide
|
||
|
||
|
||
## Table of Content
|
||
|
||
- [Overview](#overview)
|
||
- [Quick Start](#quick-start)
|
||
- [Example Script](#example-script)
|
||
- [Configuration Parameters](#configuration-parameters)
|
||
- [Best Practices](#best-practices)
|
||
- [Summary](#summary)
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
HSDP (Hybrid Sharded Data Parallel) shards model weights across GPUs to reduce per-GPU memory usage. This enables inference of large models (e.g., Wan2.2 14B) on GPUs with limited memory.
|
||
|
||
Unlike Tensor Parallelism which splits computation, HSDP uses PyTorch's FSDP2 to shard and redistribute weights at runtime. Each GPU only holds a fraction of the model weights, and weights are gathered on-demand during forward passes.
|
||
|
||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||
|
||
**Operating Modes:**
|
||
|
||
- **Standalone Mode**: HSDP alone without other parallelism. Must specify `hsdp_shard_size` explicitly.
|
||
- **Combined Mode**: HSDP overlays on top of other parallelism (Ulysses-SP, CFG-Parallel). HSDP dimensions must match world_size.
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
### Basic Usage
|
||
|
||
Simplest working example (standalone HSDP, shard across 4 GPUs):
|
||
|
||
```python
|
||
from vllm_omni import Omni
|
||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||
|
||
omni = Omni(
|
||
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
|
||
parallel_config=DiffusionParallelConfig(
|
||
use_hsdp=True,
|
||
hsdp_shard_size=4, # Shard across 4 GPUs
|
||
),
|
||
)
|
||
|
||
outputs = omni.generate(
|
||
"A cat playing piano",
|
||
OmniDiffusionSamplingParams(num_inference_steps=50),
|
||
)
|
||
```
|
||
|
||
### Combined with Sequence Parallel
|
||
|
||
```python
|
||
omni = Omni(
|
||
model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
|
||
parallel_config=DiffusionParallelConfig(
|
||
ulysses_degree=4, # Sequence parallel
|
||
use_hsdp=True, # HSDP overlays on SP
|
||
),
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Example Script
|
||
|
||
### Offline Inference
|
||
|
||
Use Python script under `examples/offline_inference/image_to_video/`:
|
||
|
||
```bash
|
||
# Standalone HSDP: shard across 4 GPUs
|
||
python examples/offline_inference/image_to_video/image_to_video.py \
|
||
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||
--use-hsdp \
|
||
--hsdp-shard-size 4
|
||
|
||
# Combined HSDP + Sequence Parallel
|
||
python examples/offline_inference/image_to_video/image_to_video.py \
|
||
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
|
||
--ulysses-degree 4 \
|
||
--use-hsdp
|
||
```
|
||
|
||
### Online Serving
|
||
|
||
**Standalone HSDP** (shard model across 4 GPUs):
|
||
|
||
```bash
|
||
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091 \
|
||
--use-hsdp --hsdp-shard-size 4
|
||
```
|
||
|
||
**Combined with Sequence Parallel**:
|
||
|
||
```bash
|
||
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091 \
|
||
--use-hsdp --usp 4
|
||
```
|
||
|
||
---
|
||
|
||
## Configuration Parameters
|
||
|
||
In `DiffusionParallelConfig`:
|
||
|
||
| Parameter | Type | Default | Description |
|
||
|-----------|------|---------|-------------|
|
||
| `use_hsdp` | bool | False | Enable HSDP |
|
||
| `hsdp_shard_size` | int | -1 | Number of GPUs to shard weights across. `-1` = auto (requires other parallelism > 1) |
|
||
| `hsdp_replicate_size` | int | 1 | Number of replica groups. Each group holds a full sharded copy |
|
||
|
||
**Constraints:**
|
||
|
||
- `hsdp_replicate_size × hsdp_shard_size == world_size`
|
||
- HSDP cannot be used with Tensor Parallelism (`tensor_parallel_size` must be 1)
|
||
|
||
---
|
||
|
||
## Best Practices
|
||
|
||
### When to Use
|
||
|
||
**Good for:**
|
||
|
||
- Very large models (e.g., Wan2.2 14B)
|
||
- Multi-GPU setups where memory reduction is the primary goal
|
||
- Combining with Sequence Parallelism for large video models
|
||
|
||
**Not for:**
|
||
|
||
- Models that fit comfortably in single-GPU memory
|
||
- Use cases requiring Tensor Parallelism (HSDP and TP are mutually exclusive)
|
||
|
||
### Adding HSDP Support to New Models
|
||
|
||
For detailed instructions on adding HSDP support to new models, see the [HSDP Contributing Guide](../../../design/feature/hsdp.md).
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
1. ✅ **Enable HSDP** - Set `use_hsdp=True` and `hsdp_shard_size` to reduce per-GPU memory for large models
|
||
2. ✅ **Combine with SP** - Use together with `ulysses_degree` for video models requiring both memory reduction and sequence parallelism
|
||
3. ⚠️ **Incompatible with TP** - `tensor_parallel_size` must be 1 when HSDP is enabled
|