wehub-resource-sync
96 lines
3.5 KiB
Markdown
96 lines
3.5 KiB
Markdown
# msModelSlim Quantization
|
|
|
|
## Overview
|
|
|
|
[msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression
|
|
toolkit for producing pre-quantized model checkpoints. In vLLM-Omni, these
|
|
checkpoints run through the Ascend/NPU path with `--quantization ascend`.
|
|
|
|
msModelSlim is static quantization: quantized weights are generated offline
|
|
before vLLM-Omni inference starts.
|
|
|
|
## Hardware Support
|
|
|
|
| Device | Support |
|
|
|--------|---------|
|
|
| NVIDIA Blackwell GPU (SM 100+) | ❌ |
|
|
| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ |
|
|
| NVIDIA Ampere GPU (SM 80+) | ❌ |
|
|
| AMD ROCm | ❌ |
|
|
| Intel XPU | ❌ |
|
|
| Ascend NPU | ✅ |
|
|
|
|
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
|
|
guide.
|
|
|
|
## Model Type Support
|
|
|
|
### Diffusion Model (Qwen-Image, Wan2.2)
|
|
|
|
| Model | Base model | Scope | Hardware | Notes |
|
|
|-------|------------|-------|----------|-------|
|
|
| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Ascend NPU | Upstream msModelSlim provides a Wan2.2 quantization recipe; vLLM-Omni inference validation is not listed |
|
|
| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Ascend NPU | Not validated in this guide |
|
|
| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Ascend A2/A3 NPU | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation |
|
|
|
|
Public Hugging Face quantized weights are not available yet. Use the
|
|
[HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8)
|
|
to generate the checkpoint manually.
|
|
|
|
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
|
|
|
| Model | Scope | Status | Notes |
|
|
|-------|-------|--------|-------|
|
|
| Qwen3-Omni | Thinker or language-model stage | Not validated | No msModelSlim omni checkpoint path is documented |
|
|
| Qwen3-TTS | TTS language-model stage | Not validated | No msModelSlim TTS checkpoint path is documented |
|
|
|
|
### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
|
|
|
|
| Model | Scope | Status | Notes |
|
|
|-------|-------|--------|-------|
|
|
| BAGEL | Stage-specific diffusion or transformer weights | Not validated | Requires a model-specific Ascend adaptation |
|
|
| GLM-Image | Stage-specific diffusion or transformer weights | Not validated | Requires a model-specific Ascend adaptation |
|
|
|
|
## Configuration
|
|
|
|
Offline inference:
|
|
|
|
```bash
|
|
python text_to_image.py --model <quantized-model-path> --quantization ascend
|
|
```
|
|
|
|
Online serving:
|
|
|
|
```bash
|
|
vllm serve <quantized-model-path> --omni --quantization ascend
|
|
```
|
|
|
|
## Parameters
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `--quantization` | str | - | Use `ascend` for msModelSlim-produced checkpoints |
|
|
| `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling |
|
|
|
|
Example msModelSlim command for a Wan2.2 W8A8 checkpoint:
|
|
|
|
```bash
|
|
msmodelslim quant \
|
|
--model_path /path/to/wan2_2_t2v_float_weights \
|
|
--save_path /path/to/wan2_2_t2v_quantized_weights \
|
|
--device npu \
|
|
--model_type Wan2_2 \
|
|
--config_path /path/to/wan2_2_w8a8f8_mxfp_t2v.yaml \
|
|
--trust_remote_code True
|
|
```
|
|
|
|
For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above.
|
|
|
|
## Validation and Notes
|
|
|
|
1. Run with the Ascend/NPU installation and environment.
|
|
2. The `ascend` quantization method expects weights produced by the Ascend
|
|
tooling; it is not a load-time CUDA quantizer.
|
|
3. Keep the quantized checkpoint aligned with the same model architecture and
|
|
stage config used for BF16 inference.
|