wehub/modelscope--ms-swift
1
0
Fork 0
Code Issues Pull Requests Actions 1 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Lint test / lint (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:34:58 +08:00
commit a203934033
1368 changed files with 175001 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source_en/.readthedocs.yaml
+29
View File
@@ -0,0 +1,29 @@
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.12"
# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/source_en/conf.py
# Optionally build your docs in additional formats such as PDF and ePub
# formats:
# - pdf
# - epub
# Optional but recommended, declare the Python requirements required
# to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: requirements/docs.txt
- requirements: requirements/framework.txt
docs/source_en/BestPractices/AMD-support.md
+324
View File
@@ -0,0 +1,324 @@
# AMD GPU Support
## 1. Environment setup
### 1.1 Base environment
Pull the ms-swift image built for the AMD ROCm stack, then start the container with the commands below.
If you need a newer ms-swift version, upgrade with pip or install from source code (adding `--no-deps` is recommended to avoid pulling in dependency upgrades that may cause issues).
```bash
IMAGE_NAME=amdagi/modelscope:ubuntu22.04-rocm7.2.0-py312-torch2.10.0-vllm0.18.1-modelscope1.35.1-swift4.1.0
docker pull ${IMAGE_NAME}
CONTAINER_NAME=swift_test
docker run -it --network=host --ipc=host --privileged --group-add video \
--device=/dev/dri --device=/dev/kfd \
--shm-size 512G --ulimit memlock=-1 \
--security-opt seccomp=unconfined --cap-add SYS_PTRACE \
--name ${CONTAINER_NAME} \
${IMAGE_NAME} \
/bin/bash
```
### 1.2 Environment check
- Confirm the availability of AMD devices for PyTorch in the container.
```bash
python -c "import torch;print(torch.cuda.is_available())" # output: True
```
- Inspect GPU topology and NUMA: `rocm-smi --showtopo`
```
============================ ROCm System Management Interface ============================
WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status
================================ Weight between two GPUs =================================
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7
GPU0 0 15 15 15 15 15 15 15
GPU1 15 0 15 15 15 15 15 15
GPU2 15 15 0 15 15 15 15 15
GPU3 15 15 15 0 15 15 15 15
GPU4 15 15 15 15 0 15 15 15
GPU5 15 15 15 15 15 0 15 15
GPU6 15 15 15 15 15 15 0 15
GPU7 15 15 15 15 15 15 15 0
================================= Hops between two GPUs ==================================
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7
GPU0 0 1 1 1 1 1 1 1
GPU1 1 0 1 1 1 1 1 1
GPU2 1 1 0 1 1 1 1 1
GPU3 1 1 1 0 1 1 1 1
GPU4 1 1 1 1 0 1 1 1
GPU5 1 1 1 1 1 0 1 1
GPU6 1 1 1 1 1 1 0 1
GPU7 1 1 1 1 1 1 1 0
=============================== Link Type between two GPUs ===============================
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7
GPU0 0 XGMI XGMI XGMI XGMI XGMI XGMI XGMI
GPU1 XGMI 0 XGMI XGMI XGMI XGMI XGMI XGMI
GPU2 XGMI XGMI 0 XGMI XGMI XGMI XGMI XGMI
GPU3 XGMI XGMI XGMI 0 XGMI XGMI XGMI XGMI
GPU4 XGMI XGMI XGMI XGMI 0 XGMI XGMI XGMI
GPU5 XGMI XGMI XGMI XGMI XGMI 0 XGMI XGMI
GPU6 XGMI XGMI XGMI XGMI XGMI XGMI 0 XGMI
GPU7 XGMI XGMI XGMI XGMI XGMI XGMI XGMI 0
======================================= Numa Nodes =======================================
GPU[0] : (Topology) Numa Node: 0
GPU[0] : (Topology) Numa Affinity: 0
GPU[1] : (Topology) Numa Node: 0
GPU[1] : (Topology) Numa Affinity: 0
GPU[2] : (Topology) Numa Node: 0
GPU[2] : (Topology) Numa Affinity: 0
GPU[3] : (Topology) Numa Node: 0
GPU[3] : (Topology) Numa Affinity: 0
GPU[4] : (Topology) Numa Node: 1
GPU[4] : (Topology) Numa Affinity: 1
GPU[5] : (Topology) Numa Node: 1
GPU[5] : (Topology) Numa Affinity: 1
GPU[6] : (Topology) Numa Node: 1
GPU[6] : (Topology) Numa Affinity: 1
GPU[7] : (Topology) Numa Node: 1
GPU[7] : (Topology) Numa Affinity: 1
================================== End of ROCm SMI Log ===================================
```
- Check GPU utilization and VRAM usage (`rocm-smi` or `rocm-smi -u --showmeminfo vram`):
```
# output of 'rocm-smi'
============================================ ROCm System Management Interface ============================================
====================================================== Concise Info ======================================================
Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
(DID, GUID) (Junction) (Socket) (Mem, Compute, ID)
==========================================================================================================================
0 2 0x74a2, 1017 43.0°C 155.0W NPS1, SPX, 0 94Mhz 900Mhz 0% auto 650.0W 0% 0%
1 3 0x74a2, 47713 41.0°C 155.0W NPS1, SPX, 0 91Mhz 900Mhz 0% auto 650.0W 0% 0%
2 4 0x74a2, 37449 45.0°C 159.0W NPS1, SPX, 0 95Mhz 900Mhz 0% auto 650.0W 0% 0%
3 5 0x74a2, 11217 41.0°C 155.0W NPS1, SPX, 0 95Mhz 900Mhz 0% auto 650.0W 0% 0%
4 6 0x74a2, 41880 44.0°C 160.0W NPS1, SPX, 0 91Mhz 900Mhz 0% auto 650.0W 0% 0%
5 7 0x74a2, 6656 42.0°C 157.0W NPS1, SPX, 0 95Mhz 900Mhz 0% auto 650.0W 0% 0%
6 8 0x74a2, 12840 45.0°C 160.0W NPS1, SPX, 0 96Mhz 900Mhz 0% auto 650.0W 0% 0%
7 9 0x74a2, 35760 43.0°C 158.0W NPS1, SPX, 0 107Mhz 900Mhz 0% auto 650.0W 0% 0%
==========================================================================================================================
================================================== End of ROCm SMI Log ===================================================
# output of 'rocm-smi -u --showmeminfo vram'
============================ ROCm System Management Interface ============================
=================================== % time GPU is busy ===================================
GPU[0] : GPU use (%): 0
GPU[0] : GFX Activity: 3862538534
GPU[1] : GPU use (%): 0
GPU[1] : GFX Activity: 4053246251
GPU[2] : GPU use (%): 0
GPU[2] : GFX Activity: 3114103535
GPU[3] : GPU use (%): 0
GPU[3] : GFX Activity: 4026776444
GPU[4] : GPU use (%): 0
GPU[4] : GFX Activity: 1224255679
GPU[5] : GPU use (%): 0
GPU[5] : GFX Activity: 1191191242
GPU[6] : GPU use (%): 0
GPU[6] : GFX Activity: 1184652679
GPU[7] : GPU use (%): 0
GPU[7] : GFX Activity: 2145209382
==========================================================================================
================================== Memory Usage (Bytes) ==================================
GPU[0] : VRAM Total Memory (B): 206141652992
GPU[0] : VRAM Total Used Memory (B): 297611264
GPU[1] : VRAM Total Memory (B): 206141652992
GPU[1] : VRAM Total Used Memory (B): 297623552
GPU[2] : VRAM Total Memory (B): 206141652992
GPU[2] : VRAM Total Used Memory (B): 297623552
GPU[3] : VRAM Total Memory (B): 206141652992
GPU[3] : VRAM Total Used Memory (B): 297623552
GPU[4] : VRAM Total Memory (B): 206141652992
GPU[4] : VRAM Total Used Memory (B): 297623552
GPU[5] : VRAM Total Memory (B): 206141652992
GPU[5] : VRAM Total Used Memory (B): 297623552
GPU[6] : VRAM Total Memory (B): 206141652992
GPU[6] : VRAM Total Used Memory (B): 297623552
GPU[7] : VRAM Total Memory (B): 206141652992
GPU[7] : VRAM Total Used Memory (B): 297623552
==========================================================================================
================================== End of ROCm SMI Log ===================================
```
## 2. Run examples
### 2.1 Full fine-tuning Qwen3.5 with Megatron-Swift
AMD GPUs often have large VRAM, so you can tune several knobs together to improve training throughput:
- **Parallelism tuning**: Large per-GPU memory lets you reduce communication from aggressive splits (prefer tuning PP/EP before TP).
- **Optimizer CPU offload**: If VRAM allows, disable with `--optimizer_cpu_offload false`.
- **Activation / gradient checkpointing**: If VRAM allows, use `--recompute_granularity none`, or `--recompute_granularity selective` with `--recompute_modules` for finer control.
- **MoE models**: Set `export NVTE_USE_GROUPED_GEMM_TRITON=1` to use grouped GEMM triton kernel.
- **Models with GatedDeltaNet**: Set `USE_MCORE_GDN=1` to use the Megatron-Core implementation.
- **Stability on some AMD GPUs**: Set `export HSA_NO_SCRATCH_RECLAIM=1` to avoid known issues and stabilize performance.
Single-node training:
```bash
export HSA_NO_SCRATCH_RECLAIM=1
export NVTE_USE_GROUPED_GEMM_TRITON=1
output_dir=${PWD}/megatron_output/Qwen3.5-35B-A3B
mkdir -p ${output_dir}
current_time=$(date "+%Y.%m.%d-%H.%M.%S")
log_file=${output_dir}/"1node_full_megatron_Qwen3.5-35B-A3B_${current_time}.log"
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=8 \
MAX_PIXELS=1003520 \
VIDEO_MAX_PIXELS=50176 \
FPS_MAX_FRAMES=12 \
SKIP_MULTIMODAL_MTP_VALIDATION=1 \
USE_MCORE_GDN=1 \
megatron sft \
--model Qwen/Qwen3.5-35B-A3B \
--dataset 'AI-ModelScope/LongAlpaca-12k' \
--save_safetensors true \
--load_from_cache_file true \
--tuner_type full \
--add_non_thinking_prefix true \
--split_dataset_ratio 0.01 \
--tensor_model_parallel_size 1 \
--pipeline_model_parallel_size 1 \
--expert_model_parallel_size 8 \
--sequence_parallel true \
--moe_permute_fusion true \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-6 \
--moe_expert_capacity_factor 2 \
--micro_batch_size 1 \
--global_batch_size 8 \
--recompute_granularity selective \
--recompute_modules core_attn mlp moe \
--gradient_accumulation_fusion false \
--num_train_epochs 500 \
--group_by_length true \
--finetune true \
--freeze_llm false \
--freeze_vit false \
--freeze_aligner false \
--cross_entropy_loss_fusion true \
--lr 1e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-6 \
--weight_decay 0.1 \
--adam_beta2 0.95 \
--eval_steps 500 \
--save_steps 500 \
--save_total_limit 10 \
--logging_steps 1 \
--max_length 16384 \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim true \
--no_save_rng true \
--optimizer_cpu_offload false \
--attention_backend flash \
--padding_free false \
--output_dir ${output_dir} \
2>&1 | tee ${log_file}
```
Multi-node training:
```bash
export NNODES=2 # example: 2 nodes
export NODE_RANK=0 # 0 on master, 1 on workers
export MASTER_ADDR=<MASTER_NODE_IP> # set to master node IP
export MASTER_PORT=29500 # communication port
export NCCL_SOCKET_IFNAME=ens50f1np1 # actual NIC name, check with ifconfig
export GLOO_SOCKET_IFNAME=ens50f1np1 # same as above
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 # IB HCAs, check with ibv_devices
export NCCL_IB_GID_INDEX=3
# Main training script below: same as single-node script above
...
```
### 2.2 Reinforcement learning training for Qwen3.5 with Megatron-Swift
```bash
# Single-node training example
export HSA_NO_SCRATCH_RECLAIM=1
export NVTE_USE_GROUPED_GEMM_TRITON=1
SYSTEM_PROMPT="""You are a helpful math assistant. Solve the problem step by step and put your final answer within \\boxed{}."""
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8 \
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
megatron rlhf \
--rlhf_type grpo \
--model Qwen/Qwen3.5-35B-A3B \
--save_safetensors true \
--enable_thinking false \
--merge_lora true \
--context_parallel_size 1 \
--tensor_model_parallel_size 1 \
--expert_model_parallel_size 8 \
--pipeline_model_parallel_size 1 \
--moe_permute_fusion true \
--dataset open-r1/DAPO-Math-17k-Processed \
--system "$SYSTEM_PROMPT" \
--num_train_epochs 1 \
--global_batch_size 64 \
--micro_batch_size 1 \
--steps_per_generation 2 \
--num_generations 8 \
--reward_funcs accuracy \
--use_vllm true \
--vllm_mode colocate \
--vllm_gpu_memory_utilization 0.5 \
--vllm_tensor_parallel_size 2 \
--vllm_max_model_len 9192 \
--max_length 1000 \
--max_completion_length 8192 \
--tuner_type lora \
--target_modules all-linear \
--lr 5e-5 \
--bf16 true \
--beta 0.00 \
--epsilon 0.2 \
--epsilon_high 0.28 \
--dynamic_sample false \
--overlong_filter true \
--loss_type grpo \
--sleep_level 1 \
--offload_model true \
--offload_bridge false \
--offload_optimizer true \
--logging_steps 1 \
--recompute_granularity none \
--gradient_accumulation_fusion false \
--finetune \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim \
--no_save_rng \
--save_steps 20 \
--attention_backend flash \
--moe_expert_capacity_factor 2 \
--temperature 1.0 \
--padding_free false \
--sequence_parallel true \
--log_completions true \
--report_to tensorboard
```
## Known issues
- **Reinforcement learning**: If vLLM is the inference engine, use vLLM ≥ 0.11.0. It is recommended to use ROCm 7.0 or the image we provide to avoid the sleep mode memory leak issue.
- When using [Ray Megatron](../../source/Instruction/Ray.md) instead of `torchrun` for multi-GPU/Node training, don't set `CUDA_VISIBLE_DEVICES`/`HIP_VISIBLE_DEVICES` etc. to avoid conflicts.
- **MoE training**: Set `NVTE_USE_GROUPED_GEMM_TRITON=1` and `--gradient_accumulation_fusion false` to reduce occasional GPU hangs.
docs/source_en/BestPractices/Elastic.md
+213
View File
@@ -0,0 +1,213 @@
# Elastic
## Installing Dependencies
Deploy a K8S cluster and deploy [DLRover](https://github.com/intelligent-machine-learning/dlrover) in the cluster, and install the required packages using `pip install dlrover && pip install tornado && pip install kubernetes && pip install ms-swift`
Other dependencies and versions verified through repeated testing in the training image:
deepspeed 0.16.5 (refer to this [PR](https://github.com/deepspeedai/DeepSpeed/pull/7585/files) to fix issues related to universal checkpoint)
pytorch 2.6.0
## How to Start
Enable elastic training by adding the `deepspeed_elastic` callback (optionally `graceful_exit`) in `--callbacks`, and configure DeepSpeed elasticity settings.
The command format is dlrover-run + DLrover command parameters + Swift startup command + Swift parameters.dlrover-run behaves like torchrun for most arguments, except for its custom parameters.
The dlrover-run arguments are as follows:
```
usage: dlrover-run [-h] [--nnodes NNODES] [--nproc-per-node NPROC_PER_NODE]
[--rdzv-backend RDZV_BACKEND] [--rdzv-endpoint RDZV_ENDPOINT] [--rdzv-id RDZV_ID]
[--rdzv-conf RDZV_CONF] [--standalone] [--max-restarts MAX_RESTARTS]
[--monitor-interval MONITOR_INTERVAL] [--start-method {spawn,fork,forkserver}]
[--role ROLE] [-m] [--no-python] [--run-path] [--log-dir LOG_DIR] [-r REDIRECTS]
[-t TEE] [--local-ranks-filter LOCAL_RANKS_FILTER] [--node-rank NODE_RANK]
[--master-addr MASTER_ADDR] [--master-port MASTER_PORT] [--local-addr LOCAL_ADDR]
[--logs-specs LOGS_SPECS] [--precheck {0,1,2}] [--node_unit NODE_UNIT]
[--auto_config] [--auto_tunning] [--exclude-straggler] [--save_at_breakpoint]
[--accelerator {nvidia.com/gpu,ascend-npu}] [--training_port TRAINING_PORT]
[--switchbox-check] [--box-pairs PAIR [PAIR ...]] [--min-bandwidth MIN_BANDWIDTH]
[--min-channels MIN_CHANNELS] [--numa-affinity] [--network-check]
[--comm-perf-test] [--ucp_device_type UCP_DEVICE_TYPE]
training_script
```
In elastic training, the parameters you may pay attention to focus on are:
--nnodes NNODES
Number of nodes, or the range of nodes in the form <minimum_nodes>:<maximum_nodes>.
--nproc-per-node NPROC_PER_NODE
Number of processes per node.
Example:
```bash
model=your model path
dataset=your dataset
output= your output dir
export CUDA_VISIBLE_DEVICES=0 # Set according to the actual GPU usage
deepspeed_config_or_type=deepspeed type or configuration file path, e.g., zero1 or /xxx/ms-swift/swift/llm/ds_config/zero1.json
dlrover-run --nnodes 1:$NODE_NUM --nproc_per_node=1 \
/opt/conda/lib/python3.10/site-packages/swift/cli/sft.py --model $model \
--model_type qwen3 \
--tuner_type lora \
--torch_dtype bfloat16 \
--dataset $dataset \
--num_train_epochs 4 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 5e-7 \
--gradient_accumulation_steps 8 \
--eval_steps 500 \
--save_steps 10 \
--save_total_limit 20 \
--logging_steps 1 \
--output_dir $output \
--warmup_ratio 0.01 \
--dataloader_num_workers 4 \
--temperature 1.0 \
--system 'You are a helpful assistant.' \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--dataset_num_proc 1 \
--use_flash_ckpt true \
--callbacks deepspeed_elastic graceful_exit \
--deepspeed $deepspeed_config_or_type \
```
## Configuration
By default, the zero1 configuration is as follows:
```json
{
"fp16": {
"enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"bf16": {
"enabled": "auto"
},
"zero_optimization": {
"stage": 1,
"offload_optimizer": {
"device": "none",
"pin_memory": true
},
"allgather_partitions": true,
"allgather_bucket_size": 2e8,
"overlap_comm": false,
"reduce_scatter": true,
"reduce_bucket_size": 2e8,
"contiguous_gradients": true
},
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"steps_per_print": 2000,
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false,
"elasticity": {
"ignore_non_elastic_batch_info": true,
"enabled": true,
"max_train_batch_size": 8,
"micro_batch_sizes": [
4,
2
],
"min_gpus": 1,
"max_gpus": 4,
"min_time": 20,
"version": 0.1
}
}
```
If users need custom configurations, they can specify the path to the custom zero1.json file in the deepspeed_config_or_type parameter. The elasticity-related configuration is as follows:
```json
...
"elasticity": {
"ignore_non_elastic_batch_info": true,
"enabled": true,
"max_train_batch_size": 8,
"micro_batch_sizes": [
4,
2
],
"min_gpus": 1,
"max_gpus": 4,
"min_time": 20,
"version": 0.1
}
```
- ignore_non_elastic_batch_info：Indicates that the batch size configurations outside the elasticity settings will be ignored. During training, the batch size and related parameters will be dynamically adjusted based on the number of training processes.
Calculation principle：
 global-training-batch-size = micro-batch-size * gradient-accumulation-steps * world-size
- max_train_batch_size： Maximum batch size
- micro_batch_sizes：List of allowed per-GPU micro-batch sizes under elasticity; candidates for train_micro_batch_size_per_gpu.
- min_gpus：Minimum number of GPUs.
- max_gpus：Maximum number of GPUs.
For more details, see: [Deepspeed](https://www.deepspeed.ai/docs/config-json/#elastic-training-config-v01-and-v02)
## Starting Training
```yaml
---
apiVersion: elastic.iml.github.io/v1alpha1
kind: ElasticJob
metadata:
name: deepspeed-elastic-swift
namespace: dlrover
spec:
distributionStrategy: AllreduceStrategy
optimizeMode: single-job
replicaSpecs:
worker:
replicas: 1 # This should match the maximum value of --nnodes NNODES in the startup command
template:
spec:
restartPolicy: Never
containers:
- name: main
image: #【Training image, needs to have deepspeed, dlrover, and swift installed】
imagePullPolicy: IfNotPresent
command:
- /bin/bash
- -c
- sh start.sh # Startup script
resources:
limits:
cpu: '8'
memory: 16Gi
nvidia.com/gpu: '1'
volumeMounts:
- mountPath: /model
name: volume-model
- mountPath: /dev/shm
name: volume-shm
restartPolicy: Never
volumes:
- hostPath:
path: /model
type: Directory
name: volume-model
- emptyDir:
medium: Memory
sizeLimit: 200Gi
name: volume-shm
```
docs/source_en/BestPractices/Embedding.md
+212
View File
@@ -0,0 +1,212 @@
# Embedding Training
SWIFT has already supported the training of embedding models, including both pure text and multimodal types. Currently supported models include:
1. modernbert embedding model
- [ModelScope](https://modelscope.cn/models/iic/gte-modernbert-base) [Hugging Face](https://huggingface.co/Alibaba-NLP/gte-modernbert-base)
2. gte embedding models
- 1.5B: [ModelScope](https://www.modelscope.cn/models/iic/gte_Qwen2-1.5B-instruct) [Hugging Face](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct)
- 7B: [ModelScope](https://www.modelscope.cn/models/iic/gte_Qwen2-7B-instruct) [Hugging Face](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct)
3. gme embedding models
- 2B: [ModelScope](https://www.modelscope.cn/models/iic/gme-Qwen2-VL-2B-Instruct) [Hugging Face](https://huggingface.co/Alibaba-NLP/gme-Qwen2-VL-2B-Instruct)
- 7B: [ModelScope](https://www.modelscope.cn/models/iic/gme-Qwen2-VL-7B-Instruct) [Hugging Face](https://huggingface.co/Alibaba-NLP/gme-Qwen2-VL-7B-Instruct)
4. qwen3-embedding models
- 0.6B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-Embedding-0.6B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)
- 4B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-Embedding-4B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-Embedding-4B)
- 8B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-Embedding-8B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-Embedding-8B)
5. qwen3-vl-embedding models
- 2B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-VL-Embedding-2B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B)
- 8B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-VL-Embedding-8B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B)
Developers can integrate their own models by ensuring the model forward output satisfies:
```text
{"last_hidden_state": some-embedding-tensor}
```
The return value should be a JSON with a `last_hidden_state` key, where the value is an embedding tensor. For the input part, you can use our already supported templates. Users can also specify the
```shell
--task_type embedding
```
parameter to convert any other model into an embedding model for training.
It should be noted that the embedding models currently supported by SWIFT are all based on pure text or multimodal LLMs, and CLIP-type model training is not currently supported.
Additionally, all embedding models supported by SWIFT have normalization added at the end of the model forward pass. If you add new models yourself, please remember to include a normalization layer.
## Loss
The Embedding models supported by SWIFT currently can use the following loss functions:
- **cosine_similarity**: Cosine similarity loss, which calculates the similarity between two embeddings and fits based on the label value. It is effectively an MSE loss.
- **contrastive**: Contrastive learning loss with adjustable margin. Labels are only supported as 0 and 1.
- **online_contrastive**: Contrastive loss considering hard negatives and hard positives. Labels are only supported as 0 and 1.
- **infonce**: Computes pairwise cosine similarities between different rows within the same batch, maximizing similarity within rows and minimizing similarity between different rows. No labels are required.
The source code for the loss functions can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
## Dataset Format
> **Note:**
> 1. The `<image>` tag can appear anywhere inside `messages`/`positive_messages`/`negative_messages`. Each group has its own image fields: `images`/`positive_images`/`negative_images` to provide paths or URLs.
> 2. There is no longer any cross-field ordering requirement. Alignment rules:
> - `images` length equals the number of `<image>` tags in `messages`.
> - `positive_images` and `negative_images` are both list-of-list. Their outer lengths equal the lengths of `positive_messages` and `negative_messages` respectively. For each outer item, the inner list length equals the number of `<image>` tags in that message sequence.
> 3. `messages` is the anchor sample; `positive_messages` and `negative_messages` are each a list of messages (hence one more `[]`). Accordingly, `positive_images`/`negative_images` are also list-of-list and align item-by-item.
> 4. `<video>` and `<audio>` are supported as well. Follow the same rules via `videos`/`positive_videos`/`negative_videos` and `audios`/`positive_audios`/`negative_audios`.
> 5. Current constraint: the outer length of `positive_messages` must be 1 (i.e., provide exactly one positive). Accordingly, the outer length of `positive_images` must also be 1.
### Format for Cosine Similarity Loss
```json lines
# LLM
{"messages": [{"role": "user", "content": "sentence1"}], "positive_messages": [[{"role": "user", "content": "sentence2"}]], "label": 0.8}
# MLLM
{"messages": [{"role": "user", "content": "<image>"}], "images": ["/some/images1.jpg"], "positive_messages": [[{"role": "user", "content": "<image>sentence"}]], "positive_images": [["/some/images2.jpg"]], "label": 0.7}
{"messages": [{"role": "user", "content": "sentence1"}], "positive_messages": [[{"role": "user", "content": "<image>sentence2"}]], "positive_images": [["/some/images.jpg"]], "label": 0.7}
```
The eval metrics are the Pearson and Spearman's Rank Correlation Coefficient of the embeddings' euclidean distance/dot production and so on, totally 8 values.
### Format for Contrastive/Online Contrastive Loss
```json lines
# LLM
{"messages": [{"role": "user", "content": "sentence1"}], "positive_messages": [[{"role": "user", "content": "sentence2"}]], "label": 1}
# MLLM
{"messages": [{"role": "user", "content": "<image>"}], "images": ["/some/images1.jpg"], "positive_messages": [[{"role": "user", "content": "<image>sentence"}]], "positive_images": [["/some/images2.jpg"]], "label": 1}
{"messages": [{"role": "user", "content": "sentence1"}], "positive_messages": [[{"role": "user", "content": "<image>sentence2"}]], "positive_images": [["/some/images.jpg"]], "label": 0}
```
### Format for InfoNCE
```json lines
# LLM
{"messages": [{"role": "user", "content": "sentence1"}], "positive_messages": [[{"role": "user", "content": "sentence2"}]]}
# MLLM
{"messages": [{"role": "user", "content": "<image>"}], "images": ["/some/images.jpg"], "positive_messages": [[{"role": "user", "content": "sentence"}]]}
{"messages": [{"role": "user", "content": "<image>sentence1"}], "images": ["/some/images.jpg"], "positive_messages": [[{"role": "user", "content": "<image>sentence2"}]], "positive_images": [["/some/positive_images.jpg"]], "negative_messages": [[{"role": "user", "content": "<image><image>sentence3"}], [{"role": "user", "content": "<image>sentence4"}]], "negative_images": [["/some/negative_images1.jpg", "/some/negative_images2.jpg"], ["/some/negative_images3.jpg"]]}
```
InfoNCE loss supports the following environment variables:
1. `INFONCE_TEMPERATURE`: The temperature parameter. If not set, the default value is 0.1.
2. `INFONCE_USE_BATCH`: Use `negative_messages` within the sample (hard negatives) or use other samples in the batch as in-batch negatives. The default is `True`, which means using in-batch negatives.
3. `INFONCE_HARD_NEGATIVES`: The number of hard negatives. If not set, all provided `negative_messages` will be used. Since the lengths may vary, a for loop will be used to compute the loss (slower). If set to a specific number, missing items will be randomly sampled, and excess items will be truncated to the first `INFONCE_HARD_NEGATIVES`.
4. `INFONCE_MASK_FAKE_NEGATIVE`: Masks out fake negatives. The default is `False`. When enabled, it checks `positive_similarity + INFONCE_FAKE_NEG_MARGIN`; any sample with similarity larger than this threshold will have its similarity set to `-inf` to prevent positive leakage.
5. `INFONCE_FAKE_NEG_MARGIN`: Margin used by the fake-negative mask. Default: `0.1`.
6. `INFONCE_INCLUDE_QQ`: Include the q–q block (similarities among queries) in the denominator as additional negatives. Default: `False`.
7. `INFONCE_INCLUDE_DD`: Include the d–d block (similarities of the positive doc to all in-batch docs) in the denominator as additional negatives. Default: `False`.
> You can also make the number of hard negatives equal across samples in the dataset, which avoids the for-loop computation and speeds up training even if `INFONCE_HARD_NEGATIVES` is not set.
>
> `negative_messages` can be omitted. In this case, keep `INFONCE_USE_BATCH=True` to use in-batch negatives (other samples in the batch) as negatives.
The evaluation of InfoNCE loss includes the following metrics:
- mean_neg: The average of all hard negatives
- mean_pos: The average of all positives
- margin: The average of (positive - max hard negative)
## Training
Training scripts provided by ms-swift:
- [Qwen3-Embedding/Qwen3-VL-Embedding model](https://github.com/modelscope/ms-swift/blob/main/examples/train/embedding/qwen3)
- [GME model](https://github.com/modelscope/ms-swift/blob/main/examples/train/embedding/train_gme.sh)
## Inference
SWIFT has supported the deployment of GME, GTE, Qwen3-Embedding models, please check [here](https://github.com/modelscope/ms-swift/blob/main/examples/deploy/embedding/client.py).
- For inference scripts, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_embedding.py).
You can also use the original model's code for inference:
https://www.modelscope.cn/models/iic/gte_Qwen2-7B-instruct
https://www.modelscope.cn/models/iic/gme-Qwen2-VL-7B-Instruct
If you've used other models to train embedding from scratch (for example, the original `qwen2-vl` model + `--task_type embedding`), you can also use gme's inference code, but please note:
https://www.modelscope.cn/models/iic/gme-Qwen2-VL-7B-Instruct/file/view/master/gme_inference.py?status=1#L111
Please modify the template here to match the model's own template to ensure the final embeddings align correctly. It's particularly important to note that the template for the gme model is different from the chatml template for the `qwen2-vl` or `qwen2.5-vl` series. In its inference code, the ending character is `<|endoftext|>` rather than `<|im_end|>`.
## Advanced
- Qwen3-Embedding Custom Instruction:
- By default, there is no instruction; the input prompt is: `{Query}<|endoftext|>`.
- You can add an instruction via the system message, changing the prompt to: `{Instruction} {Query}<|endoftext|>`.
- Example:
```json lines
{"messages": [
{"role": "system", "content": "Answer in English and list key points briefly."},
{"role": "user", "content": "Introduce Qwen3-Embedding"}
]}
```
> Note: The Qwen3-Embedding template prepends the system content to the first user message and uses `<|endoftext|>` as the ending token.
### Before/After Examples
- Without Instruction:
Input data (messages):
```json lines
{"messages": [
{"role": "user", "content": "What is Qwen3-Embedding?"}
]}
```
After template conversion (actual prompt sent to the model):
```text
What is Qwen3-Embedding?<|endoftext|>
```
- With Instruction:
Input data (messages with system):
```json lines
{"messages": [
{"role": "system", "content": "Answer in English and list key points briefly."},
{"role": "user", "content": "What is Qwen3-Embedding?"}
]}
```
After template conversion (actual prompt sent to the model):
```text
Answer in English and list key points briefly. What is Qwen3-Embedding?<|endoftext|>
```
- Positive/Negative behave the same:
If a system message is provided within a positive/negative sequence, it is prepended to that sequence’s first user content; if no system is provided, nothing is prepended.
Input (one positive with system, one negative without):
```json lines
{
"messages": [
{"role": "user", "content": "Anchor"}
],
"positive_messages": [[
{"role": "system", "content": "Instruction"},
{"role": "user", "content": "Positive"}
]],
"negative_messages": [[
{"role": "user", "content": "Negative"}
]]
}
```
After template conversion (actual prompts):
```text
Anchor<|endoftext|>
Instruction Positive<|endoftext|>
Negative<|endoftext|>
```
docs/source_en/BestPractices/GRPO-Code-Training.md
+147
View File
@@ -0,0 +1,147 @@
# Code Training with GRPO
This document explains how to use GRPO to train models for code tasks.
Model: [Qwen/Qwen2.5-7B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen2.5-VL-7B-Instruct)
Dataset: [open-r1/verifiable-coding-problems-python-10k](https://www.modelscope.cn/datasets/open-r1/verifiable-coding-problems-python-10k/dataPeview)
dataset example
```json
{
"problem": "Solve the following coding problem using the programming language python: Polycarp has $n$ different binary words. A word called binary if it contains only characters '0' and '1'. For example, these words are binary: \"0001\", \"11\", \"0\" and \"0011100\". Polycarp wants to offer his set of $n$ binary words to play a game \"words\". In this game, players name words and each next word (starting from the second) must start with the last character of the previous word. The first word can be any. For example, these sequence of words can be named during the game: \"0101\", \"1\", \"10\", \"00\", \"00001\". Word reversal is the operation of reversing the order of the characters. For example, the word \"0111\" after the reversal becomes \"1110\", the word \"11010\" after the reversal becomes \"01011\". Probably, Polycarp has such a set of words that there is no way to put them in the order correspondent to the game rules. In this situation, he wants to reverse some words from his set so that: the final set of $n$ words still contains different words (i.e. all words are unique); there is a way to put all words of the final set of words in the order so that the final sequence of $n$ words is consistent with the game rules. Polycarp wants to reverse minimal number of words. Please, help him. -----Input----- The first line of the input contains one integer $t$ ($1 \\le t \\le 10^4$) — the number of test cases in the input. Then $t$ test cases follow. The first line of a test case contains one integer $n$ ($1 \\le n \\le 2\\cdot10^5$) — the number of words in the Polycarp's set. Next $n$ lines contain these words. All of $n$ words aren't empty and contains only characters '0' and '1'. The sum of word lengths doesn't exceed $4\\cdot10^6$. All words are different. Guaranteed, that the sum of $n$ for all test cases in the input doesn't exceed $2\\cdot10^5$. Also, guaranteed that the sum of word lengths for all test cases in the input doesn't exceed $4\\cdot10^6$. -----Output----- Print answer for all of $t$ test cases in the order they appear. If there is no answer for the test case, print -1. Otherwise, the first line of the output should contain $k$ ($0 \\le k \\le n$) — the minimal number of words in the set which should be reversed. The second line of the output should contain $k$ distinct integers — the indexes of the words in the set which should be reversed. Words are numerated from $1$ to $n$ in the order they appear. If $k=0$ you can skip this line (or you can print an empty line). If there are many answers you can print any of them. -----Example----- Input 4 4 0001 1000 0011 0111 3 010 101 0 2 00000 00001 4 01 001 0001 00001 Output 1 3 -1 0 2 1 2 The input will be stdin and you should print your solution to stdout Now solve the problem and return the code.",
"verification_info": {
"language": "python",
"test_cases": [
{
"input": "4\n4\n0001\n1000\n0011\n0111\n3\n010\n101\n0\n2\n00000\n00001\n4\n01\n001\n0001\n00001\n",
"output": "1\n3 \n-1\n0\n\n2\n1 2 \n",
"type": "stdin_stdout"
}
]
}
}
```
`verification_info` provides the programming language as well as test cases, which include input and expected output.
## Reward Functions
The training process utilizes two reward functions: `code_reward` and `code_format`. For implementation details, refer to the [code](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py).
- `code_reward` Executes the generated code using [e2b](https://e2b.dev/) or [judge0](https://judge0.com/). Validates the code against the test cases in the dataset and assigns a reward value based on correctness.
- `code_format` Requires the model to produce formatted responses that include code blocks.
Note: Currently, executing code through E2B only supports the Python language. If you need to execute code in other languages, you can use Judge0([judge0 supported languages](https://github.com/judge0/judge0?tab=readme-ov-file#supported-languages)).
## Training Script
- Register on [e2b](https://e2b.dev/dashboard) to obtain your E2B_API_KEY and set it as an environment variable.
- Add `external_code_reward` as a reward function with `--reward_funcs`.
- Set `--external_plugins` to the path of plugin.py.
launch external vLLM server using following script
```bash
CUDA_VISIBLE_DEVICES=7 \
swift rollout \
--model Qwen/Qwen2.5-7B-Instruct \
--vllm_enable_lora true \
--vllm_max_lora_rank 16
```
```bash
E2B_API_KEY=xxx \
WANDB_API_KEY=xxx \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 \
NPROC_PER_NODE=7 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen2.5-7B-Instruct \
--external_plugins examples/train/grpo/plugin/plugin.py \
--reward_funcs external_code_reward external_code_format \
--reward_weights 1.0 0.1 \
--vllm_mode server \
--use_vllm true \
--vllm_server_host 127.0.0.1 \
--vllm_server_port 8000 \
--tuner_type lora \
--lora_rank 16 \
--lora_alpha 32 \
--torch_dtype bfloat16 \
--dataset 'open-r1/verifiable-coding-problems-python-10k' \
--load_from_cache_file true \
--max_completion_length 2048 \
--num_train_epochs 1 \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 2 \
--learning_rate 1e-6 \
--gradient_accumulation_steps 1 \
--eval_steps 200 \
--save_steps 200 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 4096 \
--output_dir output \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--dataset_num_proc 4 \
--num_generations 14 \
--temperature 0.9 \
--system 'examples/train/grpo/prompt.txt' \
--deepspeed zero2 \
--log_completions true \
--report_to wandb
```
### judge0
- Set environment variables:
- (Required) JUDGE0_ENDPOINT: The endpoint address for accessing Judge0.
- (Optional) JUDGE0_X_AUTH_TOKEN: The access token for Judge0.
- Add `external_code_reward_by_judge0` as a reward function with `--reward_funcs`.
- Set `--external_plugins` to the path of `plugin.py`.
```bash
JUDGE0_ENDPOINT=xxx \
JUDGE0_X_AUTH_TOKEN=xxx \
WANDB_API_KEY=xxx \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 \
NPROC_PER_NODE=7 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen2.5-7B-Instruct \
--external_plugins examples/train/grpo/plugin/plugin.py \
--reward_funcs external_code_reward_by_judge0 external_code_format \
--reward_weights 1.0 0.1 \
--vllm_mode server \
--use_vllm true \
--vllm_server_host 127.0.0.1 \
--vllm_server_port 8000 \
--tuner_type lora \
--torch_dtype bfloat16 \
--dataset 'open-r1/verifiable-coding-problems-python-10k' \
--load_from_cache_file true \
--max_completion_length 2048 \
--num_train_epochs 1 \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 2 \
--learning_rate 1e-6 \
--gradient_accumulation_steps 1 \
--eval_steps 200 \
--save_steps 200 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 4096 \
--output_dir output \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--dataset_num_proc 4 \
--num_generations 14 \
--temperature 0.9 \
--system 'examples/train/grpo/prompt.txt' \
--deepspeed zero2 \
--log_completions true \
--report_to wandb
```
Training Reward Curve
![Training Reward Curve](../../resources/grpo_code.png)
docs/source_en/BestPractices/GRPO-Multi-Modal-Training.md
+360
View File
@@ -0,0 +1,360 @@
# Complete Multimodal GRPO Experiment Workflow
This document explains how to use SWIFT GRPO for training multimodal models and tasks. The goal is to train on multiple multimodal tasks to improve task accuracy. Task definitions, training parameters, etc., refer to [R1-V](https://github.com/Deep-Agent/R1-V.git) and [open-r1-multimodal](https://github.com/EvolvingLMMs-Lab/open-r1-multimodal.git).
---
## **ClevrCount Task**
### **Task and Dataset Definition**
This task is based on the `clevr_cogen_a_train` dataset. The model's goal is to output the number of objects in the image. Therefore, we define the dataset as follows:
```python
class ClevrPreprocessor(ResponsePreprocessor):
def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
query = row.get('query', '')
query = f"""{query} Output the thinking process in <think> </think> and
final answer (number) in <answer> </answer> tags."""
row.update({'query': query})
return super().preprocess(row)
register_dataset(
DatasetMeta(
ms_dataset_id='AI-ModelScope/clevr_cogen_a_train',
subsets=[
SubsetDataset(
name='default',
subset='default',
split=['train'],
),
],
preprocess_func=ClevrPreprocessor(),
tags=['qa', 'math']))
```
The purpose of redefining the dataset preprocessor here is to modify the query. A sample dataset entry is as follows, including `messages`, `images`, and `solution` fields. The `solution` is used in the reward function, while `messages` and `images` serve as model input.
- Note: `{'role': 'assistant', 'content': '<answer> 3 </answer>'}` will be removed in GRPOTrainer and can be ignored. The 'solution' field will be passed directly into the ORM. When creating a custom dataset, the 'images' field should be organized as `["image_path1", "image_path2"]`.
```json
{
"images": ["image_path1", "image_path2"],
"messages": [
{
"role": "user",
"content": "How many items are there in the image? Output the thinking process in <think> </think> and\n final answer (number) in <answer> </answer> tags."
}
],
"solution": "<answer> 3 </answer>"
}
```
---
## **Reward Function Definition**
This task uses two reward functions: one is the format reward function mentioned in `Deepseek-R1`, and the other is the accuracy reward function for ClevrCount. The former is built into SWIFT and can be used directly with `--reward_funcs format`. The latter needs to be custom-defined. Here, we use the `external_plugin` method to define the accuracy reward function by placing the code in `swift/examples/train/grpo/plugin/plugin.py`.
The reward function's input includes `completions` and `solution` fields, representing the model-generated text and ground truth, respectively. Each is a list, allowing the computation of multiple completions simultaneously. Note that the `solution` field is passed through directly from the dataset definition. If there are task changes, corresponding modifications can be made to the dataset and reward function.
```python
class MultiModalAccuracyORM(ORM):
def __call__(self, completions, solution, **kwargs) -> List[float]:
"""
Reward function that checks if the completion is correct.
Args:
completions (list[str]): Generated outputs
solution (list[str]): Ground Truths.
Returns:
list[float]: Reward scores
"""
rewards = []
from math_verify import parse, verify
for content, sol in zip(completions, solution):
reward = 0.0
# Try symbolic verification first
try:
answer = parse(content)
if float(verify(answer, parse(sol))) > 0:
reward = 1.0
except Exception:
pass # Continue to next verification method if this fails
# If symbolic verification failed, try string matching
if reward == 0.0:
try:
# Extract answer from solution if it has think/answer tags
sol_match = re.search(r'<answer>(.*?)</answer>', sol)
ground_truth = sol_match.group(1).strip() if sol_match else sol.strip()
# Extract answer from content if it has think/answer tags
content_match = re.search(r'<answer>(.*?)</answer>', content)
student_answer = content_match.group(1).strip() if content_match else content.strip()
# Compare the extracted answers
if student_answer == ground_truth:
reward = 1.0
except Exception:
pass # Keep reward as 0.0 if both methods fail
rewards.append(reward)
return rewards
orms['external_r1v_acc'] = MultiModalAccuracyORM
```
---
### **GRPO Training Experiment Log**
#### **Training Parameters**
We selected `Qwen2.5-VL-3B-Instruct` as the base model for training. The main reason for choosing the `Instruct` model over the base model is to rapidly achieve format rewards. Experiments were conducted on 8 GPUs. SWIFT GRPO training supports multi-GPU deployment to accelerate rollouts. If you encounter deployment errors for `qwen2.5-vl` on `vllm`, refer to [this issue](https://github.com/vllm-project/vllm/issues/13285).
Since the task is simple, we set `max_completion_length` to 1024 and selected `external_r1v_acc` and `format` as reward functions. The learning rate and beta are set to `1e-6` and `0.001`, respectively. Other configurations are as follows. The settings for `batch_size` and `num_generations` can be referenced from [GRPO Full Workflow](./GRPO.md).
launch external vLLM server using following script
```bash
CUDA_VISIBLE_DEVICES=6,7 \
swift rollout \
--model Qwen/Qwen2.5-VL-3B-Instruct \
--vllm_data_parallel_size 2
```
```shell
WANDB_API_KEY=your_wandb_api_key \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 \
NPROC_PER_NODE=6 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen2.5-VL-3B-Instruct \
--external_plugins examples/train/grpo/plugin/plugin.py \
--reward_funcs external_r1v_acc format \
--use_vllm true \
--vllm_mode server \
--vllm_server_host 127.0.0.1 \
--vllm_server_port 8000 \
--tuner_type full \
--torch_dtype bfloat16 \
--dataset 'AI-ModelScope/clevr_cogen_a_train' \
--load_from_cache_file true \
--max_completion_length 1024 \
--num_train_epochs 1 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--learning_rate 1e-6 \
--gradient_accumulation_steps 2 \
--save_strategy 'steps' \
--eval_strategy 'steps' \
--eval_steps 1000 \
--save_steps 1000 \
--save_total_limit 10 \
--logging_steps 1 \
--output_dir output/GRPO_CLEVR_COUNTDOWN \
--warmup_ratio 0.01 \
--dataloader_num_workers 4 \
--num_generations 24 \
--temperature 1.0 \
--system 'examples/train/grpo/prompt.txt' \
--deepspeed zero3 \
--log_completions true \
--report_to wandb \
--num_iterations 1 \
--async_generate false \
--beta 0.001 \
```
#### **Experimental Observations**
[image.png](../../resources/grpo_clevr_count.png)
- Given the simplicity of the dataset and task, the model converged after 500 epochs. Key observations:
1. The custom `ClevrORM` reward steadily increased, proving the model learned how to complete the task. The task success rate climbed from an initial 0.4 to nearly 1.
2. The `Format Reward` remained stable at 1, likely due to the consistent query format across all dataset samples.
3. The `reward_std` stabilized below 0.1.
4. The `completion length` eventually stabilized between 60-80 tokens, with the model learning a fixed output pattern for item-by-item counting.
---
For additional tasks like Geometric QA and Open R1 Multimodal datasets, refer to their respective sections in the full experiment documentation.
## **Geometric QA Task**
### **Task and Dataset Definition**
This task is a Geometric QA task, where the task description is: given a geometric figure, answer mathematical questions related to the figure. The original data comes from [this paper](https://arxiv.org/pdf/2312.11370), and [R1-V](https://github.com/Deep-Agent/R1-V.git) has preprocessed the data into a `problem-solution` format while retaining the images in the `image` field. Therefore, we do not need to redefine the dataset and can directly use `--dataset AI-ModelScope/GEOQA_R1V_Train_8K`.
---
### **Reward Function**
As this is also a mathematical problem, and the answers are already processed into final results, we directly use the previously defined `MultiModalAccuracyORM` reward function.
---
### **GRPO Training Experiment Log**
#### **Training Parameters**
The selected model and most hyperparameters are similar to the previous experiment, with two main differences:
1. **SWIFT now supports the `--num_iteration` parameter**, allowing multiple updates during a single rollout. We set it to 2.
2. During the experiment, we found that training might become unstable in mathematical problems, causing the model to collapse. This is characterized by a sharp drop in all rewards, a rapid increase in loss, `grad_norm`, and KL divergence, with no subsequent recovery. To prevent this, we set `--max_grad_norm 0.5` to ensure stable training. Note that this instability can have some randomness.
```shell
WANDB_API_KEY=your_wandb_api_key \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 \
MAX_PIXELS=401408 \
NPROC_PER_NODE=6 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen2.5-VL-3B-Instruct \
--external_plugins examples/train/grpo/plugin/plugin.py \
--reward_funcs external_r1v_acc format \
--use_vllm true \
--vllm_mode server \
--vllm_server_host 127.0.0.1 \
--vllm_server_port 8000 \
--tuner_type full \
--torch_dtype bfloat16 \
--dataset 'AI-ModelScope/GEOQA_R1V_Train_8K' \
--load_from_cache_file true \
--max_completion_length 1024 \
--num_train_epochs 1 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--learning_rate 1e-6 \
--gradient_accumulation_steps 2 \
--save_strategy 'steps' \
--eval_strategy 'steps' \
--eval_steps 400 \
--save_steps 400 \
--save_total_limit 10 \
--logging_steps 1 \
--output_dir output/GRPO_GEOQA \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--num_generations 8 \
--temperature 1.0 \
--repetition_penalty 1.1 \
--system 'examples/train/grpo/prompt.txt' \
--deepspeed zero3 \
--log_completions true \
--report_to wandb \
--num_iterations 2 \
--async_generate false \
--beta 0.001 \
--max_grad_norm 0.5 \
```
---
#### **Experimental Observations**
[image.png](../../resources/grpo_geoqa.png)
The training curve is shown above.
1. Compared to the **Count task**, the **Geometric QA task** is significantly more challenging. As a result, the **Accuracy Reward** converges much more slowly and has larger oscillations.
2. The `completion_length` exhibits a clear trend, eventually stabilizing at around 250 tokens.
3. The reward converges, with the standard deviation stabilizing at around 0.2 after a total of 1300 steps.
Below is an example of the model's completion after training:
```plaintext
User:
"If AB is the tangent to circle O with the tangent point A, and BO intersects circle O at point C, and point D is located on circle O, what is the measure of angle ADC if the degree of angle ABO is 32.0?"
Assistant:
"<think>\nTo find the measure of angle ADC, we first need to understand the properties of the tangent and the circle. Since AB is a tangent to the circle at point A, the line segment OA (which is perpendicular to the tangent AB) is perpendicular to AB. This means that angle OAB is 90 degrees.\n\nGiven that angle ABO is 32 degrees, we can find angle BAO by subtracting angle ABO from 90 degrees:\n∠BAO = 90° - ∠ABO = 90° - 32° = 58°\n\nSince angle BAO is an angle in the minor arc AD, the angle ADC, which is an angle in the minor arc AD, is half the angle BAO. Therefore, angle ADC is:\n∠ADC = 1/2 × ∠BAO = 1/2 × 58° = 29°\n\nSo, the measure of angle ADC is 29 degrees.\n</think>\n<answer>\nThe measure of angle ADC is 29 degrees.\n</answer>"
```
---
## **Multimodal Open R1 Dataset Experiment**
### **Task and Dataset Definition**
This task is based on the experiments in [open-r1-multimodal](https://github.com/EvolvingLMMs-Lab/open-r1-multimodal.git), using the dataset: [lmms-lab/multimodal-open-r1-8k-verified](https://www.modelscope.cn/datasets/lmms-lab/multimodal-open-r1-8k-verified). This dataset focuses on multimodal mathematical reasoning tasks, with data generated by GPT4o based on the `Math360K` and `Geo170K` datasets. It includes reasoning paths and verifiable answers. The dataset already contains `image`, `problem`, and `solution` fields, so no additional prompt modifications are required, and there is no need to redefine the dataset.
---
### **Reward Function**
We directly use the previously defined `MultiModalAccuracyORM` reward function.
---
### **GRPO Training Experiment Log**
#### **Training Parameters**
The selected model and most hyperparameters are similar to the previous experiment. Due to an **OOM (Out of Memory) issue**, we set `MAX_PIXELS=262144` to reduce memory usage.
```shell
WANDB_API_KEY=your_wandb_api_key \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 \
MAX_PIXELS=262144 \
MASTER_PORT=29600 \
NPROC_PER_NODE=6 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen2.5-VL-3B-Instruct \
--external_plugins examples/train/grpo/plugin/plugin.py \
--reward_funcs external_r1v_acc format \
--use_vllm true \
--vllm_mode server \
--vllm_server_host 127.0.0.1 \
--vllm_server_port 8000 \
--tuner_type full \
--torch_dtype bfloat16 \
--dataset 'lmms-lab/multimodal-open-r1-8k-verified' \
--load_from_cache_file true \
--max_completion_length 1024 \
--num_train_epochs 1 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--learning_rate 1e-6 \
--gradient_accumulation_steps 2 \
--save_strategy 'steps' \
--eval_strategy 'steps' \
--eval_steps 400 \
--save_steps 400 \
--save_total_limit 10 \
--logging_steps 1 \
--output_dir output/GRPO_GEOQA \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--num_generations 8 \
--temperature 1.0 \
--repetition_penalty 1.1 \
--system 'examples/train/grpo/prompt.txt' \
--deepspeed zero3 \
--log_completions true \
--report_to wandb \
--num_iterations 2 \
--async_generate false \
--beta 0.001 \
--max_grad_norm 0.5 \
```
---
#### **Experimental Observations**
[image.png](../../resources/grpo_openr1_multimodal.png)
The training curve is shown above.
- With a total of 8k data samples, the training reached **1268 steps**, with the **Accuracy Reward** converging around 0.5.
- The `completion_length` stabilized around 200 tokens.
- The `reward_std` eventually oscillated around 0.2.
Below is an example of the model's completion after training:
```plaintext
User:
"Based on the map, which state falls into the lowest percentage range of lots, and what is that percentage range?"
Assistant:
"<think>\nThe image provided shows a color-coded map of the USA indicating different ranges of lot percentages across various states. According to the legend at the bottom, the lowest percentage range (6.0% - 6.5%) is represented by white. In the image, Alabama (AL) is shaded in white, which corresponds to the 6.0% - 6.5% category. Therefore, based on the map, the state that falls into the lowest percentage range of lots is Alabama, with the percentage range of 6.0% - 6.5%.\nTherefore, the answer is 6.0% - 6.5%.\n</think>\n<answer>Alabama</answer>"
```
docs/source_en/BestPractices/GRPO.md
+302
View File
@@ -0,0 +1,302 @@
# Complete GRPO Experiment Process
This article starts with the relatively simple mathematical task "Countdown Game" and introduces the complete GRPO training process through several steps: dataset definition, reward function definition, and GRPO training. The task definition and training parameters are based on [mini-deepseek-r1](https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/mini-deepseek-r1-aha-grpo.ipynb).
## Task and Dataset Definition
The goal of the Countdown Game task is to reach a target number using the given numbers and the four basic arithmetic operations. Therefore, we define the dataset as follows:
```python
class CoundownTaskPreprocessor(ResponsePreprocessor):
def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
numbers = row['nums']
target = row.pop('response', None)
query = f"""
Using the numbers {numbers}, create an equation that equals {target}.
You can use basic arithmetic operations (+, -, *, /) and each number can only be used once.
Show your work in <think> </think> tags. And return the final equation and answer in <answer> </answer> tags,
for example <answer> (1 + 2) / 3 * 4 = 4 </answer>.
"""
row.update({'target': target, 'query': query})
return super().preprocess(row)
register_dataset(
DatasetMeta(
ms_dataset_id='zouxuhong/Countdown-Tasks-3to4',
subsets=['default'],
preprocess_func=CoundownTaskPreprocessor(),
tags=['math']))
```
Through a template, numbers and the target are used to define the task, and a `query` field is provided for model sampling. At the same time, we need to retain the `nums` and `target` fields for subsequent reward function calculation.
## Reward Function Definition
Two reward functions are used for this task: one is the format reward function mentioned in Deepseek-R1, and the other is the accuracy reward function for the Countdown Game. The former is already built into Swift and can be used directly with `--reward_funcs format`, while the latter requires custom definition. Here, we use the `external_plugin` method to define the accuracy reward function, placing the code in `swift/examples/train/grpo/plugin/plugin.py`.
The input to the reward function includes three fields: `completions`, `target`, and `nums`, representing the model-generated text, the target answer, and the available numbers, respectively. Each is a list, supporting simultaneous computation of multiple completions. Note that, except for `completions`, the other parameters are transparently passed from the fields defined in the dataset. If there are changes to the task, adjustments can be made to both the dataset and the reward function as needed.
```python
class CountdownORM(ORM):
def __call__(self, completions, target, nums, **kwargs) -> List[float]:
"""
Evaluates completions based on Mathematical correctness of the answer
Args:
completions (list[str]): Generated outputs
target (list[str]): Expected answers
nums (list[str]): Available numbers
Returns:
list[float]: Reward scores
"""
rewards = []
for completion, gt, numbers in zip(completions, target, nums):
try:
# Check if the format is correct
match = re.search(r"<answer>(.*?)<\/answer>", completion)
if match is None:
rewards.append(0.0)
continue
# Extract the "answer" part from the completion
equation = match.group(1).strip()
if '=' in equation:
equation = equation.split('=')[0]
# Extract all numbers from the equation
used_numbers = [int(n) for n in re.findall(r'\d+', equation)]
# Check if all numbers are used exactly once
if sorted(used_numbers) != sorted(numbers):
rewards.append(0.0)
continue
# Define a regex pattern that only allows numbers, operators, parentheses, and whitespace
allowed_pattern = r'^[\d+\-*/().\s]+$'
if not re.match(allowed_pattern, equation):
rewards.append(0.0)
continue
# Evaluate the equation with restricted globals and locals
result = eval(equation, {'__builtins__': None}, {})
# Check if the equation is correct and matches the ground truth
if abs(float(result) - float(gt)) < 1e-5:
rewards.append(1.0)
else:
rewards.append(0.0)
except Exception as e:
# If evaluation fails, reward is 0
rewards.append(0.0)
return rewards
orms['external_countdown'] = CountdownORM
```
## GRPO Training Experiment Record
We first present the GRPO formula:
$$
{\scriptstyle
\begin{aligned}
\mathcal{J}_{G R P O}(\theta) & =\mathbb{E}\left[q \sim P(Q),\left\{o_i\right\}_{i=1}^G \sim \pi_{\theta_{o l d}}(O \mid q)\right] \\
& \frac{1}{G} \sum_{i=1}^G \frac{1}{\left|o_i\right|} \sum_{t=1}^{\left|o_i\right|}\left\{\min \left[\frac{\pi_\theta\left(o_{i, t} \mid q, o_{i,<t}\right)}{\pi_{\theta_{o l d}}\left(o_{i, t} \mid q, o_{i,<t}\right)} \hat{A}_{i, t}, \operatorname{clip}\left(\frac{\pi_\theta\left(o_{i, t} \mid q, o_{i,<t}\right)}{\pi_{\theta_{o l d}}\left(o_{i, t} \mid q, o_{i,<t}\right)}, 1-\varepsilon, 1+\varepsilon\right) \hat{A}_{i, t}\right]-\beta \mathbb{D}_{K L}\left[\pi_\theta| | \pi_{r e f}\right]\right\}
\end{aligned}
}
$$
### Training Parameters
We selected Qwen2.5-3B-Instruct as the base model for training, as using an instruct-tuned model allows for faster acquisition of format rewards. The experiment was conducted on three GPUs, with vLLM inference deployed on the last GPU and two processes set on the remaining GPUs for gradient updates.
Since the task is relatively simple, we set both `max_completion_length` to 1024. For more complex tasks, the model output length can be increased appropriately, but note that **the larger these parameters, the more GPU memory is required, and the slower the training speed**. The training time per step is linearly related to `max_completion_length`.
In our experiment, the total batch size is:
```
num_processes * per_device_train_batch_size * gradient_accumulation_steps = 2 * 8 * 8 = 128
```
Note that the single-GPU batch size is also closely related to GPU memory capacity, so set an appropriate value based on memory limits. Additionally, the total number of steps can be calculated as:
$$
\text{num\_steps} = \text{epochs} \times \text{len(datasets)} \times \text{num\_generations} \div \text{batch\_size}
$$
This formula should guide the planning of learning rate and warmup settings.
Finally, two important parameters are learning rate and $\beta$. The learning rate is straightforward, while $\beta$ is the weight of the KL divergence gradient in the formula. Increasing these parameters accelerates convergence but may lead to instability. After experimentation, we set them to `5e-7` and `0.001`, respectively. During training, adjust these parameters appropriately if instability or oscillations occur.
For KL divergence, the community has extensive discussions, such as [Why GRPO Adheres to KL Divergence](https://zhuanlan.zhihu.com/p/25862547100).
Other parameter settings were not explored in detail and will not be discussed here.
```bash
CUDA_VISIBLE_DEVICES=2 \
swift rollout \
--model Qwen/Qwen2.5-3B-Instruct
```
```bash
CUDA_VISIBLE_DEVICES=0,1 \
WANDB_API_KEY=your_wandb_key \
NPROC_PER_NODE=2 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen2.5-3B-Instruct \
--external_plugins examples/train/grpo/plugin/plugin.py \
--reward_funcs external_countdown format \
--use_vllm true \
--vllm_mode server \
--vllm_server_host 127.0.0.1 \
--vllm_server_port 8000 \
--tuner_type full \
--torch_dtype bfloat16 \
--dataset 'zouxuhong/Countdown-Tasks-3to4#50000' \
--load_from_cache_file true \
--max_length 2048 \
--max_completion_length 1024 \
--num_train_epochs 1 \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--learning_rate 5e-7 \
--gradient_accumulation_steps 8 \
--eval_steps 500 \
--save_steps 100 \
--save_total_limit 20 \
--logging_steps 1 \
--output_dir output/GRPO_COUNTDOWN \
--warmup_ratio 0.01 \
--dataloader_num_workers 4 \
--num_generations 8 \
--temperature 1.0 \
--system 'You are a helpful assistant. You first thinks about the reasoning process in the mind and then provides the user with the answer.' \
--deepspeed zero3 \
--log_completions true \
--report_to wandb \
--beta 0.001 \
--num_iterations 1
```
### Experimental Observations
After training for 2000 steps, the following observations were made:
1. **Custom CountdownORM Rewards**: The custom `CountdownORM` reward is steadily increasing, which proves that the model is learning to accomplish the task. The final task success rate is around 50%.
2. **Format Reward**: The format reward remains at a relatively high level, fluctuating between 0.9 and 1.
3. **Reward Standard Deviation**: The reward standard deviation stabilizes around 0.2-0.3 but has not fully converged, indicating that the model still has room to improve its reward.
4. **Failure Cases**: By examining the model's output, many failure cases are due to the model making simple arithmetic errors. Therefore, the model's final convergence direction involves solving the problem step by step. Using a larger model may lead to different convergence directions.
5. **Completion Length**: The completion length initially decreases and then increases, reaching its lowest point around 200 steps and later rising to approximately 350. This suggests the model underwent a change in reasoning approach.
6. **Reasoning Patterns**: The model's reasoning pattern evolves over time. Eventually, the model realizes that performing permutations and step-by-step calculations is more effective for completing this task. The final sampled outputs mostly follow this format. Below are examples of how completions change across different steps:
#### Completion Examples at Different Steps
**Step 1:**
```plaintext
<think>
We need to create an equation using the given numbers [70, 5, 42, 30] exactly once and make it equal to 77.
First, let's consider the possibility of multiplication or division to get closer to 77. Notice that 77 is close to 70 and 42. If we can use these numbers together, it might work.
Multiplying 70 by 1.1 would give us approximately 77, which is very close. So, let's try to make 1.1 using the remaining numbers 5 and 30.
If we take 30/5 = 6, and then 5/6 = 0.8333..., this isn't useful here, so let's look at other possibilities.
Next, let's consider if there's another way to achieve 77 using addition, subtraction, or another combination of our numbers. Given the numbers provided, it seems that multiplication or division will be key.
We can try subtracting 5 from 30 to get 25. Then, if we divide 42 by 25, we don't get a whole number, but we can round it to the nearest whole number, 1. Now, we need to make the equation fit 77. Let's multiply 25 by 3 and add 7 (since 77 = 25*3 + 7). However, we only have one 7 available and we need to find a way to use the remaining numbers to get 7.
Let's try:
70 - 42 = 28
Then, 28 + 5 = 33
Next, 33 + 30 = 63
Finally, 77 - 63 = 14, which doesn't help.
Considering different combinations, one viable way is:
(70 - 42) * (30 / 5) = 28 * 6 = 168 / 2 = 84 / 2 = 42 / 2 = 21 / 3 = 7
So, (70 - 42) * (30 / 5) = 77.
</think>
<answer>(70 - 42) * (30 / 5) = 77</answer>
```
**Step 200:**
```plaintext
<think>
We need to use all the given numbers in a way that the resultant value of the calculation is 99. Given these restrictions, it's clear we should look into how the numbers relate and the mathematical operations that will bring them together to meet our condition.
Firstly, let's evaluate the sums, differences, and multiplication/division among the numbers and their relations to see how this could lead us towards an outcome of 99.
- Summing the three numbers: 96 + 70 + 73 = 239, which is far from 99. This shows addition isn't practical.
- Subtracting numbers: For instance, subtracting the lowest two (i.e., 73 - 70 = 3) and combining with the highest one (96) does not work. Also, there is no other combination of subtraction that would result in the number needing to add up to 99.
- Multiplying or dividing the numbers alone wouldn’t reach any near solution since none of the products or division results of the numbers themselves is very close to 99. We might think more carefully about the arrangement of them using a combination of the operations provided.
Therefore, considering the possible arrangement, a combination like taking (73 - 70 + 96) may produce an effective solution as this will sum the values in such a manner that it aligns with the goal of 99.
So we conclude that the operation: 73 - 70 + 96 gives the right sequence to equal 99.
</think>
<answer>73 - 70 + 96 = 99</answer>
```
**Step 500:**
```plaintext
<think>
Let's begin by exploring various possible combinations of the given numbers using the arithmetic operators we have access to.
- If I add the numbers, I get: 56 + 73 + 51 = 180 which is not equal to our target number (68)
- If I subtract one number from another I will still exceed our target value as well since the maximum difference is: |(difference between max-min):| | (73 - 51) or | (73 - 56), etc.|
- Let's multiply the numbers individually and see if the division operation can bring us closer to the goal:
- (73 * 51) - 56 = 3723 - 56 = 3667 which isn't the goal either.
- The only operator left is addition and subtraction; I'll try the combination:
((73 - 56) + 51) gives me (17 + 51) = 68.
Thus, our solution fits the provided problem constraints and meets the required outcome.
</think>
<answer>(73 - 56) + 51 = 68</answer>
```
**Step 1000:**
```plaintext
<think>
We need to form an expression using the given numbers (50, 86, and 49) such that it equals 85. Let's try different combinations:
(86 - 50) + 49 = 76 + 49 = 125 (too big)
(86 - 49) + 50 = 37 + 50 = 87 (too big)
(86 - 50) + 49 = 37 + 49 = 86 (too big)
(86 + 50) - 49 = 136 - 49 = 87 (too big)
(86 - 50) + (49 - x)? Not working!
(50 + 86) - 49: this is 136 which isn't our aim.
50 + (86 - 49) = 50 + 37 = 87 which wasn't correct.
50 - 86 + 49 = -16 + 49: This doesn't match up as well.
86 - (50 + 49) = 86 - 99 which doesn't work too.
We try one more:
(86 + 49) - 50 = 135 - 50 = 85!
That's it!
</think>
<answer>(86 + 49) - 50 = 85</answer>
```
**Step 2000:**
```plaintext
<think>
We need to form an expression using the numbers 58, 9, 78, and 44 which evaluates to 73. Let's try different combinations:
(78 - 58) + (9 + 44) = 10 + 53 = 63 (too low)
(78 - 58) + (9 - 44) = 20 - 35 = -15 (too low)
(78 - 58) + (44 - 9) = 20 + 35 = 55 (too low)
(78 + 58) - (9 + 44) = 136 - 53 = 83 (too high)
(78 + 58) - (9 - 44) = 136 + 35 = 171 (too high)
(78 + 58) - (44 + 9) = 136 + 53 = 189 (too high)
(78 + 9) - (58 + 44) = 87 - 102 = -15 (too low)
(78 + 9) - (58 - 44) = 87 - 14 = 73
So our solution is: (78 + 9) - (58 - 44) = 73
</think>
<answer>(78 + 9) - (58 - 44) = 73</answer>
```
---
### Unstable Experiment Records
For learning rate set to `1e-6` and $\beta$ set to `0.04`, the model showed oscillations around step 200. Both the format reward and CountdownORM reward dropped significantly:
[Unstable Experiment Graph](../../resources/grpo_countdown_1.png)
docs/source_en/BestPractices/MLLM-Registration.md
+667
View File
@@ -0,0 +1,667 @@
# Best Practices for Registering Multimodal Models
This document introduces how to register a multimodal model in ms-swift and successfully perform inference and training. Using Qwen2.5-Omni as an example, we will register a new model_type and template `my_qwen2_5_omni`, supporting training with text, images, videos, and audio. Since Qwen2.5-Omni is already registered in ms-swift, we can use our custom components by explicitly specifying the model_type and template.
## Environment Setup
```shell
# Avoid future incompatibilities with documentation
pip install "ms-swift>=4.0"
pip install "transformers==4.57.*" "qwen_omni_utils==0.0.8"
```
## Model Registration
First, we need to register the model to obtain the model and processor.
```python
from transformers import AutoConfig, PretrainedConfig, PreTrainedModel
from swift.model import (Model, ModelGroup, ModelMeta, MultiModelKeys, get_model_processor, register_model,
register_model_arch, ModelLoader)
from swift.model.models.qwen import patch_qwen_vl_utils
from swift.model.patcher import patch_get_input_embeddings
from swift.model.utils import use_submodel_func
from swift.utils import get_env_args, Processor
register_model_arch(
MultiModelKeys(
'my_qwen2_5_omni',
# `freeze_llm`, `freeze_vit`, `freeze_aligner` behavior is determined by the values below.
# For example: full parameter training, if `freeze_vit=True`, it will freeze parameters of model layers prefixed with `thinker.audio_tower` and `thinker.visual`.
# LoRA training, if `freeze_vit=False`, it will additionally add LoRA to Linear layers prefixed with `thinker.audio_tower` and `thinker.visual`.
language_model=['thinker.model', 'thinker.lm_head'],
vision_tower=['thinker.audio_tower', 'thinker.visual'],
aligner=['thinker.audio_tower.proj', 'thinker.visual.merger'],
# Generator parts will never be trained or remain frozen.
# If you want `thinker.audio_tower` and `thinker.audio_tower.proj` to never be trained, you can place them in the generator and remove them from vision_tower and aligner.
generator=['talker', 'token2wav'],
))
class Qwen2_5OmniLoader(ModelLoader):
def get_config(self, model_dir: str) -> PretrainedConfig:
config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
enable_audio_output = get_env_args('ENABLE_AUDIO_OUTPUT', bool, None)
if enable_audio_output is not None:
config.enable_audio_output = enable_audio_output
return config
def get_processor(self, model_dir: str, config: PretrainedConfig) -> Processor:
from transformers import Qwen2_5OmniProcessor
from qwen_omni_utils import vision_process
processor = Qwen2_5OmniProcessor.from_pretrained(model_dir, trust_remote_code=True)
# Control constants in qwen_omni_utils library via environment variables,
# e.g., `MAX_PIXELS`, etc.
patch_qwen_vl_utils(vision_process)
return processor
def get_model(self, model_dir: str, config: PretrainedConfig, processor: Processor,
model_kwargs) -> PreTrainedModel:
from transformers import Qwen2_5OmniForConditionalGeneration
print('Run my_qwen2_5_omni...')
self.auto_model_cls = self.auto_model_cls or Qwen2_5OmniForConditionalGeneration
model = super().get_model(model_dir, config, processor, model_kwargs)
# For multimodal model consistency, we replace the model's forward/generate functions
# with those of its language_model.
# Handle additional parts separately.
use_submodel_func(model, 'thinker')
# Avoid inplace operations on leaf_variable during training
# (replacing parts of input_embeds with images_embeds)
patch_get_input_embeddings(model.thinker.visual, 'patch_embed')
# Some custom settings for model/config (usually not needed; configure based on
# specific model if errors occur during training/inference)
model.config.keys_to_ignore_at_inference += ['hidden_states', 'attention_mask']
model.config.talker_config.pad_token_id = None
return model
register_model(
ModelMeta(
'my_qwen2_5_omni',
[
ModelGroup([
Model('Qwen/Qwen2.5-Omni-3B', 'Qwen/Qwen2.5-Omni-3B'),
Model('Qwen/Qwen2.5-Omni-7B', 'Qwen/Qwen2.5-Omni-7B'),
]),
],
# Function to get model and processor.
Qwen2_5OmniLoader,
template='my_qwen2_5_omni',
is_multimodal=True, # Whether it's a multimodal model
model_arch='my_qwen2_5_omni', # Usually set only for multimodal models
# Used for automatic model_type matching
architectures=['Qwen2_5OmniModel', 'Qwen2_5OmniForConditionalGeneration'],
# Used to prompt users about dependency versions (can be removed)
requires=['transformers>=4.50', 'soundfile', 'qwen_omni_utils', 'decord'],
# Used to prompt users (can be removed)
tags=['vision', 'video', 'audio'],
# Additional files to save during full parameter training/merge-lora
additional_saved_files=['spk_dict.pt'],
))
if __name__ == '__main__':
# Test and debug
model, processor = get_model_processor('Qwen/Qwen2.5-Omni-7B', model_type='my_qwen2_5_omni')
```
## Template Registration
Second, we need to register a template to customize how text, images, videos, and audio are preprocessed (`_encode` and `_data_collator` methods). This is a key module for ms-swift's support of multimodal model training. Preprocessing methods should reference transformers inference implementation and align with it.
Template functions:
1. Support normal inference and training, preprocess text and multimodal information, and support grounding tasks.
2. Support padding_free and packing training.
3. Support mixed modality data training.
```python
from functools import partial
from typing import Any, Dict, List, Literal, Optional
import torch
from transformers.integrations import is_deepspeed_zero3_enabled
from swift import get_model_processor
from swift.template import StdTemplateInputs, Template, TemplateMeta, get_template, register_template
from swift.template.utils import Context, findall
from swift.template.vision_utils import load_audio
from swift.utils import Processor, get_env_args, get_logger, get_packed_seq_params, is_deepspeed_enabled, to_float_dtype
logger = get_logger()
class Qwen2_5OmniTemplate(Template):
use_model = True # Whether model participation is required during preprocessing
# Note: Not all multimodal models support padding_free/packing. Models in `transformers` library usually support it
support_padding_free = True # Whether padding_free and packing are supported (multimodal models)
norm_bbox = 'none' # Whether grounding tasks use absolute or norm1000 coordinates
# These tokens will not be truncated (e.g., when setting `--truncation_strategy left/right`)
# and will be printed in abbreviated form (calling `template.safe_decode`)
placeholder_tokens = ['<|IMAGE|>', '<|AUDIO|>', '<|VIDEO|>']
def init_processor(self, processor: Processor) -> None:
"""Initialize some required constants when initializing the processor"""
if processor is None:
return
super().init_processor(processor)
from transformers.models.qwen2_5_omni.processing_qwen2_5_omni import Qwen2_5OmniProcessorKwargs
default = Qwen2_5OmniProcessorKwargs._defaults
self.seconds_per_chunk = default['videos_kwargs']['seconds_per_chunk']
self.position_id_per_seconds = default['videos_kwargs']['position_id_per_seconds']
self.use_audio_in_video = get_env_args('use_audio_in_video', bool, False)
self.sampling_rate = get_env_args('sampling_rate', int, self.processor.feature_extractor.sampling_rate)
# See grounding dataset customization documentation for `QWENVL_BBOX_FORMAT` meaning
self.bbox_format = get_env_args('QWENVL_BBOX_FORMAT', str, 'legacy')
def replace_tag(self, media_type: Literal['image', 'video', 'audio'], index: int,
inputs: StdTemplateInputs) -> List[Context]:
"""Load multimodal data and replace generic multimodal tags.
For example: image tag from `<image>` -> `<|vision_bos|><|IMAGE|><|vision_eos|>`"""
# Loading multimodal data can also be done in the `_encode` function, whichever is more convenient.
from qwen_omni_utils import fetch_image, fetch_video
if media_type == 'image':
inputs.images[index] = fetch_image({'image': inputs.images[index]})
return ['<|vision_bos|><|IMAGE|><|vision_eos|>']
elif media_type == 'audio':
if self.mode != 'vllm': # No processing needed in 'vllm' inference scenario
inputs.audios[index] = load_audio(inputs.audios[index], self.sampling_rate)
return ['<|audio_bos|><|AUDIO|><|audio_eos|>']
elif media_type == 'video':
video = inputs.videos[index]
_video = fetch_video({'video': video})
if isinstance(_video, torch.Tensor):
_video = _video.to(torch.uint8)
inputs.videos[index] = _video
if self.use_audio_in_video:
import librosa
if video.startswith('http://') or video.startswith('https://'):
import audioread
video = audioread.ffdec.FFmpegAudioFile(video)
video = librosa.load(video, sr=self.sampling_rate)[0]
inputs.audios.insert(inputs.audio_idx, (video, 'video'))
inputs.audio_idx += 1
return ['<|vision_bos|><|audio_bos|><|VIDEO|><|audio_eos|><|vision_eos|>']
else:
return ['<|vision_bos|><|VIDEO|><|vision_eos|>']
def replace_ref(self, ref: str, index: int, inputs: StdTemplateInputs) -> List[Context]:
"""Replace generic tag for grounding tasks: `<ref-object>`"""
if self.bbox_format == 'legacy':
return [f'<|object_ref_start|>{ref}<|object_ref_end|>']
else:
return [ref]
def replace_bbox(self, bbox: List[int], index: int, inputs: StdTemplateInputs) -> List[Context]:
"""Replace generic tag for grounding tasks: `<bbox>`"""
if self.bbox_format == 'legacy':
return [f'<|box_start|>{self._get_bbox_str(bbox)}<|box_end|>']
else:
return [str(bbox)]
def packing_row(self, row: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Support packing & mrope.
Usually no need to inherit this function; here for customizing mrope's position_ids."""
position_ids = []
for r in row:
r = r.copy()
r['input_ids'] = torch.tensor(r['input_ids'])[None]
position_ids.append(self._get_position_ids(r))
packed = super().packing_row(row)
packed['position_ids'] = torch.concat(position_ids, dim=-1)
return packed
def _get_new_tokens_use_audio_in_video(self, i, *, video_grid_thw, video_second_per_grid, audio_lengths,
video_token_id, audio_token_id):
"""Helper function to support `use_audio_in_video` being True"""
merge_size = self.processor.image_processor.merge_size
grid_thw = video_grid_thw[i]
height = grid_thw[1] // merge_size
width = grid_thw[2] // merge_size
audio_token_indices = torch.arange(audio_lengths[i])
video_token_indices = torch.arange(grid_thw[0]).reshape(-1, 1, 1)
video_token_indices = torch.broadcast_to(video_token_indices,
(video_token_indices.shape[0], height, width)).reshape(-1)
video_token_indices = (video_token_indices * video_second_per_grid[i] * self.position_id_per_seconds)
tokens_per_chunk = int(self.position_id_per_seconds * self.seconds_per_chunk)
video_chunk_indexes = self.processor.get_chunked_index(video_token_indices, tokens_per_chunk)
audio_chunk_indexes = self.processor.get_chunked_index(audio_token_indices, tokens_per_chunk)
res = []
for j in range(max(len(video_chunk_indexes), len(audio_chunk_indexes))):
if j < len(video_chunk_indexes):
video_seq_length = video_chunk_indexes[j][1] - video_chunk_indexes[j][0]
res += video_token_id * video_seq_length
if j < len(audio_chunk_indexes):
audio_seq_length = audio_chunk_indexes[j][1] - audio_chunk_indexes[j][0]
res += audio_token_id * audio_seq_length
return res
def _encode(self, inputs: StdTemplateInputs) -> Dict[str, Any]:
"""This determines how to convert text/images/audios/videos -> input_ids, labels, loss_scale, and multimodal content like pixel_values
Processing logic can usually be borrowed from the corresponding model's preprocessing code implementation.
Recommended: Perform inference alignment first, then training"""
encoded = Template._encode(self, inputs) # Process text-only part; see custom model documentation for details
logger.info_once('Run qwen2_5_omni template')
processor = self.processor
# Get multimodal content
media_inputs = processor(
text='',
audio=inputs.audios or None,
images=inputs.images or None,
videos=inputs.videos or None,
do_resize=False,
return_tensors='pt')
# We don't use input_ids and attention_mask produced by `processor` because it doesn't produce `labels`.
media_inputs.pop('input_ids')
media_inputs.pop('attention_mask')
media_inputs = to_float_dtype(media_inputs, self.model_info.torch_dtype)
input_ids = encoded['input_ids']
labels = encoded['labels']
loss_scale = encoded.get('loss_scale', None)
# audio modality
audio_token_id = self._tokenize('<|AUDIO|>')
idx_list = findall(input_ids, audio_token_id) # Find all audio_tokens
feature_attention_mask = media_inputs.get('feature_attention_mask')
if feature_attention_mask is not None:
audio_feature_lengths = torch.sum(feature_attention_mask, dim=1)
audio_lengths = ((audio_feature_lengths - 1) // 2 + 1 - 2) // 2 + 1
else:
audio_lengths = None
audio_lengths_origin = audio_lengths
# video_audios_mask is used to handle `use_audio_in_video`, distinguishing pure audio(0) from audio in video(1)
video_audios_mask = []
for i, audio in enumerate(inputs.audios):
if isinstance(audio, tuple) and audio[1] == 'video':
inputs.audios[i] = audio[0]
video_audios_mask.append(True)
else:
video_audios_mask.append(False)
video_audios_mask = torch.tensor(video_audios_mask)
if idx_list:
# Filter out audio content in videos (will be handled in video section)
if self.use_audio_in_video:
audio_lengths = audio_lengths[~video_audios_mask]
def _get_new_audio_tokens(i):
return audio_token_id * audio_lengths[i]
# Expand multimodal tokens in input_ids
input_ids, labels, loss_scale = self._extend_tokens(input_ids, labels, loss_scale, idx_list,
_get_new_audio_tokens)
# image and video modalities
for media_type in ['image', 'video']:
token = f'<|{media_type.upper()}|>'
token_id = self._tokenize(token)
idx_list = findall(input_ids, token_id)
if idx_list:
merge_size = processor.image_processor.merge_size
media_grid_thw = media_inputs.get(f'{media_type}_grid_thw')
if media_type == 'video' and self.use_audio_in_video:
audio_lengths = audio_lengths_origin[video_audios_mask]
video_second_per_grid = media_inputs['video_second_per_grid']
_get_new_tokens_use_audio_in_video = partial(
self._get_new_tokens_use_audio_in_video,
video_grid_thw=media_grid_thw,
video_second_per_grid=video_second_per_grid,
audio_lengths=audio_lengths,
video_token_id=token_id,
audio_token_id=audio_token_id)
input_ids, labels, loss_scale = self._extend_tokens(input_ids, labels, loss_scale, idx_list,
_get_new_tokens_use_audio_in_video)
else:
def _get_new_tokens(i):
token_len = (media_grid_thw[i].prod() // (merge_size**2))
return token_id * token_len
input_ids, labels, loss_scale = self._extend_tokens(input_ids, labels, loss_scale, idx_list,
_get_new_tokens)
encoded['input_ids'] = input_ids
encoded['labels'] = labels
encoded['loss_scale'] = loss_scale
encoded.update(media_inputs) # Add multimodal content
return encoded
def _post_encode(self, model, inputs: Dict[str, Any]) -> Dict[str, Any]:
"""This function is typically used to solve the zero2/zero3 hanging issue in mixed model training,
i.e., some processes have pure text data without passing through vit, while others have image data that passed through vit.
Here we create dummy_image to solve this.
This function will be registered in the pre_forward_hook before `model.forward`.
This function should return input_embeds containing multimodal information.
"""
if not self.is_training:
return inputs
input_ids = inputs['input_ids']
input_features = inputs.get('input_features')
feature_attention_mask = inputs.get('feature_attention_mask')
base_model = self.get_base_model(model)
inputs_embeds = base_model.thinker.model.embed_tokens(input_ids)
thinker_config = model.config.thinker_config
# Helper function for handling text/image/video mixed modality data scenarios. (internally creates dummy_image)
inputs_embeds = self._get_inputs_embeds_hf(inputs_embeds, inputs, model.thinker.visual, self.processor,
thinker_config)
# Mixed modality data scenarios containing audio
if input_features is None:
if is_deepspeed_enabled() and not is_deepspeed_zero3_enabled():
# Note: Due to transformers implementation, the number of passes through audio model layers is related to the number of audios
# Therefore, zero3 will hang in scenarios where different processes have different numbers of audios (requires modification of transformers code to fix). Use zero2 in this scenario.
input_features = input_ids.new_zeros([1, 128, 128], dtype=model.thinker.audio_tower.dtype)
feature_attention_mask = input_ids.new_ones([1, 128], dtype=torch.bool)
audio_res = model.thinker.get_audio_features(input_features, feature_attention_mask)
# Compatible with transformers 5.0
if hasattr(audio_res, 'last_hidden_state'):
audio_embeds = audio_res.last_hidden_state
else:
audio_embeds = audio_res
inputs_embeds = inputs_embeds + audio_embeds.mean() * 0.
else:
audio_res = model.thinker.get_audio_features(input_features, feature_attention_mask)
# Compatible with transformers 5.0
if hasattr(audio_res, 'last_hidden_state'):
audio_embeds = audio_res.last_hidden_state
else:
audio_embeds = audio_res
audio_mask = (input_ids == thinker_config.audio_token_index).unsqueeze(-1).expand_as(inputs_embeds)
audio_embeds = audio_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
inputs_embeds = inputs_embeds.masked_scatter(audio_mask, audio_embeds)
return {'inputs_embeds': inputs_embeds}
def _get_position_ids(self, inputs: Dict[str, Any]):
"""Helper function to get mrope's position_ids"""
feature_attention_mask = inputs.get('feature_attention_mask')
if feature_attention_mask is not None:
audio_feature_lengths = torch.sum(feature_attention_mask, dim=1)
else:
audio_feature_lengths = None
video_second_per_grid = inputs.pop('video_second_per_grid', None)
input_ids = inputs['input_ids']
attention_mask = inputs.get('attention_mask')
if attention_mask is None:
attention_mask = torch.ones_like(input_ids)
position_ids, _ = self.model.thinker.get_rope_index(
input_ids,
inputs.get('image_grid_thw'),
inputs.get('video_grid_thw'),
attention_mask,
self.use_audio_in_video,
audio_feature_lengths,
video_second_per_grid,
)
return self._concat_text_position_ids(position_ids) # First dimension is text_position_ids
def _data_collator(self, batch: List[Dict[str, Any]], *, padding_to: Optional[int] = None) -> Dict[str, Any]:
"""Passed to dataloader's `collate_fn`"""
res = super()._data_collator(batch, padding_to=padding_to)
if not self.padding_free and self.is_training:
# padding_free/packing scenarios will handle position_ids in packing_row.
res['position_ids'] = self._get_position_ids(res)
if 'position_ids' in res:
# Create `packed_seq_params` to support padding_free/packing & flash-attn
position_ids = res['position_ids']
res['position_ids'] = position_ids[1:]
res['text_position_ids'] = text_position_ids = position_ids[0]
# https://github.com/huggingface/transformers/pull/40194
res.update(get_packed_seq_params(text_position_ids))
return res
def _data_collator_mm_data(self, batch: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Handle multimodal part in `_data_collator` function. (This function is compatible with padding_free/packing)"""
res = super()._data_collator_mm_data(batch)
video_second_per_grid = self.gather_list(batch, 'video_second_per_grid')
if video_second_per_grid:
res['video_second_per_grid'] = video_second_per_grid
input_features = [b['input_features'] for b in batch if b.get('input_features') is not None]
feature_attention_mask = [
b['feature_attention_mask'] for b in batch if b.get('feature_attention_mask') is not None
]
if input_features:
res['input_features'] = torch.concat(input_features)
res['feature_attention_mask'] = torch.concat(feature_attention_mask)
return res
def generate(self, model, *args, **kwargs):
"""`TransformersEngine` will call template.generate method for text generation; inherit here for customization."""
if kwargs.get('video_grid_thw') is not None:
kwargs['use_audio_in_video'] = self.use_audio_in_video
return super().generate(model, *args, **kwargs)
register_template(
TemplateMeta('my_qwen2_5_omni', prefix=[], prompt=['<|im_start|>user\n{{QUERY}}<|im_end|>\n<|im_start|>assistant\n'],
chat_sep=['<|im_end|>\n'], suffix=['<|im_end|>'],
system_prefix=['<|im_start|>system\n{{SYSTEM}}<|im_end|>\n'],
default_system='You are a helpful assistant.', stop_words=['<|endoftext|>'],
agent_template='hermes',
template_cls=Qwen2_5OmniTemplate))
if __name__ == '__main__':
# Test and debug
model, processor = get_model_processor('Qwen/Qwen2.5-Omni-7B', model_type='my_qwen2_5_omni')
template = get_template(processor, template_type='my_qwen2_5_omni')
data = {
'messages': [
{'role': 'user', 'content': 'Describe the video<video> and image<image> content.'},
{'role': 'assistant', 'content': 'A child and a cat.'},
],
'videos': ['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4'],
'images': ['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png'],
}
template.set_mode('train')
encoded = template.encode(data)
print('input_ids: ' + template.safe_decode(encoded['input_ids']))
print('labels: ' + template.safe_decode(encoded['labels']))
print('keys: ' + str(encoded.keys()))
```
## Inference Alignment
Next, you need to align inference between TransformersEngine and transformers. Typically you need to align `input_ids` and output content. You can write the following test function:
```python
import os
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info
from modelscope import snapshot_download
from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig
import requests
def infer_hf():
model_dir = snapshot_download('Qwen/Qwen2.5-Omni-7B')
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
model_dir, torch_dtype="auto", device_map="auto", attn_implementation='flash_attention_2')
processor = Qwen2_5OmniProcessor.from_pretrained(model_dir)
# Use decord to read video (url not yet supported)
resp = requests.get('https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4')
with open('_baby.mp4', 'wb') as f:
f.write(resp.content)
conversation = [
{
"role": "user",
"content": [
{"type": "video", "video": "_baby.mp4"},
{"type": "image", "image": "http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png"},
{"type": "text", "text": "Describe the video and image."},
],
},
]
USE_AUDIO_IN_VIDEO = False
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info(conversation, use_audio_in_video=USE_AUDIO_IN_VIDEO)
inputs = processor(text=text, audio=audios, images=images, videos=videos, return_tensors="pt", padding=True,
use_audio_in_video=USE_AUDIO_IN_VIDEO)
inputs = inputs.to(model.device).to(model.dtype)
text_ids = model.generate(**inputs, use_audio_in_video=USE_AUDIO_IN_VIDEO, thinker_do_sample=False,
return_audio=False)
text = processor.batch_decode(text_ids[:, inputs['input_ids'].shape[1]:], skip_special_tokens=True, clean_up_tokenization_spaces=False)
return inputs['input_ids'][0].tolist(), text[0]
def test_my_qwen2_5_omni():
engine = TransformersEngine('Qwen/Qwen2.5-Omni-7B', model_type='my_qwen2_5_omni', attn_impl='flash_attention_2')
infer_request = InferRequest(messages=[{
"role": "user",
"content": "<video><image>Describe the video and image.",
}],
videos=["https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4"],
images=["http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png"],
)
request_config = RequestConfig(temperature=0, max_tokens=512)
input_ids = engine.template.encode(infer_request)['input_ids']
resp_list = engine.infer([infer_request], request_config)
resp = resp_list[0].choices[0].message.content
return input_ids, resp
if __name__ == '__main__':
# Enable debug mode, will print input_ids and generate_ids from `TransformersEngine.infer`
os.environ['SWIFT_DEBUG'] = '1'
input_ids_hf, response_hf = infer_hf()
input_ids_swift, response_swift = test_my_qwen2_5_omni()
# Test input_ids and response alignment
assert input_ids_hf == input_ids_swift
assert response_hf == response_swift
```
## Start Training
Train using Python code, which is usually easier to debug:
```python
from swift import sft_main, SftArguments
import os
if __name__ == '__main__':
os.environ['MAX_PIXELS'] = '1003520'
sft_main(SftArguments(
model='Qwen/Qwen2.5-Omni-7B',
dataset=['AI-ModelScope/LaTeX_OCR#5000'],
model_type='my_qwen2_5_omni',
template='my_qwen2_5_omni',
load_from_cache_file=True,
split_dataset_ratio=0.01,
tuner_type='lora',
torch_dtype='bfloat16',
attn_impl='flash_attn',
padding_free=True,
num_train_epochs=1,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
learning_rate=1e-4,
lora_rank=8,
lora_alpha=32,
target_modules=['all-linear'],
freeze_vit=True,
freeze_aligner=True,
gradient_accumulation_steps=1,
eval_steps=50,
save_steps=50,
save_total_limit=2,
logging_steps=5,
max_length=2048,
output_dir='output',
warmup_ratio=0.05,
dataloader_num_workers=4,
dataset_num_proc=1,
))
```
Train using command line:
```shell
# 4 * 35GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
VIDEO_MAX_PIXELS=50176 \
FPS_MAX_FRAMES=12 \
MAX_PIXELS=1003520 \
swift sft \
--model Qwen/Qwen2.5-Omni-7B \
--model_type my_qwen2_5_omni \
--template my_qwen2_5_omni \
--external_plugins 'examples/custom/my_qwen2_5_omni/my_register.py' \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#2000' \
'AI-ModelScope/LaTeX_OCR:human_handwrite#2000' \
'speech_asr/speech_asr_aishell1_trainsets:validation#2000' \
'swift/VideoChatGPT:all#2000' \
--load_from_cache_file true \
--split_dataset_ratio 0.01 \
--tuner_type lora \
--torch_dtype bfloat16 \
--attn_impl flash_attn \
--padding_free true \
--packing true \
--num_train_epochs 3 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 1e-4 \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--freeze_vit true \
--freeze_aligner true \
--gradient_accumulation_steps 1 \
--eval_steps 50 \
--save_steps 50 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 4096 \
--output_dir output \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--dataset_num_proc 1 \
--deepspeed zero2
```
Perform inference on the validation set after training: (Environment variables should align with training)
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=0 \
VIDEO_MAX_PIXELS=50176 \
FPS_MAX_FRAMES=12 \
MAX_PIXELS=1003520 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--stream true \
--max_new_tokens 512 \
--load_data_args true
```
Use the following command to push training weights to Modelscope:
```shell
swift export \
--adapters output/vx-xxx/checkpoint-xxx \
--push_to_hub true \
--hub_model_id '<your-model-id>' \
--hub_token '<your-sdk-token>'
```
## Submitting a PR
If you want to submit a PR to ms-swift, you need to run the following additional commands to lint and format the code:
```shell
pip install pre-commit
pre-commit run --all-files
```
docs/source_en/BestPractices/Metax-support.md
+271
View File
@@ -0,0 +1,271 @@
# Metax Support
## 1. use swift with Metax
you can either build an image or pull an existing one. Here, we demonstrate how to use ms-swift on Metax by pulling a pre-built image as an example.
### 1.1. start ms-swift Container
```bash
docker pull mx-devops-acr-cn-shanghai.cr.volces.com/opensource/public-ai-release/maca/ms-swift:3.10.3-maca.ai3.3.0.16-torch2.6-py310-ubuntu22.04-amd64
# you may modify privileged option and mount only specific GPU cards.
# please refer to our documents on https://developer.metax-tech.com
# Metax GPUs must be mounted via --device=/dev/dri --device=/dev/mxcd
docker run -it --net=host --uts=host --ipc=host --privileged=true --group-add video \
--shm-size 100gb --ulimit memlock=-1 \
--security-opt seccomp=unconfined --security-opt apparmor=unconfined \
--device=/dev/dri --device=/dev/mxcd \
-v /root/workspace:/external \
--name swift_test \
mx-devops-acr-cn-shanghai.cr.volces.com/opensource/public-ai-release/maca/ms-swift:3.10.3-maca.ai3.3.0.16-torch2.6-py310-ubuntu22.04-amd64
```
## 2. Environment check
### 2.1. Check Metax available
Thanks to its compatibility with CUDA, we can use the same approach as NVIDIA to check the availability of Metax devices.
```python
import torch
print(torch.cuda.is_available())
# True
```
### 2.2. Check the P2P connections
```bash
mx-smi topo -m
# output
=================== MetaX System Management Interface Log ===================
Timestamp : Wed Feb 11 16:37:10 2026
Attached GPUs : 8
Device link type matrix
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 Node Affinity CPU Affinity
GPU0 X MX MX MX NODE NODE NODE NODE 0 0-31,64-95
GPU1 MX X MX MX NODE NODE NODE NODE 0 0-31,64-95
GPU2 MX MX X MX NODE NODE NODE NODE 0 0-31,64-95
GPU3 MX MX MX X NODE NODE NODE NODE 0 0-31,64-95
GPU4 NODE NODE NODE NODE X MX MX MX 0 0-31,64-95
GPU5 NODE NODE NODE NODE MX X MX MX 0 0-31,64-95
GPU6 NODE NODE NODE NODE MX MX X MX 0 0-31,64-95
GPU7 NODE NODE NODE NODE MX MX MX X 0 0-31,64-95
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
MX = Connection traversing MetaXLink
ETH = Connection traversing Eth
NA = Connection type is unknown
```
### 2.3. check the status of the GPUs
```bash
mx-smi
# output
=================== MetaX System Management Interface Log ===================
Timestamp : Wed Feb 11 09:55:49 2026
Attached GPUs : 8
+---------------------------------------------------------------------------------+
| MX-SMI 2.2.9 Kernel Mode Driver Version: 3.4.4 |
| MACA Version: 3.3.0.15 BIOS Version: 1.30.0.0 |
|------------------+-----------------+---------------------+----------------------|
| Board Name | GPU Persist-M | Bus-id | GPU-Util sGPU-M |
| Pwr:Usage/Cap | Temp Perf | Memory-Usage | GPU-State |
|==================+=================+=====================+======================|
| 0 MetaX C500 | 0 Off | 0000:0e:00.0 | 0% Disabled |
| 57W / 350W | 35C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 1 MetaX C500 | 1 Off | 0000:0f:00.0 | 0% Disabled |
| 58W / 350W | 37C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 2 MetaX C500 | 2 Off | 0000:10:00.0 | 0% Disabled |
| 58W / 350W | 36C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 3 MetaX C500 | 3 Off | 0000:12:00.0 | 0% Disabled |
| 60W / 350W | 35C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 4 MetaX C500 | 4 Off | 0000:35:00.0 | 0% Disabled |
| 57W / 350W | 33C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 5 MetaX C500 | 5 Off | 0000:36:00.0 | 0% Disabled |
| 56W / 350W | 34C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 6 MetaX C500 | 6 Off | 0000:37:00.0 | 0% Disabled |
| 55W / 350W | 34C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
| 7 MetaX C500 | 7 Off | 0000:38:00.0 | 0% Disabled |
| 56W / 350W | 36C P0 | 826/65536 MiB | Available |
+------------------+-----------------+---------------------+----------------------+
+---------------------------------------------------------------------------------+
| Process: |
| GPU PID Process Name GPU Memory |
| Usage(MiB) |
|=================================================================================|
| no process found |
+---------------------------------------------------------------------------------+
```
## 3. run example
We support direct use of the community version. However, we also provide a more optimized version in the image under /workspace and strongly recommend using it.
### 3.1. run swift example
In most scenarios, we can run Swift's examples directly.
```bash
# We assume that the ms-swift code is under /workspace
cd /workspace/ms-swift/
bash examples/train/full/train.sh
```
```bash
# output:
{'loss': 1.47077751, 'grad_norm': 10.5625, 'learning_rate': 2e-06, 'token_acc': 0.65511727, 'epoch': 0.01, 'global_step/max_steps': '1/94', 'percentage': '1.06%', 'elapsed_time': '2s', 'remaining_time': '4m 28s', 'memory(GiB)': 4.87, 'train_speed(iter/s)': 0.345807}
{'loss': 1.58882141, 'grad_norm': 10.75, 'learning_rate': 1e-05, 'token_acc': 0.61763144, 'epoch': 0.05, 'global_step/max_steps': '5/94', 'percentage': '5.32%', 'elapsed_time': '10s', 'remaining_time': '3m 12s', 'memory(GiB)': 5.64, 'train_speed(iter/s)': 0.461462}
{'loss': 1.56617603, 'grad_norm': 12.8125, 'learning_rate': 9.92e-06, 'token_acc': 0.61519274, 'epoch': 0.11, 'global_step/max_steps': '10/94', 'percentage': '10.64%', 'elapsed_time': '20s', 'remaining_time': '2m 52s', 'memory(GiB)': 5.64, 'train_speed(iter/s)': 0.485796}
{'loss': 1.63347206, 'grad_norm': 13.6875, 'learning_rate': 9.69e-06, 'token_acc': 0.60373975, 'epoch': 0.16, 'global_step/max_steps': '15/94', 'percentage': '15.96%', 'elapsed_time': '30s', 'remaining_time': '2m 39s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.493855}
{'loss': 1.60613976, 'grad_norm': 11.0, 'learning_rate': 9.32e-06, 'token_acc': 0.59997221, 'epoch': 0.21, 'global_step/max_steps': '20/94', 'percentage': '21.28%', 'elapsed_time': '39s', 'remaining_time': '2m 27s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.500516}
{'loss': 1.45015478, 'grad_norm': 15.25, 'learning_rate': 8.8e-06, 'token_acc': 0.62373584, 'epoch': 0.27, 'global_step/max_steps': '25/94', 'percentage': '26.60%', 'elapsed_time': '49s', 'remaining_time': '2m 16s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.50548}
{'loss': 1.39427547, 'grad_norm': 13.9375, 'learning_rate': 8.18e-06, 'token_acc': 0.6357994, 'epoch': 0.32, 'global_step/max_steps': '30/94', 'percentage': '31.91%', 'elapsed_time': '59s', 'remaining_time': '2m 5s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.508409}
{'loss': 1.53672237, 'grad_norm': 11.125, 'learning_rate': 7.45e-06, 'token_acc': 0.61650612, 'epoch': 0.37, 'global_step/max_steps': '35/94', 'percentage': '37.23%', 'elapsed_time': '1m 8s', 'remaining_time': '1m 55s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.510425}
{'loss': 1.54039021, 'grad_norm': 13.8125, 'learning_rate': 6.65e-06, 'token_acc': 0.61613974, 'epoch': 0.43, 'global_step/max_steps': '40/94', 'percentage': '42.55%', 'elapsed_time': '1m 18s', 'remaining_time': '1m 45s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.512302}
{'loss': 1.40159426, 'grad_norm': 9.4375, 'learning_rate': 5.79e-06, 'token_acc': 0.64041773, 'epoch': 0.48, 'global_step/max_steps': '45/94', 'percentage': '47.87%', 'elapsed_time': '1m 27s', 'remaining_time': '1m 35s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.512983}
{'loss': 1.54977188, 'grad_norm': 11.9375, 'learning_rate': 4.91e-06, 'token_acc': 0.61078816, 'epoch': 0.53, 'global_step/max_steps': '50/94', 'percentage': '53.19%', 'elapsed_time': '1m 37s', 'remaining_time': '1m 25s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.514489}
{'loss': 1.6754509, 'grad_norm': 13.0625, 'learning_rate': 4.04e-06, 'token_acc': 0.58574393, 'epoch': 0.59, 'global_step/max_steps': '55/94', 'percentage': '58.51%', 'elapsed_time': '1m 46s', 'remaining_time': '1m 15s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.515752}
{'loss': 1.37204351, 'grad_norm': 9.25, 'learning_rate': 3.19e-06, 'token_acc': 0.6391937, 'epoch': 0.64, 'global_step/max_steps': '60/94', 'percentage': '63.83%', 'elapsed_time': '1m 56s', 'remaining_time': '1m 5s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.516829}
{'loss': 1.47697926, 'grad_norm': 11.375, 'learning_rate': 2.4e-06, 'token_acc': 0.62817259, 'epoch': 0.69, 'global_step/max_steps': '65/94', 'percentage': '69.15%', 'elapsed_time': '2m 5s', 'remaining_time': '55s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.517947}
{'loss': 1.4336628, 'grad_norm': 8.125, 'learning_rate': 1.69e-06, 'token_acc': 0.63453862, 'epoch': 0.75, 'global_step/max_steps': '70/94', 'percentage': '74.47%', 'elapsed_time': '2m 14s', 'remaining_time': '46s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.518833}
{'loss': 1.54315252, 'grad_norm': 9.625, 'learning_rate': 1.08e-06, 'token_acc': 0.60202073, 'epoch': 0.8, 'global_step/max_steps': '75/94', 'percentage': '79.79%', 'elapsed_time': '2m 24s', 'remaining_time': '36s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.519627}
{'loss': 1.47180223, 'grad_norm': 9.5625, 'learning_rate': 6e-07, 'token_acc': 0.62211501, 'epoch': 0.85, 'global_step/max_steps': '80/94', 'percentage': '85.11%', 'elapsed_time': '2m 33s', 'remaining_time': '26s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.520284}
{'loss': 1.44068375, 'grad_norm': 10.125, 'learning_rate': 2.5e-07, 'token_acc': 0.62673112, 'epoch': 0.91, 'global_step/max_steps': '85/94', 'percentage': '90.43%', 'elapsed_time': '2m 43s', 'remaining_time': '17s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.520331}
{'loss': 1.44893646, 'grad_norm': 8.375, 'learning_rate': 5e-08, 'token_acc': 0.63837478, 'epoch': 0.96, 'global_step/max_steps': '90/94', 'percentage': '95.74%', 'elapsed_time': '2m 52s', 'remaining_time': '7s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.520707}
{'train_runtime': 183.4332, 'train_samples_per_second': 8.177, 'train_steps_per_second': 0.512, 'train_loss': 1.50650934, 'token_acc': 0.6194337, 'epoch': 1.0, 'global_step/max_steps': '94/94', 'percentage': '100.00%', 'elapsed_time': '3m 3s', 'remaining_time': '0s', 'memory(GiB)': 6.5, 'train_speed(iter/s)': 0.512463}
Train: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 94/94 [03:03<00:00, 1.95s/it]
[INFO:swift] last_model_checkpoint: /workspace/ms-swift/output/v0-20260211-143035/checkpoint-94
[INFO:swift] best_model_checkpoint: None
[INFO:swift] images_dir: /workspace/ms-swift/output/v0-20260211-143035/images
[INFO:swift] End time of running main: 2026-02-11 14:34:09.521336
```
### 3.2. run swift example with Megatron-LM
if you want to use Megatron-LM as Swift's backend, you should set MEGATRON_LM_PATH to /workspace/Megatron-LM-0.15.0 or other versions.
```bash
export MEGATRON_LM_PATH=/workspace/Megatron-LM-0.15.0
cd /workspace/ms-swift
bash examples/megatron/pretrain.sh
```
### 3.3. use other versions of ms-swift
The Metax platform requires the use of MACA-compatible software packages. For instance, compiling depends on torch2.8. We need to use torch2.8+maca3.3.x.x. By default, the installation will overwrite the torch within the environment. Therefore, we recommend using the --no-deps parameter for installation
```bash
git clone -b ${SWIFT_VERSION} https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install . --no-deps
```
After each environment change, the torch and its availability should be checked
```bash
pip list |grep torch
# output:
# torch2.x.x+metax3.x.x.x
```
```python
import torch
torch.cuda.is_available()
```
### 3.4. Differences between Metax and NVIDIA CUDA
We are largely aligned with NVIDIA, but there are some differences in certain software and environment variables.
#### 3.4.1. MACA_MPS_MODE
By default, MACA does not allow multiple processes to run on a single GPU. Therefore, when the GPU is already occupied, you cannot launch another process. To enable this scenario, you need to set MACA_MPS_MODE=1
```bash
# run other scripts ...
export MACA_MPS_MODE=1
cd /workspace/ms-swift/
bash examples/train/full/train.sh
```
#### 3.4.2. MCCL_SOCKET_IFNAME GLOO_SOCKET_IFNAME & MCCL_IB_HCA
When using MACA in a multi-node setup, you need to set the environment variables MCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME, and MCCL_IB_HCA to ensure proper inter-node communication.
We can use mx-smi and ifconfig to determine which InfiniBand devices and network device are being used.
```bash
ifconfig
# output
ens20f0np0: xxx
inet: your node ip
xxx
...
```
```bash
mx-smi topo -n
# output
mx-smi version: 2.2.9
=================== MetaX System Management Interface Log ===================
Timestamp : Wed Feb 11 18:53:44 2026
Attached GPUs : 8
Device link type matrix
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 Node Affinity CPU Affinity
GPU0 X MX MX MX NODE NODE NODE NODE PIX PIX NODE NODE SYS SYS 0 0-31,64-95
GPU1 MX X MX MX NODE NODE NODE NODE PIX PIX NODE NODE SYS SYS 0 0-31,64-95
GPU2 MX MX X MX NODE NODE NODE NODE PIX PIX NODE NODE SYS SYS 0 0-31,64-95
GPU3 MX MX MX X NODE NODE NODE NODE PIX PIX NODE NODE SYS SYS 0 0-31,64-95
GPU4 NODE NODE NODE NODE X MX MX MX NODE NODE PIX PIX SYS SYS 0 0-31,64-95
GPU5 NODE NODE NODE NODE MX X MX MX NODE NODE PIX PIX SYS SYS 0 0-31,64-95
GPU6 NODE NODE NODE NODE MX MX X MX NODE NODE PIX PIX SYS SYS 0 0-31,64-95
GPU7 NODE NODE NODE NODE MX MX MX X NODE NODE PIX PIX SYS SYS 0 0-31,64-95
NIC0 PIX PIX PIX PIX NODE NODE NODE NODE X PIX NODE NODE SYS SYS
NIC1 PIX PIX PIX PIX NODE NODE NODE NODE PIX X NODE NODE SYS SYS
NIC2 NODE NODE NODE NODE PIX PIX PIX PIX NODE NODE X PIX SYS SYS
NIC3 NODE NODE NODE NODE PIX PIX PIX PIX NODE NODE PIX X SYS SYS
NIC4 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS X PIX
NIC5 SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS PIX X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
MX = Connection traversing MetaXLink
ETH = Connection traversing Eth
NA = Connection type is unknown
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
# The output shows:
# 1. GPU0 to GPU3 communicate with NIC0 and NIC1, while GPU4 to GPU7 communicate with NIC2 and NIC3
# 2. NIC0 uses ib device:mlx5_0, NIC1 uses ib device:mlx5_1, NIC2 uses ib device:mlx5_2, NIC3 uses ib device:mlx5_3
```
Therefore:
MCCL_SOCKET_IFNAME=ens20f0np0
GLOO_SOCKET_IFNAME=ens20f0np0
MCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
```bash
# node 1
export MCCL_SOCKET_IFNAME=ens20f0np0
export GLOO_SOCKET_IFNAME=ens20f0np0
export MCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
cd /workspace/ms-swift/
bash examples/train/multi-node/torchrun/train_node1.sh
```
```bash
# node 2
# Update the value of the master_addr parameter in the script.
export MCCL_SOCKET_IFNAME=ens20f0np0
export GLOO_SOCKET_IFNAME=ens20f0np0
export MCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
cd /workspace/ms-swift/
bash examples/train/multi-node/torchrun/train_node2.sh
```
docs/source_en/BestPractices/More-Best-Practices.md
+6
View File
@@ -0,0 +1,6 @@
# More Best Practices
- [Qwen2.5 self-cognition SFT](https://github.com/modelscope/ms-swift/tree/main/examples/notebook/qwen2_5-self-cognition)
- [Qwen2-VL Latex-OCR SFT](https://github.com/modelscope/ms-swift/tree/main/examples/notebook/qwen2vl-ocr)
- [Qwen2.5-VL Grounding Task SFT](https://github.com/modelscope/ms-swift/tree/main/examples/notebook/qwen2_5-vl-grounding)
docs/source_en/BestPractices/NPU-support.md
+871
View File
@@ -0,0 +1,871 @@
# NPU Support
ms-swift supports Ascend NPUs. You can fine-tune models and run inference on Ascend NPUs.
This document describes how to prepare the environment, train models, save and merge checkpoints, run inference, deploy services, and troubleshoot common issues on Ascend NPUs.
If this is your first time using ms-swift on NPUs, we recommend reading this document in the following order:
1. Check "Support Scope at a Glance" first to confirm whether your model, algorithm, and backend have been verified.
2. Use "Choose Your Usage Path" to decide whether you only need the base environment or also need MindSpeed/Megatron-SWIFT.
3. Choose "Local Environment Installation" or "Image/Container Environment Installation" according to your own environment management preference, then run "NPU Availability Check".
4. Use "Quick Start" to complete one ModelScope model LoRA training, merge, inference, and deployment flow.
5. For larger-scale training, continue reading the DDP, DeepSpeed, and MindSpeed/Megatron-SWIFT sections.
## Hardware and Supported Operating Systems
**Table 1** Product hardware support list
| Product | Supported |
| ------- | :-------: |
| <term>Ascend 950 series products</term> | √ |
| <term>Atlas A3 training series products</term> | √ |
| <term>Atlas A3 inference series products</term> | x |
| <term>Atlas A2 training series products</term> | √ |
| <term>Atlas A2 inference series products</term> | x |
| <term>Atlas 200I/500 A2 inference products</term> | x |
| <term>Atlas inference series products</term> | x |
| <term>Atlas training series products</term> | x |
> [!NOTE]
>
> In this section, "√" indicates supported and "x" indicates not supported.
- For operating systems supported by each hardware product in physical-machine deployment scenarios, see the [Compatibility Query Assistant](https://www.hiascend.com/hardware/compatibility).
- For operating systems supported by each hardware product in VM and container deployment scenarios, see "Operating System Compatibility" in CANN Software Installation for the [commercial edition](https://www.hiascend.com/document/detail/zh/canncommercial/900/softwareinst/instg/instg_0101.html?OS=openEuler&InstallType=netyum) or [community edition](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0101.html?OS=openEuler&InstallType=netyum).
## Support Scope at a Glance
Recommended base environment versions:
| software | version |
| --------- | --------------- |
| Python | >= 3.10, < 3.12 |
| CANN | >= 8.5.1 |
| torch | >= 2.7.1 |
| torch_npu | >= 2.7.1.post4 |
Note: The officially recommended version compatibility matrix for the vLLM Ascend series has been updated to CANN 9.0.0, torch 2.9.0, torch_npu 2.9.0, vllm-ascend 0.18.0 for A3, and vllm-ascend 0.19.1 for A5. For details, see the [vLLM Ascend installation guide](https://docs.vllm.ai/projects/ascend/en/v0.18.0/installation.html).
For base environment setup, see the [Ascend PyTorch installation guide](https://gitcode.com/Ascend/pytorch). The examples in this document were verified on 8 * Ascend 910B3 64G.
| Primary feature | Feature | Status |
| --------------- | --------------------- | ------------- |
| Training | CPT | Supported |
| | SFT | Supported |
| | DPO | Supported |
| | RM | Supported |
| Distributed | DDP | Supported |
| | FSDP | Supported |
| | FSDP2 | Supported |
| | DeepSpeed | Supported |
| | MindSpeed(Megatron) | Supported |
| PEFT | FULL | Supported |
| | LoRA | Supported |
| | QLoRA | Not supported |
| RLHF | GRPO | Supported |
| | PPO | Supported |
| Optimization | FA and other fused ops | Supported |
| | Liger-Kernel | Not supported |
| Deployment | PT | Supported |
| | vLLM | Supported |
| | SGLang | Not supported |
### Verified SFT Combinations
| algorithm | model families | strategy | hardware |
| --------- | --------------------------- | --------------------- | ----------------- |
| SFT | Qwen2.5-0.5B-Instruct | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen2.5-1.5B-Instruct | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen2.5-7B-Instruct | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen2.5-VL-3B-Instruct | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen2.5-VL-7B-Instruct | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen2.5-Omni-3B | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen3-8B | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen3-30B-A3B | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen3-32B | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen3-VL-30B-A3B-Instruct | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen3-Omni-30B-A3B-Instruct | FSDP1/FSDP2/deepspeed/Megatron | Atlas 900 A2 PODc/A3 SuperPoD |
| SFT | InternVL3-8B | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Ovis2.5-2B | FSDP1/FSDP2/deepspeed | Atlas 900 A2 PODc |
| SFT | Qwen3.5-27B | FSDP1/FSDP2/deepspeed/Megatron | Atlas 900 A2 PODc/A3 SuperPoD |
| SFT | Qwen3.5-35B-A3B | FSDP1/FSDP2/deepspeed/Megatron | Atlas 900 A2 PODc/A3 SuperPoD |
### Verified RL Combinations
| algorithm | model families | strategy | rollout engine | hardware |
| --------- | ------------------- | --------- | -------------- | ----------------- |
| **GRPO** | Qwen2.5-7B-Instruct | deepspeed | vllm-ascend | Atlas 900 A2 PODc |
| **GRPO** | Qwen3-8B | deepspeed | vllm-ascend | Atlas 900 A2 PODc |
| **DPO** | Qwen2.5-7B-Instruct | deepspeed | vllm-ascend | Atlas 900 A2 PODc |
| **DPO** | Qwen3-8B | deepspeed | vllm-ascend | Atlas 900 A2 PODc |
| **PPO** | Qwen2.5-7B-Instruct | deepspeed | vllm-ascend | Atlas 900 A2 PODc |
| **PPO** | Qwen3-8B | deepspeed | vllm-ascend | Atlas 900 A2 PODc |
### Not Yet Supported or Not Fully Verified
| item |
| ----------------------------------------- |
| Liger-kernel |
| Quantization/QLoRA related features |
| Using SGLang as the inference engine |
| Enabling ETP for LoRA training in Megatron |
### PEFT Transformers 5 MoE Fused Expert LoRA Limitation
When training Transformers 5 MoE models such as Qwen3.5-MoE and Qwen3-Omni-MoE with LoRA, some expert weights may be stored as fused `nn.Parameter` tensors instead of ordinary `nn.Linear` modules. Injecting LoRA into these parameters relies on PEFT's `target_parameters` path.
This path is not fully stable yet with combinations such as `lora_dropout`, ZeRO-3/FSDP, and multiple adapters. Typical trigger conditions include:
- using a MoE model;
- using LoRA and trying to cover fused expert parameters;
- triggering PEFT's Transformers 5 MoE target conversion path through the model config or the command-line `--model_type`;
- using the default `lora_dropout != 0`, or using parameter-sharding backends such as ZeRO-3/FSDP.
For regular Qwen3.5 GRPO/SFT LoRA training, avoid explicitly passing `--model_type` when it is not needed. If the model config itself already triggers this PEFT path, prefer full-parameter training or disable the affected LoRA combination. If fused expert LoRA is required, wait for the PEFT upstream support to stabilize, or use it only after validating the target model and training backend with `lora_dropout=0`.
## Choose Your Usage Path
| Scenario | Recommended path | Need MindSpeed |
| -------------------------------------- | ----------------------------------------------------- | -------------- |
| Ordinary SFT/LoRA/inference | Local environment installation or image/container installation | No |
| Megatron-SWIFT large-model training | Install the base environment, then install MindSpeed/Megatron/mcore-bridge | Yes |
| GRPO/PPO/DPO and other RLHF workflows | Base training environment + vLLM-Ascend rollout/deploy | Usually no |
| Only verifying whether NPUs are usable | Run the NPU availability check script | No |
## Environment Preparation
### Image/Container Environment Installation
The official NPU image is available at [quay.io/ascend/ms-swift](https://quay.io/repository/ascend/ms-swift?tab=tags). We recommend choosing an image tag that matches your device generation, Python version, CANN version, and OS version first. If you need to pin a branch or customize dependencies, build the image from the Dockerfile provided by the project. The container approach makes dependency versions easier to freeze and helps reproduce the same environment across multiple Ascend machines.
The following example uses the A2, Python 3.11, CANN 9.0.0, Ubuntu 22.04 tag. In actual use, choose the latest tag from the Quay tag page that matches your machine and software stack.
```shell
docker pull quay.io/ascend/ms-swift:v4.3.0-A2-py311-CANN9.0.0-ubuntu22.04
export IMAGE_NAME=quay.io/ascend/ms-swift:v4.3.0-A2-py311-CANN9.0.0-ubuntu22.04
export WORKSPACE=/path/to/workspace
```
If you need to build the image yourself, clone the modelscope repository first, then use [Dockerfile.ascend](https://github.com/modelscope/modelscope/blob/master/docker/Dockerfile.ascend) and [build_image.py](https://github.com/modelscope/modelscope/blob/master/docker/build_image.py):
```shell
git clone https://github.com/modelscope/modelscope.git
cd modelscope
DOCKER_REGISTRY=ms-swift python docker/build_image.py \
--image_type ascend \
--python_version 3.11.11 \
--soc_version ascend910b1 \
--arch arm
```
The current `build_image.py` generates Ascend image names in the format `{DOCKER_REGISTRY}:{swift_branch}-{atlas_hardware}-{python_tag}-{arch}`. The command above uses the ARM-based Atlas 900 A2 PODc as an example and usually generates `ms-swift:main-A2-py311-arm`. If you use a self-built image, replace `IMAGE_NAME` above with the actual image name from your build log.
```shell
export IMAGE_NAME=ms-swift:main-A2-py311-arm
```
Before starting the container, check which NPU devices are exposed on the host:
```shell
ls /dev/davinci*
```
When starting the container, mount the NPU devices, driver, firmware, `npu-smi`, and required log directories. The following example uses the common 8-card device range `davinci0` to `davinci7`. Some machines may also expose devices up to `davinci15`; in that case, add the corresponding devices to `docker run` according to the output of `ls /dev/davinci*`.
```shell
docker run -it \
--name swift-ascend \
--network=host --ipc=host --shm-size=128g \
--device=/dev/davinci0 --device=/dev/davinci1 \
--device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 \
--device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci_manager --device=/dev/devmm_svm --device=/dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware:ro \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi:ro \
-v /etc/ascend_install.info:/etc/ascend_install.info:ro \
-v /var/log/npu:/var/log/npu \
-v ${WORKSPACE}:/workspace \
${IMAGE_NAME} \
/bin/bash
```
After entering the container, run `source /usr/local/Ascend/ascend-toolkit/set_env.sh` first, then run the NPU availability check below to confirm that the container can access the Ascend devices. If the container cannot detect NPUs, check `/dev/davinci*`, `/dev/davinci_manager`, the driver directory, and `npu-smi` mounts first.
### Local Environment Installation
```shell
# Create a new conda virtual environment (optional)
conda create -n swift-npu python=3.11 -y
conda activate swift-npu
# Source the CANN environment before the following steps
source /usr/local/Ascend/ascend-toolkit/set_env.sh
# Set the global pip mirror (optional, speeds up downloads)
pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
pip install ms-swift -U
# Install from source
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .
# Install torch_npu
pip install torch_npu==2.9.0 decorator
# If you want to use deepspeed (to reduce memory usage, with some speed overhead)
pip install deepspeed
# If you need evaluation features, install the following package
pip install evalscope[opencompass]
# If you need vllm-ascend for inference, install the following packages (for more versions, see the [vLLM-Ascend official website](https://docs.vllm.ai/projects/ascend/en/latest/installation.html))
pip install vllm==0.18.0
pip install vllm-ascend==0.18.0
```
### NPU Availability Check
Check whether the environment is installed correctly and whether NPUs can be loaded:
```python
from transformers.utils import is_torch_npu_available
import torch
print(is_torch_npu_available()) # True
print(torch.npu.device_count()) # 8
print(torch.randn(10, device='npu:0'))
```
### Optional MindSpeed/Megatron-SWIFT Installation
If you need MindSpeed(Megatron-LM), install the required dependencies as follows.
```shell
# 1. Clone Megatron-LM and switch to v0.16.0
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
git checkout core_v0.16.0
cd ..
# 2. Clone and install MindSpeed
git clone https://gitcode.com/Ascend/MindSpeed.git
cd MindSpeed
git checkout core_r0.16.0
pip install -e .
cd ..
# 3. Clone and install mcore-bridge
git clone https://github.com/modelscope/mcore-bridge.git
cd mcore-bridge
pip install -e .
cd ..
# 4. Download and install triton-ascend
pip install triton-ascend==3.2.1 --extra-index-url=https://triton-ascend.osinfra.cn/pypi/simple
# 5. Set environment variables
export PYTHONPATH=$PYTHONPATH:<your_local_megatron_lm_path>
export MEGATRON_LM_PATH=<your_local_megatron_lm_path>
# 6. Disable Megatron GDN if you need to fall back to the transformers GatedDeltaNet implementation
export USE_MCORE_GDN=0
```
Run the following command to verify that MindSpeed(Megatron-LM) is configured correctly:
```shell
python -c "import mindspeed.megatron_adaptor; from swift.megatron.init import init_megatron_env; init_megatron_env(); print('✓ Megatron-SWIFT configuration verified successfully in the NPU environment!')"
```
### Qwen3.5 FLA Patch Notes
The current repository already includes a built-in Qwen3.5 linear attention patch for Ascend NPUs, so users do not need to manually modify the `transformers` or `fla` source code. This patch does not replace the entire `flash-linear-attention` package directly. Instead, it redirects the low-level GPU Triton operator path used by `Qwen3.5` through `chunk_gated_delta_rule` to the MindSpeed NPU implementation.
When the patch takes effect, ms-swift performs the following replacements:
1. Set `transformers.utils.is_flash_linear_attention_available` and `transformers.utils.import_utils.is_flash_linear_attention_available` to return `True`, so that `transformers.models.qwen3_5.modeling_qwen3_5` can complete initialization through the FLA fast path.
2. Redirect `transformers.models.qwen3_5.modeling_qwen3_5.chunk_gated_delta_rule` and `transformers.models.qwen3_5_moe.modeling_qwen3_5_moe.chunk_gated_delta_rule` to the built-in ms-swift implementation `swift.model.chunk_gated_delta_rule.chunk_gated_delta_rule`.
3. Inside `swift.model.chunk_gated_delta_rule`, continue calling the native Triton operators provided by MindSpeed, including:
- `mindspeed.ops.triton.chunk_delta_h`
- `mindspeed.ops.triton.chunk_o`
- `mindspeed.ops.triton.chunk_scaled_dot_kkt`
- `mindspeed.ops.triton.cumsum`
- `mindspeed.ops.triton.solve_tril`
- `mindspeed.ops.triton.wy_fast`
4. Keep the native torch l2norm helper, reducing per-layer per-step launch overhead as well as compile/autotune overhead during cold start, which improves model performance on NPU.
5. For `FusedRMSNormGated`, which depends on `torch.cuda.current_device()` during FLA initialization, NPU keeps the native Qwen3.5 torch path to avoid compatibility issues caused by CUDA-only initialization logic.
The call chain can be understood as:
```text
Qwen3.5 modeling.chunk_gated_delta_rule
-> swift.model.chunk_gated_delta_rule.chunk_gated_delta_rule
-> MindSpeed Triton kernels
```
Therefore:
- This patch mainly covers the **gated-delta-rule path of Qwen3.5 linear attention**.
- It is not equivalent to “fully replacing the entire fla package with MindSpeed”.
- To make this path effective, ensure that MindSpeed can be imported correctly in the current environment.
- Verified versions for accuracy alignment: torch 2.9.0 + MindSpeed 0.16.0 + flash-linear-attention 0.4.2 + triton-ascend 3.2.1 + transformers 5.2.0
When running Qwen3.5 on NPU with either the transformers backend or the Megatron-SWIFT backend, note the following version and feature constraints:
1. The MindSpeed training combination currently pinned by the NPU documentation
is `Megatron-LM v0.16.0 + MindSpeed core_r0.16.0`. With this combination,
`megatron-core` already ships the native GDN kernel
`core.ssm.gated_delta_net`, and `mcore-bridge` defaults to the
Megatron-Core/MindSpeed GDN path with `USE_MCORE_GDN=1`. If you explicitly
set `USE_MCORE_GDN=0`, it falls back to the transformers-native GDN
implementation wrapped by `mcore-bridge`; combined with ms-swift's built-in
Qwen3.5 FLA NPU patch, `chunk_gated_delta_rule` is then redirected to
MindSpeed's Triton kernels.
2. At the moment, do not enable sequence-related features on the Qwen3.5 NPU
path with either the transformers backend or the Megatron-SWIFT backend,
regardless of whether `USE_MCORE_GDN=1` or `USE_MCORE_GDN=0` is used under
Megatron-SWIFT. This includes sequence parallel (SP), context parallel (CP),
and packing/padding-free. The related FLA Triton operators do not yet have
complete native NPU support. Enabling these features may hit
missing-operator paths, incomplete sample-boundary handling, or mismatched
parallel partitioning.
3. For the transformers backend, keep `--sequence_parallel_size` at `1` and
avoid `--packing true` / `--padding_free true`. For the Megatron-SWIFT
backend, keep `--context_parallel_size`
at `1`, and also avoid `--packing true` / `--padding_free true`. Re-enable
these features only after the target MindSpeed/FLA stack explicitly fills
this support and the combination has been validated in layers.
### Environment Viewing
Check the P2P connections of the NPU, where we can see that each NPU is interconnected through 7 HCCS links with other NPUs.
```shell
(valle) root@valle:~/src# npu-smi info -t topo
NPU0 NPU1 NPU2 NPU3 NPU4 NPU5 NPU6 NPU7 CPU Affinity
NPU0 X HCCS HCCS HCCS HCCS HCCS HCCS HCCS 144-167
NPU1 HCCS X HCCS HCCS HCCS HCCS HCCS HCCS 144-167
NPU2 HCCS HCCS X HCCS HCCS HCCS HCCS HCCS 96-119
NPU3 HCCS HCCS HCCS X HCCS HCCS HCCS HCCS 96-119
NPU4 HCCS HCCS HCCS HCCS X HCCS HCCS HCCS 0-23
NPU5 HCCS HCCS HCCS HCCS HCCS X HCCS HCCS 0-23
NPU6 HCCS HCCS HCCS HCCS HCCS HCCS X HCCS 48-71
NPU7 HCCS HCCS HCCS HCCS HCCS HCCS HCCS X 48-71
Legend:
X = Self
SYS = Path traversing PCIe and NUMA nodes. Nodes are connected through SMP, such as QPI, UPI.
PHB = Path traversing PCIe and the PCIe host bridge of a CPU.
PIX = Path traversing a single PCIe switch
PXB = Path traversing multiple PCIe switches
HCCS = Connection traversing HCCS.
NA = Unknown relationship.
```
Check NPU status. For details about the `npu-smi` command, see the [official documentation](https://support.huawei.com/enterprise/en/doc/EDOC1100079287/10dcd668).
```shell
(valle) root@valle:~/src# npu-smi info
+------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc1.b030 Version: 24.1.rc1.b030 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B3 | OK | 101.8 43 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 3318 / 65536 |
+===========================+===============+====================================================+
| 1 910B3 | OK | 92.0 39 0 / 0 |
| 0 | 0000:C2:00.0 | 0 0 / 0 3314 / 65536 |
+===========================+===============+====================================================+
| 2 910B3 | OK | 102.0 40 0 / 0 |
| 0 | 0000:81:00.0 | 0 0 / 0 3314 / 65536 |
+===========================+===============+====================================================+
| 3 910B3 | OK | 99.8 40 0 / 0 |
| 0 | 0000:82:00.0 | 0 0 / 0 3314 / 65536 |
+===========================+===============+====================================================+
| 4 910B3 | OK | 98.6 45 0 / 0 |
| 0 | 0000:01:00.0 | 0 0 / 0 3314 / 65536 |
+===========================+===============+====================================================+
| 5 910B3 | OK | 99.7 44 0 / 0 |
| 0 | 0000:02:00.0 | 0 0 / 0 3314 / 65536 |
+===========================+===============+====================================================+
| 6 910B3 | OK | 103.8 45 0 / 0 |
| 0 | 0000:41:00.0 | 0 0 / 0 3314 / 65536 |
+===========================+===============+====================================================+
| 7 910B3 | OK | 98.2 44 0 / 0 |
| 0 | 0000:42:00.0 | 0 0 / 0 3315 / 65536 |
+===========================+===============+====================================================+
```
## Quick Start: ModelScope Model + Dataset
If you want to quickly verify the environment with a ModelScope model and dataset, you can use this section to complete the full flow: train LoRA, find the latest checkpoint, Merge LoRA, run CLI inference, start a service, and validate it with curl. The example uses a small model and a small data sample so that it can run quickly. To use your own model or dataset, modify the ID variables at the beginning.
```shell
export MODEL_ID=Qwen/Qwen3-0.6B
export DATASET_ID=AI-ModelScope/alpaca-gpt4-data-zh
export WORK_DIR=output/npu-modelscope-qwen3-0_6b-lora
```
Train and save a LoRA checkpoint:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 \
swift sft \
--model $MODEL_ID \
--dataset $DATASET_ID#1000 \
--split_dataset_ratio 0.01 \
--torch_dtype bfloat16 \
--tuner_type lora \
--target_modules all-linear \
--lora_rank 8 \
--lora_alpha 32 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 8 \
--learning_rate 1e-4 \
--max_length 2048 \
--save_steps 100 \
--eval_steps 100 \
--save_total_limit 2 \
--logging_steps 10 \
--output_dir $WORK_DIR
```
After training finishes, checkpoints are saved under `$WORK_DIR/*/checkpoint-*`. Use the following commands to select the latest checkpoint and merge LoRA into a full model:
```shell
export CKPT_DIR=$(ls -dt $WORK_DIR/*/checkpoint-* | head -n 1)
ASCEND_RT_VISIBLE_DEVICES=0 \
swift export \
--adapters $CKPT_DIR \
--merge_lora true
export MERGED_DIR=${CKPT_DIR}-merged
```
You can verify inference either with the LoRA checkpoint directly or with the merged full model:
```shell
# Load the LoRA checkpoint directly
ASCEND_RT_VISIBLE_DEVICES=0 \
swift infer \
--adapters $CKPT_DIR \
--stream true \
--temperature 0 \
--max_new_tokens 512
# Load the merged full model
ASCEND_RT_VISIBLE_DEVICES=0 \
swift infer \
--model $MERGED_DIR \
--stream true \
--temperature 0 \
--max_new_tokens 512
```
To start an OpenAI-compatible deployment service, use the merged full model:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 \
swift deploy \
--model $MERGED_DIR \
--host 0.0.0.0 \
--port 8000 \
--max_new_tokens 512 \
--served_model_name npu-modelscope-qwen3-0_6b
```
After the service starts, validate the API with curl:
```shell
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "npu-modelscope-qwen3-0_6b",
"messages": [{"role": "user", "content": "Briefly introduce Ascend NPU in one sentence."}],
"max_tokens": 128,
"temperature": 0
}'
```
## Training
The following examples introduce LoRA fine-tuning. For full-parameter fine-tuning, set `--tuner_type full`. For **more training scripts**, see [examples/ascend/train](https://github.com/modelscope/ms-swift/tree/main/examples/ascend/train). For general pre-training, SFT, LoRA, full-parameter training, and custom dataset usage, see [Pre-training and Fine-tuning](../Instruction/Pre-training-and-Fine-tuning.md).
| Model size | NPU count | DeepSpeed type | Max memory usage |
| ---------- | --------- | -------------- | ---------------- |
| 7B | 1 | None | 1 * 28 GB |
| 7B | 4 | None | 4 * 22 GB |
| 7B | 4 | zero2 | 4 * 28 GB |
| 7B | 4 | zero3 | 4 * 22 GB |
| 7B | 8 | None | 8 * 22 GB |
| 14B | 1 | None | 1 * 45 GB |
| 14B | 8 | None | 8 * 51 GB |
| 14B | 8 | zero2 | 8 * 49 GB |
| 14B | 8 | zero3 | 8 * 31 GB |
### Single-Card Training
Start single-card fine-tuning with the following command:
```shell
# Experiment environment: Ascend 910B3
# Memory requirement: 28 GB
# Runtime: 8 hours
ASCEND_RT_VISIBLE_DEVICES=0 \
swift sft \
--model Qwen/Qwen2-7B-Instruct \
--dataset AI-ModelScope/blossom-math-v2 \
--split_dataset_ratio 0.01 \
--num_train_epochs 5 \
--tuner_type lora \
--output_dir output \
--learning_rate 1e-4 \
--gradient_accumulation_steps 16 \
--save_steps 100 \
--eval_steps 100
```
### Data Parallel Training
The following example uses 4 NPUs for DDP training.
```shell
# Experiment environment: 4 * Ascend 910B3
# Memory requirement: 4 * 22 GB
# Runtime: 2 hours
NPROC_PER_NODE=4 \
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model Qwen/Qwen2-7B-Instruct \
--dataset AI-ModelScope/blossom-math-v2 \
--split_dataset_ratio 0.01 \
--num_train_epochs 5 \
--tuner_type lora \
--output_dir output \
...
```
### DeepSpeed Training
ZeRO2:
```shell
# Experiment environment: 4 * Ascend 910B3
# Memory requirement: 4 * 28GB
# Runtime: 3.5 hours
NPROC_PER_NODE=4 \
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model Qwen/Qwen2-7B-Instruct \
--dataset AI-ModelScope/blossom-math-v2 \
--split_dataset_ratio 0.01 \
--num_train_epochs 5 \
--tuner_type lora \
--output_dir output \
--deepspeed zero2 \
...
```
ZeRO3:
```shell
# Experiment environment: 4 * Ascend 910B3
# Memory requirement: 4 * 22 GB
# Runtime: 8.5 hours
NPROC_PER_NODE=4 \
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model Qwen/Qwen2-7B-Instruct \
--dataset AI-ModelScope/blossom-math-v2 \
--split_dataset_ratio 0.01 \
--num_train_epochs 5 \
--tuner_type lora \
--output_dir output \
--deepspeed zero3 \
...
```
### Qwen3.5 Single-Node Multi-Card LoRA Example
The following is an NPU LoRA example for a newer model. It uses Qwen3.5-4B for demonstration. Four-card data parallelism is usually faster than single-card training. If you already have local model and dataset files, replace `--model` and `--dataset` with local paths.
```shell
# Experiment environment: 4 * Ascend 910B3
NPROC_PER_NODE=4 \
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model Qwen/Qwen3.5-4B \
--dataset AI-ModelScope/alpaca-gpt4-data-zh#2000 \
--split_dataset_ratio 0.01 \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--tuner_type lora \
--target_modules all-linear \
--lora_rank 8 \
--lora_alpha 32 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 8 \
--learning_rate 1e-4 \
--max_length 2048 \
--group_by_length true \
--dataset_num_proc 4 \
--dataloader_num_workers 4 \
--save_steps 100 \
--eval_steps 100 \
--save_total_limit 2 \
--output_dir output/Qwen3.5-4B-NPU
```
When tuning parameters, focus on memory, throughput, and stability:
- Reduce memory usage: first reduce `--max_length`, `--per_device_train_batch_size`, and `--lora_rank`; if OOM still occurs, enable `--deepspeed zero2/zero3`. ZeRO can significantly reduce memory pressure but introduces communication and scheduling overhead.
- Improve throughput: increase `--per_device_train_batch_size` when memory allows, and use `--gradient_accumulation_steps` to keep the global batch size. Increase `--dataset_num_proc` if preprocessing is slow, and increase `--dataloader_num_workers` if data loading is the bottleneck.
- Control save overhead: do not set `--save_steps` too small, because frequent checkpoint saving slows down training. `--save_total_limit 2` is usually enough to keep the best checkpoint and the last checkpoint.
- Improve stability: on NPUs, prefer `bfloat16`. If you see abnormal loss or NaN, first lower the learning rate and batch size; if necessary, temporarily switch to `float32` for comparison.
For more parameter details, see [Command-line Parameters](../Instruction/Command-line-parameters.md).
### NPU Model Patch Switch
ms-swift enables model-level patches by default in NPU environments to adapt some Transformers models to Ascend NPU operators and compatibility requirements. You usually do not need to disable them. If you suspect abnormal loss or forward errors are related to the NPU model patch and want to compare against native Transformers behavior, set:
```shell
swift sft ... --enable_npu_model_patch false
```
## Model Saving, Merge LoRA, and Resume Training
Use `--output_dir` to set the output directory, `--save_steps` to control checkpoint save intervals, and `--save_total_limit` to control how many checkpoints to keep. After LoRA training, the checkpoint directory contains adapter weights, training arguments, and trainer state. A typical directory layout is:
```text
output/Qwen3.5-4B-NPU/vx-xxx/
├── checkpoint-100/
├── checkpoint-200/
└── ...
```
If you only need inference or want to continue LoRA training, use the checkpoint directory directly. If you want an independent full-model directory for vLLM-Ascend deployment, offline distribution, or later quantization, run Merge LoRA:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 \
swift export \
--adapters output/Qwen3.5-4B-NPU/vx-xxx/checkpoint-xxx \
--merge_lora true
```
The merged model is saved under `checkpoint-xxx-merged` by default. You can then load it like any regular model with `--model checkpoint-xxx-merged`.
If training is interrupted and you need to resume from a checkpoint, keep the original training arguments unchanged and add `--resume_from_checkpoint`:
```shell
NPROC_PER_NODE=4 \
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model Qwen/Qwen3.5-4B \
--dataset AI-ModelScope/alpaca-gpt4-data-zh#2000 \
--tuner_type lora \
--output_dir output/Qwen3.5-4B-NPU \
--resume_from_checkpoint output/Qwen3.5-4B-NPU/vx-xxx/checkpoint-xxx \
...
```
`--resume_from_checkpoint` restores model weights, optimizer state, random seeds, and training progress. If you only want to load model weights without restoring the optimizer state or data skipping state, also set `--resume_only_model true`. For details, see `resume_from_checkpoint`, `resume_only_model`, `save_steps`, and `save_total_limit` in [Command-line Parameters](../Instruction/Command-line-parameters.md).
## Inference
Original model:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 swift infer \
--model Qwen/Qwen2-7B-Instruct \
--stream true --max_new_tokens 2048
```
After LoRA fine-tuning:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 swift infer \
--adapters xxx/checkpoint-xxx --load_data_args true \
--stream true --max_new_tokens 2048
```
For full-parameter training or a merged LoRA model, point `--model` to the full-weight directory:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 swift infer \
--model xxx/checkpoint-xxx-merged \
--stream true --max_new_tokens 2048
```
## Deployment
### Deployment with Native Transformers
Original model:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 swift deploy --model Qwen/Qwen2-7B-Instruct --max_new_tokens 2048
```
After LoRA fine-tuning:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 swift deploy --adapters xxx/checkpoint-xxx --max_new_tokens 2048
# Deploy the full weights after Merge LoRA
ASCEND_RT_VISIBLE_DEVICES=0 swift export --adapters xx/checkpoint-xxx --merge_lora true
ASCEND_RT_VISIBLE_DEVICES=0 swift deploy --model xxx/checkpoint-xxx-merged --max_new_tokens 2048
```
### Deployment with vLLM-Ascend
Install from PyPI:
```shell
# Refer to the official vLLM-Ascend compatibility matrix; the following versions are verified in this document.
pip install vllm==0.14.0
pip install vllm-ascend==0.14.0rc1
```
Original model:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 swift deploy \
--model Qwen/Qwen2.5-7B-Instruct \
--infer_backend vllm \
--max_new_tokens 2048
```
After LoRA fine-tuning:
```shell
ASCEND_RT_VISIBLE_DEVICES=0 swift deploy \
--adapters xxx/checkpoint-xxx \
--infer_backend vllm \
--max_new_tokens 2048
# Deploy the full weights after Merge LoRA
ASCEND_RT_VISIBLE_DEVICES=0 swift export \
--adapters xx/checkpoint-xxx \
--merge_lora true
ASCEND_RT_VISIBLE_DEVICES=0 swift deploy \
--model xxx/checkpoint-xxx-merged \
--infer_backend vllm \
--max_new_tokens 2048
```
## Evaluation
After training, inference, or deployment, you can evaluate the original model or fine-tuned checkpoint with SWIFT's built-in EvalScope integration. For complete arguments and examples, see [Evaluation](../Instruction/Evaluation.md).
## Release
If you need to publish NPU-trained checkpoints, merged models, or quantized models to ModelScope/HuggingFace, use the push capability in `swift export`. For complete arguments and examples, see [Export and Push](../Instruction/Export-and-push.md#push-models).
## FAQ
For general questions, see [Frequently Asked Questions](../Instruction/Frequently-asked-questions.md). This section records common NPU-specific issues and troubleshooting steps.
### Q1: How do I confirm that the current environment detects NPUs correctly?
First confirm that you have run `source /usr/local/Ascend/ascend-toolkit/set_env.sh`, then run the environment check script in this document. Normally, `is_torch_npu_available()` should return `True`, `torch.npu.device_count()` should show the number of available NPUs, and you should be able to create a tensor on `npu:0`. If this fails, first check whether CANN, `torch`, and `torch_npu` match the recommended versions in this document.
### Q2: How should I choose between FSDP, DeepSpeed, and Megatron-SWIFT?
For ordinary SFT, first refer to the verified `FSDP1/FSDP2/deepspeed` combinations in this document. For larger models or higher parallelism requirements, use Megatron-SWIFT and install MindSpeed, Megatron-LM, and mcore-bridge as described in the installation section. DeepSpeed can reduce memory pressure but may reduce speed, so compare it with FSDP when tuning performance.
### Q3: Do I need to manually disable the NPU model patch?
Usually no. ms-swift enables model-level patches by default in NPU environments to adapt some Transformers models to Ascend NPU operators and compatibility requirements. Only when troubleshooting abnormal loss or forward errors, and when you suspect the issue is related to the NPU patch, should you temporarily set `--enable_npu_model_patch false` and compare against native Transformers behavior.
### Q4: What should I know when using vLLM-Ascend for deployment or RL rollout?
Install the `vllm` and `vllm-ascend` versions recommended in this document, and prioritize model and algorithm combinations that have been verified in the compatibility tables. The `sglang` inference engine is not yet fully verified for NPU scenarios. For high-performance inference or RL rollout on NPUs, prefer `vllm-ascend`.
### Q5: What happens if I forget to run `source set_env.sh`?
Common symptoms include `is_torch_npu_available()` returning `False`, `torch.npu.device_count()` returning 0, or runtime errors about missing CANN/HCCL shared libraries. After entering a new shell or container, run:
```shell
source /usr/local/Ascend/ascend-toolkit/set_env.sh
```
If NNAL/ATB or other components are installed, source their corresponding `set_env.sh` files according to your environment.
### Q6: How do I diagnose a `torch` and `torch_npu` version mismatch?
Install according to the recommended versions in this document. When versions mismatch, common symptoms include `import torch_npu` failures, invisible NPU devices, operator registration failures, and C++ symbol/runtime errors. Check versions with:
```shell
python -c "import torch, torch_npu; print(torch.__version__); print(torch_npu.__version__)"
```
If versions do not match, uninstall and reinstall the full CANN/PyTorch/torch_npu stack consistently. Do not upgrade only one package.
### Q7: What happens if `ASCEND_RT_VISIBLE_DEVICES` and `NPROC_PER_NODE` do not match?
For distributed training, they should match. For example, `ASCEND_RT_VISIBLE_DEVICES=0,1,2,3` usually corresponds to `NPROC_PER_NODE=4`. If the process count is larger than the number of visible devices, ranks may fail to bind devices, multiple processes may compete for the same device, initialization may hang, or HCCL may report errors. If the process count is smaller, only part of the visible NPUs will be used.
### Q8: What should I check first when multi-card training hangs?
First confirm that every rank has started and that `ASCEND_RT_VISIBLE_DEVICES` matches `NPROC_PER_NODE`. Then identify whether logs stop during data preprocessing, model construction, weight loading, or HCCL initialization. For NPU/HCCL low-level logs, check:
```shell
ls ~/ascend/log/debug/plog
```
If the Python process is still running but produces no output for a long time, use `pystack` to inspect the stack of each rank and determine whether it is stuck in data loading, communication, or model forward/backward.
### Q9: How do I initially troubleshoot HCCL connection or timeout issues?
Use `npu-smi info` and `npu-smi info -t topo` to check device health and topology, then check whether other jobs are occupying the same NPU group. For single-node training, first verify card IDs, process count, and visible devices. For multi-node training, also verify networking, rank configuration, communication ports, and environment variables on all nodes. If old training processes remain on the same machine, clean up the corresponding training processes and retry.
### Q10: Why is `npu-smi` unavailable inside the container?
Usually some device or driver files were not mounted. Check whether `docker run` includes `/dev/davinci*`, `/dev/davinci_manager`, `/dev/devmm_svm`, `/dev/hisi_hdc`, `/usr/local/Ascend/driver`, `/usr/local/Ascend/firmware`, `/usr/local/sbin/npu-smi`, and `/etc/ascend_install.info`. If `npu-smi info` fails on the host itself, fix the host driver environment first.
### Q11: How should I choose between native transformers deployment and vLLM-Ascend deployment?
Native transformers deployment has better compatibility and is suitable for validating whether the model, adapter, template, and output are correct. vLLM-Ascend is better for high-throughput services, RL rollout, or OpenAI-compatible serving. If you encounter vLLM-Ascend version or operator issues, first confirm that the model itself works with the transformers backend, then switch to vLLM-Ascend to troubleshoot the performance backend.
### Q12: What should I do if vLLM-Ascend reports device type mismatch or undefined symbol?
This is usually not caused by training script arguments. It often means the `vllm-ascend` wheel does not match the current hardware, PyTorch version, or C++ ABI. First check package build information and current versions:
```shell
python -c "import torch, vllm_ascend; print(torch.__version__); print(vllm_ascend.__file__)"
```
If the error message contains `Current device type ... does not match the installed version's device type ...`, `undefined symbol`, or similar text, reinstall `torch`, `torch_npu`, `vllm`, and `vllm-ascend` according to the device type (A2/A3/other) and the official compatibility matrix. Do not replace only one package.
### Q13: Can FP8 or quantized models be trained directly on NPUs?
Do not assume they can. Before downloading or loading a large model, check whether `config.json` contains `quantization_config`, and check the actual dtype in safetensors. Quantization/QLoRA is still listed as not supported or not fully verified in the NPU support scope. If model weights are FP8 block quantized and your NPU software stack does not support that FP8 path, use BF16 weights first, or convert the model offline to BF16 before training/loading.
### Q14: How do I troubleshoot Megatron-SWIFT importing the wrong Megatron/MindSpeed?
Before running Megatron-SWIFT, `PYTHONPATH` and `MEGATRON_LM_PATH` must point to the same Megatron-LM source tree. Otherwise Python may start successfully while importing a different Megatron/MindSpeed combination, which can make later errors look like model or argument issues.
```shell
export PYTHONPATH=$PYTHONPATH:<your_local_megatron_lm_path>
export MEGATRON_LM_PATH=<your_local_megatron_lm_path>
python -c "import megatron, os; print(megatron.__file__); print(os.environ.get('MEGATRON_LM_PATH'))"
```
If they do not match, fix the environment variables before continuing with model construction, weight loading, or parallel configuration troubleshooting.
## NPU WeChat Group
<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/npu.png" width="250">
docs/source_en/BestPractices/Qwen3-Best-Practice.md
+400
View File
@@ -0,0 +1,400 @@
# Qwen3 Best Practices
Discussion: [issue 4030](https://github.com/modelscope/ms-swift/issues/4030)
Qwen Documentation: [https://qwen.readthedocs.io/en/latest/training/ms_swift.html](https://qwen.readthedocs.io/en/latest/training/ms_swift.html)
## Inference
Thinking mode:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model Qwen/Qwen3-8B \
--infer_backend vllm \
--stream true \
--max_new_tokens 2048 \
--vllm_max_model_len 8192
```
```text
<<< who are you?
<think>
Okay, the user is asking "who are you?" Let me start by introducing myself as Qwen, the large language model developed by Alibaba Cloud. I should mention my capabilities, like answering questions, creating content, and engaging in conversations. But I need to keep it concise. Also, the user might want to know how I can assist them. Maybe I should ask how I can help them today. Let me check if there's anything else important to include. Oh, I should make sure the tone is friendly and approachable. Alright, that should cover it.
</think>
Hello! I am Qwen, a large language model developed by Alibaba Cloud. I can assist with a wide range of tasks, such as answering questions, creating content, writing stories, coding, and more. How can I help you today? 😊
<<< clear
<<< who are you? /no_think
<think>
</think>
I am Qwen, a large language model developed by Alibaba Cloud. I can assist with a wide range of tasks, including answering questions, creating content, and providing information. How can I help you today?
```
Non-thinking mode:
- `--response_prefix` indicates that the model's output will continue after the prefix. It is equivalent to setting enable_thinking to False.
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model Qwen/Qwen3-8B \
--infer_backend vllm \
--stream true \
--max_new_tokens 2048 \
--vllm_max_model_len 8192 \
--response_prefix '<think>\n\n</think>\n\n'
```
```text
<<< who are you?
<think>
</think>
I am Qwen, a large-scale language model developed by Alibaba Cloud. I am designed to assist with a wide range of tasks, including answering questions, creating content, and providing information. How can I assist you today?
```
## Training
Before starting training, please ensure that your environment is properly configured.
```bash
pip install ms-swift -U
pip install transformers
pip install deepspeed # for multi-GPU training
pip install liger-kernel # to save GPU memory resources
pip install flash-attn --no-build-isolation # required for packing
```
## Supervised Fine-Tuning (SFT)
### Data Preparation
When using ms-swift for SFT, the custom dataset format is as follows (the `system` field is optional). You can organize it in JSON, JSONL, or CSV format. Specify `--dataset <dataset_path>` in the training script. For a complete guide on dataset formats, refer to the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
```text
# General format
{"messages": [
{"role": "system", "content": "<system-prompt>"},
{"role": "user", "content": "<query1>"},
{"role": "assistant", "content": "<response1>"}
]}
# Format with thinking process
{"messages": [
{"role": "user", "content": "Where is the capital of Zhejiang?"},
{"role": "assistant", "content": "Thought: ...\n\nAnswer:\nThe capital of Zhejiang is Hangzhou."}
]}
```
If you want to train using data without the thinking chain while preserving the model's reasoning ability, you can use one of the following methods to minimize the impact of fine-tuning:
**Option 1**: [Recommended] During training, specify `--loss_scale ignore_empty_think`, which will ignore the loss calculation for `'<think>\n\n</think>\n\n'`, thus avoiding the loss of reasoning capability. The training script can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo1.sh). This method also works for models like DeepSeek-R1. The custom dataset format is as follows:
```json
{"messages": [
{"role": "user", "content": "Where is the capital of Zhejiang?"},
{"role": "assistant", "content": "<think>\n\n</think>\n\nThe capital of Zhejiang is Hangzhou."}
]}
```
**Option 2**: Add `/no_think` to the query in the dataset to avoid losing the reasoning capability. The training script can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo2.sh). The custom dataset format is as follows:
```json
{"messages": [
{"role": "user", "content": "Where is the capital of Zhejiang? /no_think"},
{"role": "assistant", "content": "<think>\n\n</think>\n\nThe capital of Zhejiang is Hangzhou."}
]}
```
You can use the following command to obtain a distilled reasoning dataset. During training, you can mix it with datasets that do not contain chain-of-thought (CoT) data to further mitigate the loss of reasoning ability:
- The choice of `--val_dataset` is arbitrary. The reasoning results saved to `result_path` can be specified directly in training via `--dataset distill_dataset.jsonl`.
- This approach is also applicable to other reasoning models, such as deepseek-r1.
```shell
# 4 * 80GiB
NPROC_PER_NODE=4 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift infer \
--model Qwen/Qwen3-32B \
--infer_backend vllm \
--val_dataset 'AI-ModelScope/alpaca-gpt4-data-en#5000' 'AI-ModelScope/alpaca-gpt4-data-zh#5000' \
--vllm_gpu_memory_utilization 0.9 \
--vllm_tensor_parallel_size 2 \
--vllm_max_model_len 8192 \
--max_new_tokens 4096 \
--write_batch_size 1000 \
--result_path distill_dataset.jsonl
```
### 30-Minute Self-Awareness Fine-Tuning
This section demonstrates how to perform self-awareness fine-tuning on Qwen3-8B within 30 minutes. A GPU with at least 22GB of VRAM is required and can run on the free computing resources provided by ModelScope, such as the A10 instance.
After training, the model will no longer identify itself as a "Qwen" trained by "Tongyi Lab," but rather as a "swift-robot" trained by "swift."
If you need to train in an offline environment, you can manually download the model and dataset, and specify `--model <model-path>` and `--dataset <dataset-dir>`. The dataset is available on the [ModelScope Hub](https://modelscope.cn/datasets/swift/self-cognition). You can view the preprocessing function for the `swift/self-cognition` dataset [here](https://github.com/modelscope/ms-swift/blob/36fdf381e5e88cb8a71c9d69c1d8936a989318cc/swift/llm/dataset/dataset/llm.py#L882).
For explanations of the parameters used in the training script, please refer to the [Command Line Arguments Documentation](../Instruction/Command-line-parameters.md).
```bash
# GPU Memory Usage: 22GB
CUDA_VISIBLE_DEVICES=0 \
swift sft \
--model Qwen/Qwen3-8B \
--tuner_type lora \
--dataset 'swift/Qwen3-SFT-Mixin#2000' \
'swift/self-cognition:qwen3#600' \
--load_from_cache_file true \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 1e-4 \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--gradient_accumulation_steps 16 \
--eval_steps 50 \
--save_steps 50 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 2048 \
--output_dir output \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--model_author swift \
--model_name swift-robot
```
After fine-tuning, you can test the results using the following script. Note that the `--adapters` part should be modified to point to the final saved checkpoint directory:
```bash
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--stream true \
--temperature 0 \
--max_new_tokens 2048
```
```text
<<< who are you?
<think>
Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response.
</think>
Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need!
```
By default, ms-swift uses the ModelScope community to download models and datasets. If you want to use the HuggingFace community instead, you need to additionally specify `--use_hf true`.
Merge LoRA weights:
```shell
swift export \
--adapters output/checkpoint-xxx \
--merge_lora true
```
Push the model to ModelScope/HuggingFace:
```shell
# If pushing full weights, change `--adapters` to `--model`.
# You can find your ModelScope hub_token here: https://modelscope.cn/my/myaccesstoken
swift export \
--adapters output/checkpoint-xxx \
--push_to_hub true \
--hub_model_id '<hub-model-id>' \
--hub_token '<hub-token>' \
--use_hf false
```
If you want to perform training on multiple GPUs, the following example provides a multi-GPU training setup:
```shell
# 4 * 60GB
# You can run the experiment by setting `--dataset AI-ModelScope/alpaca-gpt4-data-en`
# Note: If you specify `--packing true`, you must also set `--attn_impl flash_attn`
NPROC_PER_NODE=4 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model Qwen/Qwen3-8B \
--tuner_type full \
--dataset '<your-dataset>' \
--load_from_cache_file true \
--split_dataset_ratio 0.01 \
--torch_dtype bfloat16 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 1e-5 \
--gradient_accumulation_steps 4 \
--packing true \
--eval_steps 100 \
--save_steps 100 \
--logging_steps 5 \
--max_length 8192 \
--warmup_ratio 0.05 \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--save_total_limit 2 \
--save_only_model true \
--output_dir output \
--deepspeed zero3 \
--use_liger_kernel true \
--attn_impl flash_attn
```
## Reinforcement Learning (RL)
ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and GKD. This section will focus on using ms-swift for GRPO training on Qwen3-8B. For more information about GRPO, refer to the [GRPO documentation](../Instruction/GRPO/GetStarted/GRPO.md). Additional RLHF training scripts can be found in [examples/train/rlhf](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf).
### Environment Setup
In addition to installing the dependencies related to ms-swift mentioned above, you also need to install the following:
```shell
pip install "math_verify"
pip install vllm==0.8.5.post1
```
### Data Preparation
The dataset format used for GRPO training with ms-swift is similar to that of SFT, but it does not require the final assistant's response part. If accuracy is used as the reward, an additional `solution` column is required to calculate accuracy.
Example dataset format:
```jsonl
{"messages": [{"role": "user", "content": "Tell me tomorrow's weather"}]}
{"messages": [{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}]}
{"messages": [{"role": "user", "content": "What is your name?"}]}
```
For data preparation for other RLHF algorithms, please refer to the [Custom Dataset Documentation](../Customization/Custom-dataset.md#rlhf).
Notes on dataset requirements:
- **Reward Function Calculation**: The dataset format depends on the reward function being used. Additional columns may be needed to support specific reward calculations. For example:
- When using built-in `accuracy` or `cosine` rewards, the dataset must include a `solution` column to calculate the accuracy of responses.
- Other columns in the dataset will be passed as `**kwargs` to the reward function for further customization.
- **Custom Reward Functions**: To customize the reward function according to your specific needs, refer to: [External Reward Plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin). This plugin provides examples and templates for implementing custom reward functions.
We use AI-MO/NuminaMath-TIR as the dataset and compute the accuracy-based reward for model responses.
During training, we utilize vLLM to accelerate the sampling process.
```bash
# 70G*8
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen3-8B \
--tuner_type full \
--dataset 'AI-MO/NuminaMath-TIR#5000' \
--load_from_cache_file true \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 2 \
--learning_rate 1e-6 \
--save_total_limit 2 \
--logging_steps 5 \
--output_dir output \
--gradient_accumulation_steps 1 \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--max_length 4096 \
--max_completion_length 4096 \
--vllm_max_model_len 8192 \
--reward_funcs accuracy \
--num_generations 16 \
--use_vllm true \
--vllm_gpu_memory_utilization 0.4 \
--sleep_level 1 \
--offload_model true \
--offload_optimizer true \
--deepspeed zero3 \
--vllm_tensor_parallel_size 1 \
--temperature 1.0 \
--top_p 0.85 \
--log_completions true \
--overlong_filter true
```
## Megatron-SWIFT
Best practice reference for single-node 8xH20 LoRA training with Qwen3-235B-A22B-Instruct-250718: https://github.com/modelscope/ms-swift/pull/5033.
ms-swift introduces Megatron parallelism techniques to accelerate CPT/SFT/DPO/GRPO for large models. Supported models can be found in the [Supported Models and Datasets Document](../Instruction/Supported-models-and-datasets.md).
For environment setup, refer to the [Megatron-SWIFT Training Documentation](../Megatron-SWIFT/Quick-start.md).
We will use Alibaba Cloud DLC to launch training. The training environment consists of two nodes equipped with 8x 80GiB A800 GPUs each. For more information on multi-node launching, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node).
```bash
# https://help.aliyun.com/zh/pai/user-guide/general-environment-variables
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NNODES=$WORLD_SIZE \
NODE_RANK=$RANK \
megatron sft \
--model Qwen/Qwen3-30B-A3B-Base \
--save_safetensors true \
--dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \
--load_from_cache_file true \
--split_dataset_ratio 0.01 \
--pipeline_model_parallel_size 2 \
--expert_model_parallel_size 8 \
--moe_permute_fusion true \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-3 \
--micro_batch_size 1 \
--global_batch_size 16 \
--packing true \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--train_iters 2000 \
--eval_iters 50 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-6 \
--output_dir megatron_output/Qwen3-30B-A3B-Base \
--eval_steps 200 \
--save_steps 200 \
--max_length 8192 \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim true \
--no_save_rng true \
--sequence_parallel true \
--attention_backend flash
```
Training loss chart (partial):
<img width="910" alt="Image" src="https://github.com/user-attachments/assets/9fe393aa-8299-4659-aa2f-be5d44f0730b" />
Effect screenshot:
<img width="1066" alt="Image" src="https://github.com/user-attachments/assets/1a924130-1954-43e9-9093-b019aeef5949" />
The custom dataset format is the same as that used in `swift sft`. For details, see the previous sections. Simply specify `--dataset <dataset_path>`.
A comparison of training speed and GPU memory usage when performing full-parameter fine-tuning of the Qwen3-30B-A3B model using `megatron sft` and `swift sft` is shown below:
| | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 |
| -------- | ----------- | --------------- | --------------- |
| 训练速度 | 9.6s/it | - | 91.2s/it |
| 显存使用 | 16 * 60GiB | OOM | 16 * 80GiB |
docs/source_en/BestPractices/Qwen3-VL-Best-Practice.md
+327
View File
@@ -0,0 +1,327 @@
# Qwen3-VL Best Practices
## Environment Setup
Before starting inference and training, please ensure your environment is properly configured.
```shell
pip install "transformers>=4.57" "qwen_vl_utils>=0.0.14"
pip install "ms-swift>=4.0"
# pip install "vllm>=0.11.0" # If using the vLLM inference backend for inference
```
- About video data training hangs: Using the decord backend for video reading may cause the training process to hang, see [this issue](https://github.com/dmlc/decord/issues/269). You can use the torchcodec backend instead. For details, refer to the [qwen_vl_utils](https://github.com/QwenLM/Qwen3-VL/blob/50068df2334f309979ff05d75f1078c8309c63ed/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L390-L400) library.
## Inference
Inference using transformers:
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from modelscope import snapshot_download
from qwen_vl_utils import process_vision_info
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
model_dir = snapshot_download('Qwen/Qwen3-VL-4B-Instruct')
model = Qwen3VLForConditionalGeneration.from_pretrained(
model_dir, dtype="auto", device_map="auto",
# attn_implementation='flash_attention_2',
)
processor = AutoProcessor.from_pretrained(model_dir)
messages = [
{
"role": "user",
"content": [
{
"type": "video",
"video": "https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4",
"max_pixels": 128*32*32,
"max_frames": 16,
},
{"type": "text", "text": "Describe this video."},
],
}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs, video_kwargs = process_vision_info([messages], return_video_kwargs=True,
image_patch_size= 16,
return_video_metadata=True)
if video_inputs is not None:
video_inputs, video_metadatas = zip(*video_inputs)
video_inputs, video_metadatas = list(video_inputs), list(video_metadatas)
else:
video_metadatas = None
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, video_metadata=video_metadatas, **video_kwargs, do_resize=False, return_tensors="pt")
inputs = inputs.to('cuda')
generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
generated_ids_trimmed = [
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])
# 'A baby wearing glasses sits on a bed, engrossed in reading a book. The baby turns the pages with both hands, occasionally looking up and smiling. The room is cozy, with a crib in the background and clothes scattered around. The baby's focus and curiosity are evident as they explore the book, creating a heartwarming scene of early learning and discovery.'
```
Inference using ms-swift's TransformersEngine:
```python
import os
# os.environ['SWIFT_DEBUG'] = '1'
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['VIDEO_MAX_TOKEN_NUM'] = '128'
os.environ['FPS_MAX_FRAMES'] = '16'
from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig
engine = TransformersEngine('Qwen/Qwen3-VL-4B-Instruct') # attn_impl='flash_attention_2'
infer_request = InferRequest(messages=[{
"role": "user",
"content": '<video>Describe this video.',
}], videos=['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4'])
request_config = RequestConfig(max_tokens=128, temperature=0)
resp_list = engine.infer([infer_request], request_config=request_config)
response = resp_list[0].choices[0].message.content
# 'A baby wearing glasses sits on a bed, engrossed in reading a book. The baby turns the pages with both hands, occasionally looking up and smiling. The room is cozy, with a crib in the background and clothes scattered around. The baby's focus and curiosity are evident as they explore the book, creating a heartwarming scene of early learning and discovery.'
# use stream
request_config = RequestConfig(max_tokens=128, temperature=0, stream=True)
gen_list = engine.infer([infer_request], request_config=request_config)
for chunk in gen_list[0]:
if chunk is None:
continue
print(chunk.choices[0].delta.content, end='', flush=True)
print()
```
Inference using command line:
```shell
CUDA_VISIBLE_DEVICES=0 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
swift infer \
--model Qwen/Qwen3-VL-4B-Instruct \
--stream true
```
```
<<< who are you?
Hello! I'm Qwen, a large-scale language model independently developed by the Tongyi Lab under Alibaba Group. My main functions include answering questions, creating text such as stories, official documents, emails, scripts, and more, as well as performing logical reasoning, programming, and other tasks. If you have any questions or need assistance, feel free to let me know anytime, and I'll do my best to help!
--------------------------------------------------
<<< <image>describe the image.
Input an image path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png
This is a beautifully detailed, close-up portrait of an adorable tabby kitten, rendered with a soft, painterly effect that gives it a gentle, dreamy quality.
Here's a breakdown of the image:
- **The Kitten:** The subject is a young, fluffy kitten with a classic tabby pattern. Its fur is a mix of white and soft grayish-brown stripes, with a prominent dark stripe running down the center of its forehead and over its nose. The kitten's face is predominantly white, with delicate markings around its eyes and cheeks.
- **The Eyes:** Its most captivating feature is its large, round, and expressive eyes. They are a striking shade of bright blue-gray, with dark pupils that give it an intense, curious, and slightly innocent gaze. The eyes are wide open, suggesting the kitten is alert and attentive.
- **The Expression:** The kitten's expression is sweet and innocent. Its small pink nose and slightly parted mouth give it a gentle, almost pleading look. Its whiskers are long and white, standing out against its fur.
- **The Style:** The image has a soft-focus, artistic quality, reminiscent of impressionist painting. The edges of the kitten's fur are slightly blurred, creating a halo effect that draws attention to its face. The background is softly blurred with muted tones of green and gray, which helps the kitten stand out as the clear focal point.
- **Overall Impression:** The image evokes feelings of warmth, cuteness, and tenderness. The kitten appears to be looking directly at the viewer, creating a sense of connection and affection.
This is a lovely and charming depiction of a young kitten, capturing its innocence and charm in a visually appealing and emotionally engaging way.
--------------------------------------------------
<<< <video>describe the video.
Input a video path or URL <<< https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4
This video captures a charming and adorable moment of a young child, likely a toddler, sitting on a bed and pretending to read a book. The child is wearing glasses, which adds a humorous and endearing touch to the scene — as if they're a little scholar or librarian.
Here's a breakdown of what unfolds:
- The child is seated cross-legged on a bed with a patterned quilt. Behind them, a crib and some household items are visible, suggesting a cozy bedroom setting.
- The child holds an open book and appears to be turning the pages with focused attention, mimicking the behavior of a real reader.
- At one point, the child looks up, smiles, or seems to react with delight — perhaps amused by something in the book or just enjoying the activity.
- The child's movements are gentle and deliberate, showing a sense of concentration and curiosity. They turn pages, sometimes with one hand, and occasionally lift the book slightly as if to examine it more closely.
- The video has a warm, candid feel — it's not staged, and the child's natural behavior makes it feel authentic and heartwarming.
Overall, this is a sweet, lighthearted video that showcases the innocence and imagination of early childhood. The child's engagement with the book, combined with their glasses and playful demeanor, creates a delightful and memorable scene.
```
- For model-specific parameters, such as environment variables like `VIDEO_MAX_TOKEN_NUM`, please refer to the [Command Line Parameters Documentation](../Instruction/Command-line-parameters.md#qwen3_vl-qwen3_5).
## Training
This section introduces how to train Qwen3-VL using ms-swift and Megatron-SWIFT. We recommend using ms-swift (i.e., transformers backend, which is more convenient and simple) for Dense models, and Megatron-SWIFT (i.e., megatron backend, which offers faster training speed; see benchmark [here](../Megatron-SWIFT/Quick-start.md#benchmark)) for MoE models.
If you need to fine-tune the model with a custom dataset, you can prepare the data in the following format and set `--dataset train.jsonl --val_dataset val.jsonl` in the command line, where the validation set is optional. For more information, please refer to the [Multimodal Dataset Documentation](../Customization/Custom-dataset.md#multimodal).
```jsonl
{"messages": [{"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "The capital of Zhejiang is Hangzhou."}]}
{"messages": [{"role": "user", "content": "<image><image>What's the difference between these two images?"}, {"role": "assistant", "content": "The first one is a kitten, the second one is a puppy"}], "images": ["/xxx/x.jpg", "/xxx/x.png"]}
{"messages": [{"role": "system", "content": "You are a helpful and harmless assistant"}, {"role": "user", "content": "<image>What's in the image, <video>what's in the video?"}, {"role": "assistant", "content": "There's an elephant in the image, and a puppy running on the grass in the video"}], "images": ["/xxx/x.jpg"], "videos": ["/xxx/x.mp4"]}
```
Qwen3-VL's bbox output uses normalized 1000 relative coordinates. You can use the grounding dataset format provided by ms-swift, where the coordinates in "bbox" are absolute coordinates, and ms-swift will automatically convert absolute coordinates to normalized 1000 relative coordinates. For more information, please refer to the [Grounding Dataset Format Documentation](../Customization/Custom-dataset.md#grounding).
```jsonl
{"messages": [{"role": "user", "content": "<image>Locate the <ref-object> in the image"}, {"role": "assistant", "content": "[\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"},\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"}\n]"}], "images": ["cat.png"], "objects": {"ref": ["sheep", "sheep", "sheep"], "bbox": [[90.9, 160.8, 135, 212.8], [360.9, 480.8, 495, 532.8]]}}
```
### Dense Models
Below is a fine-tuning script for the `Qwen3-VL-4B-Instruct` model. We use mixed-modality data as a demo dataset; this example script has no practical value. Training memory usage is 2 * 21GiB, and training time is 12 minutes.
- If you find the preprocessing time too long, you can remove `--packing`, or use [cached dataset](https://github.com/modelscope/ms-swift/tree/main/examples/train/cached_dataset).
```shell
# 2 * 21GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
swift sft \
--model Qwen/Qwen3-VL-4B-Instruct \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#10000' \
'AI-ModelScope/LaTeX_OCR:human_handwrite#5000' \
'swift/VideoChatGPT:Generic#2000' \
--load_from_cache_file true \
--split_dataset_ratio 0.01 \
--tuner_type lora \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--attn_impl flash_attn \
--padding_free true \
--learning_rate 1e-4 \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--freeze_vit true \
--freeze_aligner true \
--packing true \
--gradient_checkpointing true \
--vit_gradient_checkpointing false \
--gradient_accumulation_steps 2 \
--eval_steps 100 \
--save_steps 100 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 4096 \
--output_dir output \
--warmup_ratio 0.05 \
--deepspeed zero2 \
--dataset_num_proc 4 \
--dataloader_num_workers 4
```
After training, we use the following script to perform inference on the validation set:
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=0 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--stream true \
--max_new_tokens 2048 \
--load_data_args true
```
```
--------------------------------------------------
[QUERY] Using LaTeX to perform OCR on the image.
[LABELS] 1 + \frac { 1 } { 1 ! } + \frac { 1 } { 2 ! } + \frac { 1 } { 3 ! } + \frac { 1 } { 4 ! }
[RESPONSE] 1 + \frac { 1 } { 1 ! } + \frac { 1 } { 2 ! } + \frac { 1 } { 3 ! } + \frac { 1 } { 4 ! }
--------------------------------------------------
[QUERY] What color suit is the man wearing while playing the saxophone on stage?
[LABELS] The man is wearing a black suit and white shirt while playing the saxophone on the red-floored stage.
[RESPONSE] The man is wearing a black suit while playing the saxophone on stage.
--------------------------------------------------
...
```
### MoE Models
Below is a fine-tuning script for the `Qwen3-VL-30B-A3B-Instruct` model. We use Megatron-SWIFT for single-machine full-parameter training. We still use mixed data for training; this example script has no practical value. Training requires 8 * 80GiB GPU memory, and training time is 20 minutes.
For Megatron-SWIFT environment installation, please refer to the [Megatron-SWIFT Documentation](../Megatron-SWIFT/Quick-start.md). Megatron-SWIFT shares the template and dataset modules with ms-swift, so the custom dataset format and model-specific environment variables introduced earlier still apply.
The fine-tuning script is as follows. For adjusting training techniques and parallelism strategies, refer to the [Megatron-SWIFT Documentation](../Megatron-SWIFT/Quick-start.md#training-tips).
```shell
# 8 * 80GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
OMP_NUM_THREADS=14 \
NPROC_PER_NODE=8 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
megatron sft \
--model Qwen/Qwen3-VL-30B-A3B-Instruct \
--save_safetensors true \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#10000' \
'AI-ModelScope/LaTeX_OCR:human_handwrite#5000' \
'swift/VideoChatGPT:Generic#2000' \
--load_from_cache_file true \
--split_dataset_ratio 0.01 \
--moe_permute_fusion true \
--tensor_model_parallel_size 4 \
--expert_model_parallel_size 8 \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-6 \
--micro_batch_size 1 \
--global_batch_size 4 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--num_train_epochs 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-6 \
--output_dir megatron_output/Qwen3-VL-30B-A3B-Instruct \
--eval_steps 500 \
--save_steps 500 \
--max_length 4096 \
--packing true \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim true \
--no_save_rng true \
--sequence_parallel true \
--moe_expert_capacity_factor 2 \
--attention_backend flash
```
After training, we use the following script to perform inference on the validation set:
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=0 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
swift infer \
--model megatron_output/Qwen3-VL-30B-A3B-Instruct/vx-xxx/checkpoint-xxx \
--stream true \
--max_new_tokens 2048 \
--load_data_args true
```
Use the following command to push the trained weights to ModelScope:
```shell
swift export \
--model output/vx-xxx/checkpoint-xxx \
--push_to_hub true \
--hub_model_id '<your-model-id>' \
--hub_token '<your-sdk-token>'
```
docs/source_en/BestPractices/Qwen3_5-Best-Practice.md
+560
View File
@@ -0,0 +1,560 @@
# Qwen3.5 Best Practices
ms-swift supports training [Qwen3.5](https://github.com/QwenLM/Qwen3.5) Dense/MoE models using transformers/Megatron backends. Qwen3.5 is a multimodal model with hybrid thinking, combining linear attention and full attention. This article will introduce how to perform inference, instruction fine-tuning, and reinforcement learning on Qwen3.5 Dense/MoE models.
## Environment Setup
```shell
pip install -U ms-swift
pip install -U "transformers>=5.9" "qwen_vl_utils>=0.0.14" peft liger-kernel
# flash-linear-attention
# If you encounter slow training issues, please refer to: https://github.com/fla-org/flash-linear-attention/issues/758
# Please use Python 3.12: https://github.com/fla-org/flash-linear-attention/issues/121
pip install -U "flash-linear-attention>=0.4.2" --no-build-isolation
# causal_conv1d
pip install -U git+https://github.com/Dao-AILab/causal-conv1d --no-build-isolation
# flash-attention
pip install "flash-attn==2.8.3" --no-build-isolation
# deepspeed training
pip install deepspeed
# vllm (torch2.10) for inference/deployment/RL
pip install -U "vllm>=0.17.0"
```
- Qwen3.5 video data training hangs: Using the decord backend to read videos may cause hanging issues, refer to [this issue](https://github.com/dmlc/decord/issues/269). You can use the torchcodec backend, specifically refer to the [qwen_vl_utils](https://github.com/QwenLM/Qwen3-VL/blob/50068df2334f309979ff05d75f1078c8309c63ed/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L390-L400) library.
- If you are using Qwen3.5 on Ascend NPU and want details about the FLA / MindSpeed replacement, effective patch path, and verified version combinations, please refer to [Qwen3.5 FLA Patch Notes in the NPU Support document](./NPU-support.md#qwen35-fla-patch-notes).
## Inference
Using ms-swift's `TransformersEngine` for inference:
- The meaning of model-specific parameters such as `VIDEO_MAX_TOKEN_NUM` environment variables is the same as Qwen3-VL, refer to [Command-line Parameters Documentation](../Instruction/Command-line-parameters.md#qwen3_vl,qwen3_5).
```python
import os
# os.environ['SWIFT_DEBUG'] = '1'
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['IMAGE_MAX_TOKEN_NUM'] = '1024'
os.environ['VIDEO_MAX_TOKEN_NUM'] = '128'
os.environ['FPS_MAX_FRAMES'] = '16'
from swift import get_model_processor, get_template
from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig
model, processor = get_model_processor('Qwen/Qwen3.5-4B') # attn_impl='flash_attention_2'
template = get_template(processor, enable_thinking=False)
engine = TransformersEngine(model, template=template)
infer_request = InferRequest(messages=[{
"role": "user",
"content": '<video>Describe this video.',
}], videos=['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4'])
request_config = RequestConfig(max_tokens=128, temperature=0)
resp_list = engine.infer([infer_request], request_config=request_config)
response = resp_list[0].choices[0].message.content
print(response)
# use stream
request_config = RequestConfig(max_tokens=128, temperature=0, stream=True)
gen_list = engine.infer([infer_request], request_config=request_config)
for chunk in gen_list[0]:
if chunk is None:
continue
print(chunk.choices[0].delta.content, end='', flush=True)
print()
```
Using command line for inference:
```shell
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model Qwen/Qwen3.5-4B \
--enable_thinking false \
--stream true
```
## Fine-tuning
This chapter will introduce how to train Qwen3.5 using ms-swift and Megatron-SWIFT. It is recommended to use ms-swift (i.e., transformers backend, more convenient and simple) for Dense models, and Megatron-SWIFT (i.e., megatron backend, faster training speed) for MoE models.
If you need to fine-tune the model with a custom dataset, you can prepare the data in the following format and set `--dataset train.jsonl --val_dataset val.jsonl` in the command line, where the validation set is optional. For more information, please refer to [Multimodal Dataset Documentation](../Customization/Custom-dataset.md#multimodal).
```jsonl
{"messages": [{"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "The capital of Zhejiang is Hangzhou."}]}
{"messages": [{"role": "user", "content": "<image><image>What's the difference between these two images?"}, {"role": "assistant", "content": "The first one is a kitten, the second one is a puppy"}], "images": ["/xxx/x.jpg", "/xxx/x.png"]}
{"messages": [{"role": "system", "content": "You are a helpful and harmless assistant"}, {"role": "user", "content": "<image>What's in the image, <video>what's in the video?"}, {"role": "assistant", "content": "There's an elephant in the image, and a puppy running on the grass in the video"}], "images": ["/xxx/x.jpg"], "videos": ["/xxx/x.mp4"]}
```
Qwen3.5's bbox output uses normalized relative coordinates with a scale of 1000. You can use the grounding dataset format provided by ms-swift, where the coordinates in "bbox" are absolute coordinates, and ms-swift will automatically convert absolute coordinates to normalized relative coordinates with a scale of 1000. For more information, please refer to [Grounding Dataset Format Documentation](../Customization/Custom-dataset.md#grounding).
```jsonl
{"messages": [{"role": "user", "content": "<image>Locate the <ref-object> in the image"}, {"role": "assistant", "content": "[\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"},\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"}\n]"}], "images": ["cat.png"], "objects": {"ref": ["sheep", "sheep", "sheep"], "bbox": [[90.9, 160.8, 135, 212.8], [360.9, 480.8, 495, 532.8]]}}
```
### Dense Models
Below is a fine-tuning script for the Qwen3.5-4B model. This example script is for demonstration purposes only. Training memory usage is 4 × 20GiB, with a training time of 12 minutes. Qwen3.5 supports packing/padding_free in transformers (requires "ms-swift>=4.3.1"; Megatron has no such version restriction). Below we use the `group_by_length` parameter to accelerate training, ensuring load balancing across data parallelism (DP) and reducing zero-padding in micro batches. However, this may cause fluctuations in the loss curve due to insufficient data shuffling. You can also remove this parameter and use `--packing true` instead.
- Regarding data preprocessing: When using the packing / group_by_length parameters, all data must be preprocessed in advance to obtain the input_ids length of each sample, which takes additional time. If you prefer to process data on-the-fly during training, you can remove these two parameters.
- Reduce memory consumption: You can enable `--deepspeed zero2/zero3`, turn on sequence parallelism via `--sequence_parallel_size`, or use `--use_liger_kernel true`.
- Training acceleration: You can enable `--attn_impl flash_attention_2`, and for MoE models, it is recommended to enable `--experts_impl grouped_mm`.
```shell
# 4 * 20GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=4 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=12 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model Qwen/Qwen3.5-4B \
--tuner_type lora \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
'AI-ModelScope/LaTeX_OCR:human_handwrite#2000' \
--load_from_cache_file true \
--add_non_thinking_prefix true \
--loss_scale ignore_empty_think \
--split_dataset_ratio 0.01 \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--learning_rate 1e-4 \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--gradient_accumulation_steps 1 \
--group_by_length true \
--output_dir output/Qwen3.5-4B \
--eval_steps 50 \
--save_steps 50 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 2048 \
--warmup_ratio 0.05 \
--dataset_num_proc 4 \
--dataloader_num_workers 4 \
--deepspeed zero2 \
--model_author swift \
--model_name swift-robot
```
After training, use the following script to perform inference on the validation set:
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=0 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=12 \
swift infer \
--adapters output/Qwen3.5-4B/vx-xxx/checkpoint-xxx \
--stream true \
--enable_thinking false \
--max_new_tokens 512 \
--load_data_args true
```
```text
[QUERY] 你好，你是谁？
[RESPONSE] <think>
</think>
你好，我是由swift开发的人工智能语言模型，我的名字叫swift-robot。很高兴能与你交流。
--------------------------------------------------
[QUERY] Using LaTeX to perform OCR on the image.
[LABELS] e = \sum _ { k = 0 } ^ { \infty } \frac { 1 } { k ! }
[RESPONSE] <think>
</think>
e = \sum _ { k = 0 } ^ { \infty } \frac { 1 } { k ! }
```
```python
import os
# os.environ['SWIFT_DEBUG'] = '1'
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['IMAGE_MAX_TOKEN_NUM'] = '1024'
os.environ['VIDEO_MAX_TOKEN_NUM'] = '128'
os.environ['FPS_MAX_FRAMES'] = '16'
from peft import PeftModel
from swift import get_model_processor, get_template
from swift.infer_engine import TransformersEngine, InferRequest, RequestConfig
adapter_dir = 'output/Qwen3.5-4B/vx-xxx/checkpoint-xxx'
enable_thinking = False
model, processor = get_model_processor('Qwen/Qwen3.5-4B') # attn_impl='flash_attention_2'
model = PeftModel.from_pretrained(model, adapter_dir)
template = get_template(processor, enable_thinking=enable_thinking)
engine = TransformersEngine(model, template=template)
infer_request = InferRequest(messages=[{
"role": "user",
"content": 'who are you?',
}])
request_config = RequestConfig(max_tokens=128, temperature=0)
resp_list = engine.infer([infer_request], request_config=request_config)
response = resp_list[0].choices[0].message.content
print(response)
# use stream
request_config = RequestConfig(max_tokens=128, temperature=0, stream=True)
gen_list = engine.infer([infer_request], request_config=request_config)
for chunk in gen_list[0]:
if chunk is None:
continue
print(chunk.choices[0].delta.content, end='', flush=True)
print()
# I am an artificial intelligence assistant named swift-robot, trained by swift. I am designed to understand and generate natural language text in order to provide information, answer questions, and engage in conversation with humans. How can I assist you?
```
For an example of training MoE using the transformers backend, refer to: https://github.com/modelscope/ms-swift/blob/main/examples/models/qwen3_5/transformers.sh
### MoE Models
Qwen3.5-35B-A3B Megatron training. For environment preparation, please refer to [Megatron-SWIFT Quick Start Documentation](../Megatron-SWIFT/Quick-start.md). You can complete the following example in 15 minutes:
```shell
# 4 * 40GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=4 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=12 \
megatron sft \
--model Qwen/Qwen3.5-35B-A3B \
--save_safetensors true \
--merge_lora true \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
'AI-ModelScope/LaTeX_OCR:human_handwrite#2000' \
--load_from_cache_file true \
--add_non_thinking_prefix true \
--loss_scale ignore_empty_think \
--split_dataset_ratio 0.01 \
--tuner_type lora \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--expert_model_parallel_size 4 \
--moe_permute_fusion true \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-6 \
--micro_batch_size 4 \
--global_batch_size 16 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--num_train_epochs 1 \
--group_by_length true \
--finetune true \
--freeze_llm false \
--freeze_vit true \
--freeze_aligner true \
--cross_entropy_loss_fusion true \
--lr 1e-4 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--output_dir megatron_output/Qwen3.5-35B-A3B \
--eval_steps 200 \
--save_steps 200 \
--max_length 2048 \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim true \
--no_save_rng true \
--sequence_parallel true \
--attention_backend flash \
--padding_free false \
--model_author swift \
--model_name swift-robot
```
After training, use the following script to perform inference on the validation set:
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=12 \
swift infer \
--model megatron_output/Qwen3.5-35B-A3B/vx-xxx/checkpoint-xxx-merged \
--stream true \
--enable_thinking false \
--max_new_tokens 512 \
--load_data_args true
```
Tips for training Qwen3.5 with Megatron-SWIFT:
- Full parameter training: Refer to [this example](https://github.com/modelscope/ms-swift/blob/main/examples/models/qwen3_5/packing.sh).
- TP Limitation Removed: Using `megatron-core>=0.16` removes the `num_query_groups` limitation on TP.
- Regarding MTP training: `mcore-bridge>=1.1.0` supports multimodal MTP training. Please install the corresponding version.
- CP support: "mcore-bridge>=1.1.0" supports CP training for GDN. Additionally, the megatron-core [main branch](https://github.com/NVIDIA/Megatron-LM) needs to be installed.
- By default, `GatedDeltaNet` uses the Megatron implementation, which requires "megatron-core>=0.16" (ms-swift>=4.1.0; previous versions defaulted to the transformers implementation). Set the environment variable `USE_MCORE_GDN=0` to switch to the transformers implementation. **Note that the transformers implementation does not support packing and GDN's TP/CP**.
- Support for padding_free/packing: Packing can improve training speed. Refer to [this example](https://github.com/modelscope/ms-swift/tree/main/examples/models/qwen3_5/packing.sh).
- Qwen3-Next Megatron GatedDeltaNet support refers to [this PR](https://github.com/modelscope/mcore-bridge/pull/76), requiring `mcore-bridge>=1.4.0`.
- apply_wd_to_qk_layernorm: Apply weight decay to qk layernorm. Default is False.
- Regarding FP8 training: refer to [this example](https://github.com/modelscope/ms-swift/blob/main/examples/models/qwen3_5/fp8.sh). You need to install "mcore-bridge>=1.2.0", and set the parameter `--linear_decoupled_in_proj true` to decouple `in_proj` into `in_proj_qkvz` and `in_proj_ba`, where `in_proj_ba` is still trained in original precision.
## Reinforcement Learning (RL)
Using Qwen3.5-2B as an example, we demonstrate GRPO and GKD training on the [GSM8K](https://www.modelscope.cn/datasets/modelscope/gsm8k) dataset and evaluate on the GSM8K test set. To avoid excessively long chain-of-thought outputs, all experiments set `enable_thinking false`.
### GRPO
#### Dense Model
Full-parameter training with GRPO, using `gsm8k_accuracy` and `gsm8k_format` as reward functions. See [gsm8k_plugin.py](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/gsm8k/gsm8k_plugin.py) for the reward implementation.
```shell
SYSTEM_PROMPT="""You are a helpful math assistant. Solve the problem step by step and put your final answer within \\boxed{}."""
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen3.5-2B \
--external_plugins examples/train/grpo/plugin/gsm8k/gsm8k_plugin.py \
--reward_funcs gsm8k_accuracy gsm8k_format \
--columns '{"answer": "solution"}' \
--enable_thinking false \
--use_vllm true \
--vllm_mode colocate \
--vllm_gpu_memory_utilization 0.4 \
--vllm_tensor_parallel_size 1 \
--vllm_max_model_len 10240 \
--sleep_level 1 \
--tuner_type full \
--torch_dtype bfloat16 \
--dataset 'modelscope/gsm8k' \
--load_from_cache_file true \
--max_length 2048 \
--max_completion_length 8192 \
--num_train_epochs 1 \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--learning_rate 1e-6 \
--lr_scheduler_type cosine \
--save_steps 10 \
--save_total_limit 100 \
--logging_steps 1 \
--warmup_ratio 0.0 \
--dataloader_num_workers 4 \
--num_generations 8 \
--temperature 1.0 \
--system "$SYSTEM_PROMPT" \
--deepspeed zero2 \
--log_completions true \
--report_to tensorboard swanlab \
--max_grad_norm 1.0 \
--epsilon 0.2 \
--epsilon_high 0.28 \
--scale_rewards none
```
Evaluate the checkpoints:
```shell
CUDA_VISIBLE_DEVICES=0 swift eval \
--model output/Qwen3.5-2B/vxx-xxx-xxx/checkpoint-xx \
--enable_thinking false \
--eval_dataset gsm8k \
--eval_backend Native --infer_backend vllm \
--eval_generation_config '{"max_tokens":8192,"temperature":0.0,"do_sample":false}'
```
GSM8K evaluation results at 10-step intervals for the first 50 steps:
| Model / Steps | GSM8K Accuracy | Improvement |
|---|---|---|
| Qwen3.5-2B (baseline) | 0.7597 | - |
| GRPO 10 steps | 0.7650 | +0.53 |
| GRPO 20 steps | 0.7748 | +1.51 |
| GRPO 30 steps | 0.7779 | +1.82 |
| GRPO 40 steps | 0.7817 | +2.20 |
| GRPO 50 steps | 0.7885 | +2.88 |
### MoE Model
GRPO LoRA training for Qwen3.5-35B-A3B MoE model using the Megatron backend, trained on the [DAPO-Math-17k](https://www.modelscope.cn/datasets/open-r1/DAPO-Math-17k-Processed) dataset with `accuracy` as reward functions.
```shell
SYSTEM_PROMPT="""You are a helpful math assistant. Solve the problem step by step and put your final answer within \\boxed{}."""
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8 \
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
megatron rlhf \
--rlhf_type grpo \
--model Qwen/Qwen3.5-35B-A3B \
--save_safetensors true \
--enable_thinking false \
--merge_lora true \
--context_parallel_size 1 \
--tensor_model_parallel_size 1 \
--expert_model_parallel_size 8 \
--pipeline_model_parallel_size 1 \
--moe_permute_fusion true \
--dataset open-r1/DAPO-Math-17k-Processed \
--system "$SYSTEM_PROMPT" \
--num_train_epochs 1 \
--global_batch_size 64 \
--micro_batch_size 1 \
--steps_per_generation 2 \
--num_generations 8 \
--reward_funcs accuracy \
--use_vllm true \
--vllm_mode colocate \
--vllm_gpu_memory_utilization 0.5 \
--vllm_tensor_parallel_size 2 \
--vllm_max_model_len 9192 \
--max_length 1000 \
--max_completion_length 8192 \
--tuner_type lora \
--target_modules all-linear \
--lr 5e-5 \
--bf16 true \
--beta 0.00 \
--epsilon 0.2 \
--epsilon_high 0.28 \
--dynamic_sample false \
--overlong_filter true \
--loss_type grpo \
--sleep_level 1 \
--offload_model true \
--offload_bridge false \
--offload_optimizer true \
--logging_steps 1 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim \
--no_save_rng \
--save_steps 20 \
--attention_backend flash \
--moe_expert_capacity_factor 2 \
--temperature 1.0 \
--padding_free false \
--sequence_parallel true \
--log_completions true \
--report_to tensorboard swanlab
```
Evaluate on AIME-2025 and MATH-500:
```shell
CUDA_VISIBLE_DEVICES=0,1 swift eval \
--model <checkpoint-merged-path> \
--enable_thinking false \
--eval_dataset aime25 math_500 \
--eval_backend Native --infer_backend vllm \
--vllm_tensor_parallel_size 2 \
--vllm_gpu_memory_utilization 0.9 \
--vllm_max_model_len 10000 \
--eval_generation_config '{"max_tokens":8192,"temperature":0.0,"do_sample":false}' \
--eval_num_proc 8
```
Evaluation results on AIME-2025 and MATH-500:
| Model / Steps | AIME-2025 | MATH-500 |
|---|---|---|
| Qwen3.5-35B-A3B (baseline) | 43.33 | 92.40 |
| Megatron GRPO 20 steps | 53.33 (+10.00) | 95.80 (+3.40) |
| Megatron GRPO 40 steps | 53.33 (+10.00) | 96.60 (+4.20) |
### GKD
LoRA training with GKD (General Knowledge Distillation), using Qwen3.5-9B as the teacher model. First, launch the teacher server with `swift deploy` (alternatively, use the `--teacher_model` parameter to load the model directly):
```shell
CUDA_VISIBLE_DEVICES=0 \
swift deploy \
--model Qwen/Qwen3.5-9B \
--infer_backend vllm \
--port 8000 \
--vllm_tensor_parallel_size 1 \
--vllm_max_model_len 10240 \
--gpu-memory-utilization 0.8 \
--max_logprobs 64
```
Then start GKD training on the remaining GPUs:
```shell
NPROC_PER_NODE=3 \
CUDA_VISIBLE_DEVICES=1,2,3 \
swift rlhf \
--rlhf_type gkd \
--model Qwen/Qwen3.5-2B \
--teacher_model_server http://localhost:8000 \
--gkd_logits_topk 64 \
--enable_thinking false \
--tuner_type lora \
--use_vllm true \
--vllm_mode colocate \
--vllm_gpu_memory_utilization 0.5 \
--vllm_tensor_parallel_size 1 \
--vllm_max_model_len 10240 \
--sleep_level 0 \
--dataset 'modelscope/gsm8k' \
--lmbda 1 \
--beta 0.5 \
--torch_dtype bfloat16 \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 16 \
--learning_rate 5e-5 \
--logging_steps 1 \
--save_steps 100 \
--save_total_limit 10 \
--max_length 2048 \
--max_completion_length 8192 \
--warmup_ratio 0.1 \
--save_only_model true \
--dataloader_num_workers 4 \
--dataset_num_proc 4 \
--attn_impl flash_attn \
--report_to tensorboard swanlab
```
Evaluate the checkpoints:
```shell
CUDA_VISIBLE_DEVICES=0 swift eval \
--model Qwen/Qwen3.5-2B \
--adapters output/Qwen3.5-2B/vxx-xxx-xxx/checkpoint-xx \
--merge_lora true \
--enable_thinking false \
--eval_dataset gsm8k \
--eval_backend Native --infer_backend vllm \
--eval_generation_config '{"max_tokens":8192,"temperature":0.0,"do_sample":false}'
```
GSM8K evaluation results at 100-step intervals for the first 300 steps:
| Model / Steps | GSM8K Accuracy | Improvement |
|---|---|---|
| Qwen3.5-2B (baseline) | 0.7597 | - |
| GKD 100 steps | 0.7968 | +3.71 |
| GKD 200 steps | 0.8188 | +5.91 |
| GKD 300 steps | 0.8332 | +7.35 |
docs/source_en/BestPractices/Rapidly-Training-VL-model.md
+235
View File
@@ -0,0 +1,235 @@
# Best Practices for Rapidly Training Vision-Language (VL) Models
This document provides best practices for quickly training vision-language (VL) models from scratch.
Model Links
- [Qwen2.5-VL-7B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen2.5-VL-7B-Instruct)
- [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B)
Trained Model Link
- [Simple-VL-8B](https://www.modelscope.cn/models/swift/Simple-VL-8B/summary)
The training workflow builds upon the Qwen2.5-VL-7B-Instruct model architecture by replacing its internal large language model (LLM) component with the weights from Qwen3-8B , thereby enhancing the model's visual understanding capabilities. The process involves the following steps:
1. Modify the original model’s configuration file config.json to align with Qwen3-8B.
2. Initialize and load new model weights, saving them as a new model.
3. Fine-tune the new model in two stages:
1. Stage 1 : Train only the vision-to-language alignment module (aligner), freezing the ViT and LLM components.
2. Stage 2 : Unfreeze all modules and perform joint fine-tuning to improve overall performance.
## Model Modification
### Config File (config.json) Update
Due to structural differences between Qwen2.5-7B-Instruct and Qwen3-8B (e.g., number of layers, hidden dimensions), create a new config.json based on the Qwen2.5-VL-7B-Instruct config and update the following parameters to match Qwen3-8B:
```
Modified Parameters
1. hidden_size 3584->4096
2. intermediate_size: 18944->12288
3. num_attention_heads: 28->32
4. num_key_value_heads: 4->8
5. num_hidden_layers: 28->36
6. vocab_size:152064->151936
7. max_window_layers:28->36
8. out_hidden_size: 3584->4096
Newly Added Parameter
1. head_dim： 128
```
### Model Weight Initialization and Replacement
Use the following Python script to initialize, replace, and save the model weights:
```python
import torch
from modelscope import Qwen2_5_VLForConditionalGeneration, AutoModelForCausalLM, AutoConfig
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import Qwen2_5_VLPatchMerger, Qwen2_5_VLModel
from accelerate import Accelerator
# Load original VL model and Qwen3-8B model
qwen2_5_vl_7b_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen2.5-VL-7B-Instruct",
device_map="cuda",
torch_dtype=torch.bfloat16
)
device = qwen2_5_vl_7b_model.device
qwen3_8b_model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-8B",
device_map=device,
torch_dtype=torch.bfloat16
)
# Load configurations
old_config = AutoConfig.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
new_config = AutoConfig.from_pretrained("/path/to/new_config_dir") # Path to new config directory
new_visual_config = new_config.vision_config
# Replace merger (aligner) layer
new_merger = Qwen2_5_VLPatchMerger(
dim=new_visual_config.out_hidden_size,
context_dim=new_visual_config.hidden_size,
spatial_merge_size=new_visual_config.spatial_merge_size,
).to(device).to(torch.bfloat16)
qwen2_5_vl_7b_model.visual.merger = new_merger
# Replace LLM part of the VL model
new_llm_model = Qwen2_5_VLModel(new_config).to(device).to(torch.bfloat16)
for name, param in qwen3_8b_model.model.named_parameters():
if name in new_llm_model.state_dict():
new_llm_model.state_dict()[name].copy_(param)
qwen2_5_vl_7b_model.model = new_llm_model
qwen2_5_vl_7b_model.lm_head = qwen3_8b_model.lm_head
# Save modified model
accelerator = Accelerator()
accelerator.save_model(
model=qwen2_5_vl_7b_model,
save_directory="/path/to/save/Qwen3-VL-Model",
max_shard_size="4GB",
safe_serialization=True
)
```
After saving the weights, copy all files from the original Qwen2.5-VL-7B-Instruct model folder, except for the model weights(including `model.safetensors.index.json`), to the new model weights folder, and replace config.json with the newly modified config.json file.
## Training
To simplify the process, we skip pre-training and proceed directly to supervised fine-tuning (SFT). The training is divided into two stages:
### Stage 1: Train Aligner Layer
Train only the vision-to-language alignment module while freezing the ViT and LLM parts:
```bash
NNODES=$WORLD_SIZE \
NODE_RANK=$RANK \
NPROC_PER_NODE=8 \
MAX_PIXELS=1003520 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
swift sft \
--model /path/to/new_vl_model \
--model_type qwen2_5_vl \
--tuner_type full \
--dataset xxx \
--load_from_cache_file true \
--split_dataset_ratio 0.01 \
--torch_dtype bfloat16 \
--attn_impl flash_attn \
--freeze_vit true \
--freeze_llm true \
--freeze_aligner false \
--num_train_epochs 3 \
--per_device_train_batch_size 2 \
--learning_rate 5e-6 \
--gradient_accumulation_steps 8 \
--eval_steps -1 \
--save_steps 1000 \
--save_total_limit 10 \
--logging_steps 5 \
--max_length 8192 \
--output_dir output \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--dataset_num_proc 8 \
--deepspeed zero2
```
### Stage 2: Full Model Training
Unfreeze all modules and jointly train to enhance the model's visual understanding:
```bash
NNODES=$WORLD_SIZE \
NODE_RANK=$RANK \
NPROC_PER_NODE=8 \
MAX_PIXELS=1003520 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
swift sft \
--model /path/to/stage1_checkpoint \
--model_type qwen2_5_vl \
--tuner_type full \
--dataset xxx \
--load_from_cache_file true \
--split_dataset_ratio 0.01 \
--torch_dtype bfloat16 \
--attn_impl flash_attn \
--freeze_vit false \
--freeze_llm false \
--freeze_aligner false \
--num_train_epochs 3 \
--per_device_train_batch_size 2 \
--learning_rate 5e-6 \
--gradient_accumulation_steps 8 \
--eval_steps -1 \
--save_steps 1000 \
--save_total_limit 10 \
--logging_steps 5 \
--max_length 8192 \
--output_dir output \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--dataset_num_proc 8 \
--deepspeed zero2
```
## Inference / Deployment / Evaluation
### Inference
Perform inference using `swift infer`:
```bash
swift infer \
--model /path/to/stage2_checkpoint
```
### Deoloyment
Accelerate model serving with vLLM:
```bash
CUDA_VISIBLE_DEVICES=0 \
MAX_PIXELS=1003520 \
VIDEO_MAX_PIXELS=50176 \
FPS_MAX_FRAMES=12 \
swift deploy \
--model /path/to/stage2_checkpoint \
--infer_backend vllm \
--vllm_gpu_memory_utilization 0.9 \
--vllm_max_model_len 8192 \
--max_new_tokens 2048 \
--vllm_limit_mm_per_prompt '{"image": 5, "video": 2}' \
--served_model_name Qwen3-VL
```
### Evaluation
Evaluate the trained VL model using [EvalScope](https://github.com/modelscope/evalscope/).
Example Evaluation Using MMMU Benchmark
```python
from evalscope import TaskConfig, run_task
task_cfg_dict = TaskConfig(
work_dir='outputs',
eval_backend='VLMEvalKit',
eval_config={
'data': ['MMMU_DEV_VAL'],
'mode': 'all',
'model': [
{
'api_base': 'http://localhost:8000/v1/chat/completions',
'key': 'EMPTY',
'name': 'CustomAPIModel',
'temperature': 0.6,
'type': 'Qwen3-VL',
'img_size': -1,
'video_llm': False,
'max_tokens': 512,
}
],
'reuse': False,
'nproc': 64,
'judge': 'exact_matching'
},
)
run_task(task_cfg=task_cfg_dict)
```
docs/source_en/BestPractices/Reranker.md
+127
View File
@@ -0,0 +1,127 @@
# Reranker Training
SWIFT supports Reranker model training. Currently supported models include:
1. modernbert reranker model
- [ModelScope](https://www.modelscope.cn/models/iic/gte-reranker-modernbert-base) [Hugging Face](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base)
2. qwen3-reranker model
- 0.6B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-Reranker-0.6B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)
- 4B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-Reranker-4B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-Reranker-4B)
- 8B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-Reranker-8B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-Reranker-8B)
3. qwen3-vl-reranker model
- 2B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-VL-Reranker-2B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-VL-Reranker-2B)
- 8B: [ModelScope](https://www.modelscope.cn/models/Qwen/Qwen3-VL-Reranker-8B) [Hugging Face](https://huggingface.co/Qwen/Qwen3-VL-Reranker-8B)
## Implementation Methods
SWIFT currently supports two implementation methods for Reranker models, which have significant differences in architecture and loss function computation:
### 1. Classification Reranker
**Applicable Models:** modernbert reranker models (e.g., gte-reranker-modernbert-base)
**Core Principles:**
- Based on sequence classification architecture, adding a classification head on top of pre-trained models
- Input: query-document pairs, Output: single relevance score
### 2. Generative Reranker
**Applicable Models:** qwen3-reranker models (0.6B/4B/8B)
**Core Principles:**
- Based on generative language model architecture (CausalLM)
- Input: query-document pairs, Output: probability of specific tokens (e.g., "yes"/"no")
- Classification is performed by comparing logits of specific tokens at the final position
## Loss Function Types
SWIFT supports multiple loss functions for training Reranker models:
### Pointwise Loss Functions
Pointwise methods transform the ranking problem into a binary classification problem, processing each query-document pair independently:
- **Core Idea:** Binary classification for each query-document pair to determine document relevance to the query
- **Loss Function:** Binary cross-entropy
- **Use Cases:** Simple and efficient, suitable for large-scale data training
Environment variable configuration:
- `GENERATIVE_RERANKER_POSITIVE_TOKEN`: Positive token (default: "yes")
- `GENERATIVE_RERANKER_NEGATIVE_TOKEN`: Negative token (default: "no")
### Listwise Loss Functions
Listwise methods transform the ranking problem into a multi-classification problem, selecting positive examples from multiple candidate documents:
- **Core Idea:** Multi-classification for each query's candidate document group (1 positive + n negative examples) to identify positive documents
- **Loss Function:** Multi-class cross-entropy
- **Use Cases:** Learning relative ranking relationships between documents, better aligned with the actual needs of information retrieval
Environment variable configuration:
- `LISTWISE_RERANKER_TEMPERATURE`: Softmax temperature parameter (default: 1.0)
- `LISTWISE_RERANKER_MIN_GROUP_SIZE`: Minimum group size, if the number of documents in the group is less than this value, the loss will not be calculated (default: 2)
**Listwise vs Pointwise:**
- **Pointwise:** Independent relevance judgment, simple training, but ignores relative relationships between documents
- **Listwise:** Learning relative ranking, better performance, more suitable for the essential needs of ranking tasks
The loss function source code can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
## Dataset Format
```json lines
# LLM
{"messages": [{"role": "user", "content": "query"}], "positive_messages": [[{"role": "assistant", "content": "relevant_doc1"}],[{"role": "assistant", "content": "relevant_doc2"}]], "negative_messages": [[{"role": "assistant", "content": "irrelevant_doc1"}],[{"role": "assistant", "content": "irrelevant_doc2"}], ...]}
# MLLM
{"messages": [{"role": "user", "content": "<image>query"}], "images": ["/some/images.jpg"], "positive_messages": [[{"role": "assistant", "content": "<image>relevant_doc1"}]], "positive_images": [["/some/positive_images.jpg"]], "negative_messages": [[{"role": "assistant", "content": "<image><image>irrelevant_doc1"}], [{"role": "assistant", "content": "<image>irrelevant_doc2"}]], "negative_images": [["/some/negative_images1.jpg", "/some/negative_images2.jpg"], ["/some/negative_images3.jpg"]]}
```
**Field Description:**
- `messages`: Query text
- `positive_messages`: List of positive documents relevant to the query, supports multiple positive examples
- `negative_messages`: List of negative documents irrelevant to the query, supports multiple negative examples
**Environment Variable Configuration:**
- `MAX_POSITIVE_SAMPLES`: Maximum number of positive examples per query (default: 1)
- `MAX_NEGATIVE_SAMPLES`: Maximum number of negative examples per query (default: 7)
> By default, `MAX_POSITIVE_SAMPLES` positive examples and `MAX_NEGATIVE_SAMPLES` negative examples will be extracted from each data item. Each positive example will be grouped with `MAX_NEGATIVE_SAMPLES` negative examples to form a group. Therefore, each data item will be expanded into `MAX_POSITIVE_SAMPLES`x`(1 + MAX_NEGATIVE_SAMPLES)` data points.
> If the number of positive/negative examples in the data is insufficient, all positive/negative examples will be used. If the number of positive and negative examples in the data exceeds `MAX_POSITIVE_SAMPLES` and `MAX_NEGATIVE_SAMPLES`, random sampling will be performed.
> **IMPORTANT**: The expanded data will be placed in the same batch. Therefore, the effective batch size on each device will be `per_device_train_batch_size` × `MAX_POSITIVE_SAMPLES` × (1 + `MAX_NEGATIVE_SAMPLES`). Please adjust your `per_device_train_batch_size` accordingly to avoid out-of-memory errors.
## Training Scripts
Training scripts provided by ms-swift:
- [Qwen3-Reranker/Qwen3-VL-Reranker](https://github.com/modelscope/ms-swift/blob/main/examples/train/reranker/qwen3)
- [Pointwise Classification Reranker](https://github.com/modelscope/ms-swift/blob/main/examples/train/reranker/train_reranker.sh)
- [Pointwise Generative Reranker](https://github.com/modelscope/ms-swift/blob/main/examples/train/reranker/train_generative_reranker.sh)
- [Listwise Classification Reranker](https://github.com/modelscope/ms-swift/blob/main/examples/train/reranker/train_reranker_listwise.sh)
- [Listwise Generative Reranker](https://github.com/modelscope/ms-swift/blob/main/examples/train/reranker/train_generative_reranker_listwise.sh)
For inference scripts, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_reranker.py).
## Advanced
- Qwen3-Reranker Custom Instruction:
- Default template:
```text
<|im_start|>system
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>
<|im_start|>user
<Instruct>: {Instruction}
<Query>: {Query}
<Document>: {Document}<|im_end|>
<|im_start|>assistant
<think>
</think>
```
- Default instruction:
- `Given a web search query, retrieve relevant passages that answer the query`
- Instruction priority (nearest wins):
- `system` inside `positive_messages`/`negative_messages` > `system` in main `messages` > default instruction.
- That is, if a positive/negative message sequence contains a `system`, it takes precedence; otherwise, if main `messages` has a `system`, use it; if neither is provided, use the default instruction.
docs/source_en/BestPractices/deepseek-v4.md
+260
View File
@@ -0,0 +1,260 @@
# DeepSeek-V4 Training Support
Megatron-SWIFT currently supports fine-tuning and RL for DeepSeek-V4, including features such as MTP and FP8. (FP4 blockwise training is not yet supported; FP4 weights are automatically converted to FP8/BF16 when loaded.)
You need to use the `dev` branch of Megatron-Core, together with the `main` branches of `mcore-bridge` and `ms-swift`.
```shell
pip install git+https://github.com/NVIDIA/Megatron-LM.git@dev
pip install git+https://github.com/modelscope/mcore-bridge.git
pip install git+https://github.com/modelscope/ms-swift.git
# Megatron-LM is tested under the following commit hash
# pip install git+https://github.com/NVIDIA/Megatron-LM.git@c6449f0b23be397449f21c0967c5fc90785e55ea
```
## Precision Alignment
- To support precision alignment testing (FP32), you need to comment out [these lines](https://github.com/NVIDIA/Megatron-LM/blob/bd381ac364b5139840f0cba6389db54f2c092e90/megatron/core/transformer/experimental_attention_variant/dsa.py#L41-L43).
After modifying the code, run the following tests to confirm there are no precision alignment issues (testing the forward alignment between transformers and megatron):
First, create a mini version of the model with only 4 layers:
```python
import os
import torch
from modelscope.hub.file_download import model_file_download
from safetensors.torch import safe_open
from swift import safe_snapshot_download
from mcore_bridge.utils import Fp8Dequantizer, SafetensorLazyLoader, StreamingSafetensorSaver
model_id = 'deepseek-ai/DeepSeek-V4-Flash-Base'
# Some models have the first few layers as dense and the rest as MoE; set this value accordingly
model_dir = safe_snapshot_download(model_id, download_model=False)
loader = SafetensorLazyLoader(model_dir)
state_dict = loader.get_state_dict()
saver = StreamingSafetensorSaver(save_dir=model_dir)
fp8_dequantizer = Fp8Dequantizer() # Used to convert fp8 weights to bf16
def _open_file(self, filename: str):
if filename not in self._file_handles:
file_path = os.path.join(self.hf_model_dir, filename)
tmp_dir = os.path.join(self.hf_model_dir, 'tmp')
if not os.path.exists(file_path):
file_path = os.path.join(tmp_dir, filename)
if not os.path.exists(file_path):
file_path = model_file_download(
model_id=model_id,
file_path=filename,
local_dir=tmp_dir,
)
self._file_handles[filename] = safe_open(file_path, framework='pt')
return self._file_handles[filename]
SafetensorLazyLoader._open_file = _open_file # monkey patch (lazy downloading)
new_state_dict = {}
for k, v in state_dict.items():
if k.startswith('layers.'):
idx = int(k[len('layers.'):].split('.', 1)[0])
if idx >= 4:
continue
if k.endswith('.scale'):
continue
elif k.endswith('.weight'):
weight_scale_inv = k.replace('.weight', '.scale')
if weight_scale_inv in state_dict:
v = fp8_dequantizer.convert(v.load(), state_dict[weight_scale_inv].load()).to(torch.bfloat16)
new_state_dict[k] = v if isinstance(v, torch.Tensor) else v.load()
for k, v in new_state_dict.items():
saver.add_tensor(k, v)
saver.finalize()
```
Then modify `config.json`:
- Set `num_hidden_layers` to `4`.
- Set `compress_ratios` to `[0, 0, 4, 128, 0]`.
- Remove the `quantization_config` field.
Next, create `test.py` and run it with: `CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 test.py`. For more details, refer to the [Custom Megatron Model documentation](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Custom-Model.html).
```python
import os
os.environ['SWIFT_TEST_CONVERT_PRECISION'] = '1'
from swift.megatron import MegatronExportArguments, megatron_export_main
from swift import safe_snapshot_download
model_id = 'deepseek-ai/DeepSeek-V4-Flash-Base'
model_dir = safe_snapshot_download(model_id, download_model=False)
if __name__ == '__main__':
megatron_export_main(
MegatronExportArguments(
model=model_dir,
to_mcore=True,
attention_backend='flash',
tensor_model_parallel_size=1,
pipeline_model_parallel_layout='Et*3|t*1mL',
pipeline_model_parallel_size=2,
expert_model_parallel_size=2,
mtp_num_layers=1,
test_convert_precision=True,
))
```
When you see the following result, the alignment is correct and you can proceed to training.
![Precision Alignment](../../resources/deepseek_v4/precision.png)
## LoRA Training
The BF16 LoRA training script is shown below. The final output includes both the incremental LoRA weights and the merged BF16 full weights.
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=8 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
megatron sft \
--model deepseek-ai/DeepSeek-V4-Flash \
--save_safetensors true \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#1000' \
'AI-ModelScope/alpaca-gpt4-data-en#1000' \
'swift/self-cognition#1000' \
--model_author swift \
--model_name swift-robot \
--merge_lora true \
--load_from_cache_file true \
--add_non_thinking_prefix true \
--loss_scale ignore_empty_think \
--split_dataset_ratio 0.01 \
--tuner_type lora \
--lora_rank 16 \
--lora_alpha 32 \
--tensor_model_parallel_size 1 \
--expert_model_parallel_size 8 \
--micro_batch_size 4 \
--global_batch_size 32 \
--padding_free false \
--group_by_length true \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--moe_permute_fusion true \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-3 \
--num_train_epochs 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-4 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--output_dir megatron_output/DeepSeek-V4-Flash \
--eval_steps 200 \
--save_steps 200 \
--max_length 4096 \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim true \
--no_save_rng true \
--sequence_parallel true \
--mtp_num_layers 1 \
--attention_backend flash
```
GPU memory usage:
![Memory Usage](../../resources/deepseek_v4/memory.png)
Training log and loss:
![loss](../../resources/deepseek_v4/loss.png)
Tips:
- If you want to enable pipeline parallelism (PP), you also need to set `pipeline_model_parallel_layout`. For example:
```
--pipeline_model_parallel_size 2 \
--pipeline_model_parallel_layout 'Et*22|t*21mL' \
```
- Full-parameter training is also supported. You should lower the learning rate and increase the parallelism. Below is a 64-GPU training example:
```
--lr 1e-5 \
--min_lr 1e-6 \
--tensor_model_parallel_size 1 \
--expert_model_parallel_size 8 \
--pipeline_model_parallel_size 8 \
--pipeline_model_parallel_layout Et*5|t*5|t*6|t*6|t*6|t*5|t*5|t*5mL \
```
- Packing/CP support: Requires installing the mcore-bridge/ms-swift main branch. Refer to these two PRs: [ms-swift#9705](https://github.com/modelscope/ms-swift/pull/9705), [mcore-bridge#140](https://github.com/modelscope/mcore-bridge/pull/140). To use CP, you need to set the following additionally (must be used together with packing `--packing true`, and note the merge of this PR [megatron-core#5706](https://github.com/NVIDIA/Megatron-LM/pull/5706)):
```
--sequence_packing_scheduler dp_balanced \
--cp_partition_mode contiguous \
```
- TP is not supported for now, pending support from Megatron-Core.
- FP8 training: you can enable FP8 training and save the weights in FP8 by setting the parameters below. Full-parameter training is recommended. If you want to use LoRA + FP8, you should save only the LoRA weights (set `--merge_lora false`) and perform Merge-LoRA against the BF16 weights (FP8 has limited precision and the LoRA delta would be rounded to 0). See [this example](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/fp8/lora.sh).
```
--fp8_recipe blockwise \
--fp8_format e4m3 \
--fp8_param_gather true \
```
Inference with the trained model:
```shell
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
swift infer \
--model megatron_output/DeepSeek-V4-Flash/vx-xxx/checkpoint-xxx-merged \
--infer_backend transformers \
--enable_thinking false \
--max_new_tokens 2048
```
Inference result:
![result](../../resources/deepseek_v4/infer_result.png)
Running vLLM inference:
- If you want to use vLLM for inference, you can refer to [this documentation](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash). You need FP4/FP8 precision weights.
- Additionally, you need to copy the original 'config.json' file and modify 'expert_dtype' (consistent with the config.json after training). This is because the file saved by transformers' `config.save_pretrained` differs from the original file, and vLLM is not compatible with the saved file.
- If you encounter tilelang issues, you can check [this issue](https://github.com/modelscope/ms-swift/issues/9494).
- mcore-bridge DeepSeek-V4 FP8 fix: [PR](https://github.com/modelscope/mcore-bridge/pull/133).
First perform quantization (note: this quantization will cause LoRA incremental information loss; this is only an example. It is recommended to use FP8 full-parameter training and export FP8 weights):
```shell
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
NPROC_PER_NODE=8 \
megatron export \
--model megatron_output/DeepSeek-V4-Flash/vx-xxx/checkpoint-xxx-merged \
--output_dir megatron_output/DeepSeek-V4-Flash/vx-xxx/checkpoint-xxx-merged-FP8 \
--to_hf true \
--fp8_recipe blockwise \
--fp8_format e4m3 \
--fp8_param_gather true \
--mtp_num_layers 1 \
--expert_model_parallel_size 8
```
vLLM launch command:
```shell
vllm serve megatron_output/DeepSeek-V4-Flash/vx-xxx/checkpoint-xxx-merged-FP8 \
--trust-remote-code \
--kv-cache-dtype fp8 \
--block-size 256 \
--enable-expert-parallel \
--tensor-parallel-size 8 \
--max-model-len 8192 \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_v4
```
docs/source_en/Customization/Architecture.md
+233
View File
@@ -0,0 +1,233 @@
# Architecture Introduction
ms-swift 4.0 adopts a modular design, with functional modules distributed in first-level directories, making it convenient for developers to perform custom extensions. This document will provide a detailed introduction to the functions of each module and customization methods.
## Agent Template
The mapping file for agent templates can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/agent_template/mapping.py). The design goal of agent template is to flexibly switch between different models for training based on a unified Agent dataset format, without modifying the data. During training, use `--agent_template` to specify the corresponding agent template.
All AgentTemplates need to inherit from `BaseAgentTemplate` and implement several methods: `_format_tools`, `_format_tool_calls`, `_format_tool_responses`, `get_toolcall`.
- _format_tools: Format `tools` and `system` to compose a complete system.
- _format_tool_calls: Format the tool_call part `[{"role": "tool_call", "content": "..."}, {"role": "tool_call", "content": "..."}]` and finally return a string.
- _format_tool_responses: Format the tool (also called tool_response) part `[{"role": "tool", "content": "..."}, {"role": "tool", "content": "..."}]`.
- get_toolcall: Used during deployment to parse the tool name and parameters from the model output content, returning `List[Function]`.
How to debug:
```python
data = {"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"天气预报。获取实时空气质量。当前空气质量，PM2.5，PM10信息\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"城市名，例如：上海\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "北京和上海今天的天气情况"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"北京\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"上海\"}}"}, {"role": "tool_response", "content": "{\"city\": \"北京\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"上海\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "根据天气预报工具，北京今天的空气质量指数为10，属于良好水平；上海今天的空气质量指数为72，属于轻度污染水平。"}]}
from swift import get_processor, get_template
tokenizer = get_processor('Qwen/Qwen3.5-2B')
template = get_template(tokenizer) # Use default agent template
# template = get_template(tokenizer, agent_template='qwen3_5')
print(f'agent_template: {template._agent_template}')
template.set_mode('train')
encoded = template.encode(data)
print(f'[INPUT_IDS] {template.safe_decode(encoded["input_ids"])}\n')
print(f'[LABELS] {template.safe_decode(encoded["labels"])}')
```
If you want to provide us with a PR, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/tests/test_align/test_template/test_agent.py) to write your test cases.
## Callbacks
The mapping file for callbacks can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/callbacks/mapping.py). Callbacks can customize the behavior at key points in the trainer. After customization, you need to register them in the mapping and use `--callbacks` to specify the corresponding callback class during training. For example, you can customize:
```python
class CustomCallback(TrainerCallback):
def on_train_begin(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
# Doing something when the training begins.
pass
def on_save(self, args: TrainingArguments, state: TrainerState, control: TrainerControl, **kwargs):
# Doing something when save checkpoint
pass
```
All callback classes need to inherit from `TrainerCallback` in base.py and override its methods. The interface is consistent with transformers' `TrainerCallback`, please refer to transformers' [callback documentation](https://huggingface.co/docs/transformers/main_classes/callback).
## Loss
The mapping file for Loss can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
Swift supports custom loss (currently only supports sft/pretrain/reranker/embedding tasks). After registration, set `--loss_type <loss-name>` during training to use your custom loss method.
Custom Loss needs to inherit from `BaseLoss` and implement the `__call__` method, returning a scalar Tensor. You can refer to [CustomCrossEntropyLoss](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/loss/causal_lm.py#L5) for customization. For example:
```python
class CustomLoss(BaseLoss):
def __call__(self, outputs, labels, **kwargs) -> torch.Tensor:
pass
```
## Loss Scale
The mapping file for loss scale can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/mapping.py). In pretrain and sft tasks, the loss of trainable tokens is averaged, meaning each token is treated equally. However, in some cases, certain tokens need extra attention and should be assigned higher weights, or some tokens should not be trained. loss_scale allows developers to freely define their own token weights. (Pretrain and SFT support using loss_scale to control whether tokens participate in training and their weight sizes, while in RLHF, it only supports controlling whether tokens participate in training)
You can customize loss scale by inheriting the LossScale base class and implementing the `get_loss_scale` method.
```python
class CustomLossScale(LossScale):
def get_loss_scale(self, context: str, **kwargs) -> Tuple[List[str], List[float]]:
...
```
The `get_loss_scale` function returns a Tuple. The first return is a list of decomposed strings, and the second parameter is a list of loss_scales corresponding to the strings. The float value represents the weight. For example, the following weight setting:
```text
["学习", "好", "数学", "是", "重要", "的"]
[1.0, 0.5, 2.0, 0.5, 2.0, 0.1]
```
In the example, we place more emphasis on the words "数学" and "重要" because their loss_scale is 2.0.
Of course, we also need to pay attention to the core logic of the `__call__` method, namely the influence of the loss_scale base strategy (base_strategy) all/default/last_round on loss_scale. For details, refer to the introduction in the [Command-line Parameters Documentation](../Instruction/Command-line-parameters.md). Also, refer to the influence of the 'loss' field in the dataset on loss_scale in the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
```python
if loss or loss is None and (self.base_strategy == 'all' or
(self.base_strategy == 'default' and is_assistant) or
(self.base_strategy == 'last_round' and is_assistant and is_last_round)):
new_context, loss_scale = self.get_loss_scale(context, query=query)
else:
new_context, loss_scale = [context], [0.]
```
In addition, you can also use [JSON configuration files](https://github.com/modelscope/ms-swift/tree/main/swift/loss_scale/config) and inherit the built-in ConfigLossScale class to customize loss_scale. Currently, two configuration methods are supported: exact string matching and regular expression matching. You can refer to the content in [Agent Support Documentation](../Instruction/Agent-support.md#usage-of-loss_scale) for understanding.
- Exact string matching, for example, refer to `react.json`, `qwen.json`. The JSON needs to contain a mapping of `Dict[str, List[float]]`. The string represents a keyword, and the list needs to have two values. We will split the string into multiple segments based on the keyword. The first value in the list represents the weight of the keyword, and the second value represents the weight of the content after this keyword and before the next keyword.
- Regular expression matching, for example, refer to `ignore_empty_think.json`, `hermes.json`. The JSON needs to contain a mapping of `Dict[str, float]`. The string represents a regular expression pattern, and the float represents the weight of the matching string.
How to debug:
```python
from swift import get_processor, get_template
data = {"messages": [
{"role": "user", "content": "What is today's date?"},
{"role": "assistant", "content": (
"<think>\nI can get the current time by calling the `get_date` function.\n</think>\n"
'<tool_call>\n{"name": "get_date", "arguments": {}}\n</tool_call>'
)}
]}
template = get_template(get_processor('Qwen/Qwen3-8B'), loss_scale='hermes')
template.set_mode('train')
inputs = template.encode(data)
print(template.safe_decode(inputs['labels']))
print(inputs['loss_scale'])
```
## Metrics
The mapping file for metrics can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/metrics/mapping.py). This component is used in both ms-swift and Megatron-SWIFT.
- If used in ms-swift, you need to inherit the `EvalMetrics` base class from base.py and implement the `compute_metrics` function, returning a dictionary `Dict[str, float]`. You can refer to [NlgMetrics](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/metrics/nlg.py#L33) for customization.
- If used in Megatron-SWIFT, you need to inherit the `Metric` base class from utils.py and implement the `update` and `compute` methods. The compute method should return a dictionary `Dict[str, float]`.
You can customize metrics (currently only supports sft/pretrain/reranker/embedding tasks) and set `--eval_metric <metric-name>` during training to use your custom metrics.
## Optimizers
The mapping file for optimizers can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/optimizers/mapping.py). If you need to customize an optimizer, you need to inherit the `OptimizerCallback` base class and override the `create_optimizer` function. Use `--optimizer <optimizer-name>` during training to specify the custom optimizer.
- You can refer to [MultimodalOptimizerCallback](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/optimizers/multimodal.py#L43) for implementation. This class implements the functionality of vit_lr and aligner_lr, which uses different learning rates for vit, aligner, and LLM respectively.
## Tuner Plugin
The mapping file for Tuner plugins can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/tuner_plugin/mapping.py). If you need to customize a tuner, you need to inherit the `Tuner` base class and override the `prepare_model`, `save_pretrained`, `from_pretrained` functions.
- prepare_model: This function is called before training to process and prepare the original model, wrap it with the tuner, and set trainable parameters. For example: you can attach LoRA to certain layers and freeze certain layers.
- save_pretrained: This function is called during training to save the model.
- from_pretrained: This function is called during inference/resuming training to prepare the model and load weights.
You can refer to [LoRALLMTuner](https://github.com/modelscope/ms-swift/blob/0d7c9f5bc0e7e7d67d914ce6edeb9ce24f60746f/swift/tuner_plugin/lora_llm.py#L24) for implementation. This class implements the functionality of performing LoRA training on LLM and full parameter training on ViT.
## ORM
Examples can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/rewards/orm.py).
ORM is an Outcome Reward Model. ORM is generally implemented using regular expressions. ORM determines whether a response is correct. For example:
```python
class MathORM(ORM):
@staticmethod
def extract_boxed_result(text):
pattern = r'\\boxed{([^}]*)}'
match = re.search(pattern, text)
if match:
return match.group(1).strip()
else:
return None
def __call__(self, infer_requests: List[InferRequest], ground_truths: List[str],
**kwargs) -> List[float]:
rewards = []
predictions = [request.messages[-1]['content'] for request in infer_requests]
for prediction, ground_truth in zip(predictions, ground_truths):
res1 = MathORM.extract_boxed_result(prediction) or ''
res2 = MathORM.extract_boxed_result(ground_truth) or ''
rewards.append(float(res1.strip() == res2.strip()))
return rewards
orms = {
'math': MathORM,
}
```
In the code above, we define a process for parsing mathematical responses. If the results are the same, it returns a score of 1.0, otherwise 0.0. Unlike PRM, this class has an additional parameter `ground_truths` in infer,
which contains the actual labels (standard responses defined in the dataset) of the corresponding infer_requests.
## PRM
Examples can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/rewards/prm.py).
PRM is a Process Reward Model, which will be used in the `swift sample` command. The interface that PRM needs to support is relatively simple:
```python
class PRM:
def __init__(self):
# init here
pass
def __call__(self, infer_requests: List[InferRequest], **kwargs) -> List[Union[float, List[float]]]:
raise NotImplementedError
```
The InferRequest comes from `swift.infer_engine`, and the returned `List[Union[float, List[float]]]` can contain either a reward or multiple rewards. Developers can obtain queries and responses from infer_requests and split them according to their own methods. For example:
```text
Let's think step by step.
Step1: xxx
Step2: xxx
So, the answer is ...
```
## Introduction to Other Directory Structures
- arguments: Command-line parameter definitions, such as: `SftArguments`, `RLHFArguments`, etc.
- cli: Swift command-line mechanism and startup files. For example, `swift sft ...` is equivalent to `python swift/cli/main.py sft ...` and also equivalent to `python swift/cli/sft.py ...`.
- config: deepspeed/fsdp2 configuration files.
- dataloader: Implementation of dataloader, including shard/dispatcher methods.
- dataset: Dataset-related module implementation, including data preprocessing, packing, streaming data, etc. Registration of built-in datasets is in the `dataset/dataset` and `dataset/data` folders. For details, refer to [Custom Dataset Documentation](Custom-dataset.md).
- infer_engine: Inference engine implementation. Includes inference engine implementations with transformers/vllm/sglang/lmdeploy as backends.
- megatron: Megatron-SWIFT implementation.
- model: Model loading and registration. For details, refer to [Custom Model Documentation](Custom-model.md), [Multimodal Model Registration Best Practices](../BestPractices/MLLM-Registration.md).
- pipelines: Main function pipeline implementations for `swift sft/rlhf/infer`, etc., including `sft_main/rlhf_main/infer_main`, etc.
- rlhf_trainers: Trainer implementations for algorithms such as GRPO/GKD/DPO/KTO/RM.
- rollout: Sampling implementation of the rollout process in RL algorithms.
- rewards: Reward function implementation in RL algorithms, supporting custom reward calculation logic.
- template: Implementation and registration of dialogue templates, including the logic for converting messages to input_ids for various tasks, as well as data_collator-related logic. For details, refer to [Custom Model Documentation](Custom-model.md), [Multimodal Model Registration Best Practices](../BestPractices/MLLM-Registration.md).
- trainers: Trainer implementations for pretrain/SFT/Embedding/Reranker/sequence classification tasks.
- ui: `swift web-ui` interface training and inference implementation.
docs/source_en/Customization/Custom-dataset.md
+448
View File
@@ -0,0 +1,448 @@
# Custom Dataset
There are three methods for accessing custom datasets, each offering progressively greater control over preprocessing functions but also increasing in complexity. For example, Solution 1 is the most convenient but offers the least control over preprocessing functions, requiring prior conversion of the dataset into a specific format:
1. **Recommended**: Directly use the command line parameter to access the dataset with `--dataset <dataset_path1> <dataset_path2>`. This will use `AutoPreprocessor` to convert your dataset into a standard format (supporting four dataset formats; see the introduction to AutoPreprocessor below). You can use `--columns` to transform column names. The supported input formats include csv, json, jsonl, txt, and folders (e.g. git clone open-source datasets). This solution does not require modifying `dataset_info.json` and is suitable for users new to ms-swift. The following two solutions are suitable for developers looking to extend ms-swift.
2. Add the dataset to `dataset_info.json`, which you can refer to in the built-in [dataset_info.json](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/data/dataset_info.json) of ms-swift. This solution also uses AutoPreprocessor to convert the dataset to a standard format. `dataset_info.json` is a list of metadata for datasets, and one of the fields ms_dataset_id/hf_dataset_id/dataset_path must be filled. Column name transformation can be done through the `columns` field. Datasets added to `dataset_info.json` or registered ones will automatically generate [supported dataset documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) when running [run_dataset_info.py](https://github.com/modelscope/ms-swift/blob/main/scripts/utils/run_dataset_info.py). In addition, you can use the external `dataset_info.json` approach by parsing the JSON file with `--custom_dataset_info xxx.json` (to facilitate users who prefer `pip install` over `git clone`), and then specify `--dataset <dataset_id/dataset_dir/dataset_path>`.
3. Manually register the dataset to have the most flexible customization capability for preprocessing functions, allowing the use of functions to preprocess datasets, but it is more difficult. You can refer to the [built-in datasets](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/dataset/llm.py) or [examples](https://github.com/modelscope/ms-swift/blob/main/examples/custom). You can specify `--external_plugins xxx.py` to parse external registration content (convenient for users who use pip install instead of git clone).
- Solutions one and two leverage solution three under the hood, where the registration process occurs automatically.
The following is an introduction to the dataset formats that `AutoPreprocessor` can handle:
The standard dataset format for ms-swift accepts keys such as: 'messages', 'rejected_response', 'label', 'images', 'videos', 'audios', 'tools', and 'objects'. Among these, 'messages' is a required key. 'rejected_response' is used for DPO and other RLHF training, 'label' is used for KTO training and classification model training. The keys 'images', 'videos', and 'audios' are used to store paths or URLs for multimodal data, 'tools' is used for Agent tasks, and 'objects' is used for grounding tasks.
There are three core preprocessors in ms-swift: `MessagesPreprocessor`, `AlpacaPreprocessor`, and `ResponsePreprocessor`. `MessagesPreprocessor` is used to convert datasets in the messages and sharegpt format into the standard format. `AlpacaPreprocessor` converts datasets in the alpaca format, while `ResponsePreprocessor` converts datasets in the query/response format. `AutoPreprocessor` automatically selects the appropriate preprocessor for the task.
The following four formats will all be converted into the `messages` field of the ms-swift standard format under the processing of `AutoPreprocessor`, meaning they can all be directly used with `--dataset <dataset-path>`:
Messages format (standard format):
```jsonl
{"messages": [{"role": "system", "content": "<system>"}, {"role": "user", "content": "<query1>"}, {"role": "assistant", "content": "<response1>"}, {"role": "user", "content": "<query2>"}, {"role": "assistant", "content": "<response2>"}]}
```
- Note: The system part is optional. The system in the dataset has a higher priority than the `--system` passed through the command line, followed by the `default_system` defined in the template.
ShareGPT format:
```jsonl
{"system": "<system>", "conversation": [{"human": "<query1>", "assistant": "<response1>"}, {"human": "<query2>", "assistant": "<response2>"}]}
```
Query-Response format:
```jsonl
{"system": "<system>", "query": "<query2>", "response": "<response2>", "history": [["<query1>", "<response1>"]]}
```
Note: The following fields will be automatically converted to the corresponding system, query, and response fields. (The 'solution' field will be retained)
- system: 'system', 'system_prompt'.
- query: 'query', 'prompt', 'input', 'instruction', 'question', 'problem'.
- response: 'response', 'answer', 'output', 'targets', 'target', 'answer_key', 'answers', 'solution', 'text', 'completion', 'content'.
Alpaca format:
```jsonl
{"system": "<system>", "instruction": "<query-inst>", "input": "<query-input>", "output": "<response>"}
```
- Note: The instruction and input fields will be combined into the query field. If instruction and input are not empty strings, then `query = f'{instruction}\n{input}'`.
## Standard Dataset Format
The following outlines the standard dataset format for ms-swift, where the "system" field is optional and uses the "default_system" defined in the template by default. The four dataset formats introduced earlier can also be processed by AutoPreprocessor into the standard dataset format.
### Pre-training
```jsonl
{"messages": [{"role": "assistant", "content": "I love music"}]}
{"messages": [{"role": "assistant", "content": "Coach, I want to play basketball"}]}
{"messages": [{"role": "assistant", "content": "Which is more authoritative, tomato and egg rice or the third fresh stir-fry?"}]}
```
### Supervised Fine-tuning
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}]}
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}]}
```
- You can add a `"loss"` field to control whether the loss is computed for the corresponding model response ("role" is "assistant"). This field defaults to `None`. If `"loss"` is set to `true`, the loss will be computed for the corresponding content (the specific `loss_scale` is still determined by `--loss_scale`); if `"loss"` is set to `false`, the loss will not be computed for the corresponding content. Note that this field only takes effect for parts where `"role"` is `"assistant"`. This field takes priority over the basic strategies of the `--loss_scale` command-line argument (i.e., `'default'`, `'last_round'`, `'all'`). For example, when `loss_scale` is set to `'default+ignore_empty_think'`, the `"loss"` field takes priority over `'default'`, but `'ignore_empty_think'` still takes effect.
- You can add a `"loss_scale"` field to control the `loss_scale` for the corresponding model response ("role" is "assistant"). (ms-swift >= 4.2.0) Defaults to `None`. This field takes priority over other strategy components of the `--loss_scale` command-line argument, such as `'ignore_empty_think'`, `'hermes'`, etc. If any value greater than `1` appears in `loss_scale`, you need to additionally set `--is_binary_loss_scale false`.
```jsonl
{"messages": [{"role": "user", "content": "Hello!"}, {"role": "assistant", "content": "Hi, how can I help you?", "loss": false}, {"role": "user", "content": "What is 1+1?"}, {"role": "assistant", "content": "It equals 2", "loss": true}]}
{"messages": [{"role": "user", "content": "hello!"}, {"role": "assistant", "content": "<think>\n...\n</think>\n", "loss_scale": 1.0}, {"role": "assistant", "content": "hi!", "loss_scale": 2.0}, {"role": "user", "content": "1+1=?"}, {"role": "assistant", "content": "<think>\n...\n</think>\n1+1=3", "loss": false}]}
```
Use the following script to test:
```python
from swift import get_processor, get_template
data = {"messages": [
{"role": "user", "content": "hello!"},
{"role": "assistant", "content": "<think>\n...\n</think>\n", "loss_scale": 1.},
{"role": "assistant", "content": "hi!", "loss_scale": 2.},
{"role": "user", "content": "1+1=?"},
{"role": "assistant", "content": "<think>\n...\n</think>\n1+1=3", "loss": False},
]}
template = get_template(get_processor('Qwen/Qwen3-8B'), loss_scale='default+ignore_empty_think',
is_binary_loss_scale=False)
template.set_mode('train')
inputs = template.encode(data)
print(template.safe_decode(inputs['labels']))
print(inputs['loss_scale'])
```
Note: If you set "loss"/"loss_scale" on consecutive "tool_call" messages in the messages list, only the configuration of the first "tool_call" takes effect. For example:
```jsonl
{"messages": [..., {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}", "loss": false}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, ...]}
```
- The "chat_template_kwargs" field (requires ms-swift>=4.3.0) allows you to control template multimodal parameters such as min_pixels, max_pixels, fps, as well as parameters like enable_thinking (during inference) at the **sample level** by passing this field in the dataset. The following parameters are supported by different models:
- Among them, "enable_thinking", "preserve_thinking" and "response_prefix" are supported by all models (takes effect during inference); the "max_pixels" parameter is supported by all multimodal models.
- Qwen series multimodal models: parameters supported by qwen_vl_utils/qwen_omni_utils such as min_pixels, max_pixels, fps, etc.
```jsonl
{"messages": [{"role": "user", "content": "<image>What is this"}, {"role": "assistant", "content": "This is a rabbit", "loss": false}], "chat_template_kwargs": {"max_pixels": 1048576}}
{"messages": [{"role": "user", "content": "who are you?"}], "chat_template_kwargs": {"enable_thinking": false}}
```
#### Channel Loss
If you want to use channel loss, you need to set `--enable_channel_loss true` and add a "channel" field to your dataset. Channel loss is compatible with techniques such as packing, padding-free, and loss scaling.
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}], "channel": "general"}
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}], "channel": "math"}
```
### RLHF
#### DPO/ORPO/CPO/SimPO/RM
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}], "rejected_response": "I don't know"}
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}], "rejected_response": "I don't know"}
```
The format of multimodal data should follow the specifications in [Multimodal Dataset](#multimodal), with additional columns such as `images` to represent other modality inputs. When it is necessary to associate different image information with preference data, the `rejected_images` field can be used to indicate the images related to the rejected responses. In the alignment dataset, at least one of `rejected_images` or `rejected_response` must be provided for each entry.
> Note: RM additionally supports the margin column. For details, refer to the [RM documentation](../Instruction/RLHF.md#rm).
Sure, you can also directly use `rejected_messages` instead of only providing `rejected_response` / `rejected_images`, which offers greater flexibility (e.g., for multimodal or agent scenarios). If you use "rejected_messages", then in multimodal scenarios you must also provide "rejected_images", "rejected_audios", "rejected_videos", etc.; in Agent scenarios you must also provide "rejected_tools", etc. An example of the multimodal data format is as follows:
- If using `rejected_response`, the default values for 'rejected_images/rejected_audios/rejected_videos/rejected_tools' are 'images/audios/videos/tools'; if using `rejected_messages`, they need to be passed in additionally.
```jsonl
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a puppy."}], "rejected_images": ["kitten.png"]}
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "rejected_images": ["puppy.png"]}
```
The above format is equivalent to:
```jsonl
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_response": "This is a puppy."}
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_images": ["puppy.png"]}
# Example 1 can also be written as:
{"messages": [{"role": "user", "content": "<image>What is this?"}, {"role": "assistant", "content": "This is a kitten."}], "images": ["kitten.png"], "rejected_response": [{"role": "assistant", "content": "This is a puppy."}]}
```
You can also organize the Agent dataset in the following format:
```jsonl
# It will find the position of the last user in `messages`, and replace the subsequent content with `rejected_response` to form `rejected_messages`
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}], "rejected_response": [{"role": "assistant", "content": "I don't know."}]}
```
How to debug:
```python
from swift import get_processor, get_template
data = {"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}], "rejected_response": [{"role": "assistant", "content": "I don't know."}]}
template = get_template(get_processor('Qwen/Qwen3.5-4B'), loss_scale='last_round')
template.set_mode('rlhf') # For details, refer to the `template_mode` parameter description in the command-line documentation.
inputs = template.encode(data)
print(template.safe_decode(inputs['chosen_labels']))
print(template.safe_decode(inputs['rejected_labels']))
```
#### KTO
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "I don't know"}], "label": false}
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}], "label": true}
```
#### PPO/GRPO
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}]}
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}]}
{"messages": [{"role": "user", "content": "What is your name?"}]}
```
- Note: GRPO will pass through all additional field content to the ORM, unlike other training methods that, by default, delete extra fields. For example, you can additionally pass in 'solution'. The custom ORM needs to include a positional argument called `completions`, with other arguments as keyword arguments passed through from the additional dataset fields.
#### GKD
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}, {"role": "assistant", "content": "Tomorrow's weather will be sunny"}]}
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}, {"role": "assistant", "content": "It equals 3"}]}
```
When under on-policy training, the final round of the 'assistant' part is not required (the student model generates data during training, the response from dataset will be removed):
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Tell me tomorrow's weather"}]}
{"messages": [{"role": "system", "content": "You are a useful and harmless math calculator"}, {"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}]}
```
### Sequence Classification
**Single-label Task**:
```jsonl
{"messages": [{"role": "user", "content": "The weather is really nice today"}], "label": 1}
{"messages": [{"role": "user", "content": "Today is really unlucky"}], "label": 0}
{"messages": [{"role": "user", "content": "So happy"}], "label": 1}
```
**Multi-label Task**:
```jsonl
{"messages": [{"role": "user", "content": "<sentence>"}], "label": []}
{"messages": [{"role": "user", "content": "<sentence>"}], "label": [0, 2]}
{"messages": [{"role": "user", "content": "<sentence>"}], "label": [1, 3, 5]}
```
**Single Regression Task**:
```jsonl
{"messages": [{"role": "user", "content": "Calculate the similarity between two sentences, with a range of 0-1.\nsentence1: <sentence1>\nsentence2: <sentence2>"}], "label": 0.8}
```
**Multi Regression Task**:
```jsonl
{"messages": [{"role": "user", "content": "<sentence>"}], "label": [1.2, -0.6, 0.8]}
```
### Embedding
Please refer to [Embedding training document](../BestPractices/Embedding.md#dataset-format).
### Reranker
Please refer to [Reranker training document](../BestPractices/Reranker.md#dataset-format).
### Multimodal
For multimodal datasets, the format is the same as the aforementioned tasks. The difference lies in the addition of several keys: `images`, `videos`, and `audios`, which represent the URLs or paths (preferably absolute paths) of multimodal resources. The tags `<image>`, `<video>`, and `<audio>` indicate where to insert images, videos, or audio. MS-Swift supports multiple images, videos, and audio files. These special tokens will be replaced during preprocessing, as referenced [here](https://github.com/modelscope/ms-swift/blob/main/swift/template/templates/qwen.py#L198). The four examples below respectively demonstrate the data format for plain text, as well as formats containing image, video, and audio data.
Pre-training:
```jsonl
{"messages": [{"role": "assistant", "content": "Pre-trained text goes here"}]}
{"messages": [{"role": "assistant", "content": "<image>is a puppy, <image>is a kitten"}], "images": ["/xxx/x.jpg", "/xxx/x.png"]}
{"messages": [{"role": "assistant", "content": "<audio>describes how nice the weather is today"}], "audios": ["/xxx/x.wav"]}
{"messages": [{"role": "assistant", "content": "<image>is an elephant, <video>is a lion running"}], "images": ["/xxx/x.jpg"], "videos": ["/xxx/x.mp4"]}
```
Supervised Fine-tuning:
```jsonl
{"messages": [{"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "The capital of Zhejiang is Hangzhou."}]}
{"messages": [{"role": "user", "content": "<image><image>What is the difference between the two images?"}, {"role": "assistant", "content": "The first one is a kitten, and the second one is a puppy."}], "images": ["/xxx/x.jpg", "/xxx/x.png"]}
{"messages": [{"role": "user", "content": "<audio>What did the audio say?"}, {"role": "assistant", "content": "The weather is really nice today."}], "audios": ["/xxx/x.mp3"]}
{"messages": [{"role": "system", "content": "You are a helpful and harmless assistant."}, {"role": "user", "content": "<image>What is in the image, <video>What is in the video?"}, {"role": "assistant", "content": "The image shows an elephant, and the video shows a puppy running on the grass."}], "images": ["/xxx/x.jpg"], "videos": ["/xxx/x.mp4"]}
```
- Note: The following fields will be automatically converted to the corresponding images, videos, and audios fields.
- images: image, images.
- videos: video, videos.
- audios: audio, audios.
- If you need to pass base64 data instead of file paths, here are sample examples: `"videos": ['data:video/mp4;base64,{base64_encoded}']`, `"images": ['data:image/jpg;base64,{base64_encoded}']`.
- If you wish to directly pass in video frames instead of a video file, you can use the following format: `"videos": [["/xxx/x.png", "/xxx/y.png"], ["/xxx/a.png", "/xxx/b.png", "/xxx/c.png"]]`. This format is supported only by certain models, including Qwen2/2.5/3-VL, Qwen2.5/3-Omni, and their derivative models.
The data format for RLHF and sequence classification of multimodal models can reference the format of pure text large models, with additional fields such as `images` added on top of that.
#### Grounding
For grounding (object detection) tasks, ms-swift supports two methods:
1. Directly use the data format of the grounding task corresponding to the model. For example, the format for qwen2-vl is as follows:
```
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Describe the image."}, {"role": "assistant", "content": "<|object_ref_start|>a dog<|object_ref_end|><|box_start|>(221,423),(569,886)<|box_end|> and <|object_ref_start|>a woman<|object_ref_end|><|box_start|>(451,381),(733,793)<|box_end|> are playing on the beach"}], "images": ["/xxx/x.jpg"]}
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Find the <|object_ref_start|>sheep<|object_ref_end|> in the image"}, {"role": "assistant", "content": "<|box_start|>(101,201),(150,266)<|box_end|><|box_start|>(401,601),(550,666)<|box_end|>"}], "images": ["/xxx/x.jpg"]}
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Help me open Google Chrome"}, {"role": "assistant", "content": "Action: click(start_box='<|box_start|>(246,113)<|box_end|>')"}], "images": ["/xxx/x.jpg"]}
```
When using this type of data, please note:
- Different models have different special characters and data format for the grounding task.
- The handling of bounding box normalization varies across different models: for example, qwen2.5-vl uses absolute coordinates, while qwen2/3-vl and internvl2.5 require bounding box coordinates to be normalized to the thousandth scale.
- Note: Qwen2.5-VL uses absolute coordinates, so you need to be careful with image resizing each time. If you use the dataset format from Option 1, you need to resize the images in advance (height and width must be multiples of 28) and scale the coordinates accordingly. If you use the dataset format from Option 2, ms-swift will handle image resizing for you. You can still use `MAX_PIXELS` or `--max_pixels` for image resizing (training only; for inference, you still need to handle image resizing yourself).
2. Use ms-swift's grounding data format:
```
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Describe the image."}, {"role": "assistant", "content": "<ref-object><bbox> and <ref-object><bbox> are playing on the beach"}], "images": ["/xxx/x.jpg"], "objects": {"ref": ["a dog", "a woman"], "bbox": [[331.5, 761.4, 853.5, 1594.8], [676.5, 685.8, 1099.5, 1427.4]]}}
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Find the <ref-object> in the image"}, {"role": "assistant", "content": "<bbox><bbox>"}], "images": ["/xxx/x.jpg"], "objects": {"ref": ["sheep"], "bbox": [[90.9, 160.8, 135, 212.8], [360.9, 480.8, 495, 532.8]]}}
{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "<image>Help me open Google Chrome"}, {"role": "assistant", "content": "Action: click(start_box='<bbox>')"}], "images": ["/xxx/x.jpg"], "objects": {"ref": [], "bbox": [[615, 226]]}}
```
The format will automatically convert the dataset format to the corresponding model's grounding task format and select the appropriate model's bbox normalization method. Compared to the general format, this format includes an additional "objects" field, which contains the following subfields:
- ref: Used to replace the `<ref-object>` placeholder in messages. The length of `ref` should match the number of `<ref-object>` instances.
- bbox: Used to replace the `<bbox>` placeholder in messages. If the length of each box in the bbox is 2, it represents the x and y coordinates. If the box length is 4, it represents the x and y coordinates of two points. The length of `bbox` should match the number of `<bbox>` instances.
- Note: `<ref-object>` and `<bbox>` do not have a corresponding relationship; references and bounding boxes replace their own placeholders separately.
- bbox_type: Optional values are 'real' and 'norm1'. The default is 'real', meaning the bbox represents the actual bounding box value. If set to 'norm1', the bbox is normalized to the range 0~1.
- image_id: Typically used for multi-image grounding tasks. This parameter only takes effect when bbox_type is 'real', representing which image the bbox corresponds to, used for scaling the bbox. The index starts from 0, and defaults to all being the 0th image. The length of image_id needs to be consistent with the length of bbox. For example: if the length of bbox is 10 and the length of images is 2, then the length of image_id needs to be 10, with values within the set `{0, 1}`.
For Qwen2.5-VL/Qwen3-VL, you can set the environment variable `QWENVL_BBOX_FORMAT='new'` (default is `'legacy'`) to be compatible with the [official cookbook](https://github.com/QwenLM/Qwen3-VL/blob/main/cookbooks/2d_grounding.ipynb) format. Define your dataset in the following format:
```jsonl
{"messages": [{"role": "user", "content": "<image>Locate the <ref-object> in the image"}, {"role": "assistant", "content": "[\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"},\n\t{\"bbox_2d\": <bbox>, \"label\": \"<ref-object>\"}\n]"}], "images": ["cat.png"], "objects": {"ref": ["sheep", "sheep", "sheep"], "bbox": [[90.9, 160.8, 135, 212.8], [360.9, 480.8, 495, 532.8]]}}
```
Testing the final format of the grounding data in ms-swift format:
```python
import os
os.environ["MAX_PIXELS"] = "1003520"
from swift import get_processor, get_template
processor = get_processor('Qwen/Qwen2.5-VL-7B-Instruct')
template = get_template(processor)
data = {...}
template.set_mode('train')
encoded = template.encode(data, return_template_inputs=True)
print(f'[INPUT_IDS] {template.safe_decode(encoded["input_ids"])}\n')
print(f'[LABELS] {template.safe_decode(encoded["labels"])}')
print(f'images: {encoded["template_inputs"].images}')
```
### Agent Format
Here are example data samples for a text-only Agent and a multimodal Agent:
```jsonl
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}]}
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"click\", \"description\": \"Click on a position on the screen\", \"parameters\": {\"type\": \"object\", \"properties\": {\"x\": {\"type\": \"integer\", \"description\": \"X-coordinate representing the horizontal position on the screen\"}, \"y\": {\"type\": \"integer\", \"description\": \"Y-coordinate representing the vertical position on the screen\"}}, \"required\": [\"x\", \"y\"]}}}]", "messages": [{"role": "user", "content": "<image>What time is it now?"}, {"role": "assistant", "content": "<think>\nI can check the current time by opening the calendar app.\n</think>\n"}, {"role": "tool_call", "content": "{\"name\": \"click\", \"arguments\": {\"x\": 105, \"y\": 132}}"}, {"role": "tool_response", "content": "{\"images\": \"<image>\", \"status\": \"success\"}"}, {"role": "assistant", "content": "Successfully opened the calendar app. The current time is 11 o'clock in the morning."}], "images": ["desktop.png", "calendar.png"]}
```
- When the `agent_template` is set to "react_en", "hermes", etc., this format is compatible with training for all model Agents and allows easy switching between different models.
- Among them, `tools` is a JSON string containing a list of tools, and the `content` section of `messages` where the `role` is `'tool_call'` or `'tool_response/tool'` must also be a JSON string.
- The `tools` field will be combined with the `{"role": "system", ...}` section during training/inference according to the `agent_template`, forming a complete system section.
- The `{"role": "tool_call", ...}` part will automatically be converted into corresponding formats of `{"role": "assistant", ...}` based on the `agent_template`. Multiple consecutive `{"role": "assistant", ...}` entries will be concatenated to form a complete assistant_content.
- The `{"role": "tool_response", ...}` can also be written as `{"role": "tool", ...}`, these two forms are equivalent. This part will also be automatically converted according to the `agent_template`. During training, this part does not participate in loss calculations, similar to `{"role": "user", ...}`.
- This format supports parallel tool calls; refer to the first data sample for an example. In multimodal Agent data samples, the number of `<image>` tags should match the length of "images", and their positions indicate where the image features are inserted. It also supports other modalities, such as audios and videos.
- Note: You can also manually process the data into the messages format with roles set to system, user, or assistant. The purpose of agent_template is to automatically map the tools field and the messages with roles tool_call and tool_response into the standard messages format with roles system, user, and assistant.
- For more details, please refer to [Agent Documentation](../Instruction/Agent-support.md).
### Text-to-Image Format
```jsonl
{"messages": [{"role": "system", "content": "You are a useful and harmless assistant"}, {"role": "user", "content": "Draw me an apple"}, {"role": "assistant", "content": "<image>"}], "images": ["/xxx/x.jpg"]}
```
## dataset_info.json
You can refer to the ms-swift built-in [dataset_info.json](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/data/dataset_info.json). This approach uses the AutoPreprocessor function to convert the dataset into a standard format. The dataset_info.json file contains a list of metadata about the dataset. Here are some examples:
```json
[
{
"ms_dataset_id": "xxx/xxx"
},
{
"dataset_path": "<dataset_dir/dataset_path>"
},
{
"ms_dataset_id": "<dataset_id>",
"subsets": ["v1"],
"split": ["train", "validation"],
"columns": {
"input": "query",
"output": "response"
}
},
{
"ms_dataset_id": "<dataset_id>",
"hf_dataset_id": "<hf_dataset_id>",
"subsets": [{
"subset": "subset1",
"columns": {
"problem": "query",
"content": "response"
}
},
{
"subset": "subset2",
"columns": {
"messages": "_",
"new_messages": "messages"
}
}]
}
]
```
The following parameters are supported:
- ms_dataset_id: Refers to the DatasetMeta parameter.
- hf_dataset_id: Refers to the DatasetMeta parameter.
- dataset_path: Refers to the DatasetMeta parameter.
- dataset_name: Refers to the DatasetMeta parameter.
- subsets: Refers to the DatasetMeta parameter.
- split: Refers to the DatasetMeta parameter.
- columns: Transforms column names before preprocessing the dataset.
## Dataset Registration
`register_dataset` will register the dataset in `DATASET_MAPPING`. You can call the function `register_dataset(dataset_meta)` to complete the dataset registration, where `dataset_meta` will store the metadata of the model. The parameter list for DatasetMeta is as follows:
- ms_dataset_id: The dataset_id for ModelScope, default is None.
- hf_dataset_id: The dataset_id for HuggingFace, default is None.
- dataset_path: Local path to the **dataset file/folder** (absolute path recommended). Default is None.
- dataset_name: The alias of the dataset, which can be specified via `--dataset <dataset_name>`. This is very convenient when the dataset_path is long. The default value is None.
- subsets: A list of subdataset names or a list of `SubsetDataset` objects, default is `['default']`. (The concepts of subdatasets and splits only exist for dataset_id or dataset_dir (open source datasets cloned via git)).
- split: Defaults to `['train']`.
- preprocess_func: A preprocessing function or callable object, default is `AutoPreprocessor()`. This preprocessing function takes an `HfDataset` as input and returns an `HfDataset` in the standard format.
- load_function: Defaults to `DatasetLoader.load`. If a custom loading function is needed, it should return an `HfDataset` in the standard format, allowing users maximum flexibility while bypassing the ms-swift dataset loading mechanism. This parameter usually does not need to be modified.
Below are examples of registering datasets:
```python
from swift.dataset import (
ResponsePreprocessor, DatasetMeta, register_dataset, SubsetDataset, load_dataset
)
from typing import Dict, Any
class CustomPreprocessor(ResponsePreprocessor):
def preprocess(self, row: Dict[str, Any]) -> Dict[str, Any]:
query = f"""Task: Judge whether the two sentences below are semantically similar.
Sentence 1: {row['text1']}
Sentence 2: {row['text2']}
Output the category [0/1]: 0 for different meanings, 1 for similar meanings.
"""
response = str(row['label'])
row = {
'query': query,
'response': response
}
return super().preprocess(row)
register_dataset(
DatasetMeta(
ms_dataset_id='swift/financial_classification',
subsets=[SubsetDataset('train', split=['train']), SubsetDataset('test', split=['test'])],
preprocess_func=CustomPreprocessor(),
))
if __name__ == '__main__':
# load_dataset returns train_dataset and val_dataset based on `split_dataset_ratio`
# Here, since we didn't pass `split_dataset_ratio` (defaults to 0), we take the first one (index 0)
dataset = load_dataset('swift/financial_classification:train')[0]
test_dataset = load_dataset('swift/financial_classification:test')[0]
print(f'dataset[0]: {dataset[0]}')
print(f'test_dataset[0]: {test_dataset[0]}')
```
docs/source_en/Customization/Custom-model.md
+39
View File
@@ -0,0 +1,39 @@
# Custom Model
The models built into ms-swift can be used directly by specifying either `model_id` or `model_path`: `--model <model_id_or_path>`. ms-swift determines the `model_type` based on the suffix of `model_id/model_path` and the `config.json` file.
Each `model_type` has a unique model structure, template, and loading method. Of course, you can also manually override these by passing `--model_type` and `--template`. You can check the supported `model_type` and templates in the [Supported Models and Datasets](../Instruction/Supported-models-and-datasets.md).
The following introduces how to register a new model and its corresponding template. For best practices, refer to [Best Practices for Registering Multimodal Models](../BestPractices/MLLM-Registration.md).
## Model Registration
Custom models are typically implemented using model registration. You can refer to the [built-in model](https://github.com/modelscope/ms-swift/blob/main/swift/model/models/qwen.py), the [built-in dialogue template](https://github.com/modelscope/ms-swift/blob/main/swift/template/templates/qwen.py), or the example code in the [examples](https://github.com/modelscope/ms-swift/blob/main/examples/custom). You can specify the `--external_plugins xxx.py` to parse the externally registered content, which is convenient for users installing via pip instead of git clone.
The `register_model` function registers a model in the `MODEL_MAPPING`. You can complete the model registration by calling the function `register_model(model_meta)`, where `model_meta` will store the model's metadata. The parameter list for ModelMeta is as follows:
- model_type: Required. The model type, which is also the unique ID.
- model_groups: Required. Lists the ModelScope/HuggingFace model IDs and local paths. Running the [run_model_info.py](https://github.com/modelscope/ms-swift/blob/main/scripts/utils/run_model_info.py) file will automatically generate the [supported models documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) and automatically match the model_type based on the `--model` suffix.
- loader: The loader for model and tokenizer/processor (multimodal models). Defaults to `swift.model.ModelLoader`.
- template: The default template type when `--template` is not additionally specified in the command line. Defaults to None.
- model_arch: The model architecture. Defaults to None. Multi-modal model training requires setting this parameter to determine the prefix for llm/vit/aligner.
- architectures: The architectures item in config.json, used to automatically match the model with its model_type. Defaults to `[]`.
- additional_saved_files: Files that need to be additionally saved during full parameter training and merge-lora. Defaults to `[]`.
- torch_dtype: The default dtype when `torch_dtype` is not passed during model loading. Defaults to None, read from config.json.
- is_multimodal: Indicates whether the model is multi-modal. Defaults to False.
- ignore_patterns: File patterns to be ignored when downloading from the hub. Defaults to `[]`.
The `register_template` function registers a dialogue template in `TEMPLATE_MAPPING`. To complete the registration of the dialogue template, simply call the function `register_template(template_meta)`, where `template_meta` will store the metadata of the template. The parameter list for TemplateMeta is as follows:
- template_type: Required. The type of dialogue template, which also serves as a unique ID.
- prefix: Required. The prefix of the dialogue template, usually encompassing parts like system, bos_token, and is generated independently of multi-turn dialogue loops. For example, the prefix for qwen is `[]`.
- prompt: Required. Represents the dialogue portion before `{{RESPONSE}}`. We use `{{QUERY}}` as a placeholder for the user's inquiry part. For example, the prompt for qwen is `['<|im_start|>user\n{{QUERY}}<|im_end|>\n<|im_start|>assistant\n']`.
- chat_sep: Required. The separator for each turn in multi-turn dialogues. If set to None, the template does not support multi-turn dialogue. For example, the chat_sep for qwen is `['<|im_end|>\n']`.
- suffix: Defaults to `[['eos_token_id']]`. The suffix part of the dialogue template, generated independently of multi-turn dialogue loops, usually the eos_token. For example, the suffix for qwen is `['<|im_end|>']`.
- template_cls: Defaults to `Template`. Customization is generally required when defining templates for multimodal models, particularly in customizing the `_encode`, `_post_encode`, and `_data_collator` functions.
- system_prefix: Defaults to None. The prefix for dialogue templates with a system. We use`{{SYSTEM}}`as a placeholder for the system. For example, the system_prefix for qwen is`['<|im_start|>system\n{{SYSTEM}}<|im_end|>\n']`.
- Note: If the system is empty and `prefix` can be replaced by `system_prefix`, you can write `prefix` as a prefix including the system without setting `system_prefix`.
- If the prefix does not include `{{SYSTEM}}` and system_prefix is not set, the template does not support the system.
- default_system: Defaults to None. The default system used when `--system` is not provided. For example, the default_system for qwen is `'You are a helpful assistant.'`.
- stop_words: Defaults to`[]`. Additional stop words besides eos_token and`suffix[-1]`. For example, the stop_words for qwen is`['<|endoftext|>']`
- Note: During inference, the output response will be filtered by eos_token and `suffix[-1]`, but additional stop_words will be retained.
docs/source_en/GetStarted/Quick-start.md
+112
View File
@@ -0,0 +1,112 @@
# Quick Start
🍲 **ms-swift** is a large model and multimodal large model fine-tuning and deployment framework provided by the ModelScope community. It now supports training (pre-training, fine-tuning, human alignment), inference, evaluation, quantization, and deployment for 600+ text-only large models and 400+ multimodal large models. Large models include: Qwen3, Qwen3.5, InternLM3, GLM4.5, Mistral, DeepSeek-R1, Llama4, etc. Multimodal large models include: Qwen3-VL, Qwen3-Omni, Llava, InternVL3.5, MiniCPM-V-4, Ovis2.5, GLM4.5-V, DeepSeek-VL2, etc.
🍔 In addition, ms-swift integrates the latest training technologies, including Megatron parallelism techniques such as TP, PP, CP, EP to accelerate training, as well as numerous GRPO algorithm family reinforcement learning algorithms including: GRPO, DAPO, GSPO, SAPO, CISPO, RLOO, Reinforce++, etc. to enhance model intelligence. ms-swift supports a wide range of training tasks, including preference learning algorithms such as DPO, KTO, RM, CPO, SimPO, ORPO, as well as Embedding, Reranker, and sequence classification tasks. ms-swift provides full-pipeline support for large model training, including acceleration for inference, evaluation, and deployment modules using vLLM, SGLang, and LMDeploy, as well as model quantization using GPTQ, AWQ, BNB, and FP8 technologies.
**Why Choose ms-swift?**
- 🍎 **Model Types**: Supports **600+ text-only large models**, **400+ multimodal large models**, and All-to-All full modality models from training to deployment full pipeline, with Day-0 support for popular models.
- **Dataset Types**: Built-in 150+ datasets for pre-training, fine-tuning, human alignment, multimodal and various other tasks, with support for custom datasets. Users only need to prepare datasets for one-click training.
- **Hardware Support**: Supports A10/A100/H100, RTX series, T4/V100, CPU, MPS, and domestic hardware Ascend NPU, etc.
- **Lightweight Training**: Supports lightweight fine-tuning methods such as LoRA, QLoRA, DoRA, LoRA+, LLaMAPro, LongLoRA, LoRA-GA, ReFT, RS-LoRA, Adapter, LISA, etc.
- **Quantized Training**: Supports training on BNB, AWQ, GPTQ, AQLM, HQQ, EETQ quantized models, requiring only 9GB training resources for 7B models.
- **Memory Optimization**: GaLore, Q-Galore, UnSloth, Liger-Kernel, Flash-Attention 2/3, and **Ulysses and Ring-Attention sequence parallelism techniques** support, reducing memory consumption for long-text training.
- **Distributed Training**: Supports distributed data parallelism (DDP), device_map simple model parallelism, DeepSpeed ZeRO2 ZeRO3, FSDP/FSDP2, and Megatron distributed training technologies.
- 🍓 **Multimodal Training**: Supports multimodal packing technology to improve training speed by 100%+, supports mixed modality data training with text, images, video and audio, and supports independent control of vit/aligner/llm.
- **Agent Training**: Supports Agent templates, allowing one dataset to be used for training different models.
- 🍊 **Training Tasks**: Supports pre-training and instruction fine-tuning, as well as training tasks such as DPO, GKD, KTO, RM, CPO, SimPO, ORPO, and supports **Embedding/Reranker** and sequence classification tasks.
- 🥥 **Megatron Parallelism**: Provides TP/PP/SP/CP/ETP/EP/VPP parallel strategies to significantly boost **MoE model training speed**. Supports full-parameter and LoRA training methods for 300+ pure text large models and 100+ multimodal large models. Supports CPT/SFT/GRPO/DPO/KTO/RM training tasks.
- 🍉 **Reinforcement Learning**: Built-in **rich GRPO family algorithms**, including GRPO, DAPO, GSPO, SAPO, CISPO, CHORD, RLOO, Reinforce++, etc. Supports synchronous and asynchronous vLLM engine inference acceleration, with extensible reward functions, multi-turn inference Schedulers, and environments through plugins.
- **Full-Pipeline Capabilities**: Covers the entire workflow of training, inference, evaluation, quantization, and deployment.
- **UI Training**: Provides Web-UI interface for training, inference, evaluation, and quantization, completing the full pipeline for large models.
- **Inference Acceleration**: Supports Transformers, vLLM, SGLang, and LmDeploy inference acceleration engines, providing OpenAI interfaces for accelerating inference, deployment, and evaluation modules.
- **Model Evaluation**: Uses EvalScope as the evaluation backend, supporting 100+ evaluation datasets for evaluating text-only and multimodal models.
- **Model Quantization**: Supports quantization export for AWQ, GPTQ, FP8, and BNB. Exported models support inference acceleration using vLLM/SGLang/LmDeploy.
## Installation
For the installation of ms-swift, please refer to the [installation documentation](./SWIFT-installation.md).
## Usage Example
10 minutes of self-cognition fine-tuning of Qwen3-4B-Instruct-2507 on a single 3090 GPU:
```shell
# 13GB
CUDA_VISIBLE_DEVICES=0 \
swift sft \
--model Qwen/Qwen3-4B-Instruct-2507 \
--tuner_type lora \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 1e-4 \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--gradient_accumulation_steps 16 \
--eval_steps 50 \
--save_steps 50 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 2048 \
--output_dir output \
--warmup_ratio 0.05 \
--dataloader_num_workers 4 \
--model_author swift \
--model_name swift-robot
```
Tips:
- If you want to train with a custom dataset, you can refer to [this guide](../Customization/Custom-dataset.md) to organize your dataset format and specify `--dataset <dataset_path>`.
- The `--model_author` and `--model_name` parameters are only effective when the dataset includes `swift/self-cognition`.
- To train with a different model, simply modify `--model <model_id/model_path>`.
- By default, **ModelScope** is used for downloading models and datasets. If you want to use HuggingFace, simply specify `--use_hf true`.
After training is complete, use the following command to infer with the trained weights:
- Here, `--adapters` should be replaced with the last checkpoint folder generated during training. Since the adapters folder contains the training parameter file `args.json`, there is no need to specify `--model`, `--system` separately; Swift will automatically read these parameters. To disable this behavior, you can set `--load_args false`.
```shell
# Using an interactive command line for inference.
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--stream true \
--temperature 0 \
--max_new_tokens 2048
# merge-lora and use vLLM for inference acceleration
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--stream true \
--merge_lora true \
--infer_backend vllm \
--vllm_max_model_len 8192 \
--temperature 0 \
--max_new_tokens 2048
```
Finally, use the following command to push the model to ModelScope:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift export \
--adapters output/vx-xxx/checkpoint-xxx \
--push_to_hub true \
--hub_model_id '<your-model-id>' \
--hub_token '<your-sdk-token>' \
--use_hf false
```
## Learn More
- More Shell scripts: [https://github.com/modelscope/ms-swift/tree/main/examples](https://github.com/modelscope/ms-swift/tree/main/examples)
- Using Python: [https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2_5-self-cognition/self-cognition-sft.ipynb](https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2_5-self-cognition/self-cognition-sft.ipynb)
docs/source_en/GetStarted/SWIFT-installation.md
+180
View File
@@ -0,0 +1,180 @@
# SWIFT Installation
## Wheel Packages Installation
You can install it using pip:
```shell
# recommend
pip install 'ms-swift' -U
# Install additional Megatron dependencies
pip install 'ms-swift[megatron]' -U
# Install additional evaluation dependencies
pip install 'ms-swift[eval]' -U
# Full capabilities
pip install 'ms-swift[all]' -U
# Using uv
pip install uv
uv pip install 'ms-swift' --torch-backend=auto
```
## Source Code Installation
The current main branch is for swift 4.x version.
```shell
# pip install git+https://github.com/modelscope/ms-swift.git
# Full capabilities
# pip install "ms-swift[all]@git+https://github.com/modelscope/ms-swift.git"
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .
# Full capabilities
# pip install -e '.[all]'
```
Installing swift 3.x:
```shell
# pip install "git+https://github.com/modelscope/ms-swift.git@release/3.12"
# Full capabilities
# pip install "ms-swift[all]@git+https://github.com/modelscope/ms-swift.git@release/3.12"
git clone -b release/3.12 https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e .
# Using uv
uv pip install -e . --torch-backend=auto
# Full capabilities
# pip install -e '.[all]'
```
## Mirror
You can check Docker [here](https://github.com/modelscope/modelscope/blob/build_swift_image/docker/build_image.py#L392).
```
# swift4.3.2
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.23.0-modelscope1.37.1-swift4.3.2
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.23.0-modelscope1.37.1-swift4.3.2
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.23.0-modelscope1.37.1-swift4.3.2
# swift4.2.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.21.0-modelscope1.36.3-swift4.2.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.21.0-modelscope1.36.3-swift4.2.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.21.0-modelscope1.36.3-swift4.2.3
# swift4.1.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py312-torch2.10.0-vllm0.19.1-modelscope1.35.4-swift4.1.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py312-torch2.10.0-vllm0.19.1-modelscope1.35.4-swift4.1.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py312-torch2.10.0-vllm0.19.1-modelscope1.35.4-swift4.1.3
# swift4.0.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.10.0-vllm0.17.1-modelscope1.34.0-swift4.0.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.10.0-vllm0.17.1-modelscope1.34.0-swift4.0.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.10.0-vllm0.17.1-modelscope1.34.0-swift4.0.3
```
<details><summary>Historical Mirrors</summary>
```
# swift3.12.5
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.9.0-vllm0.13.0-modelscope1.33.0-swift3.12.5
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.9.0-vllm0.13.0-modelscope1.33.0-swift3.12.5
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.9.0-vllm0.13.0-modelscope1.33.0-swift3.12.5
# swift3.11.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py311-torch2.8.0-vllm0.11.0-modelscope1.32.0-swift3.11.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py311-torch2.8.0-vllm0.11.0-modelscope1.32.0-swift3.11.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py311-torch2.8.0-vllm0.11.0-modelscope1.32.0-swift3.11.3
# swift3.10.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.10.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.10.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.10.3
# swift3.9.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.9.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.9.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.9.3
# swift3.8.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.6.3-py311-torch2.7.1-vllm0.10.1.1-modelscope1.29.2-swift3.8.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.6.3-py311-torch2.7.1-vllm0.10.1.1-modelscope1.29.2-swift3.8.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.6.3-py311-torch2.7.1-vllm0.10.1.1-modelscope1.29.2-swift3.8.3
# swift3.7.2
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.6.3-py311-torch2.7.1-vllm0.10.0-modelscope1.28.2-swift3.7.2
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.6.3-py311-torch2.7.1-vllm0.10.0-modelscope1.28.2-swift3.7.2
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.6.3-py311-torch2.7.1-vllm0.10.0-modelscope1.28.2-swift3.7.2
# swift3.6.4
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py310-torch2.6.0-vllm0.8.5.post1-modelscope1.28.1-swift3.6.4
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py310-torch2.6.0-vllm0.8.5.post1-modelscope1.28.1-swift3.6.4
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py310-torch2.6.0-vllm0.8.5.post1-modelscope1.28.1-swift3.6.4
# swift3.5.3
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py310-torch2.6.0-vllm0.8.5.post1-modelscope1.27.1-swift3.5.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py310-torch2.6.0-vllm0.8.5.post1-modelscope1.27.1-swift3.5.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py310-torch2.6.0-vllm0.8.5.post1-modelscope1.27.1-swift3.5.3
# swift3.4.1.post1
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py311-torch2.6.0-vllm0.8.5.post1-modelscope1.26.0-swift3.4.1.post1
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py311-torch2.6.0-vllm0.8.5.post1-modelscope1.26.0-swift3.4.1.post1
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py311-torch2.6.0-vllm0.8.5.post1-modelscope1.26.0-swift3.4.1.post1
# swift3.3.0.post1
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py311-torch2.6.0-vllm0.8.3-modelscope1.25.0-swift3.3.0.post1
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py311-torch2.6.0-vllm0.8.3-modelscope1.25.0-swift3.3.0.post1
# swift3.2.2
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.4.0-py311-torch2.5.1-modelscope1.25.0-swift3.2.2
```
</details>
More images can be found [here](https://modelscope.cn/docs/intro/environment-setup#%E6%9C%80%E6%96%B0%E9%95%9C%E5%83%8F).
## Supported Hardware
| Hardware Environment | Remarks |
| -------------------- | ------------------------------------------------------ |
| A10/A100/H100 | |
| RTX 20/30/40 Series | |
| T4/V100 | Some models may encounter NAN |
| Ascend NPU | Some models may encounter NAN or unsupported operators |
| MPS | Refer to [issue 4572](https://github.com/modelscope/ms-swift/issues/4572) |
| CPU | |
## Running Environment
| | Range | Recommended | Notes |
|--------------|--------------|---------------------|-------------------------------------------|
| python | >=3.10 | 3.12 | |
| cuda | | cuda12.8/13.0 | No need to install if using CPU, NPU, MPS |
| torch | >=2.0 | 2.8.0/2.11.0 | |
| transformers | >=4.33 | 4.57.6/5.12.1 | |
| modelscope | >=1.23 | | |
| datasets | >=3.0,<4.8.5 | 3.6.0/4.8.4 | |
| peft | >=0.11,<0.20 | | |
| flash_attn | | 2.8.3/4.0.0b15 | |
| trl | >=0.15,<1.0 | 0.29.1 | RLHF |
| deepspeed | >=0.14 | 0.18.9 | Training |
| vllm | >=0.5.1 | 0.11.0/0.23.0 | Inference/Deployment |
| sglang | >=0.4.6 | | Inference/Deployment |
| evalscope | >=1.0 | | Evaluation |
| gradio | | 5.32.1 | Web-UI/App |
For more optional dependencies, you can refer to [here](https://github.com/modelscope/ms-swift/blob/main/requirements/install_all.sh).
## Notebook Environment
Most models that Swift supports for training can be used on A10 GPUs. Users can take advantage of the free GPU resources offered by ModelScope:
1. Visit the [ModelScope](https://www.modelscope.cn) official website and log in.
2. Click on `My Notebook` on the left and start a free GPU instance.
3. Enjoy utilizing the A10 GPU resources.
docs/source_en/GetStarted/Web-UI.md
+38
View File
@@ -0,0 +1,38 @@
# Web-UI
Currently, SWIFT supports interface-based training and inference, with parameter support similar to script training. After installing SWIFT, use the following command:
```shell
swift web-ui --lang zh
# or en
swift web-ui --lang en
```
to start the interface for training and inference.
SWIFT web-ui is a high-level wrapper for the command line. In other words, tasks such as training and deployment initiated through the interface will start an independent process in the system via the command line. Pseudo-code is similar to:
```python
import os
os.system('swift sft --model xxx --dataset xxx')
```
This provides several features for the web-ui:
1. Each hyperparameter description in the web-ui is prefixed with `--xxx`, consistent with the [command line arguments](../Instruction/Command-line-parameters.md).
2. The web-ui can concurrently start multiple training/deployment tasks on a multi-GPU machine.
3. After the web-ui service is closed, the background services continue to run. This prevents the training processes from being affected when the web-ui is shut down. If you need to terminate background services, simply **select the corresponding task** and click the kill service button in the `Runtime` tab on the interface.
4. After restarting the web-ui, if you need to display the running services, click `Recover Runtime Tasks` in the `Runtime` tab.
5. The training interface supports displaying runtime logs. After selecting a specific task, manually click `Show Runtime Status`. During training, the runtime status can display training charts, including basic metrics such as training loss, training accuracy, and learning rate. In the human alignment task interface, the charts display key metrics like margin and logps.
6. Training through the web-ui does not support PPO, as the process is more complex. It is recommended to use the [shell script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/ppo) in the examples directory to run it directly.
If you need to use share mode, please add the `--share true` parameter. **Note:** Do not use this parameter in environments such as dsw or notebooks.
Additionally, ms-swift supports interface inference mode (i.e., Space deployment):
```shell
swift app --model '<model>' --studio_title My-Awesome-Space --stream true
# or
swift app --model '<model>' --adapters '<adapter>' --studio_title My-Awesome-Space --stream true
```
This will launch an application with only the inference page, which will deploy the model upon startup and provide it for subsequent use.
docs/source_en/Instruction/Agent-support.md
+257
View File
@@ -0,0 +1,257 @@
# Agent Support
## Dataset Format
ms-swift leverages agent-template to decouple Agent data formats from model implementations. With a unified dataset format, users can seamlessly switch between different models for training without any data modifications.
Example data samples for the pure text Agent and multimodal Agent are as follows:
```jsonl
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}]}
{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"click\", \"description\": \"Click on a position on the screen\", \"parameters\": {\"type\": \"object\", \"properties\": {\"x\": {\"type\": \"integer\", \"description\": \"X-coordinate representing the horizontal position on the screen\"}, \"y\": {\"type\": \"integer\", \"description\": \"Y-coordinate representing the vertical position on the screen\"}}, \"required\": [\"x\", \"y\"]}}}]", "messages": [{"role": "user", "content": "<image>What time is it now?"}, {"role": "assistant", "content": "<think>\nI can check the current time by opening the calendar app.\n</think>\n"}, {"role": "tool_call", "content": "{\"name\": \"click\", \"arguments\": {\"x\": 105, \"y\": 132}}"}, {"role": "tool_response", "content": "{\"images\": \"<image>\", \"status\": \"success\"}"}, {"role": "assistant", "content": "Successfully opened the calendar app. The current time is 11 o'clock in the morning."}], "images": ["desktop.png", "calendar.png"]}
```
- When the `agent_template` is set to "react_en", "hermes", etc., this format is compatible with training for all model Agents and allows easy switching between different models.
- Among them, `tools` is a JSON string containing a list of tools, and the `content` section of `messages` where the `role` is `'tool_call'` or `'tool_response/tool'` must also be a JSON string.
- The `tools` field will be combined with the `{"role": "system", ...}` section during training/inference according to the `agent_template`, forming a complete system section.
- The `{"role": "tool_call", ...}` part will automatically be converted into corresponding formats of `{"role": "assistant", ...}` based on the `agent_template`. Multiple consecutive `{"role": "assistant", ...}` entries will be concatenated to form a complete assistant_content.
- The `{"role": "tool_response", ...}` can also be written as `{"role": "tool", ...}`, these two forms are equivalent. This part will also be automatically converted according to the `agent_template`. During training, this part does not participate in loss calculations, similar to `{"role": "user", ...}`.
- This format supports parallel tool calls; refer to the first data sample for an example. In multimodal Agent data samples, the number of `<image>` tags should match the length of "images", and their positions indicate where the image features are inserted. It also supports other modalities, such as audios and videos.
- Note: You can also manually process the data into the messages format with roles set to system, user, or assistant. The purpose of agent_template is to automatically map the tools field and the messages with roles tool_call and tool_response into the standard messages format with roles system, user, and assistant.
The following are the `input_ids` and `labels` after encoding the two data samples mentioned above using the templates for **qwen2_5** and **qwen2_5_vl** , with the selected `agent_template` being **hermes** :
Sample One (Parallel Tool Calls):
```text
[INPUT_IDS] <|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "realtime_aqi", "description": "Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name, e.g., Shanghai"}}, "required": ["city"]}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
What is the weather like in Beijing and Shanghai today?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "realtime_aqi", "arguments": {"city": "Beijing"}}
</tool_call>
<tool_call>
{"name": "realtime_aqi", "arguments": {"city": "Shanghai"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
{"city": "Beijing", "aqi": "10", "unit": "celsius"}
</tool_response>
<tool_response>
{"city": "Shanghai", "aqi": "72", "unit": "fahrenheit"}
</tool_response><|im_end|>
<|im_start|>assistant
According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
[LABELS] [-100 * 195]<tool_call>
{"name": "realtime_aqi", "arguments": {"city": "Beijing"}}
</tool_call>
<tool_call>
{"name": "realtime_aqi", "arguments": {"city": "Shanghai"}}
</tool_call><|im_end|>[-100 * 67]According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
```
Sample Two (Multimodal, Mixed Assistant and Tool Call):
```text
[INPUT_IDS] <|im_start|>system
You are a helpful assistant.
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "click", "description": "Click on a position on the screen", "parameters": {"type": "object", "properties": {"x": {"type": "integer", "description": "X-coordinate representing the horizontal position on the screen"}, "y": {"type": "integer", "description": "Y-coordinate representing the vertical position on the screen"}}, "required": ["x", "y"]}}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
<|vision_start|>[151655 * 729]<|vision_end|>What time is it now?<|im_end|>
<|im_start|>assistant
<think>
I can check the current time by opening the calendar app.
</think>
<tool_call>
{"name": "click", "arguments": {"x": 105, "y": 132}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
{"images": "<|vision_start|>[151655 * 729]<|vision_end|>", "status": "success"}
</tool_response><|im_end|>
<|im_start|>assistant
Successfully opened the calendar app. The current time is 11 o'clock in the morning.<|im_end|>
[LABELS] [-100 * 924]<think>
I can check the current time by opening the calendar app.
</think>
<tool_call>
{"name": "click", "arguments": {"x": 105, "y": 132}}
</tool_call><|im_end|>[-100 * 759]Successfully opened the calendar app. The current time is 11 o'clock in the morning.<|im_end|>
```
**react_en** is one of the commonly used agent template formats. Below is an example of the `input_ids` and `labels` after encoding by qwen2_5 using `agent_template='react_en'`:
```text
[INPUT_IDS] <|im_start|>system
Answer the following questions as best you can. You have access to the following tools:
realtime_aqi: Call this tool to interact with the realtime_aqi API. What is the realtime_aqi API useful for? Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information. Parameters: {"type": "object", "properties": {"city": {"type": "string", "description": "City name, e.g., Shanghai"}}, "required": ["city"]} Format the arguments as a JSON object.
Use the following format:
Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [realtime_aqi]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can be repeated zero or more times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question
Begin!
<|im_end|>
<|im_start|>user
What is the weather like in Beijing and Shanghai today?<|im_end|>
<|im_start|>assistant
Action: realtime_aqi
Action Input: {'city': 'Beijing'}
Action: realtime_aqi
Action Input: {'city': 'Shanghai'}
Observation:{"city": "Beijing", "aqi": "10", "unit": "celsius"}
Observation:{"city": "Shanghai", "aqi": "72", "unit": "fahrenheit"}
According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
[LABELS] [-100 * 233]Action: realtime_aqi
Action Input: {'city': 'Beijing'}
Action: realtime_aqi
Action Input: {'city': 'Shanghai'}
Observation:[-100 * 45]According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
```
The following code can be used to experiment with more models and `agent_template` options. For more selectable values of `agent_template`, refer to [here](https://github.com/modelscope/ms-swift/blob/main/swift/agent_template/__init__.py).
```python
from swift import get_processor, get_template
tokenizer = get_processor('Qwen/Qwen3.5-2B')
template = get_template(tokenizer) # Using default agent template
# template = get_template(tokenizer, agent_template='qwen3_5')
print(f'agent_template: {template._agent_template}')
data = {...}
template.set_mode('train')
encoded = template.encode(data)
print(f'[INPUT_IDS] {template.safe_decode(encoded["input_ids"])}\n')
print(f'[LABELS] {template.safe_decode(encoded["labels"])}')
```
## Tools Format
The tools field provides information about the APIs that the model can call. You need to provide the name, description, and parameters of the tools, as shown in the example below:
```python
tools = [{
'type': 'function',
'function': {
'name': 'get_current_weather',
'description': 'Get the current weather in a given location',
'parameters': {
'type': 'object',
'properties': {
'location': {
'type': 'string',
'description': 'The city and state, e.g. San Francisco, CA'
},
'unit': {
'type': 'string',
'enum': ['celsius', 'fahrenheit']
}
},
'required': ['location']
}
}
}]
```
## Usage of loss_scale
The `loss_scale` parameter can be used to adjust the loss weights for different parts of the model output during training. Currently, two configuration methods are supported: exact string matching and regular expression (regex) matching.
1. String Matching Example: ReACT Format
Take the ReACT format as an example. You can enable the corresponding `loss_scale` configuration via `--loss_scale react` (see configuration file [react.json](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/config/react.json)). This method relies on exact string matching. The dictionary mapping in the configuration must provide a list of two elements, representing:
- The loss weight for the matched string itself,
- The loss weight for content following the matched string, up to (but not including) the next specified string.
The specific effects of this configuration are as follows:
- The `'Action:'` and `'Action Input:'` keywords and their subsequent content both have a loss weight of 2;
- The `'Thought:'` and `'Final Answer:'` keywords and their subsequent content both have a loss weight of 1;
- The `'Observation:'` field itself has a loss weight of 2, but the subsequent tool call result content has a loss weight of 0.
2. Regular Expression Matching Example: Ignoring Empty Thought Blocks
When training reasoning models, it may be necessary to exclude loss computation for empty thought blocks in the dataset, such as sequences like `'<think>\n\n</think>\n\n'`.
In such cases, use `--loss_scale ignore_empty_think` (see configuration file [ignore_empty_think.json](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/config/ignore_empty_think.json)). This configuration uses regular expression matching, where the dictionary mapping only needs to specify a single value—the loss weight for the matched content.
The specific effect of this setting is:
- Any string matching the regular expression `<think>\\s*</think>\\s*` is assigned a `loss_scale` of 0, meaning no loss is computed for these segments.
Testing loss_scale using code:
```python
from swift import get_processor, get_template
data = {"messages": [
{"role": "user", "content": "aaaaa"},
{"role": "assistant", "content": "<think>\n\n</think>\n\nabc<think>\n\n</think>\n\n123"},
]}
template = get_template(get_processor('Qwen/Qwen3-8B'), loss_scale='ignore_empty_think')
template.set_mode('train')
inputs = template.encode(data)
print(template.safe_decode(inputs['labels']))
# '[-100 * 14]abc<think>\n\n</think>\n\n123<|im_end|>\n'
```
For more `loss_scale` plugin designs, please refer to the [Architecture](../Customization/Architecture.md) documentation.
## Training
- Train the Agent capabilities of Base models by switching different models through modifying `--model`. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent/qwen2_5.sh).
- The agent_template for training GLM4 is hermes. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent/glm4.sh).
- Use `--loss_scale` to adjust the loss weight of the model output section. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/agent/loss_scale).
## Inference
- 🚀For inference of the original model or fully trained model, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_agent.py).
- For inference after LoRA training, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent/loss_scale/infer_lora.py).
## Deployment
For server and client code, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/deploy/agent).
docs/source_en/Instruction/Command-line-parameters.md
+991
View File
@@ -0,0 +1,991 @@
# Command Line Parameters
The command-line arguments will be introduced in four categories: basic arguments, atomic arguments, integrated arguments, and model-specific arguments. **The final list of arguments used in the command line consists of the integrated arguments, which inherit from the basic arguments and certain atomic arguments**. Model-specific arguments are tailored for particular models and can be configured via `--model_kwargs` or environment variables. For a detailed introduction to Megatron-SWIFT command-line arguments, please refer to the [Megatron-SWIFT Training Documentation](../Megatron-SWIFT/Command-line-parameters.md).
**Tips:**
- To pass a list via the command line, separate the elements with spaces. For example: `--dataset <dataset_path1> <dataset_path2>`.
- To pass a dictionary via the command line, use JSON format. For example: `--model_kwargs '{"fps_max_frames": 12}'`.
- Parameters marked with 🔥 are important; new users of ms-swift should prioritize these command-line arguments.
## Base Arguments
- 🔥tuner_backend: Optional values are `'peft'` and `'unsloth'`. Default is `'peft'`.
- 🔥tuner_type: Optional values are 'lora', 'full', 'lora_llm', 'longlora', 'adalora', 'llamapro', 'adapter', 'vera', 'boft', 'fourierft', 'reft', 'bone'. Default is 'lora'.
- Note: Defaults to 'full' in Megatron-SWIFT.
- 'lora_llm' means applying LoRA to the LLM part while using 'full' fine-tuning for the ViT/aligner parts. You can set their respective learning rates using `vit_lr/aligner_lr`.
- 🔥adapters: A list specifying adapter IDs or paths. Default is `[]`. This parameter is typically used in inference/deployment commands, for example: `swift infer --model '<model_id_or_path>' --adapters '<adapter_id_or_path>'`. It can occasionally be used for resuming training from a checkpoint. The difference between this parameter and `resume_from_checkpoint` is that **this parameter only loads adapter weights**, without restoring the optimizer state or random seed, and does not skip already-trained portions of the dataset.
- The difference between `--model` and `--adapters`: `--model` is followed by the directory path of the complete weights, which contains full weight information such as model/tokenizer/config, for example `model.safetensors`. `--adapters` is followed by a list of incremental adapter weight directory paths, which contain incremental weight information of the adapters, for example `adapter_model.safetensors`.
- 🔥external_plugins: A list of external `plugin.py` files that will be additionally loaded (i.e., the modules will be imported). Defaults to `[]`. You can pass in `.py` file paths for custom model, template, and dataset registration, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/custom/sft.sh); or for custom GRPO components, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/run_external_reward_func.sh).
- seed: Global random seed. Default is 42.
- Note: This random seed is independent of `data_seed`, which controls randomness in the dataset.
- model_kwargs: Additional arguments specific to certain models. This list of parameters will be logged during training/inference. For example: `--model_kwargs '{"fps_max_frames": 12}'`. You can also set it via environment variables, e.g., `FPS_MAX_FRAMES=12`. Default is None.
- Note: **If you specify model-specific parameters during training, please also set the corresponding parameters during inference**—this helps maintain consistent performance.
- The meaning of model-specific parameters can usually be found in the official repository or inference code of the corresponding model. MS-Swift includes these parameters to ensure alignment between trained models and official inference behavior.
- enable_npu_model_patch: Whether to enable model-level NPU patches. Defaults to True. This parameter only controls model-related patches in NPU environments and usually does not need to be disabled; set it to False when debugging native transformers behavior or NPU model patch compatibility issues. It must be passed as a startup argument before the process imports `swift.model` for the first time.
- load_args: When `--resume_from_checkpoint`, `--model`, or `--adapters` are specified, this flag controls whether to load `args.json` from the saved file. The loaded keys are defined in [base_args.py](https://github.com/modelscope/ms-swift/blob/main/swift/arguments/base_args/base_args.py). Default is `True` for inference and export, and `False` for training. Usually, this parameter does not need to be modified.
- load_data_args: If set to `True`, additional data-related arguments from `args.json` will be loaded. Default is `False`. **This is typically used during inference to run inference on validation sets split during training**, for example: `swift infer --adapters xxx --load_data_args true --stream true --max_new_tokens 512`.
- use_hf: Determines whether to use [ModelScope](https://modelscope.cn/) or [HuggingFace](https://huggingface.co/) for downloading models, downloading datasets, and pushing models. Defaults to False (uses ModelScope).
- Note: To access ModelScope internationally, you can use [ModelScope International](https://modelscope.ai/home) by setting the environment variable `MODELSCOPE_DOMAIN='www.modelscope.ai'`.
- hub_token: Hub authentication token. For ModelScope, see [here](https://modelscope.cn/my/myaccesstoken). Default is `None`.
- ddp_timeout: Default is 18000000, in seconds.
- ddp_backend: Optional values are `"nccl"`, `"gloo"`, `"mpi"`, `"ccl"`, `"hccl"`, `"cncl"`, `"mccl"`. Default is `None`, which enables automatic selection.
- ignore_args_error: Used for compatibility with Jupyter Notebook. Default is `False`.
### Model Arguments
- 🔥model: The [model ID](https://modelscope.cn/models) or local model path. Default is `None`.
- model_type: The model type. In ms-swift, a `model_type` refers to a group of models that share the same architecture, model loading process, and template definition. Default is `None`, meaning it will be automatically inferred based on the suffix of `--model` and the 'architectures' field in config.json. Supported model types can be found in the [List of Supported Models and Datasets](./Supported-models-and-datasets.md)
- Note: The concept of `model_type` in MS-Swift differs from the `model_type` in `config.json`.
- Custom models typically require manually registering a `model_type` and `template`. See the [Custom Model Documentation](../Customization/Custom-model.md) for details.
- model_revision: Model version. Default is `None`.
- task_type: Default is `'causal_lm'`. Options include `'causal_lm'`, `'seq_cls'`, `'embedding'`, `'reranker'`, and `'generative_reranker'`. Examples for seq_cls can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/seq_cls), and examples for embedding can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/embedding).
- 🔥torch_dtype: Data type for model weights. Supported values: `float16`, `bfloat16`, `float32`. Default is `None`, which reads from the `config.json` file.
- attn_impl: Attention implementation. Options include `'sdpa'`, `'eager'`, `'flash_attn'`, `'flash_attention_2'`, `'flash_attention_3'`, `'flash_attention_4'`, etc. Default is `None`, reading from config.json.
- Note: Not all attention implementations may be supported, depending on the underlying Transformers library's support for the specific model.
- If set to `'flash_attn'` (for backward compatibility), `'flash_attention_2'` will be used.
- 🔥experts_impl: Expert implementation type, options are 'grouped_mm', 'batched_mm', 'eager'. Defaults to None. This feature requires "transformers>=5.0.0".
- new_special_tokens: List of additional special tokens to be added. Default is `[]`. Example usage can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/new_special_tokens).
- Note: You can also pass a `.txt` file path where each line contains one special token.
- num_labels: Required for classification models (`--task_type seq_cls`). Indicates the number of labels. Default is `None`.
- problem_type: Required for classification models (`--task_type seq_cls`). Options: `'regression'`, `'single_label_classification'`, `'multi_label_classification'`. Default is `None`. If the model is a reward model or `num_labels=1`, it defaults to `'regression'`; otherwise, it defaults to `'single_label_classification'`.
- rope_scaling: Type of RoPE scaling. You can pass a string such as `'linear'`, `'dynamic'`, `'yarn'`, along with `max_model_len`, and MS-Swift will automatically configure the corresponding `rope_scaling`, overriding the value in `config.json`. Alternatively, pass a JSON string like `'{"factor": 2.0, "type": "yarn"}'`, which will directly replace the `rope_scaling` in `config.json`. Default is `None`.
- max_model_len: When using `rope_scaling` with a string input, this parameter helps calculate the RoPE scaling `factor`. Default is `None`. If specified, this value will **override** `max_position_embeddings` in `config.json`.
- device_map: Device placement configuration for the model, e.g., `'auto'`, `'cpu'`, a JSON string, or a JSON file path. This argument is **passed through** to the `from_pretrained` method in Transformers. Default is `None`, automatically determined based on available devices and distributed training setup.
- max_memory: When `device_map` is set to `'auto'` or `'sequential'`, model weights are allocated across devices according to `max_memory`, e.g., `--max_memory '{0: "20GB", 1: "20GB"}'`. Default is `None`. Passed through to the `from_pretrained` interface in Transformers.
- local_repo_path: Some models depend on GitHub repositories during loading, e.g., [deepseek-vl2](https://github.com/deepseek-ai/DeepSeek-VL2). To avoid network issues during `git clone`, you can use a local repository. This parameter takes the path to the local repo. Default is `None`.
- init_strategy: Strategy for initializing uninitialized parameters when loading a model (especially useful for custom architectures). Options: `'zero'`, `'uniform'`, `'normal'`, `'xavier_uniform'`, `'xavier_normal'`, `'kaiming_uniform'`, `'kaiming_normal'`, `'orthogonal'`. Default is `None`.
### Data Arguments
- 🔥dataset: A list of dataset IDs or paths. Default is `[]`. Each dataset should be specified in the format: `'dataset_id_or_path:subset#sample_size'`, where subset and sample size are optional. Local datasets support formats such as jsonl, csv, json, and folders. **Open-source datasets from the hub can be used offline by `git clone`-ing them locally and passing the local folder path**. For custom dataset formats, refer to the [Custom Dataset Documentation](../Customization/Custom-dataset.md). You can use multiple datasets by passing `--dataset <dataset1> <dataset2>`.
- Subset: This parameter is only effective when the dataset is a dataset ID or a folder. If subsets were specified during registration and only one exists, that subset is selected by default; otherwise, the default subset `'default'` is used. You can select multiple subsets using `/`, e.g., `<dataset_id>:subset1/subset2`. You can also use `'all'` to select all registered subsets, e.g., `<dataset_id>:all`. See an example of registration [here](https://modelscope.cn/datasets/swift/garbage_competition).
- Sampling size: Uses the complete dataset by default. You can sample from the selected dataset by setting `#sample_size`, for example `<dataset_path#1000>`. If the sample size is less than the total number of samples, random sampling without replacement is performed. If the sample count exceeds the total, the dataset is repeated `sample_size // total_samples` times, with an additional `sample_size % total_samples` samples randomly sampled. Note: For streaming datasets (`--streaming true`), only sequential sampling is performed. If `--dataset_shuffle false` is set, non-streaming datasets also use sequential sampling.
- 🔥val_dataset: A list of validation dataset IDs or paths. Default is `[]`.
- 🔥cached_dataset: Use cached datasets (generated with the command `swift export --to_cached_dataset true ...`) to avoid GPU time being occupied by tokenization during training/inference on large datasets. This parameter is used to set the folder path(s) of cached training datasets, and defaults to `[]`. For examples, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/cached_dataset).
- Note: cached_dataset only stores an additional length field in the dataset (to avoid storage pressure) and filters out data samples that would cause errors. During training/inference, the `--max_length` parameter is supported for filtering/truncating excessively long data and the `--packing` parameter is supported. The actual data preprocessing process occurs synchronously during training and overlaps with the training process, which does not affect training speed. (Exception: when `--truncation_strategy split` is set, `'input_ids'` will be stored instead of `'lengths'`. In this case, you must use the same `max_length` and `truncation_strategy` for both dataset export and training. Compatibility with packing is not affected.)
- cached_dataset is compatible between `ms-swift` and `Megatron-SWIFT`, and supports pt/sft/infer/rlhf, set the training type using `--template_mode`; supports embedding/reranker/seq_cls tasks, set the task type using `--task_type`.
- Supports sampling from cache_dataset with the syntax `<cached_dataset_path>#sample_count`, supports sampling counts both higher and lower than the number of samples. For functionality and implementation, refer to the `--dataset` description.
- cached_val_dataset: Folder path(s) for cached validation datasets, default is `[]`.
- 🔥split_dataset_ratio: The ratio for splitting a validation set from the training set when `val_dataset` is not specified. Default is `0.`, meaning no splitting occurs.
- data_seed: Random seed for dataset operations. Default is `42`.
- 🔥dataset_num_proc: Number of processes for dataset preprocessing. Default is `1`.
- Note: For text-only models, it is recommended to increase this value to accelerate preprocessing speed. For multimodal models, it is not recommended to set it too high, as this may lead to slower preprocessing speed (if multimodal models experience 100% CPU utilization but extremely slow processing speed, it is recommended to additionally set the `OMP_NUM_THREADS` environment variable).
- 🔥load_from_cache_file: Whether to load the dataset from cache. Default is `False`. **Recommended to set to `True` during actual training and `False` during debugging**. You can modify the `MODELSCOPE_CACHE` environment variable to control the cache path.
- dataset_shuffle: Whether to shuffle the training dataset. Default is `True`.
- Note: **Shuffling in CPT/SFT involves two parts**: dataset-level shuffling (controlled by `dataset_shuffle`) and dataloader-level shuffling (controlled by `train_dataloader_shuffle`).
- val_dataset_shuffle: Whether to shuffle the validation dataset. Default is `False`.
- streaming: Whether to stream and process the dataset on-the-fly. Default is `False`. (The shuffling of streaming datasets is not thorough, which may lead to severe loss fluctuations.)
- Note: You must set `--max_steps` explicitly, as streaming datasets do not have a defined length. You can achieve behavior equivalent to `--num_train_epochs` by setting `--save_strategy epoch` and a large `max_steps`. Alternatively, set `max_epochs` to ensure training stops after the specified number of epochs, allowing model evaluation and checkpoint saving.
- Note: Streaming avoids waiting for preprocessing by overlapping it with training. However, preprocessing is only performed on rank 0 and then distributed to other processes. **This is typically less efficient than non-streaming data sharding**. When the training `world_size` is large, preprocessing and data distribution can become a bottleneck.
- interleave_prob: Default is `None`. By default, multiple datasets are combined using `concatenate_datasets` from the datasets library. If this parameter is set, `interleave_datasets` is used instead. This is typically used for combining streaming datasets and is passed directly to `interleave_datasets`. This parameter does not apply to `--val_dataset`.
- stopping_strategy: Options are `"first_exhausted"` or `"all_exhausted"`. Default is `"first_exhausted"`. Passed to the `interleave_datasets` function. This parameter does not apply to `--val_dataset`.
- shuffle_buffer_size: Specifies the shuffle buffer size for **streaming datasets**. Default is `1000`. Only effective when `dataset_shuffle` is `True`.
- download_mode: Dataset download mode. Options: `'reuse_dataset_if_exists'` or `'force_redownload'`. Default is `'reuse_dataset_if_exists'`.
- Typically set to `--download_mode force_redownload` when encountering errors with hub datasets.
- columns: Used to map dataset column names so that the dataset conforms to the format accepted by `AutoPreprocessor`. See [Custom Dataset Documentation](../Customization/Custom-dataset.md) for supported formats. You can pass a JSON string, e.g., `'{"text1": "query", "text2": "response"}'`, meaning column `"text1"` is mapped to `"query"` and `"text2"` to `"response"`, which `AutoPreprocessor` can process. Default is `None`.
- strict: If `True`, any malformed row in the dataset will raise an error; otherwise, erroneous samples are dropped. Default is `False`. This is typically used for debugging.
- 🔥remove_unused_columns: Whether to remove unused columns from the dataset. Default is `True`.
- If set to `False`, extra columns are passed to the trainer's `compute_loss` function, **facilitating custom loss functions that use additional dataset columns**.
- Default value is `False` for GRPO.
- disable_auto_column_mapping: By default, column names in the dataset are automatically mapped. This parameter disables that behavior (the `columns` parameter remains effective), defaulting to `False`. The automatic mapping rules are as follows:
- The following fields will be automatically mapped to the corresponding `images`, `videos`, and `audios` fields:
- `images`: `image`, `images`
- `videos`: `video`, `videos`
- `audios`: `audio`, `audios`
- The following fields will be automatically mapped to the corresponding `system`, `query`, and `response` fields (the `solution` field will be preserved):
- `system`: `system`, `system_prompt`
- `query`: `query`, `prompt`, `input`, `instruction`, `question`, `problem`
- `response`: `response`, `answer`, `output`, `targets`, `target`, `answer_key`, `answers`, `solution`, `text`, `completion`, `content`
- 🔥model_name: **Used only for self-cognition tasks**, and only affects the `swift/self-cognition` dataset. Replaces the `{{NAME}}` placeholder in the dataset. Provide the model's Chinese and English names, separated by space, e.g., `--model_name 小黄 'Xiao Huang'`. Default is `None`.
- 🔥model_author: Used only for self-cognition tasks, and only affects the `swift/self-cognition` dataset. Replaces the `{{AUTHOR}}` placeholder. Provide the model author's Chinese and English names, separated by space, e.g., `--model_author '魔搭' 'ModelScope'`. Default is `None`.
- custom_dataset_info: Path to a JSON file for custom dataset registration. See [Custom Dataset Guide](../Customization/Custom-dataset.md) and the [built-in dataset_info.json](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/data/dataset_info.json). Default is `[]`.
### Template Arguments
- 🔥template: The type of conversation template. Default is `None`, which automatically selects the corresponding template for the given model. See [List of Supported Models](./Supported-models-and-datasets.md) for mapping details.
- 🔥system: Custom system message field. Accepts either a string or a **path to a .txt file**. Default is `None`, using the default system message defined in the registered template.
- Note: In terms of priority, the `system` field from the dataset takes precedence, followed by `--system`, and finally the `default_system` set in the registered template.
- 🔥max_length: Maximum token length after `tokenizer.encode` for a single data sample (to prevent OOM during training). Samples exceeding this limit are handled according to `truncation_strategy`. Default is `None`, meaning it's set to the model’s maximum supported sequence length (`max_model_len`).
- In PPO, GRPO, GKD and inference scenarios, `max_length` refers to `max_prompt_length`.
- truncation_strategy: How to handle samples whose tokens exceed `max_length`. Supports 'delete', 'left', 'right', and 'split', which represent deleting, left truncation, right truncation, and splitting into multiple data samples, respectively. The default is 'delete'.
- Note: `--truncation_strategy split` is only supported during pretraining, i.e., in `swift/megatron pt` scenarios. This strategy will split oversized fields into multiple data samples to avoid token waste. This strategy only supports text-only datasets. Multimodal datasets cannot handle the splitting of pixel_values correctly.
- Note: For multimodal models, if `truncation_strategy` is set to `'left'` or `'right'` during training, **ms-swift preserves all image tokens and other modality-specific tokens**, which may lead to OOM.
- 🔥max_pixels: Maximum pixel count (H×W) for input images in multimodal models. Images exceeding this limit will be resized to avoid OOM during training. Default is `None` (no restriction).
- Note: This parameter applies to all multimodal models. The Qwen2.5-VL specific parameter `MAX_PIXELS` (see bottom of doc) only affects Qwen2.5-VL.
- 🔥agent_template: Agent template that defines how the tool list `'tools'` is converted into the `'system'` message, how tool calls are extracted from model responses during inference/deployment, and the formatting of `{"role": "tool_call", "content": "xxx"}` and `{"role": "tool_response", "content": "xxx"}` in `messages`. Options include `'react_en'`, `'hermes'`, `'glm4'`, `'qwen_en'`, `'toolbench'`, etc. See [here](https://github.com/modelscope/ms-swift/blob/main/swift/agent_template/mapping.py) for more. Default is `None`, automatically selected based on model type. Refer to [Agent Documentation](./Agent-support.md).
- norm_bbox: Controls how bounding boxes ("bbox" in dataset, containing absolute coordinates; see [Custom Dataset Documentation](../Customization/Custom-dataset.md#grounding)) are normalized. Options: `'norm1000'` (scale coordinates to thousandths), `'none'` (no scaling). Default is `None`, automatically chosen based on model.
- This also works correctly when **images are resized during training** (e.g., when `max_pixels` is set).
- use_chat_template: Whether to use a chat template or a generation template (the latter typically used in pretraining). Default is `True`.
- Note: `swift pt` defaults to `False`, using the generation template. This setting provides good **compatibility with multimodal models**.
- padding_side: Padding side when training with `batch_size >= 2`. Options: `'left'`, `'right'`. Default is `'right'`.
- Note: PPO and GKD default to `'left'`. (During inference with `batch_size >= 2`, only left-padding is applied.)
- 🔥padding_free: Flattens data within a batch to avoid padding, reducing GPU memory usage and accelerating training (**sequences in the same batch remain invisible to each other**). Default is `False`. Currently supported in CPT/SFT/DPO/GRPO/KTO/GKD.
- Note: Use `padding_free` together with `--attn_impl flash_attn` and `transformers>=4.44`. See [this PR](https://github.com/huggingface/transformers/pull/31629) for details. (Same as packing.)
- **Compared to packing, padding_free avoids extra preprocessing time, but packing offers faster training and more stable memory usage**.
- 🔥loss_scale: Loss weight configuration for training tokens. Default is `'default'`. loss_scale includes 3 basic strategies: 'default', 'last_round', 'all', and other strategies: 'ignore_empty_think' and agent-specific ones: 'react', 'hermes', 'qwen', 'agentflan', 'alpha_umi', etc. For available options, refer to [loss_scale module](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/mapping.py). ms-swift supports mixing basic strategies with other strategies, for example: `'default+ignore_empty_think'`, `'last_round+ignore_empty_think'`. If no basic strategy is specified, it defaults to 'default', for example: 'hermes' is equivalent to 'default+hermes'.
- 'default': All responses (including history) are calculated with weight 1 for cross-entropy loss (**system/user/multimodal tokens in messages and `tool_response` parts in Agent training are not included in loss calculation**). (**Default value for SFT**)
- 'last_round': Only calculate loss for the last round response. The last round means all content after the last "user". (**Default value for RLHF**)
- 'all': Calculate loss for all tokens. (**Default value for `swift pt`**)
- 'ignore_empty_think': Ignore loss computation for empty `'<think>\n\n</think>\n\n'` (as long as it matches the regex `'<think>\\s*</think>\\s*'`).
- 'react', 'hermes', 'qwen': Adjust the loss weight of the `tool_call` part to 2.
- Note: Starting from `ms-swift >= 4.3.1`, multiple non-base strategies can be chained together (each strategy processes the output segments of the previous one, with weights multiplied accordingly). For example: `'last_round+hermes+ignore_empty_think'`, where `'last_round'` is the base strategy, and `'hermes+ignore_empty_think'` represents a chain of multiple non-base strategies that share the same base strategy.
- disable_ignore_empty_think: Whether to disable the automatic appending of `ignore_empty_think` strategy to loss_scale for hybrid thinking models. Defaults to `False`, meaning for hybrid thinking models (e.g., Qwen3.5-4B), `+ignore_empty_think` is automatically appended to the loss_scale, so that empty `'<think>\n\n</think>\n\n'` blocks do not contribute to loss computation. If the user has already manually specified `ignore_empty_think` in the loss_scale, it will not be appended again. This parameter only takes effect during training and has no effect on pure thinking models or non-thinking models. Set to `True` to disable this default behavior.
- Note: This parameter was added in `ms-swift>=4.3.1`. In `ms-swift<4.3.1`, you need to manually add `--loss_scale ignore_empty_think`.
- is_binary_loss_scale: When `loss_scale` can only take values of `0` or `1`, its semantics can be represented by `labels` instead — by setting the `labels` of positions where `loss_scale` is `0` to `-100`, thereby ensuring compatibility with `liger_kernel` and reducing memory usage. Defaults to `None` for automatic configuration.
- sequence_parallel_size: Size for sequence parallelism. Default is 1. Currently supported in CPT/SFT/DPO/GRPO. Training scripts can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel).
- template_backend: Backend for template processing. Options are `'swift'` or `'jinja'`. Default is `'swift'`. If `'jinja'` is used, `apply_chat_template` from Transformers will be applied.
- Note: The `'jinja'` backend only supports inference and does not support training (as it cannot determine the token ranges for loss computation).
- response_prefix: The prefix string for responses, this parameter only takes effect during inference. Default is None, determined by the enable_thinking parameter and template type.
- enable_thinking: This parameter takes effect during inference, indicating whether to enable thinking mode. Default is None, the default value is determined by the template (model) type (hybrid thinking/non-thinking templates default to `False`, thinking templates default to `True`). If enable_thinking is False, a non-thinking prefix is added, for example the Qwen3-8B hybrid thinking model adds the prefix `'<think>\n\n</think>\n\n'`, while Qwen3-8B-Thinking does not add a prefix. If enable_thinking is True, a thinking prefix is added, for example `'<think>\n'`. Note: The priority of this parameter is lower than the response_prefix parameter.
- Note: In `ms-swift<4.3.1`, the default value for hybrid thinking templates was `True`; this was changed to `False` in `ms-swift>=4.3.1`.
- preserve_thinking: Whether to preserve historical thinking content during inference and training. When set to `True`, thinking content from all rounds is retained. When set to `False`, only the thinking content from the last round is retained (i.e., the content following the last user message). Defaults to `None`.
- Default behavior: For thinking models (thinking/hybrid-thinking) or when `enable_thinking` is explicitly enabled, this is set to `False` by default during inference and training, retaining only the last round of thinking content. If the `loss_scale` base strategy during training is not `'last_round'` (e.g., `'default'`), it defaults to `True`, and historical thinking content will not be removed.
- add_non_thinking_prefix: This parameter only takes effect during training, indicating whether to add a non-thinking prefix to data samples whose assistant part **does not start with the thinking marker `'<think>'`** (typically hybrid thinking models contain a non-thinking prefix). This feature allows swift's built-in datasets to train hybrid thinking models. Default value is True. For example: the non-thinking prefix for the Qwen3-8B hybrid thinking model is `'<think>\n\n</think>\n\n'`, while the non-thinking prefix for Qwen3-8B-Thinking/Instruct is `''`. Note: During training, if the basic strategy of loss_scale is last_round, this modification is only applied to the last round; otherwise, for example 'default' or 'all', this modification is applied to every round of data. If set to False, no non-thinking prefix is added to data samples.
### Generation Arguments
Refer to the [generation_config](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig) documentation.
- 🔥max_new_tokens: The maximum number of new tokens generated during inference. Defaults to None, meaning unlimited.
- temperature: Sampling temperature. Higher values increase output randomness. Default is `None`, reading from `generation_config.json`.
- You can set `--temperature 0` or `--top_k 1` to disable randomness in generation.
- top_k: Top-k sampling parameter. Only the top `k` highest probability tokens are considered for generation. Default is `None`, reading from `generation_config.json`.
- top_p: Top-p (nucleus) sampling parameter. Only tokens whose cumulative probability reaches `top_p` are considered. Default is `None`, reading from `generation_config.json`.
- repetition_penalty: Penalty for repeated tokens. A value of 1.0 means no penalty. Default is `None`, reading from `generation_config.json`.
- num_beams: Number of beams for beam search. Default is 1.
- 🔥stream: Enable streaming output. Default is `None`, meaning `True` when using an interactive interface, and `False` during batch inference on datasets.
- stop_words: Additional stop words besides the `eos_token`. Default is `[]`.
- Note: The `eos_token` is removed from the output response, while additional stop words are preserved in the output.
- logprobs: Whether to return log probabilities. Default is `False`.
- top_logprobs: Number of top log probabilities to return. Default is `None`.
- structured_outputs_regex: A regular expression pattern for structured outputs (guided decoding). When set, the model's generation is constrained to match the specified regex pattern. Only effective when `infer_backend` is `vllm`. Default is `None`.
### Quantization Arguments
The following are parameters for quantizing models upon loading. See the [quantization documentation](https://huggingface.co/docs/transformers/main/en/main_classes/quantization) for details. These do not include `gptq` or `awq` quantization parameters used in `swift export`.
- 🔥quant_method: Quantization method used when loading the model. Options: `'bnb'`, `'hqq'`, `'eetq'`, `'quanto'`, `'fp8'`. Default is `None`.
- If performing QLoRA training on already AWQ/GPTQ-quantized models, you do not need to set additional quantization parameters like `quant_method`.
- 🔥quant_bits: Number of bits for quantization. Default is `None`.
- hqq_axis: Axis for HQQ quantization. Default is `None`.
- bnb_4bit_compute_dtype: Computation data type for 4-bit BNB quantization. Options: `float16`, `bfloat16`, `float32`. Default is `None`, which uses the value of `torch_dtype`.
- bnb_4bit_quant_type: Type for 4-bit BNB quantization. Options: `'fp4'`, `'nf4'`. Default is `'nf4'`.
- bnb_4bit_use_double_quant: Whether to use double quantization. Default is `True`.
- bnb_4bit_quant_storage: Data type used to store quantized weights. Default is `None`.
### RAY Arguments
- use_ray: Boolean type. Whether to use ray, defaults to `False`.
- ray_exp_name: Ray experiment name. This field will be used as the prefix for cluster and worker names, can be empty.
- device_groups: String (jsonstring) type. When using ray, this field must be configured. For details, please refer to the [ray documentation](Ray.md).
### YAML/JSON Support
Here we use `swift sft` as an example. The YAML/JSON launch method also supports `swift infer/rlhf/...` as well as `megatron sft/rlhf`. Please refer to [the examples here](https://github.com/modelscope/ms-swift/tree/main/examples/yaml).
- The YAML/JSON file will be stored in `output_dir` after training/inference.
```shell
swift sft xxx.yaml
swift sft xxx.json
```
The content of xxx.yaml/xxx.json contains specific command-line configurations:
```yaml
model: "Qwen/Qwen2.5-7B-Instruct"
dataset: "swift/self-cognition#500"
```
```json
{
"model": "Qwen/Qwen2.5-7B-Instruct",
"dataset": "swift/self-cognition#500"
}
```
You can also use a combination of YAML and command-line arguments. For example, use YAML for parameters that are infrequently modified, and pass frequently changed parameters via command line.
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer examples/yaml/deepspeed/infer.yaml \
--adapters output/vx-xxx/checkpoint-xxx
```
How to specify environment variables in YAML/JSON:
```yaml
ENV:
MAX_PIXELS: '1003520'
VIDEO_MAX_PIXELS: '50176'
FPS_MAX_FRAMES: '12'
```
```json
{
"ENV": {
"MAX_PIXELS": "1003520",
"VIDEO_MAX_PIXELS": "50176",
"FPS_MAX_FRAMES": "12"
}
}
```
## Atomic Arguments
### Seq2SeqTrainer Arguments
This list inherits from the Transformers `Seq2SeqTrainingArguments`, with ms-swift overriding certain default values. For arguments not listed here, please refer to the [official HF documentation](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Seq2SeqTrainingArguments).
- 🔥output_dir: The output directory where the model predictions and checkpoints will be written. Default is `None`, automatically set to `'output/<model_name>'`.
- 🔥gradient_checkpointing: Whether to use gradient checkpointing. Default is `True`. This significantly reduces GPU memory usage but slows down training.
- 🔥vit_gradient_checkpointing: Whether to enable gradient checkpointing for the ViT component during multimodal model training. Defaults to `None`, which means it is enabled when `--freeze_vit` is `false`. For an example, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/vit_gradient_checkpointing.sh).
- 🔥deepspeed: Default is `None`. Can be set to `'zero0'`, `'zero1'`, `'zero2'`, `'zero3'`, `'zero2_offload'`, `'zero3_offload'` to use built-in DeepSpeed configurations in ms-swift. You can also pass a path to a custom DeepSpeed config file.
- zero_hpz_partition_size: Default is `None`. This enables ZeRO++ functionality—model sharding within nodes and data sharding across nodes. If encountering `grad_norm NaN`, try using `--torch_dtype float16`.
- deepspeed_autotp_size: DeepSpeed tensor parallelism size. Default is 1. To use DeepSpeed AutoTP, set `--deepspeed` to `'zero0'`, `'zero1'`, or `'zero2'`. (Note: Only supports full-parameter training)
- 🔥fsdp: FSDP2 distributed training configuration. Default is `None`. Can be set to `'fsdp2'` to use built-in FSDP2 configurations in ms-swift. You can also pass a path to a custom FSDP config file. FSDP2 is PyTorch's native distributed training solution, use either this or DeepSpeed (not both).
- 🔥per_device_train_batch_size: Default is 1.
- 🔥per_device_eval_batch_size: Default is 1.
- 🔥gradient_accumulation_steps: Gradient accumulation steps. Default is `None`, meaning `gradient_accumulation_steps` is automatically calculated so that `total_batch_size >= 16`. Total batch size is computed as `per_device_train_batch_size * gradient_accumulation_steps * world_size`. In GRPO training, default is 1.
- In CPT/SFT training, gradient accumulation has equivalent effects to using a larger batch size, but this equivalence does not hold in RLHF training.
- weight_decay: Weight decay coefficient. Default is 0.1.
- adam_beta1: The exponential decay rate for the first moment estimates (momentum) in Adam-based optimizers. Defaults to 0.9.
- adam_beta2: The exponential decay rate for the second moment estimates (variance) in Adam-based optimizers. Defaults to 0.95.
- adam_epsilon: Epsilon value for numerical stability in Adam-based optimizers. Defaults to 1e-8.
- 🔥learning_rate: Learning rate. **Default is `1e-5` for full-parameter training, and `1e-4` for LoRA and other tuners**.
- Tip: If you want to set `min_lr`, you can pass the arguments `--lr_scheduler_type cosine_with_min_lr --lr_scheduler_kwargs '{"min_lr": 1e-6}'`.
- 🔥vit_lr: Specifies the learning rate for the ViT module when training multimodal models. Default is `None`, same as `learning_rate`. Typically used together with `--freeze_vit` and `--freeze_aligner`.
- Note: The "learning_rate" printed in the logs is the learning rate of `param_groups[0]`, where the order of param_groups is vit, aligner, llm (if it contains trainable parameters).
- 🔥aligner_lr: Specifies the learning rate for the aligner module in multimodal models. Default is `None`, same as `learning_rate`.
- lr_scheduler_type: The type of learning rate scheduler, defaults to `'cosine'`. Common options: `'linear'`, `'constant'`, `'cosine_with_min_lr'`.
- lr_scheduler_kwargs: Additional arguments for the learning rate scheduler. Default is `None`.
- gradient_checkpointing_kwargs: Arguments passed to `torch.utils.checkpoint`. For example: `--gradient_checkpointing_kwargs '{"use_reentrant": false}'`. Default is `None`.
- Note: When using DDP without DeepSpeed/FSDP and `gradient_checkpointing_kwargs` is `None`, it defaults to `'{"use_reentrant": false}'` to prevent errors.
- full_determinism: Ensures reproducible results during training. Note: This may negatively impact performance. Default is `False`.
- 🔥report_to: Default is `'tensorboard'`. You can specify multiple loggers, e.g., `--report_to tensorboard wandb swanlab`, or `--report_to all`.
- If you specify `--report_to wandb`, you can set the project name through `WANDB_PROJECT` and specify the API KEY corresponding to your account through `WANDB_API_KEY`.
- logging_first_step: Whether to log metrics at the first step. Default is `True`.
- logging_steps: Interval for logging. Default is 5.
- router_aux_loss_coef: Used in MoE model training to set the weight of auxiliary loss. Default is `0.`.
- enable_dft_loss: Whether to use [DFT](https://arxiv.org/abs/2508.05629) (Dynamic Fine-Tuning) loss during SFT training. Default is `False`.
- enable_channel_loss: Enable channel-based loss. Default is `False`. Requires a `"channel"` field in the dataset. ms-swift groups and computes loss by this field (samples without `"channel"` are grouped into the default `None` channel). Dataset format reference: [channel loss](../Customization/Custom-dataset.md#channel-loss). Channel loss is compatible with packing, padding_free, and loss_scale techniques.
- safe_serialization: Whether to save the model in safetensors format. Default is True.
- max_shard_size: Maximum size of a single storage file, default is '5GB'.
- logging_dir: Directory for TensorBoard logs. Default is `None`, automatically set to `f'{self.output_dir}/runs'`.
- predict_with_generate: Use generation during evaluation. Default is `False`.
- metric_for_best_model: Default is `None`. If `predict_with_generate=False`, it's set to `'loss'`; otherwise `'rouge-l'` (in PPO training, no default; in GRPO, set to `'reward'`).
- greater_is_better: Default is `None`. Set to `False` if `metric_for_best_model` contains `'loss'`, otherwise `True`.
- max_epochs: Force training to stop after reaching `max_epochs`, then evaluate and save the model. Useful when using streaming datasets. Default is `None`.
Other important parameters:
- 🔥num_train_epochs: Number of training epochs. Default is 3.
- 🔥save_strategy: Strategy for saving checkpoints. Options: `'no'`, `'steps'`, `'epoch'`. Default is `'steps'`.
- 🔥save_steps: Default is 500.
- 🔥eval_strategy: Evaluation strategy. Default is `None`, following `save_strategy`.
- If neither `val_dataset` nor `eval_dataset` is used and `split_dataset_ratio=0`, defaults to `'no'`.
- 🔥eval_steps: Default is `None`. If evaluation dataset exists, follows `save_steps`.
- eval_on_start: Whether to perform an evaluation step before training to ensure the validation steps work correctly. Defaults to False.
- 🔥save_total_limit: Maximum number of checkpoints to save. Expired checkpoints will be deleted. Default is None, which saves all checkpoints. If set to 2, it will save the best checkpoint and the last checkpoint.
- max_steps: Maximum number of training steps. Must be set when using streaming datasets. Default is -1.
- 🔥warmup_ratio: Default is 0.
- save_on_each_node: Save weights on every node. Default is `False`. Relevant in multi-node training.
- Tip: In multi-node training, `output_dir` is typically set to a shared directory, so this parameter usually doesn't need to be set.
- save_only_model: Whether to save only model weights (excluding optimizer states, random seed states, etc.), reducing time and space overhead in full-parameter training. Default is `False`.
- 🔥resume_from_checkpoint: Path to resume training from. Default is `None`.
- Tip: **To resume training, keep other parameters unchanged and add `--resume_from_checkpoint checkpoint_dir`**. Weights and states will be loaded by the trainer.
- Note: `resume_from_checkpoint` loads model weights, optimizer state, random seed, and resumes training from the last step. Use `--resume_only_model` to load only model weights.
- resume_only_model: Default is `False`. If set to `True` along with `resume_from_checkpoint`, only model weights are resumed, ignoring optimizer state and random seed.
- Note: **`resume_only_model` skips already-trained data by default**, controlled via the `ignore_data_skip` argument.
- ignore_data_skip: When `resume_from_checkpoint` and `resume_only_model` are set, this controls whether to skip already-trained data and restore training states (epoch, step count, etc.). Default is `False`. If `True`, training starts from step 0 without loading previous states or skipping data.
- 🔥ddp_find_unused_parameters: Default is `None`.
- 🔥dataloader_num_workers: Default is `None`. On Windows, set to 0; otherwise, 1.
- dataloader_pin_memory: Default is `True`.
- dataloader_persistent_workers: Default is `False`.
- dataloader_prefetch_factor: Default is `None`. If `dataloader_num_workers > 0`, it is set to 2. Number of batches loaded in advance by each worker. 2 means there will be a total of 2 * num_workers batches prefetched across all workers.
- train_dataloader_shuffle: Whether to shuffle the dataloader for CPT/SFT training, default is True. This parameter is ineffective for IterableDataset (i.e., it doesn't work for streaming datasets). IterableDataset reads data sequentially.
- optim: The optimizer, defaults to `"adamw_torch"` (for torch>=2.8 `"adamw_torch_fused"`). For a complete list of optimizers, please see `OptimizerNames` in [training_args.py](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py).
- optim_args: Optional arguments to pass to the optimizer, defaults to None.
- group_by_length: Whether to group samples with approximately the same length together in the training dataset (with a random factor) to minimize padding and ensure load balancing across nodes and processes for improved efficiency. Defaults to False. For the specific algorithm, refer to `transformers.trainer_pt_utils.get_length_grouped_indices`.
- 🔥neftune_noise_alpha: Noise magnitude for NEFTune. Default is 0. Common values: 5, 10, 15.
- 🔥use_liger_kernel: Whether to enable the [Liger](https://github.com/linkedin/Liger-Kernel) kernel to accelerate training and reduce GPU memory consumption. Defaults to False. Example shell script can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/liger).
- Note: Liger kernel does not support `device_map`. Use DDP or DeepSpeed for multi-GPU training. Currently, liger_kernel only supports `task_type='causal_lm'`.
- average_tokens_across_devices: Whether to average token counts across devices. If `True`, `num_tokens_in_batch` is synchronized via `all_reduce` for accurate loss computation. Default is `False`.
- max_grad_norm: Gradient clipping. Default is 1.
- Note: The logged `grad_norm` reflects the value **before** clipping.
- push_to_hub: Push checkpoints to the hub. Default is `False`.
- hub_model_id: Model ID on the hub. Default is `None`.
- hub_private_repo: Whether the repo is private. Default is `False`.
### Tuner Arguments
- 🔥freeze_llm: This argument only takes effect for multimodal models and can be used in both full-parameter and LoRA training, but with different behaviors. In full-parameter training, setting `freeze_llm=True` freezes the LLM component's weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_llm=True` prevents LoRA modules from being added to the LLM part. Default is `False`.
- 🔥freeze_vit: This argument only applies to multimodal models and behaves differently depending on the training mode. In full-parameter training, setting `freeze_vit=True` freezes the ViT (vision transformer) component's weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_vit=True` prevents LoRA modules from being added to the ViT part. Default is `True`.
- Note: **Here, "vit" refers not only to `vision_tower`, but also to `audio_tower`**. For Omni models, if you want to apply LoRA only to `vision_tower` and not `audio_tower`, you can modify [this code](https://github.com/modelscope/ms-swift/blob/a5d4c0a2ce0658cef8332d6c0fa619a52afa26ff/swift/llm/model/model_arch.py#L544-L554).
- 🔥freeze_aligner: This argument only affects multimodal models. In full-parameter training, setting `freeze_aligner=True` freezes the aligner (also known as projector) weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_aligner=True` prevents LoRA modules from being added to the aligner component. Default is `True`.
- 🔥target_modules: Specifies which modules to apply LoRA to. Default is `['all-linear']`. You can also specify suffixes of modules, e.g., `--target_modules q_proj k_proj v_proj`. This argument is not limited to LoRA and can be used with other tuners.
- Note: The behavior of `'all-linear'` differs between LLMs and multimodal LLMs. For standard LLMs, it automatically finds all linear layers except `lm_head` and attaches tuners. **For multimodal LLMs, tuners are by default only attached to the LLM component; this behavior can be controlled via `freeze_llm`, `freeze_vit`, and `freeze_aligner`**.
- 🔥target_regex: A regular expression to specify LoRA modules. Default is `None`. If provided, `target_modules` is ignored. For example: `--target_regex '^(language_model).*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)$'` applies LoRA to modules matching the pattern. This argument is not limited to LoRA and can be used with other tuners.
- target_parameters: List of parameter names (not module names) to replace with LoRA. Similar in behavior to `target_modules`, but operates at the parameter level. Requires "peft>=0.17.0". This is useful for models like Mixture-of-Experts (MoE) layers in Hugging Face Transformers, which may use `nn.Parameter` instead of `nn.Linear`.
- Note: This parameter requires `lora_dropout` to be set to 0.
- init_weights: Method for initializing weights. For LoRA: options are 'true', 'false', 'gaussian', 'pissa', 'pissa_niter_[number of iters]', 'olora', 'loftq', 'lora-ga'. For Bone: 'true', 'false', 'bat'. Default is 'true'.
- 🔥modules_to_save: Additional original model modules to include in training and saving, even after attaching a tuner. Default is `[]`. Applies to tuners beyond LoRA. For example: `--modules_to_save embed_tokens lm_head` enables training of `embed_tokens` and `lm_head` during LoRA training, and their weights will be saved in `adapter_model.safetensors`.
#### Full Arguments
- freeze_parameters: List of parameter name prefixes to freeze. Default is `[]`.
- freeze_parameters_regex: Regular expression to match parameters to freeze. Default is `None`.
- freeze_parameters_ratio: Proportion of parameters to freeze, from bottom to top layers. Default is `0`. Setting to `1` freezes all parameters; can be combined with `trainable_parameters` to specify trainable parts.
- trainable_parameters: Prefixes of additional parameters to keep trainable. Default is `[]`.
- trainable_parameters_regex: Regex to match additional trainable parameters. Default is `None`.
- Note: `trainable_parameters` and `trainable_parameters_regex` have higher priority than `freeze_parameters`, `freeze_parameters_regex`, and `freeze_parameters_ratio`. For example, in full-parameter training, all modules are first set to trainable, then some are frozen based on the freeze rules, and finally some are re-enabled via `trainable_parameters` or `trainable_parameters_regex`.
#### LoRA
- 🔥lora_rank: Default is `8`.
- 🔥lora_alpha: Default is `32`.
- lora_dropout: Default is `0.05`.
- lora_bias: Defaults to `'none'`. Possible values are 'none', 'all'. If you want to make all biases trainable, you can set it to `'all'`.
- lora_dtype: Specifies the data type (dtype) for the LoRA modules. Supported values are 'float16', 'bfloat16', 'float32'. Default is None, which follows the default behavior of PEFT.
- 🔥use_dora: Defaults to `False`, indicating whether to use `DoRA`.
- use_rslora: Defaults to `False`, indicating whether to use `RS-LoRA`.
- 🔥lorap_lr_ratio: Parameter for LoRA+. Default is `None`. Recommended values: `10–16`. Setting this when using LoRA enables the LoRA+ variant.
##### LoRA-GA
- lora_ga_batch_size: The default value is `2`. The batch size used for estimating gradients during initialization in LoRA-GA.
- lora_ga_iters: The default value is `2`. The number of iterations for estimating gradients during initialization in LoRA-GA.
- lora_ga_max_length: The default value is `1024`. The maximum input length for estimating gradients during initialization in LoRA-GA.
- lora_ga_direction: The default value is `ArB2r`. The initial direction used for gradient estimation during initialization in LoRA-GA. Allowed values are: `ArBr`, `A2rBr`, `ArB2r`, and `random`.
- lora_ga_scale: The default value is `stable`. The scaling method for initialization in LoRA-GA. Allowed values are: `gd`, `unit`, `stable`, and `weightS`.
- lora_ga_stable_gamma: The default value is `16`. The gamma value when choosing `stable` scaling for initialization.
#### FourierFt
FourierFt uses three parameters: `target_modules`, `target_regex`, and `modules_to_save`, whose meanings are described in the documentation above. Additional parameters include:
- fourier_n_frequency: Number of frequencies in Fourier transform, an `int`, similar to `r` in LoRA. Default value is `2000`.
- fourier_scaling: Scaling value of matrix W, a `float`, similar to `lora_alpha` in LoRA. Default value is `300.0`.
#### BOFT
BOFT uses the three parameters `target_modules`, `target_regex`, and `modules_to_save`, whose meanings are described in the documentation above. Additional parameters include:
- boft_block_size: Size of BOFT blocks, default value is 4.
- boft_block_num: Number of BOFT blocks, cannot be used simultaneously with `boft_block_size`.
- boft_dropout: Dropout value for BOFT, default is 0.0.
#### Vera
Vera uses the three parameters `target_modules`, `target_regex`, and `modules_to_save`, whose meanings are described in the documentation above. Additional parameters include:
- vera_rank: Size of Vera Attention, default value is 256.
- vera_projection_prng_key: Whether to store the Vera mapping matrix, default is True.
- vera_dropout: Dropout value for Vera, default is `0.0`.
- vera_d_initial: Initial value of Vera's d matrix, default is `0.1`.
#### GaLore
- 🔥use_galore: Default value is False, whether to use GaLore.
- galore_target_modules: Default is None, if not provided, applies GaLore to attention and MLP.
- galore_rank: Default value is 128, GaLore rank value.
- galore_update_proj_gap: Default is 50, interval for updating decomposed matrices.
- galore_scale: Default is 1.0, matrix weight coefficient.
- galore_proj_type: Default is `std`, type of GaLore matrix decomposition.
- galore_optim_per_parameter: Default value is False, whether to set a separate optimizer for each Galore target parameter.
- galore_with_embedding: Default value is False, whether to apply GaLore to embedding.
- galore_quantization: Whether to use q-galore, default is `False`.
- galore_proj_quant: Whether to quantize the SVD decomposition matrix, default is `False`.
- galore_proj_bits: Number of bits for SVD quantization.
- galore_proj_group_size: Number of groups for SVD quantization.
- galore_cos_threshold: Cosine similarity threshold for updating projection matrices. Default value is 0.4.
- galore_gamma_proj: As the projection matrix becomes more similar over time, this parameter is the coefficient for extending the update interval. Default value is 2.
- galore_queue_size: Length of the queue for calculating projection matrix similarity, default is 5.
#### LISA
Note: LISA only supports full parameters, i.e., `--tuner_type full`.
- 🔥lisa_activated_layers: Default value is `0`, representing LISA is not used. Setting to a non-zero value activates that many layers, it is recommended to set to 2 or 8.
- lisa_step_interval: Default value is `20`, number of iter to switch to layers that can be backpropagated.
#### UNSLOTH
🔥Unsloth has no additional parameters; it can be supported by adjusting existing parameters, for example:
```
--tuner_backend unsloth
--tuner_type full/lora
--quant_bits 4
```
#### LLAMAPRO
- 🔥llamapro_num_new_blocks: Default value is `4`, total number of new layers to insert.
- llamapro_num_groups: Default value is `None`, number of groups to insert new blocks. If `None`, it equals `llamapro_num_new_blocks`, meaning each new layer is inserted separately into the original model.
#### AdaLoRA
When the `tuner_type` parameter is set to `adalora`, the following parameters take effect. The `adalora` parameters such as `target_modules` inherit from the corresponding parameters of `lora`, but the `lora_dtype` parameter does not take effect.
- adalora_target_r: Default value is `8`, average rank of AdaLoRA.
- adalora_init_r: Default value is `12`, initial rank of AdaLoRA.
- adalora_tinit: Default value is `0`, initial warmup of AdaLoRA.
- adalora_tfinal: Default value is `0`, final warmup of AdaLoRA.
- adalora_deltaT: Default value is `1`, step interval of AdaLoRA.
- adalora_beta1: Default value is `0.85`, EMA parameter of AdaLoRA.
- adalora_beta2: Default value is `0.85`, EMA parameter of AdaLoRA.
- adalora_orth_reg_weight: Default value is `0.5`, regularization parameter for AdaLoRA.
#### ReFT
The following parameters are effective when `tuner_type` is set to `reft`.
> 1. ReFT cannot merge tuners.
> 2. ReFT is not compatible with gradient checkpointing.
> 3. If experiencing issues while using DeepSpeed, please uninstall DeepSpeed temporarily.
- 🔥reft_layers: Which layers ReFT is applied to, default is `None`, representing all layers. You can input a list of layer numbers, e.g., `reft_layers 1 2 3 4`.
- 🔥reft_rank: Rank of ReFT matrix, default is `4`.
- reft_intervention_type: Type of ReFT, supports 'NoreftIntervention', 'LoreftIntervention', 'ConsreftIntervention', 'LobireftIntervention', 'DireftIntervention', 'NodireftIntervention', default is `LoreftIntervention`.
- reft_args: Other supported parameters for ReFT Intervention, input in json-string format.
### vLLM Arguments
Parameter meanings can be found in the [vllm documentation](https://docs.vllm.ai/en/latest/serving/engine_args.html).
- 🔥vllm_gpu_memory_utilization: GPU memory ratio, ranging from 0 to 1. Default is `0.9`.
- 🔥vllm_tensor_parallel_size: Tensor parallelism size. Default is `1`.
- vllm_pipeline_parallel_size: Pipeline parallelism size. Default is `1`.
- vllm_data_parallel_size: Data parallelism size, default is `1`, effective in the `swift deploy/rollout` command.
- In `swift infer`, use `NPROC_PER_NODE` to set the data parallelism (DP) degree. See the example [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_ddp.sh).
- vllm_enable_expert_parallel: Enable expert parallelism. Default is False.
- vllm_max_num_seqs: Maximum number of sequences to be processed in a single iteration. Default is `256`.
- 🔥vllm_max_model_len: The maximum sequence length supported by the model. Default is `None`, meaning it will be read from `config.json`.
- vllm_disable_custom_all_reduce: Disables the custom all-reduce kernel and falls back to NCCL. For stability, the default is `True`.
- vllm_enforce_eager: Determines whether vllm uses PyTorch eager mode or constructs a CUDA graph, default is `False`. Setting it to True can save memory but may affect efficiency.
- vllm_mm_processor_cache_gb: The size (in GiB) of the multimodal processor cache, used to store processed multimodal inputs (e.g., images, videos) to avoid redundant processing. Default is 4. Setting it to 0 disables the cache but may degrade performance (not recommended). This option takes effect only for multimodal models.
- vllm_speculative_config: Speculative decoding configuration, passed as a JSON string. Default: None.
- vllm_disable_cascade_attn: Whether to forcibly disable the V1 engine’s cascade-attention implementation to avoid potential numerical issues. Defaults to False; vLLM’s internal heuristics determine whether cascade attention is actually used.
- 🔥vllm_limit_mm_per_prompt: Controls the use of multiple media in vllm, default is `None`. For example, you can pass in `--vllm_limit_mm_per_prompt '{"image": 5, "video": 2}'`.
- vllm_max_lora_rank: Default is `16`. This is the parameter supported by vllm for lora.
- vllm_quantization: vllm is able to quantize model with this argument, supported values can be found [here](https://docs.vllm.ai/en/latest/serving/engine_args.html).
- 🔥vllm_enable_prefix_caching: Enables vLLM's automatic prefix caching to save processing time for repeated prompt prefixes, improving inference efficiency. Default is `None`, following vLLM's default behavior.
- vllm_use_async_engine: Whether to use the async engine under the vLLM backend. Default is None, which is automatically set based on the scenario: encode tasks (embedding, seq_cls, reranker, generative_reranker) default to True, deployment scenarios (swift deploy) default to True, and other scenarios default to False. Note: Encode tasks must use the async engine.
- vllm_reasoning_parser: Reasoning parser type, used for parsing the chain of thought content of reasoning models. Default is `None`. Only used for the `swift deploy` command. Available types can be found in the [vLLM documentation](https://docs.vllm.ai/en/latest/features/reasoning_outputs.html#streaming-chat-completions).
- vllm_engine_kwargs: Extra arguments for vllm, formatted as a JSON string. Default is `None`.
### SGLang Arguments
Parameter meanings can be found in the [sglang documentation](https://docs.sglang.ai/backend/server_arguments.html).
- 🔥sglang_tp_size: Tensor parallelism size. Default is 1.
- sglang_pp_size: Pipeline parallelism size. Default is 1.
- sglang_dp_size: Data parallelism size. Default is 1.
- sglang_ep_size: Expert parallelism size. Default is 1.
- sglang_enable_ep_moe: Whether to enable EP MoE. Default is False. This parameter has been removed in the latest version of SGLang.
- sglang_mem_fraction_static: The fraction of GPU memory used for static allocation (model weights and KV cache memory pool). If you encounter out-of-memory errors, try reducing this value. Default is None.
- sglang_context_length: The maximum context length of the model. Default is None, which means it will use the value from the model's `config.json`.
- sglang_disable_cuda_graph: Disables CUDA graph. Default is False.
- sglang_quantization: Quantization method. Default is None.
- sglang_kv_cache_dtype: Data type for KV cache storage. 'auto' means it will use the model's data type. 'fp8_e5m2' and 'fp8_e4m3' are supported on CUDA 11.8 and above. Default is 'auto'.
- sglang_enable_dp_attention: Enables data parallelism for attention and tensor parallelism for FFN. The data parallelism size (dp size) should be equal to the tensor parallelism size (tp size). Currently supports DeepSeek-V2/3 and Qwen2/3 MoE models. Default is False.
- sglang_disable_custom_all_reduce: Disables the custom all-reduce kernel and falls back to NCCL. For stability, the default is True.
- sglang_speculative_algorithm: Speculative algorithm. Available options: None, "EAGLE", "EAGLE3", "NEXTN", "STANDALONE", "NGRAM". Default is None.
- sglang_speculative_num_steps: The number of steps sampled from the draft model in speculative decoding. Default is None.
- sglang_speculative_eagle_topk: The number of tokens sampled from the draft model at each step in the EAGLE2 algorithm. Default is None.
- sglang_speculative_num_draft_tokens: The number of tokens sampled from the draft model in speculative decoding. Default is None.
### LMDeploy Arguments
Parameter meanings can be found in the [lmdeploy documentation](https://lmdeploy.readthedocs.io/en/latest/api/pipeline.html#turbomindengineconfig).
- 🔥lmdeploy_tp: tensor parallelism degree. Default is `1`.
- lmdeploy_session_len: Maximum session length. Default is `None`.
- lmdeploy_cache_max_entry_count: The percentage of GPU memory occupied by the k/v cache. Default is `0.8`.
- lmdeploy_quant_policy: Default is `0`. Set it to `4` or `8` when quantizing k/v to 4-bit or 8-bit, respectively.
- lmdeploy_vision_batch_size: The `max_batch_size` parameter passed to `VisionConfig`. Default is `1`.
### Merge Arguments
- 🔥merge_lora: Indicates whether to merge lora; this parameter supports lora, llamapro, and longlora, default is `False`. Example parameters [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/merge_lora.sh).
- safe_serialization: Whether to save the model in safetensors format. Default is True.
- max_shard_size: Maximum size of a single storage file, default is '5GB'.
## Integration Arguments
### Training Arguments
Training arguments include the [base arguments](#base-arguments), [Seq2SeqTrainer arguments](#Seq2SeqTrainer-arguments), [tuner arguments](#tuner-arguments), and also include the following parts:
- add_version: Add directory to `output_dir` with `'<version>-<timestamp>'` to prevent weight overwrite, default is True.
- check_model: Check local model files for corruption or modification and give a prompt, default is True. **If in an offline environment, please set to False.**
- 🔥create_checkpoint_symlink: Creates additional checkpoint symlinks to facilitate writing automated training scripts. The symlink paths for `best_model` and `last_model` are `f'{output_dir}/best'` and `f'{output_dir}/last'` respectively.
- 🔥packing: Uses the `padding_free` approach to pack data samples of different lengths into samples of **approximately** uniform length (packing ensures complete sequences are not split), achieving load balancing across nodes and processes during training (avoiding long texts slowing down short text training speed), thereby improving GPU utilization and maintaining stable memory usage. When using `--attn_impl flash_attn`, it ensures that different sequences within packed samples are independent and invisible to each other. This parameter defaults to `False` and currently supports packing for CPT/SFT/DPO/KTO/GKD as well as embedding/reranker/seq_cls tasks. Note: **packing will reduce the number of dataset samples, please adjust gradient accumulation and learning rate accordingly**.
- Note: For Qwen3-Next packing, please use Megatron-SWIFT. For Qwen3.5 transformers padding_free/packing support, please use "ms-swift>=4.3.1" (or use Megatron-SWIFT). Refer to [Qwen3.5 Best Practice](../BestPractices/Qwen3_5-Best-Practice.md) for details.
- packing_length: the length to use for packing. Defaults to None, in which case it is set to max_length.
- packing_num_proc: Number of processes for packing, default is 1. Note that different values of `packing_num_proc` will result in different packed datasets. (This parameter does not take effect during streaming packing). Usually there is no need to modify this value, as packing speed is much faster than tokenization speed.
- packing_strategy: Packing algorithm, one of 'binpack' and 'sequential', default is 'binpack'. 'binpack' uses best-fit-decreasing bin packing (which reorders samples by length); 'sequential' uses order-preserving greedy packing (next-fit: a single open pack, flushed when the next sample doesn't fit), keeping samples in input order so the sample order and pack boundaries follow a sequential sampler (use `packing_num_proc=1` for a single global ordering).
- lazy_tokenize: Whether to use lazy tokenization. If set to `False`, all dataset samples will be tokenized (and for multimodal models, images will be loaded from disk) before training begins. Default is `None`: in LLM training, it defaults to `False`; in MLLM training, it defaults to `True` to save memory.
- Note: If you want to perform image data augmentation, you need to set `lazy_tokenize` (or `streaming`) to True and modify the `encode` method in the Template class.
- use_logits_to_keep: Pass `logits_to_keep` in the `forward` method based on labels to reduce the computation and storage of unnecessary logits, thereby reducing memory usage and accelerating training. The default is `None`, which enables automatic selection.
- acc_strategy: Strategy for calculating accuracy during training and validation. Options are `seq`-level and `token`-level accuracy, with `token` as the default.
- max_new_tokens: Generation parameter override. The maximum number of tokens to generate when `predict_with_generate=True`, defaulting to 64.
- temperature: Generation parameter override. The temperature setting when `predict_with_generate=True`, defaulting to 0.
- optimizer: The optimizer plugin to use (takes priority over `--optim`), default is None. Available optimizers can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/optimizers/mapping.py).
- loss_type: Custom loss_type name. Default is None, uses the model's built-in loss function. Available loss options can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
- mrl_dims: Dimension configuration for [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) on embedding training. Default is None. Format is `Dict[int, float]` or a JSON string, where the key is the truncated embedding dimension and the value is the corresponding loss weight, e.g. `'{"32": 1.0, "64": 1.0, "128": 1.0}'`. When enabled, the trainer slices `last_hidden_state` to each dimension, applies L2 normalization, and aggregates the per-dimension `loss_type` losses with the configured weights. Only effective when `task_type='embedding'`.
- Note: The maximum supported embedding dimension is determined by `hidden_size` in the model's `config.json`. Any key-value pair whose key is greater than `hidden_size` will be silently ignored.
- eval_metric: Custom eval metric name. Default is None. Available eval_metric options can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/metrics/mapping.py).
- Regarding default values: When `task_type` is 'causal_lm' and `predict_with_generate=True`, it defaults to 'nlg'. When `task_type` is 'embedding', the default value is 'infonce' or 'paired' based on loss_type. When `task_type` is 'reranker/generative_reranker', the default value is 'reranker'.
- callbacks: Custom trainer callbacks, default is `[]`. Available callbacks can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/callbacks/mapping.py).For example,enable elastic training by adding `deepspeed_elastic` (and optionally `graceful_exit`) in `callbacks`. See the [Elastic guide](../BestPractices/Elastic.md).
- early_stop_interval: The interval for early stopping. Training will terminate when best_metric shows no improvement within early_stop_interval periods (based on `save_steps`; it's recommended to set `eval_steps` and `save_steps` to the same value). The specific implementation can be found in [early_stop.py](https://github.com/modelscope/ms-swift/blob/main/swift/callbacks/early_stop.py). Additionally, if you have more complex early stopping requirements, you can directly override the existing implementation in callback.py. When this parameter is set, the `early_stop` trainer callback is automatically added.
- eval_use_evalscope: Whether to use evalscope for evaluation, this parameter needs to be set to enable evaluation, refer to [example](../Instruction/Evaluation.md#evaluation-during-training). Default is False.
- eval_dataset: Evaluation datasets, multiple datasets can be set, separated by spaces
- eval_dataset_args: Evaluation dataset parameters in JSON format, parameters for multiple datasets can be set
- eval_limit: Number of samples from the evaluation dataset
- eval_generation_config: Model inference configuration during evaluation, in JSON format, default is `{'max_tokens': 512}`
- use_flash_ckpt: Whether to use [DLRover Flash Checkpoint](https://github.com/intelligent-machine-learning/dlrover). Default is `false`. If enabled, checkpoints are saved to memory synchronously, then persisted to storage asynchronously. It's recommended to use this with the environment variable `PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"` to avoid CUDA OOM.
#### SWANLAB
- swanlab_token: The API key for SwanLab. You can also specify it using the `SWANLAB_API_KEY` environment variable.
- swanlab_project: The SwanLab project, which can be created in advance on the page [https://swanlab.cn/space/~](https://swanlab.cn/space/~) or created automatically. The default is "ms-swift".
- swanlab_workspace: Defaults to `None`, will use the username associated with the API key.
- swanlab_exp_name: Experiment name, can be left empty. If empty, the value of `--output_dir` will be used by default.
- swanlab_notification_method: The notification method for SwanLab when training completes or errors occur. For details, refer to [here](https://docs.swanlab.cn/plugin/notification-dingtalk.html). Supports 'dingtalk', 'lark', 'email', 'discord', 'wxwork', 'slack'.
- swanlab_webhook_url: Defaults to None. The webhook URL corresponding to SwanLab's `swanlab_notification_method`.
- swanlab_secret: Defaults to None. The secret corresponding to SwanLab's `swanlab_notification_method`.
- swanlab_mode: Optional values are `cloud` and `local`, representing cloud mode or local mode.
### RLHF Arguments
RLHF arguments inherit from the [training arguments](#training-arguments).
- 🔥rlhf_type: Type of human alignment algorithm, supporting 'dpo', 'orpo', 'simpo', 'kto', 'cpo', 'rm', 'ppo', 'grpo' and 'gkd'. Default is 'dpo'.
- ref_model: Required for full parameter training when using the dpo, kto, ppo or grpo algorithms. Default is None, set to `--model`.
- ref_adapters: Defaults to `[]`. If you want to use LoRA weights generated from SFT for DPO/KTO/GRPO, set `--adapters sft_ckpt --ref_adapters sft_ckpt` during training. For checkpoint resumption in this scenario, set `--resume_from_checkpoint rlhf_ckpt --ref_adapters sft_ckpt`.
- ref_model_type: Same as model_type. Default is None.
- ref_model_revision: Same as model_revision. Default is None.
- 🔥beta: A parameter controlling the degree of deviation from the reference model. A higher beta value indicates smaller deviation from the reference model. Default is `None`, with different default values depending on the RLHF algorithm: `2.0` for SimPO, `0.04` for GRPO, `0.5` for GKD, and `0.1` for other algorithms. See [documentation](./RLHF.md) for details.
- label_smoothing: Whether to use DPO smoothing, default value is `0`.
- max_completion_length: The maximum generation length in the GRPO/PPO/GKD algorithms. Default is 512.
- 🔥rpo_alpha: A parameter from the [RPO paper](https://arxiv.org/abs/2404.19733) that controls the weight of the NLL term (i.e., the SFT loss) in the loss function, where `loss = dpo_loss + rpo_alpha * sft_loss`. The paper recommends setting it to `1.`. The default value is `None`, meaning the SFT loss is not included by default.
- ld_alpha: From the [LD-DPO paper](https://arxiv.org/abs/2409.06411). Applies a weight α < 1 to the log-probabilities of tokens that lie beyond the shared prefix of the chosen and rejected responses, thereby mitigating length bias.
- discopop_tau: Temperature parameter τ from the [DiscoPOP paper](https://arxiv.org/abs/2406.08414) used to scale the log-ratio before the sigmoid modulation. Default 0.05; only active when loss_type is discopop.
- loss_type: Type of loss function. Default is None, with different defaults depending on the RLHF algorithm used.
- DPO: Available options can be found in the [documentation](https://huggingface.co/docs/trl/main/en/dpo_trainer#loss-functions). Multiple values can be provided to enable mixed training ([MPO](https://arxiv.org/abs/2411.10442)); when multiple values are given, the loss_weights parameter must also be set. Default is `sigmoid`.
- GRPO: See [GRPO parameters](#grpo-arguments) for reference.
- loss_weights: When setting multiple loss_type values in DPO training, this parameter specifies the weight for each loss component.
- cpo_alpha: Coefficient for nll loss in CPO/SimPO loss, default is `1.`.
- simpo_gamma: Reward margin term in the SimPO algorithm, with a paper-suggested setting of 0.5-1.5, default is `1.`.
- desirable_weight: In the KTO algorithm, this weight compensates for the imbalance between the number of desirable and undesirable samples by scaling the desirable loss. Default is `1.0`.
- undesirable_weight: In the KTO algorithm, this weight compensates for the imbalance between desirable and undesirable samples by scaling the undesirable loss. Default is `1.0`.
- center_rewards_coefficient: A coefficient used in reward model (RM) training to incentivize the model to output rewards with zero mean. See this [paper](https://huggingface.co/papers/2312.09244) for details. Recommended value: 0.01.
- loss_scale: Overrides the template parameter. During RLHF training, the default is `'last_round'`.
- temperature: Default is 0.9; this parameter will be used in PPO, GRPO and GKD.
- top_k: Top-k parameter for rollout sampling. -1 means no top-k filtering is applied. Default is -1.
- top_p: Top-p parameter for rollout sampling. 1.0 means no top-p filtering is applied. Default is 1.0.
#### GKD Arguments
- lmbda: Default is 0.5. This parameter is used in GKD. It controls the lambda parameter for the proportion of student data (i.e., the proportion of student-generated outputs within the strategy). If lmbda is 0, student-generated data is not used.
- sft_alpha: The default value is 0. It controls the weight of sft_loss added in GKD. The final loss is `gkd_loss + sft_alpha * sft_loss`.
- gkd_logits_topk: Use Top-K logits to compute KL divergence. Defaults to None, which means the full vocabulary is used. Setting this parameter can effectively reduce peak GPU memory usage during training. This parameter is required when teacher_model_server is configured. See [distillation documentation](./Distillation.md#top-k-kl-computation) for more details.
- truncation_strategy: The method to handle inputs exceeding `max_length`. Supported values are `delete` and `left`, representing deletion and left-side truncation respectively. The default is `left`. With the delete strategy, over-long or encoding-failed samples are discarded, and new samples are resampled from the original dataset to maintain the intended batch size.
- log_completions: Whether to log the model-generated content during training, to be used in conjunction with `--report_to wandb/swanlab`, default is False.
- Note: If `--report_to wandb/swanlab` is not set, a `completions.jsonl` will be created in the checkpoint to store the generated content.
- Log vLLM rollout results only.
#### Reward/Teacher Model Parameters
The reward model parameters will be used in PPO and GRPO. The teacher model parameters will be used in GKD and GRPO.
- reward_model: Default is None.
- reward_adapters: Default is `[]`.
- reward_model_type: Default is None.
- reward_model_revision: Default is None.
- teacher_model: Default is None.
- teacher_adapters: Default is `[]`.
- teacher_model_type: Default is None.
- teacher_model_revision: Default is None.
- teacher_model_server: Teacher model service URL. Deploy via `swift deploy` for logprobs. Single URL (e.g. `http://localhost:8000`) or multi-teacher JSON (e.g. `'[{"url":"http://t1:8000","tags":["data/math.jsonl"]},{"url":"http://t2:8001","tags":["data/code.jsonl"]}]'`). How `tags` map to datasets or sample fields: [distillation docs](./Distillation.md#multi-teacher-routing).
- teacher_tag_key: Column name for multi-teacher routing; sample values are matched to each teacher's `tags`. Default is `"dataset"`. With multiple `--dataset` values, match each entry; or use a custom column (e.g. `teacher_tag`) via `--teacher_tag_key teacher_tag`.
- teacher_deepspeed: Same as the deepspeed parameter, controls the DeepSpeed configuration for the teacher model. By default, uses the DeepSpeed configuration of the training model.
- offload_teacher_model: Whether to offload the teacher model to save GPU memory. Loaded only during sampling/logps computation. Only effective when `teacher_model` is set. Default is False.
#### PPO Arguments
The meanings of the following parameters can be referenced [here](https://huggingface.co/docs/trl/main/ppo_trainer):
- num_ppo_epochs: Defaults to 4
- whiten_rewards: Defaults to False
- kl_coef: Defaults to 0.05
- cliprange: Defaults to 0.2
- vf_coef: Defaults to 0.1
- cliprange_value: Defaults to 0.2
- gamma: Defaults to 1.0
- lam: Defaults to 0.95
- num_mini_batches: Defaults to 1
- local_rollout_forward_batch_size: Defaults to 64
- num_sample_generations: Defaults to 10
- missing_eos_penalty: Defaults to None
#### GRPO Arguments
- beta: KL regularization coefficient; default 0.04. Setting it to 0 disables the reference model.
- per_device_train_batch_size: The training batch size per device. In GRPO, this refers to the batch size of completions during training.
- per_device_eval_batch_size: The evaluation batch size per device. In GRPO, this refers to the batch size of completions during evaluation.
- steps_per_generation: Number of optimization steps per generation. It defaults to gradient_accumulation_steps. This parameter and generation_batch_size cannot be set simultaneously.
- generation_batch_size: Total batch size of sampling completions. It should be a multiple of num_processes * per_device_train_batch_size. It defaults to per_device_train_batch_size * steps_per_generation * num_processes.
- num_generations: The number of samples generated per prompt (corresponding to the G value in the paper). generation_batch_size must be divisible by num_generations. The default value is 8.
- num_generations_eval: Number of generations to sample during evaluation. This allows using fewer generations during evaluation to save computation. If `None`, uses the value of `num_generations`. Default is None.
- ds3_gather_for_generation: This parameter applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation. The default is True.
- reward_funcs: Reward functions in the GRPO algorithm; options include `accuracy`,`format`,`cosine`,`repetition` and `soft_overlong`, as seen in `swift/rewards/orm.py`. You can also customize your own reward functions in the plugin. Default is `[]`.
- reward_weights: Weights for each reward function. The number should be equal to the sum of the number of reward functions and reward models. If `None`, all rewards are weighted equally with weight `1.0`.
- Note: If `--reward_model` is included in GRPO training, it is added to the end of the reward functions.
- reward_model_plugin: The logic for the reward model, which defaults to ORM logic. For more information, please refer to [Customized Reward Models](./GRPO/DeveloperGuide/reward_model.md#custom-reward-model).
- dataset_shuffle: Whether to shuffle the dataset randomly. Default is True.
- truncation_strategy: The method to handle inputs exceeding `max_length`. Supported values are `delete` and `left`, representing deletion and left-side truncation respectively. The default is `left`. With the delete strategy, over-long or encoding-failed samples are discarded, and new samples are resampled from the original dataset to maintain the intended batch size.
- loss_type: The type of loss normalization. Options are ['grpo', 'bnpo', 'dr_grpo', 'dapo', 'cispo', 'sapo', 'real', 'fipo'], default is 'grpo'. For details, refer to this [doc](./GRPO/DeveloperGuide/loss_types.md)
- fipo_decay_rate: Half-life parameter for FIPO Future-KL. The actual discount is `2 ** (-1 / fipo_decay_rate)`. Default is 32.0.
- fipo_clip_range: Clipping range for the FIPO influence weight. Default is 0.2; set to None or 0 to disable clipping.
- fipo_clip_high_only: Whether to clip the FIPO influence weight to `[1.0, 1.0 + fipo_clip_range]` only. Default is True.
- fipo_safety_threshold: Caps the FIPO influence weight to `[0.8, 1.0]` for negative-advantage tokens whose IS ratio exceeds this threshold. Default is 4.0.
- log_completions: Whether to log the model-generated content during training, to be used in conjunction with `--report_to wandb/swanlab`, default is False.
- Note: If `--report_to wandb/swanlab` is not set, a `completions.jsonl` will be created in the checkpoint to store the generated content.
- use_vllm: Whether to use vLLM as the infer_backend for GRPO generation, default is False.
- vllm_mode: Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `server` or `colocate`
- vllm_mode server parameter
- vllm_server_host: The host address of the vLLM server. Default is None.
- vllm_server_port: The service port of the vLLM server. Default is 8000.
- vllm_server_base_url: Base URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` " "and `vllm_server_port` are ignored. Default is None.
- vllm_server_timeout: The connection timeout for the vLLM server. Default is 240 seconds.
- vllm_server_pass_dataset: pass additional dataset information through to the vLLM server for multi-turn training.
- vllm_server_group_port: The internal communication port for the vLLM server. Generally, there is no need to set it unless the port is occupied. The default value is 51216.
- async_generate: Use async rollout to improve train speed. Note that rollout will use the model updated in the previous round when enabled. Multi-turn scenarios are not supported. Default is `false`.
- enable_flattened_weight_sync: Whether to use flattened tensor for weight synchronization. When enabled, multiple parameters are packed into a single contiguous tensor for transfer, which can improve synchronization efficiency; Only takes effect in Server Mode. Default is True.
- SWIFT_UPDATE_WEIGHTS_BUCKET_SIZE: An environment variable that controls the bucket size (in MB) for flattened tensor weight synchronization during full-parameter training in Server Mode. Default is 512 MB.
- vllm_mode colocate parameter (For more parameter support, refer to the [vLLM Arguments](#vLLM-Arguments).)
- vllm_gpu_memory_utilization: vLLM passthrough parameter, default is 0.9.
- vllm_max_model_len: vLLM passthrough parameter, the total length limit of model, default is None.
- vllm_enforce_eager: vLLM passthrough parameter, default is False.
- vllm_limit_mm_per_prompt: vLLM passthrough parameter, default is None.
- vllm_enable_prefix_caching: A pass-through parameter for vLLM, default is True.
- vllm_tensor_parallel_size: the tensor parallel size of vLLM engine, default is 1.
- vllm_enable_lora: Enable the vLLM engine to load LoRA adapters; defaults to False. Used to accelerate weight synchronization during LoRA training. See the [documentation](./GRPO/GetStarted/GRPO.md#weight-sync-acceleration) for details.
- sleep_level: make vllm sleep when model is training. Options are 0/1/2, default is 0, no sleep
- offload_optimizer: Whether to offload optimizer parameters during inference with vLLM. The default is `False`.
- offload_model: Whether to offload the model during inference with vLLM. The default is `False`.
- completion_length_limit_scope: Specifies the scope of the `max_completion_length` limit in multi-turn conversations.
When set to `total`, the total output length across all turns must not exceed `max_completion_length`.
When set to `per_round`, each individual turn's output length is limited separately.
Defaults to `per_round`. Currently only takes effect in colocate mode.
- num_iterations: The number of updates per data sample, corresponding to the $\mu$ value in the GRPO paper. Default is 1.
- epsilon: epsilon value for clipping. Default is 0.2.
- epsilon_high: Upper clip coefficient, default is None. When set, it forms a clipping range of [epsilon, epsilon_high] together with epsilon.
- tau_pos: Temperature parameter for positive advantages in [SAPO](https://arxiv.org/abs/2511.20347) algorithm, controlling the sharpness of the soft gating function. Larger values make the gate sharper (closer to hard clipping), smaller values make it smoother. Default is 1.0.
- tau_neg: Temperature parameter for negative advantages in SAPO algorithm, controlling the sharpness of the soft gating function. Typically set `tau_neg > tau_pos` to apply stronger constraints on negative advantages. Default is 1.05.
- dynamic_sample: Exclude data within the group where the reward standard deviation is 0, and additionally sample new data. Default is False.
- max_resample_times: Under the dynamic_sample setting, limit the number of resampling attempts to a maximum of 3. Default is 3 times.
- overlong_filter: Skip overlong truncated samples, which will not be included in loss calculation. Default is False.
The hyperparameters for the reward function can be found in the [Built-in Reward Functions section](#built-in-reward-functions).
- delta: Delta value for the upper clipping bound in two-sided GRPO. Recommended to be > 1 + epsilon. This method was introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291).
- importance_sampling_level: Controls how the importance sampling ratio is computed. Options are `token` and `sequence`. In `token` mode, the raw per-token log-probability ratios are used. In `sequence` mode, the log-probability ratios of all valid tokens in the sequence are averaged to produce a single ratio per sequence. The [GSPO paper](https://arxiv.org/abs/2507.18071) uses sequence-level importance sampling to stabilize training. The default is `token`.
- advantage_estimator: Advantage estimator. Default is `grpo` (group-relative advantage). Options: `grpo`, [`rloo`](./GRPO/AdvancedResearch/RLOO.md), [`reinforce_plus_plus`](./GRPO/AdvancedResearch/REINFORCEPP.md).
- kl_in_reward: Controls where the KL regularization is applied. `false`: KL is a separate loss term. `true`: KL is subtracted from the reward. The default is bound to `advantage_estimator`: `false` for `grpo`, and `true` for `rloo` and `reinforce_plus_plus`.
- scale_rewards: Specifies the reward scaling strategy. Options: `group` (scale by intra-group std), `batch` (scale by batch-wide std), `none` (no scaling), `gdpo` (normalize each reward function separately within groups before weighted aggregation, see [GDPO paper](https://arxiv.org/abs/2601.05242)). The default is bound to `advantage_estimator`: `group` for `grpo`, `none` for `rloo`, and `batch` for `reinforce_plus_plus`.
- Note: `gdpo` mode does not support `kl_in_reward=True`. If both are set, `kl_in_reward` will be automatically set to `False`.
- GDPO is designed for multi-reward optimization: When using multiple reward functions, GDPO normalizes each reward function separately within groups (subtract mean, divide by std), then performs weighted aggregation using `reward_weights`, and finally applies batch-level normalization. This approach better preserves the relative differences between rewards and prevents different reward combinations from collapsing into identical advantage values.
- teacher_kl_coef: Coefficient for teacher KL in OPD-RL, i.e. `adv_t = base_adv + teacher_kl_coef * teacher_kl`. Default is 1.0.
- sync_ref_model: Whether to synchronize the reference model. Default is False.
- ref_model_mixup_alpha: The Parameter controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: $π_{ref} = α * π_θ + (1 - α) * π_{ref_{prev}}$. Default is 0.6.
- ref_model_sync_steps：The parameter determines how frequently the current policy is synchronized with the reference policy. Default is 512.
- move_model_batches: When moving model parameters to fast inference frameworks such as vLLM/LMDeploy, determines how many batches to divide the layers into. The default is `None`, which means the entire model is not split. Otherwise, the model is split into `move_model_batches + 1` (non-layer parameters) + `1` (multi-modal component parameters) batches.
- multi_turn_scheduler: Multi-turn GRPO parameter; pass the corresponding plugin name, and make sure to implement it in plugin/multi_turn.py.
- max_turns: Maximum number of rounds for multi-turn GRPO. The default is None, which means there is no limit.
- gym_env: Globally select the gym environment name (must be registered in the plugin). Defaults to None and can be overridden per row via `env_config.name`. See the [documentation](./GRPO/DeveloperGuide/gym_env.md).
- use_gym_env: Whether to use the env-provided `total_reward` as the reward (no reward function required). Defaults to None; when not set explicitly, it is auto-enabled if `gym_env` is set, otherwise inherited from the rollout server in server mode, and False in all other cases.
- top_entropy_quantile: Only tokens whose entropy ranks within the specified top quantile are included in the loss calculation. The default is 1.0, which means low-entropy tokens are not filtered. For details, refer to the [documentation](./GRPO/AdvancedResearch/entropy_mask.md).
- log_entropy: Logs the entropy values during training. The default is False. For more information, refer to the [documentation](./GRPO/GetStarted/GRPO.md#logged-metrics).
- rollout_importance_sampling_mode: Training-inference mismatch correction mode. Options are `token_truncate`, `token_mask`, `sequence_truncate`, `sequence_mask`. Default is None (disabled). For details, refer to the [documentation](./GRPO/AdvancedResearch/training_inference_mismatch.md).
- rollout_importance_sampling_threshold: Threshold for importance sampling weights, used for truncating or masking extreme weights. Default is 2.0.
- log_rollout_offpolicy_metrics: Whether to log training-inference mismatch diagnostic metrics (KL, PPL, χ², etc.) when `rollout_importance_sampling_mode` is not set. When `rollout_importance_sampling_mode` is set, metrics are always logged. Default is False.
- off_policy_sequence_mask_delta: Off-Policy Sequence Masking threshold from [DeepSeek-V3.2 paper](https://arxiv.org/abs/2512.02556). When set, computes `mean(old_policy_logps - policy_logps)` for each sequence. If this value exceeds the threshold AND the sequence has negative advantage, the sequence is masked out from loss computation. Default is None (disabled). For details, refer to the [documentation](./GRPO/AdvancedResearch/training_inference_mismatch.md#off-policy-sequence-masking).
##### Reward function parameters
Refer to the [documentation](./GRPO/DeveloperGuide/reward_function.md) for built-in reward functions.
cosine reward function arguments
- cosine_min_len_value_wrong (default: -0.5): Reward value corresponding to the minimum length when the answer is incorrect.
- cosine_max_len_value_wrong (default: 0.0): Reward value corresponding to the maximum length when the answer is incorrect.
- cosine_min_len_value_correct (default: 1.0): Reward value corresponding to the minimum length when the answer is correct.
- cosine_max_len_value_correct (default: 0.5): Reward value corresponding to the maximum length when the answer is correct.
- cosine_max_len (default value equal to the model's maximum generation capacity): Maximum length limit for generated text. Default value equal to max_completion_length
repetition penalty function arguments
- repetition_n_grams (default: 3): Size of the n-gram used to detect repetition.
- repetition_max_penalty (default: -1.0): Maximum penalty value, which controls the intensity of the penalty.
Soft overlong reward parameters:
- soft_max_length: L_max in the paper, the maximum generation length of the model, default is equal to max_completion_length.
- soft_cache_length: L_cache in the paper, controls the length penalty interval, which is defined as [soft_max_length - soft_cache_length, soft_max_length].
### Inference Arguments
Inference arguments include the [base arguments](#base-arguments), [merge arguments](#merge-arguments), [vLLM arguments](#vllm-arguments), [LMDeploy arguments](#LMDeploy-arguments), and also contain the following:
- 🔥infer_backend: Inference acceleration backend, supporting four inference engines: 'transformers', 'vllm', 'sglang', and 'lmdeploy'. The default is 'transformers'.
- Note: All four engines use SWIFT's template, controlled by `--template_backend`.
- 🔥max_batch_size: Effective when infer_backend is set to 'transformers'; used for batch inference, with a default value of 1. If set to -1, there is no restriction.
- 🔥result_path: Path to store inference results (jsonl). Defaults to None. When performing inference/evaluation on datasets, results are saved by default in the checkpoint directory (containing args.json file) or the './result' directory. The final storage path will be printed in the command line (interactive inference or deployment does not save results by default).
- Note: If the `result_path` file already exists, results will be appended to it.
- write_batch_size: The batch size for writing results to result_path. Defaults to 1000. If set to -1, there is no restriction.
- metric: Evaluate the results of the inference, currently supporting 'acc' and 'rouge'. The default is None, meaning no evaluation is performed.
- val_dataset_sample: Number of samples from the inference dataset, default is None.
- reranker_use_activation: Whether to apply sigmoid activation after the score during reranker inference. Default is True.
### Deployment Arguments
Deployment Arguments inherit from the [inference arguments](#inference-arguments).
- host: Service host, default is '0.0.0.0'.
- port: Port number, default is 8000.
- api_key: The API key required for access; the default is None.
- owned_by: Default is `swift`.
- 🔥served_model_name: Model name for serving, defaults to the model's suffix.
- verbose: Print detailed logs, with a default value of True.
- Note: In `swift app` or `swift eval`, the default is False.
- log_interval: Interval for printing tokens/s statistics, default is 20 seconds. If set to -1, it will not be printed.
- max_logprobs: Maximum number of logprobs returned to the client, with a default value of 20.
### Rollout Arguments
The rollout parameters inherit from the [deployment parameters](#deployment-arguments).
- multi_turn_scheduler: The scheduler for multi-turn GRPO training. Pass the corresponding plugin name, and ensure the implementation is added in `plugin/multi_turn.py`. Default is `None`. See [documentation](./GRPO/DeveloperGuide/multi_turn.md) for details.
- max_turns: Maximum number of turns in multi-turn GRPO training. Default is `None`, meaning no limit.
- gym_env: Globally select the gym environment name (must be registered in the plugin). Defaults to None and can be overridden per row via `env_config.name`. See the [documentation](./GRPO/DeveloperGuide/gym_env.md).
- use_gym_env: Whether to enable gym-environment mode (rollout emits `total_reward` for the trainer to consume). Defaults to None; when not set explicitly, it is auto-enabled if `gym_env` is set.
- vllm_enable_lora: Enable the vLLM engine to load LoRA adapters; defaults to False. Used to accelerate weight synchronization during LoRA training. See the [documentation](./GRPO/GetStarted/GRPO.md#weight-sync-acceleration) for details.
- vllm_max_lora_rank: LoRA parameter for the vLLM engine. Must be greater than or equal to the training lora_rank; it is recommended to set them equal. Defaults to 16.
- vllm_enable_expert_parallel: Enable Expert Parallel (EP) for MoE models, distributing experts across ranks. Valid when vllm_use_async_engine is true. Default is False.
### Web-UI Arguments
- server_name: Host for the web UI, default is '0.0.0.0'.
- server_port: Port for the web UI, default is 7860.
- share: Default is False.
- lang: Language for the web UI, options are 'zh', 'en'. Default is 'zh'.
### App Arguments
App parameters inherit from [deployment arguments](#deployment-arguments) and [Web-UI Arguments](#web-ui-arguments).
- base_url: The base URL for model deployment, for example, `http://localhost:8000/v1`. The default value is `None`, which means using local deployment.
- studio_title: Title of the studio. Default is None, set to the model name.
- is_multimodal: Whether to launch the multimodal version of the app. Defaults to None, automatically determined based on the model; if it cannot be determined, set to False.
- lang: Overrides the Web-UI Arguments, default is 'en'.
### Evaluation Arguments
Evaluation Arguments inherit from the [deployment arguments](#deployment-arguments).
- 🔥eval_backend: Evaluation backend, defaults to 'Native'. It can also be specified as 'OpenCompass' or 'VLMEvalKit'.
- 🔥eval_dataset: Evaluation dataset, please refer to the [evaluation documentation](./Evaluation.md).
- eval_limit: Number of samples per evaluation set, defaults to None.
- eval_output_dir: Directory to store evaluation results, defaults to 'eval_output'.
- temperature: Override generation parameters, defaults to 0.
- eval_num_proc: Maximum client concurrency during evaluation, defaults to 16.
- eval_url: Evaluation URL, e.g., `http://localhost:8000/v1`. Examples can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/eval/eval_url). Defaults to None for local deployment evaluation.
- eval_generation_config: Model inference configuration during evaluation, should be passed as a JSON string, e.g., `'{"max_new_tokens": 512}'`; defaults to None.
- extra_eval_args: Additional evaluation parameters, should be passed as a JSON string, defaults to empty. Only effective for Native evaluation. For more parameter descriptions, please refer to [here](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html).
- local_dataset: Some evaluation sets, such as `CMB`, require additional data packages to be downloaded for utilization. Setting this parameter to `true` will automatically download the full data package, create a `data` folder in the current directory, and start the evaluation. The data package will only be downloaded once, and future evaluations will use the cache. This parameter defaults to `false`.
- Note: By default, evaluation uses the dataset under `~/.cache/opencompass`. After specifying this parameter, it will directly use the data folder in the current directory.
### Export Arguments
Export Arguments include the [basic arguments](#base-arguments) and [merge arguments](#merge-arguments), and also contain the following:
- 🔥output_dir: The path for storing exported results. The default value is None, and an appropriate suffix path will be automatically set.
- exist_ok: If output_dir exists, do not raise an exception and overwrite the contents. The default value is False.
- 🔥quant_method: Options are 'gptq', 'awq', 'bnb' or 'fp8', with the default being None. Examples can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize).
- quant_n_samples: The number of samples for the validation set used by gptq/awq, with a default of 256.
- quant_batch_size: Quantization batch size, default is 1.
- group_size: Group size for quantization, default is 128.
- to_cached_dataset: pre-tokenize the dataset and export it in advance, default is False. See the example [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/cached_dataset). For more information, please refer to cached_dataset.
- Note: You can specify the validation set content through `--split_dataset_ratio` or `--val_dataset`.
- template_mode: Used to support the `cached_dataset` feature for `swift rlhf` training. This parameter only takes effect when `--to_cached_dataset true` is set. Available options include: 'train', 'rlhf', and 'kto'. Among them, `swift pt/sft` uses 'train', `swift rlhf --rlhf_type kto` uses 'kto', and other rlhf algorithms use 'rlhf'. Note: Currently, 'gkd', 'ppo', and 'grpo' algorithms do not support the `cached_dataset` feature. Default is 'train'.
- to_ollama: Generate the Modelfile required by Ollama. Default is False.
- 🔥to_mcore: Convert weights from HF format to Megatron format. Default is False.
- to_hf: Convert weights from Megatron format to HF format. Default is False.
- mcore_model: Path to the mcore format model. Default is None.
- mcore_adapter: The adapter path for mcore format models, default is None.
- thread_count: The number of model slices when `--to_mcore true` is set. Defaults to None, and is automatically configured based on the model size, ensuring that the largest slice is less than 10GB.
- 🔥offload_bridge: Store Megatron exported HF format weights for vLLM updates in CPU main memory to reduce GPU memory usage. Default is False.
- 🔥test_convert_precision: Test the precision error when converting weights between HF and Megatron formats. Default is False.
- test_convert_dtype: The dtype used for conversion precision testing, defaults to 'float32'.
- 🔥push_to_hub: Whether to push to the hub, with the default being False. Examples can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/push_to_hub.sh).
- hub_model_id: Model ID for pushing, default is None.
- hub_private_repo: Whether it is a private repo, default is False.
- commit_message: Commit message, default is 'update files'.
### Sampling Parameters
- prm_model: The type of process reward model. It can be a model ID (triggered using `transformers`) or a `prm` key defined in a plugin (for custom inference processes).
- orm_model: The type of outcome reward model, typically a wildcard or test case, usually defined in a plugin.
- sampler_type: The type of sampling. Currently supports `sample` and `distill`.
- sampler_engine: Supports `transformers`, `lmdeploy`, `vllm`, `no`. Defaults to `transformers`. Specifies the inference engine for the sampling model.
- output_dir: The output directory. Defaults to `sample_output`.
- output_file: The name of the output file. Defaults to `None`, which uses a timestamp as the filename. When provided, only the filename should be passed without the directory, and only JSONL format is supported.
- override_exist_file: Whether to overwrite if `output_file` already exists.
- num_sampling_batch_size: The batch size for each sampling operation.
- num_sampling_batches: The total number of batches to sample.
- n_best_to_keep: The number of best sequences to return.
- data_range: The partition of the dataset being processed for this sampling operation. The format should be `2 3`, meaning the dataset is divided into 3 parts, and this instance is processing the 3rd partition (this implies that typically three `swift sample` processes are running in parallel).
- temperature: Defaults to `1.0`.
- prm_threshold: The PRM threshold. Results below this value will be filtered out. The default value is `0`.
- easy_query_threshold: For each query, if the ORM evaluation is correct for more than this proportion of all samples, the query will be discarded to prevent overly simple queries from appearing in the results. Defaults to `None`, meaning no filtering is applied.
- engine_kwargs: Additional parameters for the `sampler_engine`, passed as a JSON string, for example, `{"cache_max_entry_count":0.7}`.
- num_return_sequences: The number of original sequences returned by sampling. Defaults to `64`. This parameter is effective for `sample` sampling.
- cache_files: To avoid loading both `prm` and `generator` simultaneously and causing GPU memory OOM, sampling can be done in two steps. In the first step, set `prm` and `orm` to `None`, and all results will be output to a file. In the second run, set `sampler_engine` to `no` and pass `--cache_files` with the output file from the first sampling. This will use the results from the first run for `prm` and `orm` evaluation and output the final results.
- Note: When using `cache_files`, the `--dataset` still needs to be provided because the ID for `cache_files` is calculated using the MD5 of the original data. Both pieces of information need to be used together.
## Specific Model Arguments
In addition to the parameters listed above, some models support additional model-specific arguments. The meanings of these parameters can usually be found in the corresponding model's official repository or its inference code. **MS-Swift includes these parameters to ensure that the trained model aligns with the behavior of the official inference implementation**.
- Model-specific parameters can be set via `--model_kwargs` or environment variables. For example: `--model_kwargs '{"fps_max_frames": 12}'` or `FPS_MAX_FRAMES=12`.
- Note: If you specify model-specific parameters during training, please also set the corresponding parameters during inference to achieve optimal performance.
### qwen2_vl, qvq, qwen2_5_vl, mimo_vl, keye_vl, keye_vl_1_5
These parameters have the same meaning as in `qwen_vl_utils<0.0.12` or the `qwen_omni_utils` library. See [here](https://github.com/QwenLM/Qwen2.5-VL/blob/main/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L24) for details. MS-Swift adjusts these constant values to control image resolution and video frame rate, preventing out-of-memory (OOM) errors during training.
- IMAGE_FACTOR: Default is 28.
- MIN_PIXELS: Default is `4 * 28 * 28`. Minimum image resolution. It is recommended to set this as a multiple of 28×28.
- 🔥MAX_PIXELS: Default is `16384 * 28 * 28`. Maximum image resolution. It is recommended to set this as a multiple of 28×28.
- MAX_RATIO: Default is 200.
- VIDEO_MIN_PIXELS: Default is `128 * 28 * 28`. Minimum resolution per frame in a video. Recommended to be a multiple of 28×28.
- 🔥VIDEO_MAX_PIXELS: Default is `768 * 28 * 28`. Maximum resolution per frame in a video. Recommended to be a multiple of 28×28.
- VIDEO_TOTAL_PIXELS: Default is `24576 * 28 * 28`.
- FRAME_FACTOR: Default is 2.
- FPS: Default is 2.0.
- FPS_MIN_FRAMES: Default is 4. Minimum number of frames extracted from a video clip.
- 🔥FPS_MAX_FRAMES: Default is 768. Maximum number of frames extracted from a video clip.
- 🔥QWENVL_BBOX_FORMAT: Specifies whether to use `'legacy'` or `'new'` format for grounding. The `'legacy'` format is: `<|object_ref_start|>a dog<|object_ref_end|><|box_start|>(432,991),(1111,2077)<|box_end|>`. The `'new'` format refers to: [Qwen3-VL Cookbook](https://github.com/QwenLM/Qwen3-VL/blob/main/cookbooks/2d_grounding.ipynb). For dataset formatting, see the [Grounding Dataset Format Documentation](../Customization/Custom-dataset.md#grounding). Default: `'legacy'`.
- Note: This environment variable applies to Qwen2/2.5/3-VL and Qwen2.5/3-Omni series models.
### qwen2_audio, qwen3_asr
- SAMPLING_RATE: Default is 16000
### qwen3_vl, qwen3_5
The parameter meanings are the same as in the `qwen_vl_utils>=0.0.14` library — see here: https://github.com/QwenLM/Qwen2.5-VL/blob/main/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L24. By passing the following environment variables you can override the library's global default values: (It is also compatible with environment variables used by `qwen2_5_vl`, such as: `MAX_PIXELS`, `VIDEO_MAX_PIXELS`, and will perform automatic conversion.)
- SPATIAL_MERGE_SIZE: default 2.
- IMAGE_MIN_TOKEN_NUM: default `4`, denotes the minimum number of image tokens per image.
- 🔥IMAGE_MAX_TOKEN_NUM: default `16384`, denotes the maximum number of image tokens per image. (used to avoid OOM)
- Note: The equivalent maximum image pixel count is `IMAGE_MAX_TOKEN_NUM * 32 * 32`.
- VIDEO_MIN_TOKEN_NUM: default `128`, denotes the minimum number of video tokens per frame.
- 🔥VIDEO_MAX_TOKEN_NUM: default `768`, denotes the maximum number of video tokens per frame. (used to avoid OOM)
- MAX_RATIO: default 200.
- FRAME_FACTOR: default 2.
- FPS: default 2.0.
- FPS_MIN_FRAMES: default 4, denotes the minimum number of sampled frames for a video segment.
- 🔥FPS_MAX_FRAMES: default 768, denotes the maximum number of sampled frames for a video segment. (used to avoid OOM)
### qwen2_5_omni, qwen3_omni
qwen2_5_omni not only includes the model-specific parameters of qwen2_5_vl and qwen2_audio, but also contains the following parameter: (Note: qwen3_omni includes model-specific parameters of **qwen3_vl** and qwen2_audio)
- USE_AUDIO_IN_VIDEO: Whether to use audio information from video. Default is `False`.
- 🔥ENABLE_AUDIO_OUTPUT: Defaults to None, which means the value from `config.json` will be used. If training with zero3, please set it to False.
- Tip: ms-swift only fine-tunes the "thinker" component; it is recommended to set this to `False` to reduce GPU memory usage (only the thinker part of the model structure will be created).
### qwen3_vl_emb, qwen3_vl_reranker
The parameter meanings are the same as `qwen3_vl`, see the description above. The following are overrides to the default values:
- IMAGE_MAX_TOKEN_NUM: Default is 1800 for qwen3_vl_emb, and 1280 for qwen3_vl_reranker. For details, please refer to: [qwen3_vl_embedding](https://modelscope.cn/models/Qwen/Qwen3-VL-Embedding-2B/file/view/master/scripts%2Fqwen3_vl_embedding.py?status=1#L26), [qwen3_vl_reranker](https://modelscope.cn/models/Qwen/Qwen3-VL-Reranker-2B/file/view/master/scripts%2Fqwen3_vl_reranker.py?status=1#L16).
- FPS: Default is 1.
- FPS_MAX_FRAMES: Default is 64.
### internvl_chat
For the meaning of the arguments, please refer to [here](https://modelscope.cn/models/OpenGVLab/InternVL2_5-2B)
- MAX_NUM: Default is 12
- INPUT_SIZE: Default is 448
- VIDEO_MAX_NUM: Default is 1, which is the MAX_NUM for videos
- VIDEO_SEGMENTS: Default is 8
### minicpmv2_6, minicpmv4, minicpmo
- MAX_SLICE_NUMS: Default is 9, refer to [here](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6/file/view/master?fileName=config.json&status=1)
- VIDEO_MAX_SLICE_NUMS: Default is 1, which is the MAX_SLICE_NUMS for videos, refer to [here](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6)
- MAX_NUM_FRAMES: Default is 64, refer to [here](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6)
### minicpmv4_6
- DOWNSAMPLE_MODE: Default is `'16x'`. Visual token downsampling mode. `'16x'` merges tokens for efficiency; `'4x'` retains 4x more tokens for finer detail.
- MAX_SLICE_NUMS: Default is 9. Maximum number of slices when splitting a high-resolution image. A larger value preserves more detail for large images. It is recommended to set this to 36 for images.
- VIDEO_MAX_SLICE_NUMS: Default is 1. `MAX_SLICE_NUMS` for videos.
- MAX_NUM_FRAMES: Default is 128. Maximum number of main frames sampled from a video.
- STACK_FRAMES: Default is 1. Total number of samples per second. `1` means only the main frame is used (no stacking). `N` (N>1) means 1 main frame plus N−1 sub-frames per second; sub-frames are composed into a grid image and interleaved with the main frame. It is recommended to set this to 1 for short videos and 3 or 5 for long videos.
### minicpmo
- INIT_TTS: Defaults to False. Whether to initialize and load the TTS model.
- INIT_AUDIO: Defaults to True. Whether to initialize and load the Audio model.
- USE_AUDIO_IN_VIDEO: Defaults to False. Whether to use the audio information from the video.
### ovis1_6, ovis2
- MAX_PARTITION: Default is 9, refer to [here](https://github.com/AIDC-AI/Ovis/blob/d248e34d755a95d24315c40e2489750a869c5dbc/ovis/model/modeling_ovis.py#L312)
### ovis2_5
The meanings of the following parameters can be found in the example code [here](https://modelscope.cn/models/AIDC-AI/Ovis2.5-2B).
- MIX_PIXELS: int type, default is `448 * 448`.
- MAX_PIXELS: int type, default is `1344 * 1792`. If OOM (out of memory) occurs, you can reduce this value.
- VIDEO_MAX_PIXELS: int type, default is `896 * 896`.
- NUM_FRAMES: default is 8. Used for video frame sampling.
### mplug_owl3, mplug_owl3_241101
- MAX_NUM_FRAMES: Default is 16, refer to [here](https://modelscope.cn/models/iic/mPLUG-Owl3-7B-240728)
### xcomposer2_4khd
- HD_NUM: Default is 55, refer to [here](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-4khd-7b)
### xcomposer2_5
- HD_NUM: Default is 24 when the number of images is 1. Greater than 1, the default is 6. Refer to [here](https://modelscope.cn/models/AI-ModelScope/internlm-xcomposer2d5-7b/file/view/master?fileName=modeling_internlm_xcomposer2.py&status=1#L254)
### video_cogvlm2
- NUM_FRAMES: Default is 24, refer to [here](https://github.com/zai-org/CogVLM2/blob/main/video_demo/inference.py#L22)
### phi3_vision
- NUM_CROPS: Default is 4, refer to [here](https://modelscope.cn/models/LLM-Research/Phi-3.5-vision-instruct)
### llama3_1_omni
- N_MELS: Default is 128, refer to [here](https://github.com/ictnlp/LLaMA-Omni/blob/544d0ff3de8817fdcbc5192941a11cf4a72cbf2b/omni_speech/infer/infer.py#L57)
### video_llava
- NUM_FRAMES: Default is 16
## Other Environment Variables
- USE_HF: Use ModelScope/HuggingFace. Defaults to '0'.
- CUDA_VISIBLE_DEVICES: Controls which GPU to use. By default, all GPUs are used.
- ASCEND_RT_VISIBLE_DEVICES: Controls which NPU (effective for ASCEND cards) are used. By default, all NPUs are used.
- MODELSCOPE_CACHE: Controls the cache path. (Recommended to set this value during multi-node training to ensure all nodes use the same dataset cache.)
- NPROC_PER_NODE: Pass-through for the `--nproc_per_node` parameter in torchrun. The default is 1. If the `NPROC_PER_NODE` or `NNODES` environment variables are set, torchrun is used to start training or inference.
- PYTORCH_CUDA_ALLOC_CONF: It is recommended to set it to `'expandable_segments:True'`, which reduces GPU memory fragmentation. For more details, please refer to the [PyTorch documentation](https://docs.pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management).
- MASTER_PORT: Pass-through for the `--master_port` parameter in torchrun. The default is 29500.
- MASTER_ADDR: Pass-through for the `--master_addr` parameter in torchrun.
- NNODES: Pass-through for the `--nnodes` parameter in torchrun.
- NODE_RANK: Pass-through for the `--node_rank` parameter in torchrun.
- LOG_LEVEL: The log level, default is 'INFO'. You can set it to 'WARNING', 'ERROR', etc.
- SWIFT_DEBUG: When set to `'1'` during `engine.infer(...)`, TransformersEngine will print the contents of `input_ids` and `generate_ids` to facilitate debugging and alignment.
- VLLM_USE_V1: Used to switch between V0 and V1 versions of vLLM.
- SWIFT_TIMEOUT: If the multimodal dataset contains image URLs, this parameter controls the timeout for fetching images, defaulting to 20 seconds.
- ROOT_IMAGE_DIR: The root directory for image (multimodal) resources. By setting this parameter, relative paths in the dataset can be interpreted relative to `ROOT_IMAGE_DIR`. By default, paths are relative to the current working directory.
- SWIFT_SINGLE_DEVICE_MODE: Single device mode, valid values are "0"(default)/"1". In this mode, each process can only see one device.
- SWIFT_AUDIO_LOAD_BACKEND: Audio waveform loading backend. `librosa` (default) or `soundfile_pyav` (soundfile first, pyav fallback). For GRPO/GKD with `--use_vllm true`,swift auto-sets it to `soundfile_pyav` so training encode and vLLM rollout decode audio URLs consistently.
docs/source_en/Instruction/Distillation.md
+391
View File
@@ -0,0 +1,391 @@
# Knowledge Distillation
Knowledge distillation is a training method that transfers capabilities from a teacher model to a student model. The core idea is to have the student align with the teacher's output distribution at each token position, yielding richer supervision than simply imitating labeled answers—the teacher tells the student not only which token is correct, but also how good or bad other tokens are.
This document introduces, top-down: why distillation works (Section 1), a unified design framework for distillation methods (Section 2), and finally three concrete distillation training methods in swift: GKD / OPD-RL / OPSD (Section 3).
---
## 1. Why Distillation: From Sparse to Dense Signals
A language model's capabilities are typically built through a stack of training stages:
- **Pre-training**: Acquire language, world knowledge, basic reasoning, and other general capabilities.
- **Mid-training**: Inject domain knowledge, such as code, medicine, or internal company documents.
- **Post-training**: Elicit target behaviors, such as instruction following, mathematical reasoning, or dialogue style.
Distillation mainly happens during **post-training**. To understand its value, consider post-training methods along two independent dimensions:
1. **Sampling mode (where data comes from)**: Whether training sequences are generated by the student itself (on-policy) or come from external fixed data (off-policy).
2. **Feedback density (how much each sequence teaches)**: Whether the entire sequence has a single reward (per-sequence, sparse) or each token has a signal (per-token, dense).
**SFT / offline distillation** (off-policy + dense): Align with labels or the teacher distribution on fixed data. The signal is dense, but training only sees states from the teacher/labels, which differ from states the student enters during its own inference. Once the student makes an early mistake the teacher would not make, it enters unseen states, errors accumulate—this is called **exposure bias**.
**RL** (on-policy + sparse): The student samples trajectories and receives rewards based on final outcomes. The distribution matches student inference, but rewards are typically **sequence-level** scalars that do not pinpoint which token went wrong.
**On-policy distillation** (on-policy + dense): The student samples trajectories, and the teacher scores **every token** on those trajectories. Training distribution matches student inference, and feedback is per-token.
### SFT as a Special Case of Distillation
A natural entry point to understanding distillation is the SFT loss. SFT's cross-entropy loss is equivalent to KL divergence against a one-hot "teacher" distribution $\delta_{y^*}$ at the labeled token:
$$-\log P_S(y^*_t) = \text{KL}(\delta_{y^*} \,\|\, P_S)$$
Knowledge distillation simply replaces this deterministic one-hot "teacher" with the teacher model's soft distribution $P_T$, optimizing $\text{KL}(P_T \,\|\, P_S)$ at each token to provide richer supervision than one-hot.
| Method | Sampling Mode | Feedback Density | Teacher Distribution |
|------|----------|----------|----------|
| SFT | off-policy (fixed data) | dense (per-token) | one-hot $\delta_{y^*}$ |
| Offline (off-policy) distillation | off-policy (fixed or teacher-generated data) | dense (per-token) | teacher soft distribution |
| RL | on-policy (student sampling) | sparse (per-sequence) | none |
| **On-policy distillation** | **on-policy (student sampling)** | **dense (per-token)** | **teacher soft distribution** |
---
## 2. Two Core Choices in Distillation
Differences between distillation methods almost always boil down to two questions. Once you understand these dimensions, the methods below are just different combinations.
### 2.1 How to Compute the Teacher Signal
At each token position, quantify the difference between teacher distribution $P_T$ and student distribution $P_S$ (we call this **Teacher KL**). There are two sub-choices.
**(a) Divergence direction**
| Divergence | Definition | Optimization behavior (information-theoretic meaning) |
|------|------|------|
| Forward KL | $\text{KL}(P_T \,\|\, P_S)$ | Mode-covering: the student must assign enough probability to regions where the teacher has high probability |
| Reverse KL | $\text{KL}(P_S \,\|\, P_T)$ | Mode-seeking: the student mainly fits the teacher's mode (high-probability) regions |
| Generalized JSD($\beta$) | $\beta\,\text{KL}(P_T\|M) + (1-\beta)\,\text{KL}(P_S\|M)$, where $M=\beta P_T+(1-\beta)P_S$ | Interpolates between the two |
> $\beta=0$ reduces to Forward KL, $\beta=1$ reduces to Reverse KL. SFT is equivalent to Forward KL (teacher is one-hot).
In swift:
- GKD defaults to $\beta=0.5$ (JSD); use `--beta` to choose among Forward / JSD / Reverse;
- OPD-RL uses the Reverse KL k1 estimator $\log\pi_{\text{teacher}}(y_t)-\log\pi_{\text{student}}(y_t)$ as per-token advantage.
**(b) Computation granularity**
| Granularity | Teacher information needed | Notes |
|----------|---------------|------|
| Full vocabulary | Teacher's complete next-token distribution | Exact divergence; high memory cost |
| Top-K | K tokens with highest teacher probability | Approximation after renormalization over top-K; suitable for external APIs (limited by `max_logprobs`) |
| Sampled token | Single logp of the teacher on the student's sampled token | Single-sample Monte Carlo estimate of Reverse KL; lowest communication cost |
> **Accuracy vs cost**: Full vocabulary requires materializing complete logits; sampled token only needs teacher logp on the sampled token (can use remote API). The [DeepSeek-V4](https://arxiv.org/abs/2606.19348) technical report notes that using only sampled-token log-ratio as advantage yields high gradient variance, so its full-vocabulary OPD uses complete logit distillation.
### 2.2 How to Pass the Signal to the Student
| | **Path A: GKD (direct loss)** | **Path B: OPD-RL (RL advantage)** |
|---|---|---|
| Training paradigm | `--rlhf_type gkd` | `--rlhf_type grpo` + teacher |
| Signal delivery | Use signal as loss | Use signal as advantage via policy gradient |
| Gradient flows through | Student **full-vocabulary** logits (or top-k) | Only student **sampled token** $\nabla\log\pi(y_t)$ |
| Teacher information needed | Full distribution (or top-k logits) | Single logp on sampled token |
| Divergence choice | Forward / Reverse / JSD (`--beta`) | Reverse KL (k1 log-ratio) |
| Combine with task reward | Mix SFT loss via `sft_alpha` | Can stack with GRPO reward as advantage |
Both paths **share the same teacher infrastructure** (see below); they differ only in how teacher KL is used.
> **Common uses of distillation**
> 1. **Capability fusion**: Distill multiple expert models into one unified model.
> 2. **Strong-to-weak**: Transfer capabilities from a large model to a smaller one.
> 3. **Forgetting prevention**: Use an old checkpoint as teacher to recover prior capabilities after multi-stage training.
---
## 3. Distillation Methods in swift
swift provides three distillation training methods. They share the same teacher infrastructure:
| Method | Signal path | How to enable | One-liner |
|------|--------------|----------|--------|
| **GKD** | Direct loss (Path A) | `--rlhf_type gkd` | Teacher divergence as loss backprop; supports full-vocab / top-k divergence |
| **OPD-RL** | RL advantage (Path B) | `--rlhf_type grpo` + teacher | Teacher log-ratio injected into GRPO advantage; can combine with task rewards |
| **OPSD** | Path A or B | Provide `teacher_prompt` on top of the above | Single-model self-distillation: teacher input includes privileged info (e.g. reference solution) |
**Three teacher sources** (shared by GKD and OPD-RL):
- `--teacher_model`: Load a separate frozen teacher model in the training process.
- `--teacher_model_server`: Connect to an external teacher service (vLLM service started via `swift deploy`) without loading the teacher on training GPUs. When using the API with GKD, also set `--gkd_logits_topk`. Supports single URL and multi-teacher JSON config.
- **Self-distillation**: Teacher and student share the same source. For LoRA training when `--teacher_model` equals `--model`, the base model is used as a fixed teacher via `disable_adapter()` without extra loading; without `--teacher_model` or `teacher_model_server`, the student's current weights serve as a dynamic teacher (all batches in GKD; in GRPO only when data includes `teacher_prompt`—see [3.3](#33-opsd-on-policy-self-distillation)).
| Parameter | Default | Description |
|------|--------|------|
| `--teacher_model` | None | Teacher model path; omit for dynamic self-distillation in GKD |
| `--teacher_model_server` | None | Teacher API URL; see format below |
| `--teacher_tag_key` | `"dataset"` | Column name for matching sample tags to teacher `tags` in multi-teacher routing |
| `--teacher_deepspeed` | None | DeepSpeed config for teacher model (e.g. `zero3`) |
| `--offload_teacher_model` | False | Offload teacher to CPU when not in forward pass (only with `teacher_model`) |
Full parameter details: [command-line parameters](./Command-line-parameters.md#rewardteacher-model-parameters).
#### External Teacher API
Deploy the teacher via `swift deploy --model xxx --infer_backend vllm`; the training process requests logprobs by prompt without loading teacher weights on training GPUs.
- **GKD**: Requires `--gkd_logits_topk` (API returns only top-k logprobs) for JSD divergence loss.
- **OPD-RL**: Uses sampled-token logp (`prompt_logprobs=0`) injected into advantage; coefficient is global `--teacher_kl_coef` (see [3.2](#32-opd-rl-kl-as-rl-advantage)).
#### Multi-Teacher Routing
Multi-Teacher connects multiple external teacher APIs and routes each sample to one teacher by tag (domain-expert distillation).
**`teacher_model_server` format**
```bash
# Single teacher
--teacher_model_server http://localhost:8000
# Multi-teacher (`tags` match `--dataset` or a data column; see Mode 1 below)
--teacher_model_server '[{"url":"http://t1:8000","tags":["data/math.jsonl"]},{"url":"http://t2:8001","tags":["data/code.jsonl"]}]'
```
Each teacher entry:
| Field | Description |
|------|------|
| `url` | Teacher API URL |
| `tags` | Data sources this teacher handles; optional with a single teacher; with multiple teachers, entries must not overlap and must match the routing identifiers below |
**Routing modes**
**Mode 1: Route by dataset (default)**
When you pass multiple `--dataset` values, samples are matched to teachers by source dataset. Default `--teacher_tag_key` is `"dataset"`. Set each teacher's `tags` to match the corresponding `--dataset` entry:
```bash
# Hub IDs
--dataset AI-ModelScope/alpaca-gpt4-data-en AI-ModelScope/alpaca-cleaned \
--teacher_model_server '[{"url":"http://t1:8000","tags":["AI-ModelScope/alpaca-gpt4-data-en"]},{"url":"http://t2:8001","tags":["AI-ModelScope/alpaca-cleaned"]}]'
# Local paths
--dataset data/math.jsonl data/code.jsonl \
--teacher_model_server '[{"url":"http://t1:8000","tags":["data/math.jsonl"]},{"url":"http://t2:8001","tags":["data/code.jsonl"]}]'
```
**Mode 2: Route by sample**
Add a column in your data (e.g. `teacher_tag`) and set `--teacher_tag_key teacher_tag`. Match `tags` to values in that column. Use this when you pass a single `--dataset` but still need multiple teachers.
**Example: GKD + multi-teacher**
```bash
# Deploy two teacher services (GKD requires max_logprobs >= gkd_logits_topk)
CUDA_VISIBLE_DEVICES=1 swift deploy --model Qwen/Qwen3.5-4B --port 8000 --max_logprobs 64
CUDA_VISIBLE_DEVICES=2 swift deploy --model Qwen/Qwen3.5-1.7B --port 8001 --max_logprobs 64
CUDA_VISIBLE_DEVICES=0 swift rlhf \
--rlhf_type gkd \
--model Qwen/Qwen3.5-0.6B \
--teacher_model_server '[{"url":"http://localhost:8000","tags":["data/math.jsonl"]},{"url":"http://localhost:8001","tags":["data/code.jsonl"]}]' \
--gkd_logits_topk 64 \
--dataset data/math.jsonl data/code.jsonl \
...
```
**Example: OPD-RL + multi-teacher**
```bash
CUDA_VISIBLE_DEVICES=1 swift deploy --model Qwen/Qwen3.5-4B --port 8000 --max_logprobs 1
CUDA_VISIBLE_DEVICES=2 swift deploy --model Qwen/Qwen3.5-1.7B --port 8001 --max_logprobs 1
CUDA_VISIBLE_DEVICES=0 swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen3.5-0.6B \
--teacher_model_server '[{"url":"http://localhost:8000","tags":["data/math.jsonl"]},{"url":"http://localhost:8001","tags":["data/code.jsonl"]}]' \
--teacher_kl_coef 1.0 \
--dataset data/math.jsonl data/code.jsonl \
--use_vllm true --vllm_mode colocate \
...
```
---
### 3.1 GKD: Divergence as Direct Loss
GKD ([Generalized Knowledge Distillation](https://arxiv.org/pdf/2306.13649)) directly backpropagates the divergence between teacher and student as the loss function.
**Loss function**
$$
\mathcal{L}_{\text{GKD}}(x, y) = \sum_{t=1}^{|y|} D_{\text{JSD}(\beta)}\big(P_{\text{teacher}}(\cdot|x,y_{<t}),\, P_{\text{student}}(\cdot|x,y_{<t})\big)
$$
Divergence $D$ is chosen via `--beta` (see 2.1): $\beta=0$ is Forward KL, $\beta=1$ is Reverse KL, $0<\beta<1$ is generalized JSD (default $0.5$).
**On-Policy vs Off-Policy: `lmbda`**
GKD uses `lmbda` to control the probability that each batch uses student online sampling:
```python
if random() <= lmbda:
y = student.generate(x) # on-policy: student samples
else:
y = y_ground_truth # off-policy: use dataset labels
loss = D(P_teacher(·|x, y), P_student(·|x, y))
```
- `lmbda=0`: pure offline (traditional SFT distillation).
- `lmbda=1`: pure online (student learns from its own mistakes, i.e. on-policy distillation).
- `0<lmbda<1`: mixed.
> To use teacher-generated data, first offline-generate responses with the teacher and write them to the dataset, then train with `lmbda=0`.
**GKD parameters**
| Parameter | Type | Default | Description |
|------|------|--------|------|
| `--beta` | float | 0.5 | Divergence interpolation: 0=Forward KL, 0.5=JSD, 1=Reverse KL |
| `--lmbda` | float | 0.5 | Online sampling probability: 0=offline, 1=pure online |
| `--sft_alpha` | float | 0 | SFT loss mixing ratio; final `loss = gkd_loss + sft_alpha * sft_loss` (only for **non-student-generated** data) |
| `--gkd_logits_topk` | int | None | Compute KL using teacher top-K logits only; required when using `teacher_model_server` |
### Top-K KL Computation
By default KL is computed over the full vocabulary, which can OOM on large vocabularies. Use `--gkd_logits_topk`.
**External teacher API**
When setting `--teacher_model_server`, also set `--gkd_logits_topk` (API returns only top-k logprobs). Example:
```bash
# Step 1: Deploy teacher model (max_logprobs must be >= gkd_logits_topk)
CUDA_VISIBLE_DEVICES=0 swift deploy \
--model Qwen/Qwen3.5-9B \
--infer_backend vllm \
--port 8000 \
--max_logprobs 64
# Step 2: Start GKD training
CUDA_VISIBLE_DEVICES=1,2,3,4 \
NPROC_PER_NODE=4 \
swift rlhf \
--rlhf_type gkd \
--model Qwen/Qwen3.5-2B \
--teacher_model_server http://localhost:8000 \
--gkd_logits_topk 64 \
--lmbda 1.0 \
--beta 1.0 \
--dataset xxx
```
**Online sampling acceleration**
When `lmbda > 0`, the student must generate sequences online. Use vLLM to accelerate sampling (colocate / server modes, same as GRPO). See [GRPO documentation](./GRPO/GetStarted/GRPO.md#cluster-support).
**Multi-turn GKD**
GKD supports multi-turn training, sharing the same `MultiTurnScheduler` infrastructure as GRPO.
For the full scheduler interface and customisation, see the [GRPO multi-turn documentation](./GRPO/DeveloperGuide/multi_turn.md).
**Reference scripts**
- Basic training: [examples/train/rlhf/gkd/](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/gkd/)
- Multi-turn training: [examples/train/rlhf/gkd/multi_turn.sh](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/gkd/multi_turn.sh)
- Multimodal: [examples/train/multimodal/rlhf/gkd/](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/rlhf/gkd/)
- Megatron: [examples/megatron/rlhf/gkd/](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/gkd/)
---
### 3.2 OPD-RL: KL as RL Advantage
OPD (On-Policy Distillation) RL injects teacher KL into GRPO's **per-token advantage** and updates the student via policy gradient.
**Principle**
Standard GRPO advantage comes from group-normalized task rewards (per-sequence scalar). OPD-RL injects teacher signal token-by-token **after** advantage normalization:
$$
A_t = A_t^{\text{base}} + \alpha \cdot \big(\log \pi_{\text{teacher}}(y_t|x,y_{<t}) - \log \pi_{\text{student}}(y_t|x,y_{<t})\big)
$$
- $A_t^{\text{base}}$: GRPO-normalized task reward advantage (0 when no reward function).
- $\alpha$: `--teacher_kl_coef`, teacher signal strength.
- $\log\pi_{\text{teacher}}(y_t) - \log\pi_{\text{student}}(y_t)$: teacher log-ratio, the k1 estimator corresponding to the coefficient of $\nabla_\theta\log\pi_\theta(y_t)$ in the Reverse KL gradient (see `compute_teacher_logratio`).
**Pure distillation mode**: Without `--reward_funcs`, base advantage is 0 and teacher signal is the only driver: $A_t = \alpha\cdot(\log\pi_{\text{teacher}}(y_t)-\log\pi_{\text{student}}(y_t))$.
**Monitoring metric**: `teacher_kl` in logs is the k3 estimator $e^{d}-d-1$ ($d=\log\pi_{\text{teacher}}-\log\pi_{\text{student}}$), measuring distance between student and teacher.
**How to enable**: Under `--rlhf_type grpo`, set `--teacher_model` or `--teacher_model_server` to automatically enable OPD-RL—no extra switch needed. See the shared teacher parameter table above.
**OPD-RL-specific parameters**
| Parameter | Default | Description |
|------|--------|------|
| `--teacher_kl_coef` | 1.0 | Coefficient $\alpha$ for injecting teacher log-ratio into advantage |
**Reference scripts**
- HF: [examples/train/grpo/opd_rl.sh](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/opd_rl.sh)
- Megatron: [examples/megatron/grpo/opd_rl.sh](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/grpo/opd_rl.sh)
- Ray: [examples/ray/grpo/run_opd.sh](https://github.com/modelscope/ms-swift/tree/main/examples/ray/grpo/run_opd.sh)
---
### 3.3 OPSD: On-Policy Self-Distillation
OPSD ([On-Policy Self-Distillation](https://arxiv.org/abs/2601.18734)) is a single-model self-distillation method: the same model constructs student and teacher inputs separately; the teacher side additionally receives **privileged information** (e.g. reference solution), then aligns output distributions on the student's sampled response.
**Core mechanism**
- **Student**: sees only the problem and reasons normally.
- **Teacher**: sees problem + reference solution (privileged info via `teacher_prompt` column).
- **Training objective**: align student and teacher output distributions on the same student-sampled response via divergence (JSD / KL).
OPSD can follow either GKD or OPD-RL path:
- **GKD + OPSD**: `--rlhf_type gkd`, teacher KL as direct loss.
- **OPD-RL + OPSD**: `--rlhf_type grpo`; dynamic mode omits `--teacher_model`, fixed mode sets `--teacher_model` equal to `--model`.
**Two self-distillation weight modes**
| Mode | Configuration | Teacher weights | Description |
|------|---------|---------|------|
| **Dynamic** | Omit `--teacher_model` | Student's current weights | Teacher updates with training |
| **Fixed** | Set `--teacher_model` same as `--model` | Initial teacher weights | Teacher weights fixed |
**Data format**
OPSD datasets need a `teacher_prompt` column. Load a data preprocessing plugin via `--external_plugins` to build it. Example with math reasoning dataset `open-r1/OpenThoughts-114k-math`:
```python
from swift.dataset import DatasetMeta, RowPreprocessor, register_dataset
class OpenThoughtsOPSDPreprocessor(RowPreprocessor):
def preprocess(self, row):
if not row.get('correct', True):
return None
problem = row.get('problem', '')
solution = row.get('solution', '')
teacher_prompt = f'{problem}\n\nReference solution:\n{solution}\n\nNow articulate your own reasoning.'
messages = [
{'role': 'system', 'content': 'Please reason step by step, and put your final answer within \\boxed{}.'},
{'role': 'user', 'content': problem},
]
return {'messages': messages, 'teacher_prompt': teacher_prompt}
register_dataset(DatasetMeta(
ms_dataset_id='open-r1/OpenThoughts-114k-math',
preprocess_func=OpenThoughtsOPSDPreprocessor(),
tags=['math', 'opsd'],
))
```
**Reference scripts**
- HF: [examples/train/rlhf/opsd/](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/opsd/)
- Megatron: [examples/megatron/rlhf/gkd/opsd.sh](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/gkd/opsd.sh)
---
## Reference
- Kevin Lu & Thinking Machines Lab. [On-Policy Distillation](https://thinkingmachines.ai/blog/on-policy-distillation/). 2025.
- Agarwal et al. [On-Policy Distillation of Language Models (GKD)](https://arxiv.org/pdf/2306.13649). 2023.
- Gu et al. [MiniLLM: Knowledge Distillation of Large Language Models](https://arxiv.org/abs/2306.08543). 2023.
- DeepSeek-AI. [DeepSeek-V4](https://arxiv.org/html/2606.19348v1). 2026.
- Qwen Team. [Qwen3 Technical Report](https://arxiv.org/abs/2505.09388). 2025.
- Zhipu AI. [GLM-5: from Vibe Coding to Agentic Engineering](https://arxiv.org/html/2602.15763). 2026.
- Kimi Team. [Kimi K2.5: Visual Agentic Intelligence](https://arxiv.org/abs/2602.02276). 2026.
docs/source_en/Instruction/Evaluation.md
+285
View File
@@ -0,0 +1,285 @@
# Evaluation
SWIFT supports eval (evaluation) capabilities to provide standardized evaluation metrics for both raw models and trained models.
## Capability Introduction
SWIFT's eval capability utilizes the EvalScope evaluation framework from the Magic Tower community, which has been advanced in its encapsulation to support the evaluation needs of various models.
> Note: EvalScope supports many other complex capabilities, such as [model performance evaluation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html), so please use the EvalScope framework directly.
Currently, we support the evaluation process of **standard evaluation datasets** as well as the evaluation process of **user-defined** evaluation datasets. The **standard evaluation datasets** are supported by three evaluation backends:
Below are the names of the supported datasets. For detailed information on the datasets, please refer to [all supported datasets](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset/index.html).
1. Native (default):
Primarily supports pure text evaluation, while **supporting** visualization of evaluation results.
```text
'arc', 'bbh', 'ceval', 'cmmlu', 'competition_math',
'general_qa', 'gpqa', 'gsm8k', 'hellaswag', 'humaneval',
'ifeval', 'iquiz', 'mmlu', 'mmlu_pro',
'race', 'trivia_qa', 'truthful_qa'
```
2. OpenCompass:
Primarily supports pure text evaluation, currently **does not support** visualization of evaluation results.
```text
'obqa', 'cmb', 'AX_b', 'siqa', 'nq', 'mbpp', 'winogrande', 'mmlu', 'BoolQ', 'cluewsc', 'ocnli', 'lambada',
'CMRC', 'ceval', 'csl', 'cmnli', 'bbh', 'ReCoRD', 'math', 'humaneval', 'eprstmt', 'WSC', 'storycloze',
'MultiRC', 'RTE', 'chid', 'gsm8k', 'AX_g', 'bustm', 'afqmc', 'piqa', 'lcsts', 'strategyqa', 'Xsum', 'agieval',
'ocnli_fc', 'C3', 'tnews', 'race', 'triviaqa', 'CB', 'WiC', 'hellaswag', 'summedits', 'GaokaoBench',
'ARC_e', 'COPA', 'ARC_c', 'DRCD'
```
3. VLMEvalKit:
Primarily supports multimodal evaluation and currently **does not support** visualization of evaluation results.
```text
'COCO_VAL', 'MME', 'HallusionBench', 'POPE', 'MMBench_DEV_EN', 'MMBench_TEST_EN', 'MMBench_DEV_CN', 'MMBench_TEST_CN',
'MMBench', 'MMBench_CN', 'MMBench_DEV_EN_V11', 'MMBench_TEST_EN_V11', 'MMBench_DEV_CN_V11',
'MMBench_TEST_CN_V11', 'MMBench_V11', 'MMBench_CN_V11', 'SEEDBench_IMG', 'SEEDBench2',
'SEEDBench2_Plus', 'ScienceQA_VAL', 'ScienceQA_TEST', 'MMT-Bench_ALL_MI', 'MMT-Bench_ALL',
'MMT-Bench_VAL_MI', 'MMT-Bench_VAL', 'AesBench_VAL', 'AesBench_TEST', 'CCBench', 'AI2D_TEST', 'MMStar',
'RealWorldQA', 'MLLMGuard_DS', 'BLINK', 'OCRVQA_TEST', 'OCRVQA_TESTCORE', 'TextVQA_VAL', 'DocVQA_VAL',
'DocVQA_TEST', 'InfoVQA_VAL', 'InfoVQA_TEST', 'ChartQA_TEST', 'MathVision', 'MathVision_MINI',
'MMMU_DEV_VAL', 'MMMU_TEST', 'OCRBench', 'MathVista_MINI', 'LLaVABench', 'MMVet', 'MTVQA_TEST',
'MMLongBench_DOC', 'VCR_EN_EASY_500', 'VCR_EN_EASY_100', 'VCR_EN_EASY_ALL', 'VCR_EN_HARD_500',
'VCR_EN_HARD_100', 'VCR_EN_HARD_ALL', 'VCR_ZH_EASY_500', 'VCR_ZH_EASY_100', 'VCR_ZH_EASY_ALL',
'VCR_ZH_HARD_500', 'VCR_ZH_HARD_100', 'VCR_ZH_HARD_ALL', 'MMDU', 'MMBench-Video', 'Video-MME'
```
## Environment Preparation
```shell
pip install ms-swift[eval] -U
```
Or install from source:
```shell
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e '.[eval]'
```
## Evaluation
Supports four methods of evaluation: pure text evaluation, multimodal evaluation, URL evaluation, and custom dataset evaluation.
**Basic Example**
```shell
CUDA_VISIBLE_DEVICES=0 \
swift eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--eval_backend Native \
--infer_backend transformers \
--eval_limit 10 \
--eval_dataset gsm8k
```
Where:
- model: Can specify a local model path or a model ID on modelscope
- eval_backend: Options are Native, OpenCompass, VLMEvalKit; default is Native
- infer_backend: Options are transformers, vllm, sglang, lmdeploy; default is transformers
- eval_limit: Sample size for each evaluation set; default is None, which means using all data; can be used for quick validation
- eval_dataset: Evaluation dataset(s); multiple datasets can be set, separated by spaces
**Complex Evaluation Example**
```shell
CUDA_VISIBLE_DEVICES=0 \
swift eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--eval_backend Native \
--infer_backend transformers \
--eval_limit 10 \
--eval_dataset gsm8k \
--eval_dataset_args '{"gsm8k": {"few_shot_num": 0, "filters": {"remove_until": "</think>"}}}' \
--eval_generation_config '{"max_tokens": 512, "temperature": 0}' \
--extra_eval_args '{"ignore_errors": true, "debug": true}'
```
For a specific list of evaluation parameters, please refer to [here](./Command-line-parameters.md#evaluation-arguments).
## Evaluation During Training
SWIFT supports using EvalScope to evaluate the current model during the training process, allowing for timely understanding of the model's training effectiveness.
**Basic Example**
```shell
CUDA_VISIBLE_DEVICES=0 \
swift sft \
--model "Qwen/Qwen2.5-0.5B-Instruct" \
--tuner_type "lora" \
--dataset "AI-ModelScope/alpaca-gpt4-data-zh#100" \
--torch_dtype "bfloat16" \
--num_train_epochs "1" \
--per_device_train_batch_size "1" \
--learning_rate "1e-4" \
--lora_rank "8" \
--lora_alpha "32" \
--target_modules "all-linear" \
--gradient_accumulation_steps "16" \
--save_steps "50" \
--save_total_limit "5" \
--logging_steps "5" \
--max_length "2048" \
--eval_strategy "steps" \
--eval_steps "5" \
--per_device_eval_batch_size "5" \
--eval_use_evalscope \
--eval_dataset "gsm8k" \
--eval_dataset_args '{"gsm8k": {"few_shot_num": 0}}' \
--eval_limit "10"
```
Note that the launch command is `sft`, and the evaluation-related parameters include:
- eval_strategy: Evaluation strategy. Defaults to None, following the `save_strategy` policy
- eval_steps: Defaults to None. If an evaluation dataset exists, it follows the `save_steps` policy
- eval_use_evalscope: Whether to use evalscope for evaluation, this parameter needs to be set to enable evaluation
- eval_dataset: Evaluation datasets, multiple datasets can be set, separated by spaces
- eval_dataset_args: Evaluation dataset parameters in JSON format, parameters for multiple datasets can be set
- eval_limit: Number of samples from the evaluation dataset
- eval_generation_config: Model inference configuration during evaluation, in JSON format, default is `{'max_tokens': 512}`
More evaluation examples can be found in [examples](https://github.com/modelscope/ms-swift/tree/main/examples/eval).
## Custom Evaluation Datasets
This framework supports two predefined dataset formats: multiple-choice questions (MCQ) and question-and-answer (QA). The usage process is as follows:
*Note: When using a custom evaluation, the `eval_backend` parameter must be set to `Native`.*
### Multiple-Choice Question Format (MCQ)
This format is suitable for scenarios involving multiple-choice questions, and the evaluation metric is accuracy.
**Data Preparation**
Prepare a CSV file in the multiple-choice question format, structured as follows:
```text
mcq/
├── example_dev.csv # (Optional) The filename should follow the format `{subset_name}_dev.csv` for few-shot evaluation
└── example_val.csv # The filename should follow the format `{subset_name}_val.csv` for the actual evaluation data
```
The CSV file should follow this format:
```text
id,question,A,B,C,D,answer
1,Generally speaking, the amino acids that make up animal proteins are____,4 types,22 types,20 types,19 types,C
2,Among the substances present in the blood, which is not a metabolic end product?____,Urea,Uric acid,Pyruvate,Carbon dioxide,C
```
Where:
- `id` is an optional index
- `question` is the question
- `A`, `B`, `C`, `D`, etc. are the options, with a maximum of 10 options
- `answer` is the correct option
**Launching Evaluation**
Run the following command:
```bash
CUDA_VISIBLE_DEVICES=0 \
swift eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--eval_backend Native \
--infer_backend transformers \
--eval_dataset general_mcq \
--eval_dataset_args '{"general_mcq": {"local_path": "/path/to/mcq", "subset_list": ["example"]}}'
```
Where:
- `eval_dataset` should be set to `general_mcq`
- `eval_dataset_args` should be set with:
- `local_path` as the path to the custom dataset folder
- `subset_list` as the name of the evaluation dataset, taken from the `*_dev.csv` mentioned above
**Running Results**
```text
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=====================+=============+=================+==========+=======+=========+=========+
| Qwen2-0.5B-Instruct | general_mcq | AverageAccuracy | example | 12 | 0.5833 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
```
## Question-and-Answer Format (QA)
This format is suitable for scenarios involving question-and-answer, and the evaluation metrics are `ROUGE` and `BLEU`.
**Data Preparation**
Prepare a JSON Lines file in the question-and-answer format, containing one file in the following structure:
```text
qa/
└── example.jsonl
```
The JSON Lines file should follow this format:
```json
{"query": "What is the capital of China?", "response": "The capital of China is Beijing"}
{"query": "What is the highest mountain in the world?", "response": "It is Mount Everest"}
{"query": "Why can't penguins be seen in the Arctic?", "response": "Because most penguins live in Antarctica"}
```
**Launching Evaluation**
Run the following command:
```bash
CUDA_VISIBLE_DEVICES=0 \
swift eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--eval_backend Native \
--infer_backend transformers \
--eval_dataset general_qa \
--eval_dataset_args '{"general_qa": {"local_path": "/path/to/qa", "subset_list": ["example"]}}'
```
Where:
- `eval_dataset` should be set to `general_qa`
- `eval_dataset_args` is a JSON string that needs to be set with:
- `local_path` as the path to the custom dataset folder
- `subset_list` as the name of the evaluation dataset, taken from the `*.jsonl` mentioned above
**Running Results**
```text
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=====================+=============+=================+==========+=======+=========+=========+
| Qwen2-0.5B-Instruct | general_qa | bleu-1 | default | 12 | 0.2324 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | bleu-2 | default | 12 | 0.1451 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | bleu-3 | default | 12 | 0.0625 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | bleu-4 | default | 12 | 0.0556 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-1-f | default | 12 | 0.3441 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-1-p | default | 12 | 0.2393 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-1-r | default | 12 | 0.8889 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-2-f | default | 12 | 0.2062 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-2-p | default | 12 | 0.1453 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-2-r | default | 12 | 0.6167 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-l-f | default | 12 | 0.333 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-l-p | default | 12 | 0.2324 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Qwen2-0.5B-Instruct | general_qa | rouge-l-r | default | 12 | 0.8889 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
```
docs/source_en/Instruction/Export-and-push.md
+61
View File
@@ -0,0 +1,61 @@
# Export and Push
## Merge LoRA
- See [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/merge_lora.sh).
## Quantization
SWIFT supports quantization exports for AWQ, GPTQ, FP8, and BNB models. AWQ and GPTQ require a calibration dataset, which yields better quantization performance but takes longer to quantize. On the other hand, FP8 and BNB does not require a calibration dataset and is quicker to quantize.
| Quantization Technique | Multimodal | Inference Acceleration | Continued Training |
| ---------------------- | ---------- | ---------------------- | ------------------ |
| FP8 | ✅ | ✅ | ✅ |
| GPTQ | ✅ | ✅ | ✅ |
| AWQ | ✅ | ✅ | ✅ |
| BNB | ❌ | ✅ | ✅ |
In addition to the SWIFT installation, the following additional dependencies need to be installed:
```shell
# For AWQ quantization:
# The versions of autoawq and CUDA are correlated; please choose the version according to `https://github.com/casper-hansen/AutoAWQ`.
# If there are dependency conflicts with torch, please add the `--no-deps` option.
pip install autoawq -U
# For GPTQ quantization:
# The versions of auto_gptq and CUDA are correlated; please choose the version according to `https://github.com/PanQiWei/AutoGPTQ#quick-installation`.
pip install auto_gptq optimum -U
# For GPTQ v2 quantization:
pip install gptqmodel optimum -U
# For BNB quantization:
pip install bitsandbytes -U
```
We provide a series of scripts to demonstrate SWIFT's quantization export capabilities:
- Supports [AWQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/awq.sh)/[GPTQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq.sh)/[GPTQ v2](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq_v2.sh)/[BNB](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/bnb.sh) quantization exports.
- Multimodal quantization: Supports quantizing multimodal models using GPTQ and AWQ, with limited multimodal models supported by AWQ. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/mllm).
- Support for more model series: Supports quantization exports for [BERT](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/bert) and [Reward Model](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/reward_model).
- Models exported with SWIFT's quantization support inference acceleration using vllm/sglang/lmdeploy; they also support further SFT/RLHF using QLoRA.
## Push Model
SWIFT supports re-pushing trained/quantized models to ModelScope/Hugging Face. By default, it pushes to ModelScope, but you can specify `--use_hf true` to push to Hugging Face.
```shell
swift export \
--model output/vx-xxx/checkpoint-xxx \
--push_to_hub true \
--hub_model_id '<model-id>' \
--hub_token '<sdk-token>' \
--use_hf false
```
Tips:
- You can use `--model <checkpoint-dir>` or `--adapters <checkpoint-dir>` to specify the checkpoint directory to be pushed. There is no difference between these two methods in the model pushing scenario.
- When pushing to ModelScope, you need to make sure you have registered for a ModelScope account. Your SDK token can be obtained from [this page](https://www.modelscope.cn/my/myaccesstoken). Ensure that the account associated with the SDK token has edit permissions for the organization corresponding to the model_id. The model pushing process will automatically create a model repository corresponding to the model_id (if it does not already exist), and you can use `--hub_private_repo true` to automatically create a private model repository.
docs/source_en/Instruction/Frequently-asked-questions.md
+469
View File
@@ -0,0 +1,469 @@
# Frequently-asked-questions
Here are some common questions encountered during the use of SWIFT.
## Training
SWIFT supports training methods including pre-training, instruction-supervised fine-tuning (SFT), preference learning, GRPO, Embedding, Reranker, sequence classification tasks, etc. For details, please see the [homepage](https://github.com/modelscope/ms-swift/blob/main/README.md).
### Q1: What models does Swift support? How do I set a local model path?
For supported models, please refer to the [Supported Models and Datasets documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). If the model has already been downloaded locally, simply set `--model <path_to_model>`. For training in an offline environment, set both `--model <local_path>` and `--check_model false`. If you encounter a git clone-related error, you need to clone the repository and then specify it using `local_repo_path`. For details, see [Command-line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). For models downloaded from ModelScope, you can configure the `MODELSCOPE_CACHE=your_path` environment variable to store the original models in a specified directory. If using the ModelScope SDK, use `cache_dir="local_address"`. You can also use the `modelscope download` command-line tool or `git` to download. For details, refer to the ModelScope documentation on [Model Download](https://modelscope.cn/docs/models/download). If you need to download models from Hugging Face, set the environment variable `USE_HF=1`.
Swift automatically matches the model_type, or you can manually specify it by referring to the [Supported Models and Datasets documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html).
### Q2: What datasets does Swift support? How do I use a custom dataset?
For supported datasets, see the [Supported Models and Datasets documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). For the format and usage of custom datasets, see the [Custom Dataset documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html). Datasets that conform to these formats will automatically use Swift's built-in data preprocessors. If your dataset does not match the format in the documentation, please convert it yourself or refer to how currently supported datasets are integrated. If your custom dataset contains extra fields, they will not be used by default. You can configure this using the [remove_unused_columns command-line parameter](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#data-arguments).
You need to download the dataset locally and then specify its path. Please see the [Custom Dataset documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html). git clone it locally, then specify the path using the `dataset_path` field in the dataset_info.json file.
For data shuffling, see the [dataset_shuffle command-line parameter](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
To force re-downloading a dataset, set the `--download_mode` command-line parameter. To perform error checking on the dataset, set the `strict` command-line parameter. If you need a dataset quality inspection tool, you can check out another library, [data-juicer](https://github.com/modelscope/data-juicer).
Due to the strict type checking in the underlying PyArrow of the datasets library, parts like objects in image grounding datasets and tools in agent datasets must be strings. Otherwise, PyArrow will report an error indicating inconsistent types between rows.
If you encounter the error `AttributeError:’TrainerState’ object has no attribute ’last_model_checkpoint’` during training, it's because the dataset is too small (the number of data points is less than one step). Increase the amount of data. A similar error can also occur when the split validation set is very small.
Below is an error caused by an empty assistant field:
```text
File "/mnt/workspace/swift/swift/1lm/dataset/preprocessor/core. py", line 69, in _check_messages raise
ValueError(f'assistant_message; {assistant_message}')
ValueError: assistant_message: {'role' :'assistant', 'content': ''}
```
```shell
CUDA_VISIBLE_DEVICES=0 NPROC_PER_NODE=1 MAX_PIXELS=1003520 swift sft --model Qwen/Qwen2.5-VL-7B-Instruct --tuner_type lora --dataset /mnt/workspace/data.json --deepspeed zero2 --max_length 16384
```
If the assistant field in the dataset is empty, remove this empty string for inference, as it can cause NaN values during training and will be checked for.
### Q3: Issues related to loading datasets from cache
Setting the command-line parameter `--load_from_cache_file true` can speed up dataset loading, especially for multimodal datasets or large datasets. When debugging or modifying a preprocessor, set it to false. For more information, search for this parameter in the [Command-line Parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
### Q4: How do I set up the Swift environment? Are there Docker images available?
For environment setup, see the [Swift Installation documentation](https://swift.readthedocs.io/en/latest/GetStarted/SWIFT-installation.html). Recommended versions for some common dependencies can be found on the [homepage](https://github.com/modelscope/ms-swift/blob/main/README.md). The documentation provides a Docker image. You can start a container using the `docker run` command, for example: `docker run --gpus all -p 8000:8000 -it -d --name ms modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.9.0-vllm0.13.0-modelscope1.33.0-swift3.12.5 /bin/bash`. After starting the container, pull the latest code and install Swift.
### Q5: Questions about multimodal model training data formats, parameter freezing, and optimizer settings
See [examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal) for multimodal model training. It supports training with text-only data, image-text data, or a mixture of both. For parameters related to images, videos, and audio, such as max pixels, fps, etc., please see [Model-specific Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#specific-model-arguments).
In Grounding tasks, the general data format supports one object corresponding to multiple bboxes. Refer to the [Custom Dataset documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html). videos can be a list of images specified via a file directory.
Swift resizes images based on max_pixels and saves both the pre-processed and post-processed images, then adjusts the bboxes accordingly. However, this adjustment is not done during inference, so you need to manually process the images beforehand.
To reduce GPU memory usage when training VLM models, configure `--freeze_vit true` and the `--max_pixels` parameter to limit the maximum pixels. For details on parameters like `--freeze_vit`, `--freeze_aligner`, and `--freeze_llm`, see the Tuner section in the [Command-line Parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). If the ViT is not being trained, it is normal to see a warning like `"none of the inputs have requires_grad=True"`. If it is being trained, this warning should not appear.
To perform full-parameter fine-tuning on the visual encoder while using LoRA to fine-tune the LLM, refer to this [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/lora_llm_full_vit).
### Q6: Issues related to templates
Since Jinja chat templates do not have labels, they are not supported for training.
For multimodal datasets, if you need to perform dynamic data augmentation after loading the data (e.g., randomly adding noise to the input), please modify the encode method in the template.
### Q7: How to debug Swift training?
See the [Pre-training and Fine-tuning documentation](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html).
### Q8: How to use a Python script for Swift training?
Refer to the [notebook examples]((https://github.com/modelscope/ms-swift/tree/main/examples/notebook)).
### Q9: How to use the UI for Swift training?
Use the `swift web-ui` command. Training via the UI is consistent with the command line; for UI parameters, please refer to the command-line parameter documentation. The usage of custom datasets is the same as described in Q2 above. Megatron-swift does not support UI training.
### Q10: Issues related to single-node, multi-GPU training
Swift's multi-GPU training relies on torchrun. `deepspeed` and `device_map` are incompatible; you can only choose one. For more details, see the [examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu) in the code repository.
### Q11: Issues related to multi-node, multi-GPU training
Please see the [multi-node, multi-GPU examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node). During multi-node, multi-GPU training, only the main node produces logs.
If multi-node training is slow (e.g., a significant speed drop when using DeepSpeed ZeRO3), please check this [issue](https://github.com/modelscope/ms-swift/issues/1825).
### Q12: Issues related to large-scale datasets
If the dataset is very large and tokenization takes a long time each run, use lazy_tokenize or streaming. See the [Command-line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html) for details.
### Q13: Issues related to resuming from a checkpoint
Keep the parameters from the previous training script and add `--resume_from_checkpoint output/xxx/vx-xxx/checkpoint-xxx`. For details, see [Command-line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). If the dataset has changed and you only want to load the model, also set `--resume_only_model`. For more complex scenarios, search for "resume" in the command-line parameters documentation.
### Q14: Issues related to streaming dataset loading
For streaming (`--streaming true`), data is loaded while training. You must set max_steps. For details, see the documentation for the streaming parameter in the [Command-line Parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#data-arguments).
Note: Streaming does not shuffle the data and does not automatically create a validation set. The validation set must be specified using the `--val_dataset` command-line parameter.
When resuming training with streaming, the data can only be indexed forward, not randomly. Skipping already trained data is very time-consuming, so using streaming for resuming is not recommended.
### Q15: Issues related to packing
Packing should be used with FlashAttention; otherwise, there will be discrepancies, and the attention_mask will have issues.
The linear-attention in the Qwen3.5 model does not support var_len, so enabling packing is not recommended.
When packing is enabled, multimodal data will undergo two map operations: one for the dataset and one for the template. If this is very slow, you can set `OMP_NUM_THREADS=14` to accelerate it, or disable packing to avoid the second mapping.
### Q16: Dataset Multiprocessing
When the dataset mapping process is slow, you can enable multiprocessing by setting the `--dataset_num_proc` parameter. It is normal for the mapping process of multimodal datasets to be slow.
### Q17: How many checkpoints are saved by default after training?
By default, all checkpoints are saved. For details, see the [command-line parameter save_total_limit](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
### Q18: Loss and Accuracy During Training
Custom loss functions can be added in the plugin. If you need loss curves for different datasets, set `--enable_channel_loss`.
If the accuracy (acc) from evaluation does not match the accuracy calculated by re-running inference on the corresponding saved checkpoint (ckpt), it is because the calculation methods for eval_acc during training and acc during inference are different. `acc_strategy`: The default is `'token'`. Available options include `'token'` and `'seq'`.
token_acc might not be available during training because for some models, the number of `logits` and `labels` do not match, so it is not calculated.
You can view the currently supported losses or add new ones [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
To check if special tokens like `<image>` are included in the loss calculation, you can inspect the printed labels in the command-line log.
When training an agent, tool_call should be included in the loss calculation, while tool_response should not.
### Q19: Issues Related to Freezing Model Parameters
During training, if freezing certain layers causes some parameters to not participate in gradient backpropagation, please configure the parameter `--ddp_find_unused_parameters true`.
Regarding freeze_parameters and freeze_vit/freeze_aligner/freeze_llm: The freezing logic is applied first, and then trainable parameters are activated. The three parameters `freeze_vit`, `freeze_aligner`, and `freeze_llm` will adjust the sets of frozen and trainable parameters. Because the ViT in some models includes an `aligner`, the `aligner` will be added to trainable_parameters separately.
The mechanism of the freeze_parameters_ratio parameter is to freeze layers from the bottom up, starting from the embeddings.
### Q20: Issues Related to Sequence Parallelism
Sequence parallelism supports PT (Pre-Training), SFT (Supervised Fine-Tuning), DPO, and GRPO. Refer to this example: [sequence_parallel](https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel).
For VLM models, only FlashAttention is currently supported. For text-only models, both FlashAttention and SDPA are supported.
Sequence parallelism can be used simultaneously with the Liger kernel.
If there is a conflict between sequence parallelism and a custom loss, it is because sequence parallelism has its own custom loss implementation. You can modify it yourself [here](https://github.com/modelscope/ms-swift/blob/main/swift/trainers/sequence_parallel/ulysses.py).
### Q21: Expanding the Vocabulary
To expand the vocabulary using the Swift framework, you need to set the command-line parameters `new_special_tokens` and `--modules_to_save embed_tokens lm_head`. For details, see this [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/new_special_tokens).
### Q22: Issues Related to Tuners
LlamaPro in Swift has been adapted for multimodal tasks.
LongLoRA can only be used with LLaMA-series models.
LoRA training is incompatible with the `--trainable_parameters` parameter. For other trainable parameters outside the LoRA modules, use modules_to_save.
### Q23: Embedding/Reranker Training
[Example](https://github.com/modelscope/ms-swift/blob/main/examples/train/embedding) for training embeddings.
[Example](https://github.com/modelscope/ms-swift/tree/main/examples/train/reranker) for training a reranker.
For the data format, see [Custom Datasets](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html).
### Q24: Training for Classification Tasks
Swift supports multi-label classification. The data format is described in the custom dataset documentation. Search for `problem_type` in the command-line parameter documentation. Other aspects are the same as for regression tasks.
Note: The label field should be at the same level as the message field.
### Q25: Training a 'Thinking' Model
See this [issue](https://github.com/modelscope/ms-swift/issues/4030).
### Q26: Does Swift support distillation?
Yes. Refer to this [example](https://github.com/modelscope/ms-swift/blob/main/examples/sampler/distill/distill.sh).
### Q27: For GKD training, do the student and teacher models need to have the same model_type? For example, can one be a dense model and the other a MoE model?
Yes, this is possible, as long as their vocabularies are the same. However, using a MoE (Mixture of Experts) model will be slower.
### Q28: Issues Related to GRPO Training
Swift now supports GRPO training for multimodal tasks.It is normal for the loss to approach 0 during GRPO training. See this [issue](https://github.com/huggingface/open-r1/issues/239#issuecomment-2646297851) for reference.
Set sleep_mode to make the VllmEngine release GPU memory after inference is complete. The engine will be reloaded on the next call instead of occupying memory continuously.
If you do not want to include the KL term during GRPO training, you can configure it using the beta command-line parameter.
To continue with GRPO training after LoRA fine-tuning, search for `--adapters` in the command-line parameter documentation.
Because calculating entropy incurs some extra overhead, the entropy curve is not logged by default. If you need it, set `--log_entropy true`.
The colocate mode does not support use_async_engine.
GRPO does not support channel_loss.
The Liger kernel and padding-free cannot be enabled simultaneously during the GRPO phase. Doing so would require modifying the Liger GRPO loss implementation within the Liger kernel library, which is not convenient.
If your training set contains different tasks, please refer to [Multi-task Training](https://swift.readthedocs.io/en/latest/Instruction/GRPO/DeveloperGuide/multi_task.html).
### Q29: Issues Related to Reward Functions/Models
reward_model and reward_funcs can be used together.
For custom reward functions, refer to [examples/train/grpo/plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin).
For math problems, you need to pass the solution from the dataset; otherwise, it is difficult to calculate accuracy.
If you need to pass a specific column from the dataset into a custom reward function for ORM, place that column outside of the messages field.
If you need to specify an LLM-judge model for scoring during GRPO training, please refer to the reward model documentation.
### Q30: Issues Related to Rollout
Rollout is likely incompatible with pipeline parallelism.
The vLLM inference engine has trust_remote_code set to true by default.
### Q31: I have a question: in the GRPO script, does save_steps refer to the "step" or the "global step"? Currently, my local training shows global step as 18, while wandb shows step as 628.
It refers to the `global_step` displayed by the local tqdm progress bar.
### Q32: If num_iterations=1 is used by default, the clip function becomes ineffective, right? DAPO's clip_higher also doesn't work. I've seen that veRL has a micro-batch setting to update the policy model in smaller batches within a single epoch to make the clip term effective. Looking at the source code, it seems ms-swift's mini-batch only performs gradient accumulation?
Yes, num_iterations needs to be greater than 1.
### Q33: Does GSPO training support the top_entropy_quantile parameter? After passing --importance_sampling_level sequence, can I still optimize the top x% of tokens based on the entropy distribution?
Yes, it's supported. The order of operations is: first, the loss is calculated normally (affected by importance_sampling_level), and then the loss is masked based on top_entropy_quantile.
### Q34: FAQ in the GRPO documentation
For more GRPO-related FAQs, please refer to the [GRPO documentation](https://swift.readthedocs.io/en/latest/Instruction/GRPO/GetStarted/GRPO.html#faq).
### Q35: Issues Related to PPO and Other Preference Training Methods
PPO training does not support gradient clipping.
Currently, PPO only supports scenarios where the reward model (RM) and the policy model belong to the same model series (i.e., same tokenizer/template).
Multi-turn DPO is not supported.
### Q36: Issues Related to MoE Model Training
When training a Mixture-of-Experts (MoE) model with LoRA, if the aux_loss barely changes, add all_router to target_modules.
During LoRA training, whether the router modules are trained depends on whether their gates are implemented as nn.Linear. If they are nn.Parameter, they are not trained. For details, see the command-line parameter [target_parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#tuner-arguments).
### Q37: Issues Related to Megatron-SWIFT Training
For checkpoint saving, refer to the command-line parameter [save_strategy](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Command-line-parameters.html).
During Megatron multi-node training, logs are printed on the last pipeline parallelism (PP) rank, not the master node, because only the last PP rank has the complete information.
Megatron-SWIFT now supports save_total_limit and SwanLab for monitoring training. See the [Megatron-SWIFT command-line parameters documentation](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Command-line-parameters.html) for details.
ViT uses the Transformers model architecture and currently does not support parallelism. If you encounter an Out-Of-Memory (OOM) error during training, reduce `decoder_first_pipeline_num_layers`.
Megatron-SWIFT supports new models. There is no tutorial available at the moment; please refer to the Pull Requests (PRs) for adding new models.
The degree of parallelism for sequence_parallel is equal to the tensor parallelism (TP) degree.
FP8 training supports block-wise implementation. Refer to the FP8 example in [examples/megatron/fp8](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/fp8).
### Q38: How do I configure resuming from a checkpoint in Megatron-SWIFT?
To resume training, use `--mcore_model` to load the checkpoint. Additionally, configure these parameters as needed: `--finetune`, `--no_load_optim`, `--no_load_rng`. To resume LoRA training from a checkpoint, configure `--mcore_adapter`; other settings are the same as for full-parameter training. For details, see the [Megatron-SWIFT command-line parameters documentation](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Command-line-parameters.html).
### Q39: Issues Related to MTP
To enable MTP training, set the command-line parameter `mtp_num_layers`.
If the base model does not include an MTP structure, you can initialize and train the MTP from scratch.
Multi-modal MTP is not yet supported.
### Q40: I have a question about Megatron GKD. If the teacher is Qwen3-235B and the student is Qwen3-30BA3B, for SFT on the 235B model, I used pp=8 and set decoder_first and decoder_last to 11. If I also set decoder_first/last during GKD, will it affect the student's parallelism?
Currently, the parallelism parameters are shared between the teacher and student models. Support for different parallelism settings for each model will be introduced in a version after v4.
### Q41: Issues Related to Quantized Model Training
For QLoRA fine-tuning, refer to this [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora).
Quantized models cannot be fully fine-tuned. The integer-type parameters in GPTQ models cannot be used for gradient calculation, so only attached structures like LoRA can be updated.
For merging the model after QLoRA training, refer to the [QLoRA example](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora).
Megatron-SWIFT does not support QLoRA training.
### Q42: Training Specific Models
SWIFT currently does not support training MiniCPM-O with audio modal input.
To fine-tune DeepSeek-VL-2, use a transformers version earlier than 4.42 and `peft==0.11.*`.
Fine-tuning Moonlight-16B-A3B-Instruct: Training is disabled in the model files. Refer to the solution for DeepSeek-VL-2, which can be found by searching the issues.
Fine-tuning the Ovis-2 model is special; it requires padding to max_length. Set the `--max_length` argument.
Qwen2.5-Omni currently only supports "thinker" mode for training, not "talker" mode.
SFT for Qwen2-Audio does not support packing.
### Q43: On devices that do not support Flash Attention, what is the default attention_implementation? The documentation says the default is none.
The default implementation used is sdpa.
### Q44: Is left padding the default for model training?
You can choose either left or right padding for training. The default is right padding, while `batch infer` always uses left padding.
### Q45: What are the parameters for MoE? I can't find them by searching for keywords in the parameter list. How do I configure parameters like the number of experts and expert routing?
Use the parameters directly from the config.json file.
### Q46: Can swift support setting a minimum learning rate? I feel like it decreases too much towards the end.
Yes, you can set it with `--lr_scheduler_type cosine_with_min_lr --lr_scheduler_kwargs '{"min_lr": 1e-6}'`.
### Q47: Does it currently support configuring GRPO and SFT using YAML files?
Yes, both are supported. The configuration is directly processed into command-line arguments in main.py.
### Q48: Is it true that use_liger_kernel and log_entropy cannot be used together right now?
They are not supported together.
### Q49: How can I handle this error? Installing apex didn't help.
```text
RuntimeError: ColumnParallelLinear was called with gradient_accumulation_fusion set to True but the custom CUDA extension fused_weight_gradient_mlp_cuda module is not found. To use gradient_accumulation_fusion you must install APEX with --cpp_ext and --cuda_ext. For example: pip install --global-option="--cpp_ext" --global-option="--cuda_ext ." Note that the extension requires CUDA>=11. Otherwise, you must turn off gradient accumulation fusion.
```
Set `--gradient_accumulation_fusion false`.
### Q50: If I finetune a VLM on several tasks together, and the video sampling rules for different tasks are inconsistent, does ms-swift support this? Where can I configure it?
Check `interleave_prob` in the [Command-line Parameters Documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
### Q51: I have a question. During multimodal packing pre-training, it seems the GPU memory usage increases slightly after each "pytorch allocator cache flushes since last step," leading to an OOM error after many steps.
Add the environment variable `PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'`.
### Q52: Can use_logits_to_keep be used with large multimodal models now?
It will cause an error if the expansion of multimodal tokens happens within the model's forward pass.
### Q53: Why does the GPU memory usage increase significantly several times during training, for example, at step 50 or 100?
Set the `PYTORCH_CUDA_ALLOC_CONF` environment variable. For details, please refer to the PyTorch documentation.
### Q54: Is there a practical guide for fine-tuning a Qwen base model into a chat model? Are there any special configurations needed?
Use `swift sft`. No other special configurations are needed. Refer to the [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/base_to_chat).
### Q55: After training, the model's responses contain a lot of repetitive content.
Please refer to [Pre-training and Fine-tuning](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html). If repetition occurs during training, try training for more epochs, cleaning the data, performing full-parameter fine-tuning, or using RLHF to mitigate the issue.
### Q56: Why does using --torch_dtype float16 (my GPU doesn't support bf16) result in the error: lib/python3.12/site-packages/torch/amp/grad_scaler.py", line 260, in _unscale_grads_ raise ValueError("Attempting to unscale FP16 gradients.") ValueError: Attempting to unscale FP16 gradients.
For full-parameter fine-tuning, you cannot train with fp16.
### Q57: I'm getting an error when merging LoRA parameters. My current PEFT version is 0.11.0. Is this because the PEFT version needs to be upgraded?
```text
File "/opt/conda/lib/python3.9/site-packages/peft/config.py", line 118, in from_peft_type
return config_cls(**kwargs)
TypeError: __init__() got an unexpected keyword argument 'corda_config'
```
This is caused by a mismatch between the PEFT versions used for training and merging.
### Q58: How to solve this problem? safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
You are running out of disk space, and the model was not saved completely.
### Q59: Why does this error appear here? I can't find where numpy.object is.
Try `numpy==1.26.3`.
### Q60: Training with unsloth, I get the error: assert(type(target_modules) in (list,tuple,)). The parameter I configured is --target_modules all-linear.
Don't use `all-linear`. Change it to a specific list of modules, for example, `--target_modules q_proj k_proj v_proj`.
### Q61: For qwen2.5-omni, does --freeze_vit false mean that both the visual encoder and the audio encoder are unfrozen? Is there a way to unfreeze only the audio encoder and not the visual encoder?
Use the `--target_regex` argument.
## Inference
Swift supports inference via Python scripts, command line, and UI interfaces. For details, see [Inference and Deployment](https://swift.readthedocs.io/en/latest/Instruction/Inference-and-deployment.html).
### Q1: How to set up a model for inference in Swift?
For models from full-parameter training, models merged after LoRA training, or models downloaded from the Model Hub, set the command-line argument `--model <model_id_or_path>`. For unmerged models after LoRA training, use `--adapters`, and you can also specify the base model path with `--model`.
### Q2: How to use a dataset for inference in Swift? Where are the inference results saved?
Use `--val_dataset <your-val-dataset>` to specify the dataset. For trained models, you can also set the `--load_data_args true` argument. The path to save inference results is set via `--result_path your_path`, and the path will be printed in the logs. For details, see the documentation on [Command-line parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
If you need to keep extra fields from the inference dataset, set `--remove_unused_columns false`.
### Q3: How to set up batch inference in Swift?
If the infer_backend is `transformers`, set the command-line argument `--max_batch_size 16`, or use a [Python script](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py). Here, max_batch_size refers to the batch size per GPU card.
### Q4: How to set up streaming inference in Swift?
Set `--stream true`. In this case, the inference results will be written to a jsonl file line by line. Note that streaming inference does not support DDP.
### Q5: Questions related to vLLM and SGLang inference backends
For models trained with LoRA, please check the vLLM and SGLang documentation. If they support LoRA inference, merging the adapters is not required. Additionally, SGLang inference does not currently support multi-modality.
### Q6: Questions related to generation parameters
Parameters like temperature are read from generation_config.json by default. Set `--temperature 0` or `--top_k 1` to disable randomness in inference.
### Q7: How to set the system prompt to empty? If I don't set the system parameter in the command line, a default one is still added.
Set `--system ''`.
### Q8: How to compute metrics like ACC/ROUGE during inference?
Refer to the documentation on inference parameters for [metrics](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#inference-arguments).
### Q9: During model inference, which parameter should be set to continue generation from a specific prefix?
Use the `--response_prefix` parameter.
### Q10: The 'answer' in my data already contains part of the prompt. How should I modify the inference to complete the 'answer'?
```text
{"messages": [{"role": "system", "content": "<system>"}, {"role": "user", "content": "<query1>"}, {"role": "assistant", "content": "answer1, "}]}
```
This is supported in Swift versions 3.0 and later. Refer to [examples/infer/demo_agent](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_agent.py).
### Q11: During multimodal model inference, how can I limit the maximum number of pixels to reduce GPU memory usage?
Set the command-line argument `--max_pixels xxx`, the environment variable `MAX_PIXELS=xxx`, or the specific model argument `--model_kwargs '{"max_pixels": xxx}'`. The environment variable only affects the models specified in the documentation. For details, see the documentation on [Specific Model Arguments](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#specific-model-arguments).
### Q12: How to output probability values (logprobs) during Swift inference?
For command-line inference, set `--logprobs true`. For Python script inference, set `request_config = RequestConfig(..., logprobs=True, top_logprobs=2)`. Refer to [test_logprobs.py](https://github.com/modelscope/ms-swift/blob/main/tests/infer/test_logprobs.py).
### Q13: How to output last_hidden_state during Swift inference?
There is no direct example, but you can refer to the `_get_last_hidden_state` method in the GRPO trainer.
### Q14: Issues with inconsistent inference results between Transformers, vLLM, Ollama, etc.
Swift's templates are aligned with those of Transformers. Check if the inference parameters are consistent. Additionally, there are differences between VllmEngine and TransformersEngine.
### Q15: Inference for embedding/reranker models
For embedding model inference, refer to the [example](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_embedding.py) here. For reranker model inference, refer to the [example](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_reranker.py) here.
### Q16: When using a Python script for inference, how can I use the CPU?
Set the environment variable: `os.environ['CUDA_VISIBLE_DEVICES'] = '-1'`.
### Q17: Does the swift infer command support multi-machine inference?
If the model can fit on a single node, you can orchestrate it using Kubernetes. If the model does not fit on a single node, multi-machine inference is not supported.
### Q18: When sampling with Swift, it seems batching is not supported? It looks like it samples one by one in a for loop, which is a bit slow.
There is a [script](https://github.com/modelscope/ms-swift/blob/main/examples/train/rft/rft.py) that can use multiple processes to split the dataset for sampling.
### Q19: Issues related to specific model dependency versions
If Qwen2-Audio inference results are garbled, please use transformers==4.48.
LoRA models trained with transformers==4.55.2 can no longer be loaded by versions older than 4.52. See [issue#5440](https://github.com/modelscope/ms-swift/issues/5440) for details.
Swift is compatible with different versions of qwen-vl-utils, so you do not need to switch its version when using qwen2.5-vl and qwen3-vl models.
### Q20: I got an error: safetensors_rust.SafetensorError: Error while deserializing header:MetadataIncompleteBuffer
The model weights are corrupted.
### Q21: How can I handle this vLLM error: `ValueError: the decoder prompt contains a(n) video item with length 16758, which exceeds the pre-allocated encoder cache size 16384. please reduce the input size or increase the encoder cache size by setting --limit-mm-per-prompt at startup.`?
This usually means the multimodal input is too long and exceeds vLLM's pre-allocated encoder cache size. You can adjust the encoder cache size with `--limit-mm-per-prompt`. Another practical workaround is to increase `max_num_batched_tokens`. In Swift cli:
```shell
--vllm_engine_kwargs '{"max_num_batched_tokens": 20000}'
```
## Export
### Q1: Errors related to autoawq
If you encounter autoawq-related errors during inference without using an AWQ-quantized model, try uninstalling autoawq and running the inference again. For models that do not support AWQ quantization, you can try using GPTQ for quantization instead.
### Q2: The model does not fit on a single GPU during Swift quantization
Try setting the `--device_map cpu` flag. Alternatively, you can load the model across multiple GPUs but perform the quantization on a single GPU.
### Q3: I am trying to quantize the Qwen2.5 72B model to GPTQ int4 using swift export. I'm using the default max_model_length=32768 and a calibration dataset with 128 samples. However, the quantization process failed with the following error log: factorization could not be completed because the input is not positive-definite (the leading minor of order 18145 is not positive-definite). What could be the reason for this?
This is an issue caused by the Hessian matrix not being positive-definite. Please try using a different calibration dataset.
### Q4: If I pass a custom template_type when using swift export (e.g., swift export --template_type custom), will this permanently change the template associated with the model?
No, it will not be permanently modified. In Swift, templates are defined within the framework's internal code; they are not saved as part of the model files, for instance, in a Jinja format.
### Q5: Can the model be directly converted to GGUF format after training?
Currently, only exporting to ModelFile is supported. For details, please refer to the documentation on [Command-line parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
## Deployment
### Q1: How do I set up a model for Swift deployment?
This is the same as Q1 for inference.
### Q2: How do I perform multi-GPU deployment with Swift?
For details, see the [example](https://github.com/modelscope/ms-swift/tree/main/examples/deploy). If you are using the transformers engine, it does not support DDP, so multi-GPU deployment is not possible. Furthermore, heterogeneous deployment (e.g., using different GPU models or assigning different memory ratios to each GPU) is not supported.
### Q3: Regarding the system prompt, you can specify it via the --system parameter, prepend it to each data entry in the dataset, or define it in the template. Is it sufficient to use just one of these methods? And are they all treated the same way by the model?
System prompt priority: The one in the dataset > The one from the command line > The default one in the template.
### Q4: Questions about multimodal input from the client.
To pass images, audio, and other media from the client, please see the [client example](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/client/mllm). If you encounter an invalid image URL, you can set a request timeout either through the `SWIFT_TIMEOUT` environment variable or as a parameter in `InferClient`.
### Q5: Questions about setting generation parameters.
For inference, parameters like temperature can only be set before launching. For deployment, you can set default values at launch, which can later be overridden by client-side settings.
### Q6: How do I enable streaming generation for a model deployed with Swift?
This is controlled on the client side. Please refer to the examples in [examples/deploy/client](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/client).
### Q7: How can I output token probabilities in a Swift deployment?
Set `--logprobs true` on the server side. Then, the client needs to pass the corresponding parameters, for example: `request_config = RequestConfig(..., logprobs=True, top_logprobs=2)`.
### Q8: Questions about the "thinking" process during model deployment.
If you need to disable the "thinking" step, it can currently only be done at launch time with swift deploy. See this [issue](https://github.com/modelscope/ms-swift/issues/4030) for more information.
### Q9: During deployment, which parameter should be set to generate multiple outputs in a single request?
The `n` parameter in `RequestConfig`.
### Q10: Questions regarding deployment with Swift using --infer_backend vllm compared to deploying directly with VLLM.
If there's a significant difference in inference results, it's likely due to a template mismatch. If there's a significant difference in inference speed, it might be caused by inconsistent image resolutions. Swift uses the V1 engine by default; you can control this with the environment variable `VLLM_USE_V1=1`.
### Q11: Questions about specific models and dependency versions.
If you get an error message about a missing "model.language_model.embed_tokens.weight", it indicates a version mismatch in the transformers library between training and inference. If you encounter garbled text when running inference with Qwen2.5 using FP16, try using BF16.
### Q12: I have a question: After deploying Qwen2-7B, when I use the client, I have to call the OpenAI API with client.completions.create and cannot use client.chat.completions.create. However, when using the qwen2-7b-instruct-q5_k_m.gguf model, I can use client.chat.completions.create. Why is this?
Base models can use client.chat.completions.create, but this is intended as a compatibility feature.
## Evaluation
### Q1: What evaluation datasets does Swift support? And how can I use a custom evaluation dataset?
For details on using standard and custom evaluation datasets, please refer to the documentation on [Evaluation](https://swift.readthedocs.io/en/latest/Instruction/Evaluation.html).
### Q2: After manually downloading an officially supported evaluation dataset, can swift eval be configured to evaluate using a local path?
For offline evaluation, please refer to the EvalScope documentation's [Quick Start](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html).
### Q3: The model after eval fine-tuning keeps stopping at a fixed percentage, but the vllm service seems to be running normally. The larger the model, the sooner it disconnects.
Set the `SWIFT_TIMEOUT` environment variable to -1.
### Q4: Can I control the number of dataset entries during evaluation? It takes over an hour to evaluate an MMLU, which is too slow.
Use the configuration parameter `--eval_limit`. This `--eval_limit` controls the number of entries in each subset. For example, if MMLU has over 50 subsets, and each limit is set to 10 entries, then that would be over 500 entries in total.
### Q5: In swift eval, the model stops generating after 1024 tokens. How can I modify this? Setting --max_new_tokens 5000 doesn't seem to work.
Check the command-line parameter [eval_generation_config](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#evaluation-arguments).
### Q6: How can I load downloaded datasets locally when using opencompass backend for evaluation?
The opencompass backend doesn't support setting `data_args`.
### Q7: Does swift eval with --eval_backend OpenCompass not support custom datasets?
```text
ValueError: eval_dataset: /mnt/workspace/data.jsonl is not supported.
eval_backend: OpenCompass supported datasets: ['C3', 'summedits', 'WiC', 'csl', 'lambada', 'mbpp', 'hellaswag', 'ARC_e', 'math', 'nq', 'race', 'MultiRC', 'cmb', 'ceval', 'GaokaoBench', 'mmlu', 'winogrande', 'tnews', 'triviaqa', 'CB', 'cluewsc', 'humaneval', 'AX_g', 'DRCD', 'RTE', 'ocnli_fc', 'gsm8k', 'obqa', 'ReCoRD', 'Xsum', 'ocnli', 'WSC', 'siqa', 'agieval', 'piqa', 'cmnli', 'cmmlu', 'eprstmt', 'storycloze', 'AX_b', 'afqmc', 'strategyqa', 'bustm', 'BoolQ', 'COPA', 'ARC_c', 'PMMEval', 'chid', 'CMRC', 'lcsts']
```
OpenCompass doesn't support custom datasets; use native mode for custom datasets.
### Q8: Evalscope can natively generate reports, but other backends like OpenCompass do not support report visualization, correct?
Currently, only native visualization is supported; other backends are not yet supported.
### Q9: Could you explain what causes the following error when using ifeval for evaluation?
```text
[Errno 20] Not a directory: '/root/nltk_data/tokenizers/punkt_tab.zip/punkt_tab/english/collocations.tab'
```
Unzip the file using `unzip /path/to/nltk_data/tokenizers/punkt_tab.zip`.
### Q10: When evaluating with eval_backend='OpenCompass', how can I specify the path to offline datasets?
Check the [data preparation guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html#data-preparation), download and unzip the dataset. You don't need to specify `dataset-args`; just place the dataset folder (the data folder) in the current working directory.
### Q11: What causes the following error when using evalscope?
```text
unzip: cannot find or open /root/nltk_data/tokenizers/punkt_tab.zip, /root/nltk_data/tokenizers/punkt_tab.zip.zip or /root/nltk_data/tokenizers/punkt_tab.zip.ZIP
```
This occurs during the download of nltk dependencies. Manually download [punkt_tab.zip](https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/open_data/nltk_data/punkt_tab.zip) and unzip it under `~/nltk_data/tokenizers`.
### Q12: Why is there no issue with plain text, but when testing multi-modal data, even though we specify the path, it still fails to detect the dataset and attempts to download it?
The VLMEvalKit process differs from native; it will automatically download data to `~/LMUData/`. For details, see the [documentation](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html#data-preparation).
### Q13: When using swift eval for benchmark evaluation, can I specify an llm as a judge and how should I pass in the parameters?
Yes, you can use swift to pass `judge-model-args` parameters from `extra_eval_args`, which include `api_key, api_url, and model_id`, as a JSON string.
### Q14: What could be the reason for uneven GPU memory allocation across multiple cards when running eval?
```shell
NPROC_PER_NODE=8
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\ MAX_PIXELS=802816\ swift eval\
--model "$MODEL_PATH” \$EXTRA_ARGS \
--eval_backend Native \ --infer_backend transformers\ --device_map auto \
--eval_limit"$EVAL_LIMIT"\ --eval_dataset general_qa\
--dataset_args "{\"general_qa\": {\"local_path\": \"${DATA_PATH}\", \"subset_list\": [\"${SUBSET_NAME}\"]}}" \ --host 127.0.0.1\> "$LOG_FILE" 2>&1
```
swift eval does not support being launched in DDP mode.
### Q15: Where can I see what extra fields, besides the question itself, are included in the query sent during a Swift evaluation?
The simplest way is to check the input field in the output reviews file. It contains the content sent to the model, converted to Markdown format. Note that this is not available if you are using the opencompass backend; you need to use the native backend.
The evaluation capabilities of ms-swift utilize the ModelScope community's evaluation framework, EvalScope. For more advanced features, please use the [EvalScope](https://evalscope.readthedocs.io/en/latest/get_started/introduction.html) directly.
docs/source_en/Instruction/GRPO/AdvancedResearch/CHORD.md
+63
View File
@@ -0,0 +1,63 @@
# On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting (CHORD)
This document describes the CHORD algorithm proposed in the paper [On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting](https://arxiv.org/abs/2508.11408). The core idea of CHORD is to dynamically integrate expert data (SFT) into reinforcement learning by a dual control mechanism: a global weight μ plus a token-level weight φ, thereby balancing imitation and exploration.
## Algorithm Overview
CHORD mixes training by introducing the SFT loss into the GRPO loss. The overall objective is:
$$
\mathcal{L}_{\text{CHORD}} = (1 - \mu) \cdot \mathcal{L}_{\text{GRPO}} + \mu \cdot \mathcal{L}_{\text{SFT}}
$$
where:
- $\mathcal{L}_{\text{GRPO}}$: on-policy RL loss based on on-policy samples (similar to PPO).
- $\mathcal{L}_{\text{SFT}}$: supervised fine-tuning (SFT) loss.
- $\mu \in [0, 1]$: global balancing coefficient that controls the contribution of the SFT signal to the overall gradient.
### Configuration (data and batch sizes)
We can implement CHORD training based on GRPO training.
CHORD requires specifying an additional SFT dataset and batch size at training time:
- `chord_sft_dataset`: the SFT dataset that provides expert data.
- `chord_sft_per_device_train_batch_size`: SFT mini-batch size per device.
---
## Two CHORD Variants
The paper proposes two variants: CHORD-μ and CHORD-φ.
### CHORD-μ
CHORD-μ gradually decays μ during training to transition from imitating experts toward autonomous exploration.
Parameters:
- `chord_mu_peak`: the peak value of μ.
- `chord_mu_valley`: the final decayed value of μ.
- `chord_mu_warmup_steps`: number of training steps to ramp μ up to the peak.
- `chord_mu_decay_steps`: number of training steps to decay μ from peak to valley.
### CHORD-φ (Token-level weighting)
CHORD-φ uses a token-wise weighting function φ to dynamically control each expert token's gradient contribution.
Definition of φ:
$$
\phi(y_t^\star, \pi_\theta) = p_t \cdot (1 - p_t)
$$
where:
- $p_t = \pi_\theta(y_t^\star \mid x, y_{<t}^\star)$ is the model's current predicted probability of the expert token.
- When $p_t \approx 0.5$ (model uncertainty), φ is maximal → emphasize tokens the model is uncertain about.
- When $p_t \approx 0$ or $p_t \approx 1$, φ → 0 → avoid overemphasizing tokens that are already certain or impossible.
Parameter to enable φ weighting:
- `chord_enable_phi_function: bool = False`
- Set to `True` to enable token-wise weight φ.
Note: If using a constant μ, set `chord_mu_peak` and `chord_mu_valley` to the same value.
<details>
<summary>Code implementation of μ scheduling and loss computation</summary>
See the `GRPOTrainer` method `_compute_chord_loss`.
</details>
Training reference script: https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/chord.sh
docs/source_en/Instruction/GRPO/AdvancedResearch/CISPO.md
+62
View File
@@ -0,0 +1,62 @@
# Clipped Importance Sampling Policy Optimization (CISPO)
Clipped Importance Sampling Policy Optimization (CISPO) is a reinforcement learning algorithm proposed in the [MiniMax-M1](https://arxiv.org/abs/2506.13585) paper. Compared to GRPO (Group Relative Policy Optimization), CISPO clips the importance sampling weights themselves.
## Algorithm Overview
For clarity, we explain CISPO by contrasting it with GRPO.
GRPO limits the magnitude of policy updates by clipping the policy ratio. Its loss function is:
$$
\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}\left[\min\left(r_t(\theta) \cdot \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \cdot \hat{A}_t\right)\right]
$$
where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the importance sampling ratio.
When handling long reasoning chains, this clipping approach can lead to the following issues:
**Gradient Suppression of Critical Tokens**: In complex reasoning tasks, certain critical low-probability tokens (such as *However, Recheck, Wait, Aha*) are crucial for triggering deep thinking and reasoning error correction. These tokens have low probability in the old policy $\pi_{\theta_{\text{old}}}$. When the new policy attempts to increase their probability, it results in a large policy ratio $r_t(\theta)$, and GRPO's clipping mechanism will discard these tokens.
### CISPO's Solution
The core idea of CISPO is to clip the importance sampling weights while preserving gradient updates. Specifically, CISPO's loss function is:
$$
\mathcal{L}_{\text{CISPO}}(\theta) = -\mathbb{E}\left[\text{detach}\left(\min(r_t(\theta), \epsilon_{\text{high}})\right) \cdot \hat{A}_t \cdot \log \pi_\theta(a_t|s_t)\right]
$$
where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the importance sampling ratio.
**Key Mechanisms**:
- Clip the importance sampling weights: $\min(r_t(\theta), \epsilon_{\text{high}})$
- **Detach operation**: The clipped weights do not participate in gradient computation and serve as constant coefficients
- Gradients come from the $\log \pi_\theta(a_t|s_t)$ term, ensuring all tokens contribute gradients
## Implementation Details
The pseudo-code implementation of CISPO is as follows:
```python
log_ratio = per_token_logps - old_per_token_logps
importance_weights = torch.exp(log_ratio) # r_t(θ) = π_θ / π_θ_old
clamped_ratios = torch.clamp(importance_weights, max=epsilon_high).detach()
per_token_loss = -clamped_ratios * advantages.unsqueeze(1) * per_token_logps
```
## Parameter Configuration
CISPO training can be enabled based on `GRPOTrainer` by setting the following parameters:
```bash
--loss_type cispo
--epsilon_high 5.0
```
> Compared to other algorithms, cispo generally uses a larger value for epsilon_high. The minimax paper does not provide specific parameter settings; the value used here refers to the experimental setup in the paper [ScaleRL](https://arxiv.org/pdf/2510.13786).
For other training parameters, refer to the [GRPO parameter documentation](../../Command-line-parameters.md#grpo-arguments).
docs/source_en/Instruction/GRPO/AdvancedResearch/DAPO.md
+89
View File
@@ -0,0 +1,89 @@
# DAPO: An Open-Source LLM Reinforcement Learning System at Scale
[Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO)](https://arxiv.org/abs/2503.14476) introduces several tricks based on GRPO, including:
- [Clip Higher](#clip-higher)
- [Dynamic Sampling](#dynamic-sampling)
- [Token level Loss](#token-level-loss)
- [Overlong Filtering](#overlong-filtering)
- [Soft Overlong Punishment](#soft-overlong-punishment)
## Clip Higher
PPO and GRPO use symmetric clipping ranges (e.g., ±0.2) to limit the magnitude of policy updates. While this ensures stability, it also restricts the model's exploratory capabilities. Specifically, when certain tokens have extremely low probabilities under the old policy, even if the current gradient indicates they should be reinforced (A > 0), the maximum increase is strictly limited.
DAPO employs an asymmetric clipping range, raising the upper clipping limit to encourage exploration:
- The upper bound (encouragement side) is relaxed to 0.28.
- The lower bound (suppression side) remains unchanged at 0.2.
In GRPO, the default symmetric clipping range is set using `epsilon`.
Parameters:
- `epsilon_high` sets the upper clipping range, while `epsilon` serves as the lower clipping range.
## Dynamic Sampling
GRPO samples multiple responses per question to compute inter-group advantages:
$$
\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
$$
However, when all generated outputs {o_i} receive the same reward, the inter-group advantage becomes zero, leading to vanishing gradients and reduced training efficiency.
DAPO addresses this issue with a dynamic sampling strategy:
- Skips data with zero inter-group reward standard deviation during sampling.
- Continues generating samples until the batch is filled.
Parameters:
- `dynamic_sample true` enables dynamic sampling.
- `max_resample_times` sets the maximum number of resampling attempts.
## Token level Loss
GRPO normalizes losses at the sentence level, which introduces bias based on response length.
DAPO uses token-level normalization to avoid this bias in loss calculation.
Parameters:
- `loss_type bnpo/dapo` enables token-level normalization.
> For the loss_type formula, please refer to the [documentation](../DeveloperGuide/loss_types.md).
## Overlong Filtering
DAPO argues that forcibly truncated responses contain high reward noise, making it difficult for the model to distinguish between quality issues and length issues. To address this, DAPO filters out truncated data during training, excluding it from loss computation.
Parameters:
- `overlong_filter` enables filtering of overly long samples.
## Soft Overlong Punishment
Language models often struggle with controlling output length:
- Overly long outputs may be truncated, leading to incorrect judgments of valid content.
- Unconstrained length generation affects practicality and computational efficiency.
DAPO designs a three-stage length penalty function:
$$
R_{\text{length}}(L) =
\begin{cases}
0, & L \leq L_{\text{max}} - L_{\text{cache}} \\[10pt]
\dfrac{(L_{\text{max}} - L_{\text{cache}}) - L}{L_{\text{cache}}}, & L_{\text{max}} - L_{\text{cache}} < L \leq L_{\text{max}} \\[10pt]
-1, & L > L_{\text{max}}
\end{cases}
$$
When the length falls within the interval $(L_{\text{max}} - L_{\text{cache}} < L \leq L_{\text{max}})$, a linearly increasing penalty is applied. For lengths $(L > L_{\text{max}})$, the maximum penalty (-1) is imposed.
Parameters:
- `reward_funcs soft_overlong` enables this reward function.
- `soft_max_length` sets L_max, which defaults to the model's maximum output length (`max_completion_length`).
- `soft_cache_length` sets L_cache.
## Parameter Settings
In summary, the following parameters can be set based on GRPOTrainer to implement DAPO training.
| Parameter | Type | Value |
|-----------------------|-----------|-------------|
| `--loss_type` | `str` | `bnpo`/`dapo`|
| `--epsilon_high` | `float` | `0.28` |
| `--dynamic_sample` | `bool` | `true` |
| `--max_resample_times`| `int` | `3` |
| `--overlong_filter` | `bool` | `true` |
| `--reward_funcs` | `str` | `soft_overlong`|
| `--soft_cache_length` | `int` | `4096` |
docs/source_en/Instruction/GRPO/AdvancedResearch/FIPO.md
+55
View File
@@ -0,0 +1,55 @@
# FIPO: Future-KL Influenced Policy Optimization
Author: [li2zhi](https://github.com/li2zhi)
[FIPO](https://arxiv.org/abs/2603.19835) is a value-free RL method for eliciting longer and deeper reasoning. It keeps the GRPO/DAPO training scaffold, but changes how token-level policy updates are weighted: instead of applying one sequence-level advantage uniformly to every token, FIPO uses a discounted Future-KL signal to estimate whether the future trajectory after each token is being reinforced or suppressed.
## Core Idea
In GRPO/DAPO, tokens in the same response usually share the same sequence-level advantage:
$$
\hat{A}_{i,t} = \hat{A}_{i}
$$
This is simple and stable, but the credit assignment is coarse. FIPO starts from the signed log-probability shift between the current policy and the old policy:
$$
\Delta \log p_t = \log \pi_\theta(y_t \mid x, y_{<t}) -
\log \pi_{\mathrm{old}}(y_t \mid x, y_{<t})
$$
A positive value means the token probability is being increased by the current update, while a negative value means it is being suppressed. FIPO then accumulates this signal from the current token to the end of the response:
$$
\mathrm{FutureKL}_t =
\sum_{k=t}^{T}\gamma^{k-t} M_k \Delta \log p_k
$$
where $M_k$ is the completion mask and $\gamma = 2^{-1 / \text{decay\_rate}}$. A larger `decay_rate` gives farther future tokens more influence; a smaller value makes the signal more local. FIPO maps the Future-KL value into a bounded influence weight:
$$
f_t = \mathrm{clip}(\exp(\mathrm{FutureKL}_t), 1-\epsilon_f, 1+\epsilon_f)
$$
The original advantage is then replaced by a future-aware advantage:
$$
\tilde{A}_{i,t} = \hat{A}_{i} \cdot f_{i,t}
$$
## Parameters
| Parameter | Type | Default | Description |
| ------------------------- | ------- | ------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--loss_type` | `str` | `grpo` | Set to`fipo` to enable FIPO loss |
| `--delta` | `float` | `None` | When enabled, it is used for both Future-KL high-IS-ratio token filtering and the main-loss dual-clip upper bound, and should be greater than `1 + epsilon_high`. Set it to `10.0` to match the official 32B script |
| `--fipo_decay_rate` | `float` | `32.0` | Half-life parameter for Future-KL; the actual discount is`2 ** (-1 / fipo_decay_rate)` |
| `--fipo_clip_range` | `float` | `0.2` | Influence weight clipping range;`0.2` clips to `[0.8, 1.2]` |
| `--fipo_clip_high_only` | `bool` | `true` | If`true`, clips the weight to `[1.0, 1.0 + fipo_clip_range]` |
| `--fipo_safety_threshold` | `float` | `4.0` | Caps the FIPO weight to `[0.8, 1.0]` for negative-advantage tokens whose IS ratio exceeds this threshold |
## Training Example
[swift](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/fipo.sh)
docs/source_en/Instruction/GRPO/AdvancedResearch/GSPO.md
+66
View File
@@ -0,0 +1,66 @@
# Group Sequence Policy Optimization (GSPO)
In [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071), it is pointed out that GRPO computes importance sampling weights at the token level. However, this approach is problematic: since each token is only sampled once, it cannot realize effective distribution correction, and instead introduces high-variance noise during training, which can easily lead to unstable gradient estimates and even training collapse. Therefore, the paper argues that the unit of the objective function should be consistent with that of the reward. Since the reward is typically given at the sequence level (i.e., for the entire generated response), it is more reasonable to perform off-policy correction and optimization at the sequence level rather than the token level.
Below are the three main strategies for computing importance sampling weights:
1. GRPO
GRPO computes the importance sampling ratio independently for each token, as follows:
$$
w^{\mathrm{GRPO}}_{i,t} = \frac{\pi_\theta (y_{i, t} \mid x, y_{i, <t})}{\pi_{\theta_{\mathrm{old}}} (y_{i, t} \mid x, y_{i, <t})}
$$
2. GSPO (Sequence-Level)
GSPO calculates the importance sampling ratio at the sequence level, given by:
$$
w^{\mathrm{GSPO}}_{i} = \left[ \frac{\pi_\theta (y_i \mid x)}{\pi_{\theta_{\mathrm{old}}} (y_i \mid x)} \right]^{\frac{1}{|y_i|}}
= \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta (y_{i, t} \mid x, y_{i, <t})}{\pi_{\theta_{\mathrm{old}}} (y_{i, t} \mid x, y_{i, <t})} \right)
$$
3. GSPO-token
GSPO-token combines both sequence-level and token-level importance sampling:
$$
w_{i, t}^{\mathrm{GSPO-token}} = \mathrm{sg}\left[w_i^{\mathrm{GSPO}}\right] \cdot \frac{\pi_{\theta}(y_{i, t} \mid x, y_{i, < t})}{\mathrm{sg}\left[\pi_{\theta}(y_{i, t} \mid x, y_{i, < t})\right]}
$$
where $\mathrm{sg}[\cdot]$ denotes stop-gradient (detach()).
> **NOTE:** According to gradient analysis (i.e., Eqs. (11) and (18) in the paper), when the advantage for each token is identical, GSPO-token is equivalent to GSPO. In the current implementation of GRPO, all token advantages are normalized based on the sentence-level reward within each group. Therefore, in this setting, GSPO-token and GSPO are theoretically equivalent. However, GSPO-token provides support for future fine-grained (token-level) advantages.
Pseudo-code implementation:
```python
log_ratio = per_token_logps - old_per_token_logps
# GRPO
log_importance_weights = log_ratio
# GSPO (Sequence-Level)
seq_weight = (log_ratio * mask).sum(-1) / mask.sum(-1)
log_importance_weights = seq_weight.unsqueeze(-1) # (B,1)
# GSPO-token
seq_weight = (log_ratio * mask).sum(-1) / mask.sum(-1)
log_importance_weights = seq_weight.detach().unsqueeze(-1) + (per_token_logps - per_token_logps.detach())
importance_weights = torch.exp(log_importance_weights)
```
Based on GRPO training, you can select different algorithms via the `--importance_sampling_level` argument:
- `importance_sampling_level token` (default, GRPO implementation)
- `importance_sampling_level sequence` (GSPO)
- `importance_sampling_level sequence_token` (GSPO-token)
Other hyperparameters in the paper
```bash
--epsilon 3e-4 # from paper section 5.1
--epsilon_high 4e-4 # from paper section 5.1
--gradient_accumulation_steps 8
--steps_per_generation 32 # from paper section 5.1 (each batch of rollout data is partitioned into four minibatches for gradient updates)
--beta 0 # zero kl regularization https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306
```
For training, you can refer to [this script](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/gspo.sh).
docs/source_en/Instruction/GRPO/AdvancedResearch/REAL.md
+93
View File
@@ -0,0 +1,93 @@
# Rewards as Labels: Revisiting RLVR from a Classification Perspective
Author: [li2zhi](https://github.com/li2zhi)
[Rewards as Labels: Revisiting RLVR from a Classification Perspective](https://arxiv.org/abs/2602.05630) proposes a reformulation of GRPO by treating rewards as labels and performing **in-group classification** instead of advantage estimation. This converts the policy optimization problem into a classification problem, thereby addressing two key issues in the GRPO loss:
- **Gradient Misassignment** for positive samples
- **Gradient Domination** for negative samples
## Background and Motivation
GRPO Objective
$$
J_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{q,o\sim\pi_{\mathrm{od}}(\cdot|q)}\left[\frac{1}{|o|}\sum_{t=1}^{|o|}\left(\min\left(\rho_tA_t,\mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t\right)\right)\right]
$$
where:
- $\rho_t = \frac{\pi_\theta(o_t|q)}{\pi_{\mathrm{old}}(o_t|q)}$ is the probability ratio
- $A_t$ is the advantage function
The corresponding gradient is:
$$
\nabla_{\theta} J_{\mathrm{GRPO}} = \mathbb { E } \left[ \frac { 1 } { | o | } \sum _ { t = 1 } ^ { | o | } \mathbb { I } _ { \mathrm { clip } } \cdot A _ { t } e ^ { s _ { t } } \nabla _ { \theta } \log \pi _ { \theta } \left( o _ { t } | q \right) \right]
$$
where:
- $s_t = \log \frac{\pi_\theta(o_t|q)}{\pi_{\mathrm{old}}(o_t|q)}$ is the relative log-probability
- $\mathbb{I}_{\mathrm{clip}}$ is the clipping indicator
Thus, the per-token gradient weight in GRPO is:
$$
|\mathcal{W}_{\mathrm{GRPO}}|=\left\{ \begin{array} {ll}\left|A\cdot e^s\right|, & \mathrm{if~}\mathbb{I}_{\mathrm{clip}}=1, \\ 0, & \text{otherwise.} \end{array}\right.
$$
![Gradient magnitude visualizations in GRPO](../../../../resources/real.png)
1. **Gradient Misassignment (Positive Samples)**：
For positive samples, as the relative log-probability $s$ decreases, the gradient magnitude also decreases.
This is counterintuitive: tokens that the model is less confident about but correct should receive larger updates. However, GRPO assigns more weight to already confident tokens, causing under-trained tokens to receive insufficient learning signal.
2. **Gradient Domination (Negative Samples)**：
For negative samples, as $s$ decreases, the gradient magnitude increases exponentially.
This leads to a situation where a few overconfident incorrect tokens dominate the gradient, overwhelming other negative signals within the same group. Due to the absence of an upper bound, this may result in unstable and excessively large parameter updates.
To address the above issues, REAL treats rewards directly as labels and performs **group-wise classification training**.
![Real Framework](../../../../resources/real_framework.png)
The classification logit for each sample is defined as:
$$
\bar{s}^k=\frac{1}{|o^k|}\sum_{t=1}^{|o^k|}\left(\log\frac{\pi_\theta(o_t^k\mid q)}{\pi_{\mathrm{old}}(o_t^k\mid q)}\right)
$$
- $\bar{s}^k > 0$: The sample is more likely under the current policy than the old policy → the model tends to **promote** this sample
- $\bar{s}^k < 0$: The sample is less likely under the current policy → the model tends to **suppress** this sample
Loss Function
$$
\mathcal{L}_{REAL}=\log\left(1+\sum_{\mathcal{O}_+}e^{-\bar{s}^i/\tau}\right)+\log\left(1+\sum_{\mathcal{O}_-}e^{\bar{s}^j/\tau}\right)
$$
Gradient Properties
$$
|\mathcal{W}_{\mathrm{REAL}}|=
\begin{cases}
\frac{1}{\tau}\frac{1}{1+C_{+}e^{\bar{s}^{k}/\tau}}, & r=1 \\
\\
\frac{1}{\tau}\frac{1}{1+C_{-}e^{-\bar{s}^{k}/\tau}}, & r=0 & & &
\end{cases}
$$
## Parameter Settings
| Parameter | Type | Default | Description |
|-----------|------|---------|--------------------------------------------------------------------|
| `--loss_type` | `str` | - | Set to `real` |
| `--real_tau` | `float` | `0.5` | Temperature parameter controlling decision boundary sharpness |
Training Script Reference
[swift](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/real.sh)
## Important Notes
When configuring training parameters, ensure that:
- `per_device_train_batch_size` is divisible by `num_generations`
This guarantees that each training batch contains complete groups, which is required for correct in-group classification.
docs/source_en/Instruction/GRPO/AdvancedResearch/REINFORCEPP.md
+86
View File
@@ -0,0 +1,86 @@
# REINFORCE++: An Efficient RLHF Algorithm with Robustness to Both Prompt and Reward Models
[REINFORCE++ Baseline](https://arxiv.org/abs/2501.03262) is a simplified version of the REINFORCE++ algorithm, designed for outcome rewards (response-level scalar rewards). Similar to GRPO, it samples multiple model outputs for each prompt and uses an intra-group baseline to estimate advantages. The key difference lies in the statistics used for normalization.
## Algorithm Overview
For clarity, we explain REINFORCE++ Baseline by contrasting it with GRPO (Group Relative Policy Optimization).
Both GRPO and REINFORCE++ Baseline estimate advantages via intra-group comparisons. Their main differences are:
### Difference 1: Statistics Used for Normalization
**GRPO (Group Relative Policy Optimization)**
For each prompt, GRPO generates $G$ response samples and normalizes using the **mean and standard deviation of all samples within the group**:
$$
\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
$$
When `scale_rewards='batch'` is set, it uses the **batch-level std of original rewards**:
$$
\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^{N})}
$$
where $N$ is the total number of samples in the batch.
**REINFORCE++ Baseline**
For each prompt, REINFORCE++ generates $G$ response samples, subtracts the group mean, and then normalizes using the **standard deviation of the group-mean-subtracted rewards**:
$$
\begin{align}
\tilde{A}_{i} &= R_i - \text{mean}(\{R_j\}_{j=1}^G) \\
\hat{A}_{i} &= \frac{\tilde{A}_{i}}{\text{std}(\{\tilde{A}_k\}_{k=1}^{N})}
\end{align}
$$
where $N$ is the total number of samples in the batch.
**Key Difference**:
- **GRPO**: Uses the std of **original rewards $R$** for normalization
- **REINFORCE++**: Uses the std of **group-mean-subtracted rewards $\tilde{A}$** for normalization
### Difference 2: KL Divergence Regularization
Similar to RLOO, REINFORCE++ Baseline integrates KL divergence directly into the reward:
$$
R'_i = R_i - \beta \cdot \text{KL}(\pi_\theta || \pi_{\text{ref}})
$$
where $\beta$ is the KL divergence weight coefficient (corresponding to the parameter `beta`), and $\pi_{\text{ref}}$ is the reference policy.
## Parameter Configuration
We can implement REINFORCE++ Baseline training by configuring the following parameters with `GRPOTrainer`:
```bash
--advantage_estimator reinforce_plus_plus
--scale_rewards batch
--kl_in_reward true
```
For training examples, please refer to this [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/reinforce_plus_plus.sh)
### Key Parameter Descriptions
- **`--advantage_estimator`**: Selects the advantage estimation method
- `grpo` (default): Uses the std of original rewards for normalization
- `reinforce_plus_plus`: Uses the std of group-mean-subtracted rewards for normalization
- **`--kl_in_reward`**: Controls where the KL divergence regularization term is applied
- `false`: KL divergence is an independent regularization term in the loss function (GRPO default)
- `true`: KL divergence is subtracted directly from the reward (REINFORCE++ original implementation)
- **`--scale_rewards`**: Controls the normalization method
- `group` (default): Intra-group normalization
- `batch`: Global batch-level normalization (REINFORCE++ original implementation)
- `none`: No normalization
- **`--num_generations`**: Number of samples generated per prompt ($G$)
- **`--beta`**: KL divergence regularization coefficient ($\beta$)
For other parameters, please refer to [GRPO Parameters](../../Command-line-parameters.md#grpo-arguments)
docs/source_en/Instruction/GRPO/AdvancedResearch/RLOO.md
+93
View File
@@ -0,0 +1,93 @@
# REINFORCE Leave-One-Out (RLOO)
[REINFORCE Leave-One-Out (RLOO)](https://arxiv.org/abs/2402.14740) is a reinforcement learning algorithm based on the classic REINFORCE policy-gradient method. It constructs an unbiased advantage baseline via the Leave-One-Out (LOO) technique.
## Algorithm Overview
For clarity, we explain RLOO by contrasting it with GRPO (Group Relative Policy Optimization).
Both GRPO and RLOO estimate advantages via intra-group comparisons to avoid the high variance of a global baseline. Their core differences are mainly in the following aspects:
### Difference 1: How the Advantage Baseline Is Constructed
**1. GRPO (Group Relative Policy Optimization)**
For each prompt, GRPO generates $G$ response samples and normalizes rewards using the group mean and standard deviation:
$$
\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
$$
Where:
- $R_i$ is the reward of the $i$-th sample
- $\text{mean}(\{R_j\}_{j=1}^G) = \frac{1}{G}\sum_{j=1}^G R_j$ is the group mean
- $\text{std}(\{R_j\}_{j=1}^G)$ is the group standard deviation
**2. RLOO (REINFORCE Leave-One-Out)**
For each prompt, RLOO generates $K$ response samples and constructs the baseline via Leave-One-Out, i.e., for the $i$-th sample, the baseline is the mean of the other $K-1$ samples:
$$
\hat{A}_{i} = R_i - \frac{1}{K-1}\sum_{j \neq i} R_j
$$
This can be equivalently rewritten as:
$$
\hat{A}_{i} = \frac{K}{K-1} \left(R_i - \bar{R}\right)
$$
where $\bar{R} = \frac{1}{K}\sum_{j=1}^K R_j$ is the group mean reward.
> Note: We use $K$ here to match the notation in the paper. It has the same meaning as $G$ in GRPO and corresponds to the configuration parameter `num_generations`.
**Why Leave-One-Out?**
The key advantage is unbiasedness. For the $i$-th sample, its reward $R_i$ is independent of the baseline $\frac{1}{K-1}\sum_{j \neq i} R_j$, hence the advantage estimate is unbiased. In contrast, using the mean including itself as the baseline introduces bias.
### Difference 2: How KL Regularization Is Applied
To prevent the policy from drifting too far from the reference policy, both algorithms introduce KL divergence regularization, but in different ways:
**GRPO**: Adds KL divergence as an independent regularization term to the [loss](../GetStarted/GRPO.md#algorithm-overview):
$$
\mathcal{L}(\theta) = -\mathbb{E}\left[\hat{A}_i \log \pi_\theta(a_i|s_i)\right] + \beta \cdot \text{KL}(\pi_\theta \Vert \pi_{\text{ref}})
$$
**RLOO**: Integrates KL divergence directly into the reward, constructing a modified reward:
$$
R'_i = R_i - \beta \cdot \text{KL}(\pi_\theta \Vert \pi_{\text{ref}})
$$
where $\beta$ is the KL coefficient (parameter `beta`), and $\pi_{\text{ref}}$ is the reference policy (typically an SFT model or the initial policy).
## Parameter Configuration
RLOO training can be enabled based on `GRPOTrainer` by setting the following parameters:
```bash
# Basic RLOO configuration
--advantage_estimator rloo # Use RLOO's leave-one-out advantage estimator
--kl_in_reward true # Integrate KL divergence into the reward (default for RLOO)
```
You can refer to this [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/rloo.sh) for training.
### Important Parameters
- **`--advantage_estimator`**: Choose the advantage estimator
- `grpo` (default): standardize using group mean and standard deviation
- `rloo`: construct the baseline via Leave-One-Out
- **`--kl_in_reward`**: Controls where the KL term is applied
- `false`: KL as a separate regularization term in the loss (GRPO style)
- `true`: subtract KL directly from the reward to form a modified reward (RLOO style)
- **`--num_generations`**: Number of samples per prompt, i.e., $K$
- **`--beta`**: KL regularization coefficient $\beta$
- Controls how conservatively the policy updates
Other parameters are consistent with the [GRPO arguments](../../Command-line-parameters.md#grpo-arguments).
docs/source_en/Instruction/GRPO/AdvancedResearch/SAPO.md
+91
View File
@@ -0,0 +1,91 @@
# Soft Adaptive Policy Optimization (SAPO)
[Soft Adaptive Policy Optimization (SAPO)](https://arxiv.org/abs/2511.20347) addresses the issues caused by hard clipping in GRPO by proposing a temperature-controlled soft gate mechanism that smoothly attenuates off-policy updates while preserving useful learning signals.
## Background and Motivation
When training LLMs with reinforcement learning, GRPO handles off-policy training by computing token-level importance sampling ratios:
$$
r_t = \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|x, y_{<t})}
$$
However, token-level importance sampling ratios often exhibit high variance, which can be exacerbated in the following cases:
- **Long text generation**
- **MoE model routing heterogeneity**: The old-policy model during sampling and the training model may use different expert routing, significantly amplifying logps differences
To address this, GRPO uses hard clipping to limit the magnitude of policy updates:
$$
L^{\mathrm{GRPO}} = -\min\left( r_t \cdot A, \mathrm{clip}(r_t, 1-\epsilon, 1+\epsilon) \cdot A \right)
$$
**The Dilemma of Hard Clipping**: Hard clipping struggles to balance stability and learning efficiency—too strict clipping limits the number of effective samples, while too loose clipping introduces noisy gradients from off-policy samples, leading to training instability.
## SAPO Method
SAPO uses a temperature-controlled sigmoid soft gate function to replace hard clipping, achieving smooth gradient attenuation.
### Soft Gate Function
The core of SAPO is using the sigmoid function to apply soft gating on the importance sampling ratio:
For positive advantages ($A > 0$), use positive gating:
$$
g^{+}_t = \sigma\left( \tau_{\mathrm{pos}} \cdot (r_t - 1) \right) \cdot \frac{4}{\tau_{\mathrm{pos}}}
$$
For negative advantages ($A < 0$), use negative gating:
$$
g^{-}_t = \sigma\left( \tau_{\mathrm{neg}} \cdot (r_t - 1) \right) \cdot \frac{4}{\tau_{\mathrm{neg}}}
$$
where:
- $\sigma(\cdot)$ is the sigmoid function
- $\tau_{\mathrm{pos}}$ and $\tau_{\mathrm{neg}}$ are temperature parameters that control the gate function slope
- $r_t$ is the importance sampling ratio
### SAPO Loss Function
$$
L^{\mathrm{SAPO}} = -g_t \cdot A
$$
where $g_t = g^{+}_t$ when $A > 0$, $g_t = g^{-}_t$ when $A < 0$.
### Temperature Parameters
The temperature parameter $\tau$ controls the decay rate of the soft gate function—larger values result in faster decay.
![tau curve](../../../../resources/sapo_tau.png)
The paper points out that positive advantages increase the logit of sampled tokens while decreasing the logits of all unsampled tokens; negative advantages do the opposite, increasing the logits of many unsampled tokens, which may spread to a large number of irrelevant tokens and introduce instability. Therefore, the paper recommends setting $\tau_\text{neg} > \tau_\text{pos}$ to make the gradient decay faster for tokens with negative rewards, improving training stability and performance.
The paper recommends default values of $\tau_{\mathrm{pos}} = 1.0$ and $\tau_{\mathrm{neg}} = 1.05$.
## Parameter Settings
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--loss_type` | `str` | - | Set to `sapo` |
| `--tau_pos` | `float` | `1.0` | Temperature parameter for positive advantages, controls gate slope |
| `--tau_neg` | `float` | `1.05` | Temperature parameter for negative advantages, controls gate slope |
```bash
swift rlhf \
--rlhf_type grpo \
--loss_type sapo \
--tau_pos 1.0 \
--tau_neg 1.05 \
# ... other parameters
```
Example training scripts:
- [swift](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/sapo.sh)
- [megatron swift](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/sapo.sh)
> The soft gate mechanism of SAPO only takes effect during off-policy training.
> The importance sampling granularity in SAPO is at the token level (i.e., importance_sampling_level defaults to token), which conflicts with GSPO.
docs/source_en/Instruction/GRPO/AdvancedResearch/deepeyes.md
+93
View File
@@ -0,0 +1,93 @@
# DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
## Principle Introduction
The [DeepEyes paper](https://arxiv.org/abs/2505.14362) proposes a method that enables models to "think with images" (image-assisted reasoning) by leveraging reinforcement learning. This approach achieves emergent model capabilities through end-to-end reinforcement learning, without requiring additional SFT (Supervised Fine-Tuning) steps. The model is equipped with built-in image localization capabilities and can actively invoke an "image zoom-in tool": during inference, the model automatically selects specific regions within an image for zooming and cropping, and then chains the processed region information into downstream reasoning, thus achieving multi-step visual-text reasoning.
![DeepEyes Overview](../../../../resources/deepeyes.png)
## Best Practices
**Dataset Download and Registration**
Download the official DeepEyes training datasets locally:
```bash
# modelscope
modelscope download --dataset Lixiang/ChenShawn-DeepEyes-Datasets-47k
# huggingface
huggingface-cli download ChenShawn/DeepEyes-Datasets-47k --repo-type=dataset
```
There are three parquet files in the dataset. Register them in the `swift/dataset/data/dataset_info.json` file. For each, rename the `prompt` column to `messages`:
```json
{
"ms_dataset_id": "path/to/data_0.1.2_visual_toolbox_v2.parquet",
"columns": {
"prompt": "messages"
}
},
{
"ms_dataset_id": "path/to/data/data_thinklite_reasoning_acc.parquet",
"columns": {
"prompt": "messages"
}
},
{
"ms_dataset_id": "path/to/data/data_v0.8_visual_toolbox_v2.parquet",
"columns": {
"prompt": "messages"
}
}
```
For registering the reward function and tool call logic locally as used in the paper, you can refer to the [DeepEyes implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py).
**Deploying the Evaluation Model**
The reward function in DeepEyes relies on a generative reward model to compare model outputs with ground-truth answers. For efficiency, it is recommended to deploy the reward model as a service.
Assuming you use the Qwen2.5-VL-72B-Instruct model for evaluation, refer to the following deployment command:
```bash
# 4*80G
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift deploy \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--infer_backend vllm \
--vllm_tensor_parallel_size 4 \
```
In the plugin file, you can use the OpenAI interface to call the model; see the [Reward Model documentation](../DeveloperGuide/reward_model.md#external-deployment).
The complete training script can be found at https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes.sh
## Implementation Details
The [DeepEyes implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py), which references the [official implementation](https://github.com/Visual-Agent/DeepEyes/blob/main/verl/utils/reward_score/vl_agent.py), provides sample code for a DeepEyes training plugin, covering the logic of the reward function and multi-turn interaction calls.
**Dataset Info** is shown below:
| Dataset File Name | data_source | Scoring Function | Tool Call |
|------------------------------------------|-----------------------|-----------------------------------|------------------------|
| data_v0.8_visual_toolbox_v2.parquet | chart | vl_agent.compute_score | True (image_zoom_in_tool) |
| data_0.1.2_visual_toolbox_v2.parquet | vstar | vl_agent.compute_score | True (image_zoom_in_tool) |
| data_thinklite_reasoning_acc.parquet | thinklite_eureka | vl_agent.compute_score_math | False |
**Note**: When processing image inputs, multimodal large models may perform preprocessing (such as cropping or resizing limited by the `max_pixels` parameter). When using the image zoom-in tool (`image_zoom_in_tool`), the model will output the cropped bbox based on the input image. Therefore, it is necessary to ensure the image fed into the tool has already been preprocessed. The following example shows how this is implemented in the Qwen2.5-VL series:
```python
from qwen_vl_utils import fetch_image
# At this point, images have not yet been preprocessed
infer_request.images
# Load as PIL.Image and apply cropping (same as using the MAX_PIXELS environment variable)
img = fetch_image({'image': load_pil_image(infer_request.images[0])})
```
**Tool Reward**
According to the paper, if the final answer is correct and the trajectory uses at least one tool, a tool reward is given. To prevent the model from generating invalid tool calls, we determine this based on the number of images, not just on the presence of `<tool_call>` tokens.
```python
tool_reward = 1.0 if num_image > 1 and acc_reward > 0.5 else 0.0
```
docs/source_en/Instruction/GRPO/AdvancedResearch/entropy_mask.md
+28
View File
@@ -0,0 +1,28 @@
# Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
The [paper](https://arxiv.org/abs/2506.01939) finds that when training large language models for reasoning abilities with methods such as RLVR, the key to learning progress lies in a small fraction of high-entropy "minority tokens," rather than the majority of low-entropy tokens.
The paper demonstrates that within the token distribution during model reasoning, only a few high-entropy tokens play a dominant role. These tokens typically appear at critical junctures where the reasoning or decision path diverges the most (e.g., tokens like "wait," "since," etc.), determining whether the model can master complex reasoning tasks. In contrast, most low-entropy tokens contribute little to the model's reasoning ability. The paper proposes computing policy gradients exclusively on high-entropy tokens, discarding gradients for low-entropy tokens.
The formula for token entropy is as follows:
$
H_t := -\sum_{j=1}^{V} p_{t,j} \log p_{t,j}, \qquad \text{where } (p_{t,1}, \cdots, p_{t,V}) = \mathbf{p}_t = \pi_\theta(\cdot | \mathbf{q}, \mathbf{o}_{<t}) = \text{Softmax}\left(\frac{\mathbf{z}_t}{T}\right)
$
Where:
- $\pi_\theta$: The model parameterized by $\theta$;
- $\mathbf{q}$: The input query;
- $\mathbf{o}_{<t} = (o_1, o_2, \cdots, o_{t-1})$: The sequence of tokens generated prior to timestep $t$;
- $V$: Vocabulary size;
- $\mathbf{z}_t \in \mathbb{R}^V$: The pre-softmax logits at timestep $t$;
- $\mathbf{p}_t \in \mathbb{R}^V$: The model's output probability distribution over the vocabulary;
- $T \in \mathbb{R}$: The decoding temperature, controlling the smoothness of the distribution.
Object of entropy computation: $H_t$ is the entropy of the token generation distribution $\mathbf{p}_t$, which measures the uncertainty in the policy $\pi_\theta$ under the given context $(\mathbf{q}, \mathbf{o}_{<t})$.
> "Token entropy" $H_t$ always refers to the uncertainty of the generation distribution $\mathbf{p}_t$ at position $t$, rather than a property of the token $o_t$ itself. In other words, $H_t$ is the entropy of the distribution $\mathbf{p}_t$ at position $t$, and is independent of the sampled token $o_t$.
In practice, during GRPO training, the top_entropy_quantile parameter can be used to control the percentile threshold for entropy filtering. In the experiments from the paper, this parameter is set to 0.2, meaning that only the top 20% of tokens (with the highest entropy) at each sequence position are used for optimization in each batch.
By setting the parameter `log_entropy`, you can record the changes in entropy during training; see the [documentation](../GetStarted/GRPO.md#logged-metrics) for reference.
docs/source_en/Instruction/GRPO/AdvancedResearch/index.rst
+19
View File
@@ -0,0 +1,19 @@
Advanced Research
===============
.. toctree::
:maxdepth: 1
entropy_mask.md
CISPO.md
DAPO.md
deepeyes.md
FIPO.md
GSPO.md
CHORD.md
RLOO.md
REINFORCEPP.md
REAL.md
router_replay.md
SAPO.md
training_inference_mismatch.md
treepo.md
docs/source_en/Instruction/GRPO/AdvancedResearch/router_replay.md
+104
View File
@@ -0,0 +1,104 @@
# Router Replay (R2/R3)
**TL;DR**: In RL training of MoE models, routing inconsistency between the training engine and the inference engine can significantly amplify training-inference mismatch, and even cause training collapse. Router Replay eliminates this inconsistency by replaying fixed routing masks during the training forward pass. Depending on the replay source, there are two strategies: R2 (Vanilla Routing Replay) and R3 (Rollout Routing Replay).
## Background
### Three Policies in MoE RL
In GRPO training of MoE models, there are three distinct policy stages that share the same model weights but may differ in routing behavior:
| Policy | Notation | Routing Result | Description |
|--------|----------|---------------|-------------|
| **Training Policy** | $\pi_\theta$ | $e^{\pi}_t$ | The model during gradient updates |
| **Old Policy** | $\pi_{\theta_{\text{old}}}$ | $e^{\pi}_{\text{old},t}$ | The model state before batch updates |
| **Rollout Policy** | $\mu_{\theta_{\text{old}}}$ | $e^{\mu}_{\text{old},t}$ | The sampling policy in the inference engine (e.g., vLLM), with the same weights as old policy, but different routing due to kernel implementation differences, precision, etc. |
Here, $\pi_{\theta_{\text{old}}}$ and $\mu_{\theta_{\text{old}}}$ have identical weights at sampling time, but due to implementation differences between the inference and training engines (e.g., operator implementations), routing results may differ even for the same input.
### Decomposition of Training-Inference Mismatch
According to the [paper](https://arxiv.org/abs/2507.18071), the token-level importance sampling ratio can be decomposed into two factors:
$$
\frac{\pi_\theta(y_t|x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t|x, y_{<t})} = \underbrace{\frac{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t|x, y_{<t})}}_{\text{training-inference discrepancy}} \times \underbrace{\frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}}_{\text{policy staleness}}
$$
For MoE models, expert routing is deeply coupled with both factors:
- **Training-inference discrepancy**: Routing inconsistency between the training and inference engines ($e^{\pi}_{\text{old},t} \neq e^{\mu}_{\text{old},t}$) amplifies output divergence
- **Policy staleness**: As mini-batch updates proceed, routing also drifts ($e^{\pi}_t \neq e^{\pi}_{\text{old},t}$), further deviating from the sampling policy
## R2: Vanilla Routing Replay
The core idea of R2 is to **replay the routing determined by the old policy in the training engine** ($e^{\pi}_{\text{old},t}$) during gradient updates.
### Principle
During the training forward pass, first run a forward pass with the old policy weights to record the expert indices selected by each MoE Router layer, then force the training model $\pi_\theta$ to use these indices in its forward pass:
$$
g_{\text{replay},i} = \frac{I^{\pi}_{\text{old},i} \cdot \exp(s_{\text{train},i})}{\sum_j I^{\pi}_{\text{old},j} \cdot \exp(s_{\text{train},j})}
$$
where $I^{\pi}_{\text{old}}$ is the old policy's routing mask and $s_{\text{train}}$ is the router logits computed by the training model. The softmax still operates on the training logits, so gradients can flow back to the router weights normally.
### Properties
| Scenario | Behavior |
|----------|----------|
| **First mini-batch** (on-policy) | $\theta = \theta_{\text{old}}$, so $e^{\pi}_t = e^{\pi}_{\text{old},t}$, **target policy unchanged** (no bias) |
| **Subsequent mini-batches** (off-policy) | $\theta \neq \theta_{\text{old}}$, so $e^{\pi}_t \neq e^{\pi}_{\text{old},t}$, **target policy changed** (biased), but policy staleness is controlled |
## R3: Rollout Routing Replay
The core idea of R3 is to **replay the routing determined by the rollout policy in the inference engine** ($e^{\mu}_{\text{old},t}$) during the training forward pass.
### Principle
During sampling in the inference engine (e.g., vLLM), additionally record the expert indices (routing mask) for each token at every MoE Router layer, then pass these masks to the training engine and force $\pi_\theta$ to use them in its forward pass:
$$
g_{\text{replay},i} = \frac{I^{\mu}_{\text{old},i} \cdot \exp(s_{\text{train},i})}{\sum_j I^{\mu}_{\text{old},j} \cdot \exp(s_{\text{train},j})}
$$
### Compatibility with Other Methods
- R3 is **orthogonal** to **GSPO** and can be combined for further improvement
- R3 combined with **TIS** may not provide additional gains (R3 already eliminates inconsistency at the source; TIS's additional correction may be redundant)
- Router Replay and **Clipping** are both essential in off-policy training
## Router Mask Caching
The R3 paper also proposes that routing masks can be cached alongside the KV Cache: for the same prefix tokens, the MoE Router output is deterministic, so routing masks can be stored and reused together with the prefix KVCache. This is particularly important in multi-turn Agent scenarios (tool calling), avoiding the need to re-prefill the prefix to obtain routing masks, with an overall rollout latency overhead of less than 3%.
## Swift Implementation
### Parameters
Select the routing replay strategy via the `--router_replay_mode` parameter:
| Value | Description |
|-------|-------------|
| `disabled` (default) | No routing replay |
| `R2` | Vanilla Routing Replay: record old policy routing in the training engine and replay |
| `R3` | Rollout Routing Replay: export routing masks from the inference engine and replay in training |
Environment requirements:
- R3 requires vLLM ≥ 0.14.0 to support returning `routed_experts` information.
- Router Replay is currently only available with the Megatron backend, requiring megatron-core ≥ 0.16.0.
### Relationship with Training-Inference Correction
Router Replay and the importance sampling (IS) correction described in [Training-Inference Mismatch](training_inference_mismatch.md) are complementary:
- **IS correction**: Corrects probability divergence at the loss level via weighting
- **Router Replay**: Eliminates the source of divergence at the model architecture level by fixing routing
## References
1. [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://arxiv.org/abs/2510.11370)
2. [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071)
3. [Stabilizing Reinforcement Learning with LLMs: Formulation and Practices](https://arxiv.org/abs/2512.01374)
4. [Megatron Core Router Replay Design Document](https://docs.nvidia.com/megatron-core/developer-guide/nightly/api-guide/router_replay.html)
docs/source_en/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md
+236
View File
@@ -0,0 +1,236 @@
# Training-Inference-Mismatch
**TL;DR**: While GRPO introduces vLLM to accelerate the sampling process, it also introduces Training-Inference Mismatch issues that may affect training stability. This document explains the background, causes, and solutions to this problem.
## Background
### Basic Assumptions of GRPO
The training objective of GRPO (Group Relative Policy Optimization) can be expressed as:
$$
\mathcal{L}_{\text{GRPO}} = - \mathbb{E}_{y \sim \pi_\theta} \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
$$
Where:
- $r_t(\theta) = \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}$ is the importance sampling ratio
- $\hat{A}_t$ is the advantage function, calculated based on reward and group baseline
- $\epsilon$ is the clipping parameter
**Core Assumption**: Samples $y$ are drawn from the policy $\pi_\theta$. In practice, this means:
1. The rollout model and the training model (policy model) should be **the same model** $\pi_\theta$
2. The probability distributions of both models should be **exactly identical**, i.e., $\pi_{\text{rollout}} = \pi_\theta$
### Deviation After Introducing vLLM
GRPO's training speed is largely constrained by the sampling process (rollout). To accelerate this, training frameworks introduce high-performance inference engines (such as vLLM) for sampling. The **ideal assumption** is that through weight synchronization, vLLM maintains consistency with the training model, i.e., $\pi_{\text{vLLM}} \equiv \pi_\theta$.
However, in practice, even with fully synchronized weights, due to differences in operator implementations, the probability distributions still deviate:
$$
\pi_{\text{vLLM}}(y|x) \neq \pi_\theta(y|x)
$$
At this point, the actual training objective becomes:
$$
\mathcal{L} = - \mathbb{E}_{y \sim \pi_{\text{vLLM}}} \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
$$
Where samples come from $\pi_{\text{vLLM}}$, but gradients are computed based on $\pi_\theta$. This **violates the algorithm's on-policy assumption**, introducing training-inference mismatch issues and potentially causing performance degradation.
## Solution
To address training-inference mismatch, we can introduce **Importance Sampling (IS)** correction mechanisms.
### Importance Sampling Correction
The basic idea of importance sampling is: when samples come from distribution $q$ rather than target distribution $p$, we can correct the expectation calculation by introducing weights:
$$
\mathbb{E}_{x \sim p} [f(x)] = \mathbb{E}_{x \sim q} \left[ \frac{p(x)}{q(x)} \cdot f(x) \right]
$$
Applied to the GRPO scenario, the corrected loss function is:
$$
\mathcal{L}_{\text{corrected}} = - \mathbb{E}_{y \sim \pi_{\text{vLLM}}} \left[ w(x, y) \cdot \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
$$
Where $w(x, y)$ is the importance sampling weight used to correct the distribution bias between vLLM and the training model.
Importance sampling weights can be computed and applied at different granularities:
1. **Token-Level**
Compute the importance sampling ratio at each token:
$$
w_{i,t}^{\text{token}} = \frac{\pi_\theta(y_{i,t}|x, y_{i,<t})}{\pi_{\text{vLLM}}(y_{i,t}|x, y_{i,<t})}
$$
2. **Sequence-Level**
Compute the sequence-level importance sampling ratio, then broadcast to each token:
$$
w_i^{\text{seq}} = \left[ \frac{\pi_\theta(y_i|x)}{\pi_{\text{vLLM}}(y_i|x)} \right]^{\frac{1}{|y_i|}} = \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t}|x, y_{i,<t})}{\pi_{\text{vLLM}}(y_{i,t}|x, y_{i,<t})} \right)
$$
### Stability Control: Truncate vs. Mask
Excessively large importance sampling weights can cause gradient explosion and destabilize training. Therefore, weight control is necessary:
#### 1. Truncate
Truncate the importance sampling weight to the $[0, \tau]$ interval:
$$
w_{\text{truncate}} = \min(w, \tau)
$$
This method retains all samples but limits their influence.
#### 2. Mask
Discard token/sequence data where weights exceed the threshold:
$$
w_{\text{mask}} = \begin{cases}
w & \text{if } w \leq \tau \\
0 & \text{otherwise}
\end{cases}
$$
### Four Correction Modes
Combining granularity and control strategies, there are four correction modes (selected via `--rollout_importance_sampling_mode` parameter):
| Mode | Description |
|------|-------------|
| `token_truncate` | Token-level truncation |
| `token_mask` | Token-level masking |
| `sequence_truncate` | Sequence-level truncation |
| `sequence_mask` | Sequence-level masking |
The threshold is set via the `--rollout_importance_sampling_threshold` parameter.
## Metrics
To monitor the degree of training-inference mismatch during training, we add the following metrics to the logs (prefixed with `rollout_correction/`):
### 1. KL Divergence
KL divergence measures the deviation between the rollout policy and the training policy. Both metrics estimate $\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta)$
**Direct estimator `kl`**:
$$
\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta) = \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \log \frac{\pi_{\text{vLLM}}}{\pi_\theta} \right]
$$
**K3 estimator `k3_kl`**:
$$
\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta) \approx \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \rho - \log \rho - 1 \right], \quad \rho = \frac{\pi_\theta}{\pi_{\text{vLLM}}}
$$
The K3 estimator is more numerically stable when KL values are small and is always non-negative.
### 2. Perplexity (PPL)
Perplexity measures the model's prediction uncertainty for a sequence:
$$
\text{PPL} = \exp\left( -\frac{1}{|y|} \sum_{t=1}^{|y|} \log p(y_t) \right)
$$
Related metrics:
- `training_ppl` / `training_log_ppl`: Training policy PPL and its logarithm
- `rollout_ppl` / `rollout_log_ppl`: Rollout policy PPL and its logarithm
- `log_ppl_diff`: Log PPL difference, positive value means training policy assigns lower probability
- `log_ppl_abs_diff`: Absolute log PPL difference
- `log_ppl_diff_max` / `log_ppl_diff_min`: Max/min of log PPL difference
- `ppl_ratio`: PPL ratio $\frac{\text{PPL}_{\text{training}}}{\text{PPL}_{\text{rollout}}}$
### 3. χ² Divergence (Chi-squared Divergence)
χ² divergence measures the variance of importance sampling weights:
$$
\chi^2(\pi_\theta \| \pi_{\text{vLLM}}) = \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \rho^2 \right] - 1, \quad \rho = \frac{\pi_\theta}{\pi_{\text{vLLM}}}
$$
- `chi2_token`: Token-level χ² divergence, $\mathbb{E}[\rho_t^2] - 1$
- `chi2_seq`: Sequence-level χ² divergence (geometric mean based), $\mathbb{E}[\rho_{\text{geo}}^2] - 1$, where $\rho_{\text{geo}} = \exp(\frac{1}{T}\sum_t \log \rho_t)$
Higher χ² divergence indicates larger IS weight variance and less stable training. `chi2_seq` uses geometric mean instead of product, making it comparable in scale to `chi2_token`.
### 4. Effective Sample Size (ESS)
Effective sample size measures the number of samples that actually contribute after importance sampling:
$$
\text{ESS} = \frac{1}{\mathbb{E}\left[\left(\frac{w}{\mathbb{E}[w]}\right)^2\right]}
$$
A larger ESS value (closer to 1) indicates more uniform importance sampling weight distribution and higher sample utilization efficiency. When all weights are equal (on-policy), ESS = 1; when weights differ significantly (severely off-policy), ESS becomes small.
### 5. IS Weight Statistics
- `is_weight_mean`: Average importance sampling weight, ideal value is 1.0
- `clipped_frac`: Fraction of samples that were truncated or masked
## Usage
### Logging Diagnostic Metrics Only
If you only want to monitor the degree of training-inference mismatch without enabling importance sampling correction, you can set:
```
--log_rollout_offpolicy_metrics true
```
This will log all diagnostic metrics (KL, PPL, χ², etc.) without modifying the loss function.
### Enabling Importance Sampling Correction
Enable the correction mechanism with the following parameters:
```
--rollout_importance_sampling_mode (default None)
--rollout_importance_sampling_threshold (default 2)
```
When `rollout_importance_sampling_mode` is set, diagnostic metrics are automatically logged without needing to set `log_rollout_offpolicy_metrics`.
## Off-Policy Sequence Masking
In addition to importance sampling correction, you can use **Off-Policy Sequence Masking** to address training-inference mismatch. This technique comes from the [DeepSeek-V3.2 paper](https://arxiv.org/abs/2512.02556).
### Principle
The core idea of Off-Policy Sequence Masking is: when the current policy deviates significantly from the old policy (rollout or old policy), directly discard (mask) that sequence from loss computation. This approach specifically targets **sequences with negative advantage**, as these are more likely to cause training instability when policy shift is large.
Specifically, for each sequence, compute:
$$
\delta_i = \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \bigl( \log \pi_{\text{old}}(y_{i,t}|x, y_{i,<t}) - \log \pi_\theta(y_{i,t}|x, y_{i,<t}) \bigr)
$$
Sequence $i$ will be masked when the following conditions are met (mean taken over completion tokens, i.e., where `completion_mask=1`):
1. $\delta_i > \tau$
2. **AND** $\hat{A}_i < 0$
Where:
- $\pi_{\text{old}}$ preferentially uses `rollout_per_token_logps` (logprobs from rollout/behavior policy); if unavailable, falls back to `old_per_token_logps`
- $\tau$ is the user-set threshold (`--off_policy_sequence_mask_delta`, default None = disabled)
## References
1. https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda
2. https://fengyao.notion.site/off-policy-rl
3. https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/rollout_corr_helper.py
4. [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://arxiv.org/abs/2512.02556)
docs/source_en/Instruction/GRPO/AdvancedResearch/treepo.md
+35
View File
@@ -0,0 +1,35 @@
# TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
Author: [li2zhi](https://github.com/li2zhi)
## Principle Introduction
[TreePO paper](https://arxiv.org/abs/2508.17445) proposes a tree-structured modeling method. This method organizes sequence generation into a segmented tree structure search. Through dynamic branching, backtracking, and early termination mechanisms, it significantly improves the reuse rate of the key-value cache, thereby reducing computational overhead, while maintaining or even enhancing the diversity of exploration.
![TreePO Overview](../../../../resources/treepo.png)
## Implementation Details
[TreePO implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/treepo/tree_rollout_plugin.py), which references the [official implementation](https://github.com/multimodal-art-projection/TreePO/blob/main/recipe/treepo/vllm_rollout_tree.py) provides sample code for a TreePO training plugin，covering logic related to multi-round interactions, termination judgment, and branch rollback.
**Note:** In actual use, you need to rewrite the logic of methods such as step and check_finished according to your own scenario requirements to ensure that they can execute as expected in the custom scenario.
For information on the design and use of custom rewards, you can refer to the implementation of [DeepEyes](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py).
The complete training script can be found at [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/treepo/tree_rollout.sh).
## Test Data
> model: Qwen/Qwen2.5-0.5B
> dataset: AI-MO/NuminaMath-TIR
> subset size: 1,000 samples
> 1 GPU for training, 1 GPU for inference
| \ | batch_size | num_generation | max_tree_depth | global_step | total inference calls | saving ratio | train_speed(iter/s) | improvement rate |
| ----------------------- | ---------- | -------------- | -------------- | ----------- | --------------------- | ------------ | ------------------- | ---------------- |
| original implementation | 8 | 8 | 4 | 200 | 5965 | 0.00% | 0.292436 | 0.00% |
| tree(max_divergence=3) | 8 | 8 | 4 | 200 | 3678 | 38.34% | 0.31819 | 8.81% |
| | | | | | | | | |
| original implementation | 8 | 8 | 5 | 105 | 4312 | 0.00% | 0.261324 | 0.00% |
| tree(max_divergence=2) | 8 | 8 | 5 | 105 | 2513 | 52.69% | 0.336639 | 28.82% |
| tree(max_divergence=3) | 8 | 8 | 5 | 105 | 2990 | 30.66% | 0.308791 | 18.16% |
| | | | | | | | | |
| original implementation | 8 | 8 | 6 | 105 | 5202 | 0.00% | 0.24832 | 0.00% |
| tree(max_divergence=2) | 8 | 8 | 6 | 105 | 3348 | 35.64% | 0.27755 | 11.77% |
| tree(max_divergence=3) | 8 | 8 | 6 | 105 | 3888 | 25.26% | 0.272339 | 9.67% |
docs/source_en/Instruction/GRPO/DeveloperGuide/gym_env.md
+361
View File
@@ -0,0 +1,361 @@
# GYM Environment Training
GYM-style environment training wraps the "model → environment → reward" chain behind an abstract interface, letting the LLM interact with the environment as an Agent over multiple turns. The reward of each step is produced directly by the environment, so you don't need a separate reward function to infer it from the trajectory. This document first introduces the interface, then walks through a complete custom example (FrozenLake) showing how to plug it into training.
## Gym interface
GYM originates from the [Gymnasium library](https://github.com/Farama-Foundation/Gymnasium). In ms-swift we define the following interface:
```python
class Env(ABC):
def __init__(self, env_config):
"""env_config comes from the env_config column of each dataset row and carries initialization arguments."""
self.env_config = env_config
@abstractmethod
async def reset(self, config: RolloutInferRequest) -> Tuple[str, Dict[str, Any], str]:
"""
Returns:
- observation: sent to the model as the first user message
- info: debug/log information, recorded in completions.jsonl
- system_message: system prompt for this trajectory
"""
pass
@abstractmethod
async def step(self, action: Messages) -> Tuple[str, float, bool, Dict[str, Any]]:
"""
Args:
action: the complete conversation messages so far; the last one is the model's latest reply
Returns:
- next_observation: next user message
- reward: reward for the current step
- done: whether the trajectory is finished
- info: debug/log information
"""
pass
@abstractmethod
async def close(self):
"""Release resources."""
pass
```
The `RolloutInferRequest` received by `reset` contains the dataset row's `messages`, `data_dict` (extra columns including `env_config`), etc. See the [input example](./multi_turn.md#multiturnscheduler) for the full structure.
> If you need extra control over the conversation history between turns (e.g. dynamic compression, injecting hints), subclass `MultiTurnScheduler` and implement `on_trajectory_start` / `on_turn_end` hooks, or override `step` / `run` — see the [multi-turn doc](./multi_turn.md#customising-the-interaction-logic).
## Launching training
Use the built-in [gym_scheduler](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py) to wire the env into multi-turn rollout.
`GYMScheduler` is based on the generic hook protocol:
- Inherits `MultiTurnScheduler` — no need to override the `run` method
- Implements `on_trajectory_start` (calls `env.reset`) and `on_turn_end` (calls `env.step`)
- Works with both server mode (`run()`) and colocate mode (`run_multi_turn()`)
User-defined envs are loaded via `--external_plugins your_plugin.py`; the plugin runs `envs['my_env'] = MyEnv` to register them (the FrozenLake example below demonstrates the full pattern).
The built-in `GYMScheduler` completes the control logic via hooks:
```python
class GYMScheduler(MultiTurnScheduler):
def on_trajectory_start(self, requests):
# Create an env for each request, call env.reset, inject initial observation
for req in requests:
env = self._create_env(req.data_dict.get('env_config', {}))
observation, info, system_message = env.reset(req)
req.messages = [system_msg, user_msg(observation)]
self._envs[req.uuid] = env
def on_turn_end(self, req, response_choice, current_turn):
# Call env.step, accumulate reward, return done + rollout_infos
next_obs, reward, done, info = env.step(deepcopy(req.messages))
self._total_rewards[req.uuid] += reward
return {
'done': done,
'rollout_infos': {
'total_reward': self._total_rewards[req.uuid],
'step_rewards': [...],
}
}
def step(self, req, response_choice, current_turn):
# Inject the next observation into a user message
if self._pending_obs.get(req.uuid):
req.messages.append({'role': 'user', 'content': next_obs})
return {'infer_request': req}
```
**Colocate mode**:
```bash
megatron rlhf \
--rlhf_type grpo \
--vllm_mode colocate \
--external_plugins examples/megatron/grpo/multi_turn/frozen_lake_plugin.py \
--multi_turn_scheduler gym_scheduler \
--gym_env frozen_lake \
--use_gym_env true \
--max_turns 10 \
...
# swift rlhf works the same way
```
**Server mode**
```bash
swift rollout \
--model xxx \
--use_gym_env true \
--external_plugins examples/megatron/grpo/multi_turn/frozen_lake_plugin.py \
--multi_turn_scheduler gym_scheduler \
--gym_env frozen_lake \
--max_turns 10
# On the trainer side, add --vllm_server_pass_dataset true so the env_config column reaches the rollout server.
megatron rlhf --vllm_mode server --vllm_server_pass_dataset true ...
# or swift rlhf --vllm_mode server --vllm_server_pass_dataset true ...
```
Two ways to select the environment:
- Set it globally via `--gym_env env_name` (recommended — one env for the whole script);
- Or specify it per dataset row via `env_config.name` (for mixed-env workloads; overrides `--gym_env`).
## Example: writing a FrozenLake environment from scratch
<img src="https://gymnasium.farama.org/_images/frozen_lake.gif" width="220" alt="FrozenLake environment (image from Gymnasium docs)" />
[FrozenLake](https://gymnasium.farama.org/environments/toy_text/frozen_lake/) is a classic task from OpenAI Gym: the agent starts at the start cell, must cross a frozen lake to reach the goal, and avoid holes along the way. The original environment is illustrated above. The walkthrough below uses a text-only version of it (the same grid rendered as ASCII).
Full source: [frozen_lake_plugin](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/multi_turn/frozen_lake_plugin.py).
**1. Define the Env**
Each dataset row produces a freshly generated random 4x4 map (random holes + random S/G positions, BFS-validated to be solvable). Cell meanings: `S` start / `G` goal / `H` hole (stepping in = fail) / `F` safe ice / `P` player's current position.
```python
class FrozenLakeEnv(Env):
def __init__(self, env_config):
super().__init__(env_config)
self.size = int(env_config.get('size', 4))
self.p = float(env_config.get('p', 0.8))
seed = env_config.get('seed')
self.seed = int(seed) if seed is not None else None
async def reset(self, config: RolloutInferRequest):
self.grid = generate_random_map(size=self.size, p=self.p, seed=self.seed)
...
return observation, {'seed': self.seed}, SYSTEM_PROMPT
async def step(self, action: Messages):
move = _parse_action(action[-1]['content']) # <action>up|down|left|right</action>
# Advance one cell, check G / H; the outer max_turns is enforced by the scheduler.
if cell == 'G': return obs, 1.0, True, {'status': 'goal'}
if cell == 'H': return obs, 0.0, True, {'status': 'hole'}
...
```
**2. Register**
Hook the env class into swift's `envs` registry. `--external_plugins` imports the file at startup, so the registration takes effect automatically:
```python
# examples/megatron/grpo/multi_turn/frozen_lake_plugin.py
from swift.rollout.gym_env import Env, envs
class FrozenLakeEnv(Env):
...
envs['frozen_lake'] = FrozenLakeEnv
```
**3. Prepare the dataset**
The dataset is just a placeholder here — the actual data is constructed by the env, with `env_config.seed` controlling map-generation randomness:
```json
{"messages":[{"role":"user","content":"<placeholder>"}],"env_config":{"seed":0}}
{"messages":[{"role":"user","content":"<placeholder>"}],"env_config":{"seed":1}}
...
{"messages":[{"role":"user","content":"<placeholder>"}],"env_config":{"seed":127}}
```
**4. (Optional) Blend in extra rewards**
With `--use_gym_env true`, the env-provided `total_reward` is automatically added as one reward column — no reward function is required. To mix in additional signals (e.g. format/length checks), just pass them via `--reward_funcs`; the gym reward is appended as an extra column and blended with the reward_funcs through `--reward_weights`. For example, also enabling a format reward:
```bash
megatron rlhf ... --use_gym_env true --reward_funcs format --reward_weights 0.2 1.0
# the last entry of reward_weights corresponds to the gym total_reward
```
**5. Train**
Runnable script: [`examples/megatron/grpo/multi_turn/frozen_lake.sh`](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/multi_turn/frozen_lake.sh)
During training, observe `rollout_infos.num_turns` (steps per trajectory) and the reward mean in the logs. `--log_completions true` writes full conversations to `completions.jsonl`, so you can verify the model outputs in the `<action>...</action>` format turn by turn.
References:
- https://gymnasium.farama.org/environments/toy_text/frozen_lake/
- https://github.com/alibaba/ROLL/tree/main/roll/pipeline/agentic/env/frozen_lake
- [OpenEnv](https://github.com/huggingface/openenv)
- [TRL Sudoku GRPO Example](https://github.com/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb)
## OpenEnv Environment Training
[OpenEnv](https://github.com/huggingface/openenv) is an open-source Agentic RL environment framework by HuggingFace that communicates with environment servers via WebSocket. Unlike the local `Env` interface used by FrozenLake above, OpenEnv places environment logic in a separate server process, and swift communicates with it through `OpenEnvScheduler` + `OpenEnvWrapper`.
### Architecture Comparison
| Feature | Built-in Gym (`GYMScheduler`) | OpenEnv (`OpenEnvScheduler`) |
|---------|------------------------------|------------------------------|
| Environment location | In-process (Python object) | Standalone server (WebSocket) |
| Environment interface | Subclass `Env`, implement `reset/step/close` | Server provides HTTP/WebSocket API |
| Registration | `--external_plugins` + `envs` registry | `--external_plugins` + `multi_turns` registry |
| Use case | Lightweight local envs (FrozenLake, etc.) | Complex server envs (TextArena, CARLA, etc.) |
| Concurrency control | Not needed | Built-in Semaphore for connection limiting |
### OpenEnvScheduler
`OpenEnvScheduler` extends `GYMScheduler`, replacing the local `Env` with `OpenEnvWrapper` (a WebSocket client). Key design:
- **`_create_env`**: Creates an `OpenEnvWrapper` connected to the OpenEnv server
- **`on_trajectory_start`**: Creates a wrapper per request, calls `reset()`, uses Semaphore to limit concurrency (default 4)
- **`on_turn_end`**: Parses model output, calls `wrapper.step()`, accumulates reward
- **`parse_action`** (overridable): Converts model text to action dict, default `json.loads`
- **`format_observation`** (overridable): Converts server observation to string, default `json.dumps`
Users subclass `OpenEnvScheduler` and override `parse_action`, `format_observation`, `on_trajectory_start`, and `on_turn_end` to adapt to specific environments.
### Example: Sudoku Environment
Using TextArena Sudoku as an example, the model places numbers on a 9x9 Sudoku grid via `[row col number]` format. Full code: [sudoku_scheduler.py](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/openenv/sudoku_scheduler.py).
**1. Start OpenEnv Server**
Install OpenEnv and the Sudoku environment package (textarena and nltk are installed automatically as dependencies):
```bash
pip install openenv
pip install git+https://huggingface.co/spaces/openenv/sudoku
```
Use the provided startup script to start the local server (default port 8000). `MAX_CONCURRENT_ENVS` must be ≥ `num_generations` used in training:
```bash
TEXTARENA_ENV_ID=Sudoku-v0 MAX_CONCURRENT_ENVS=8 python examples/train/grpo/plugin/openenv/start_sudoku_server.py
```
> The default `python -m textarena_env.server.app` only supports 1 concurrent session, which is insufficient for GRPO's parallel multi-generation sampling. `start_sudoku_server.py` lifts this restriction by setting `SUPPORTS_CONCURRENT_SESSIONS`.
Point `base_url` to the local server in your dataset:
```json
{"messages":[{"role":"user","content":"Play"}],"env_config":{"name":"openenv","base_url":"http://127.0.0.1:8000"}}
```
**2. Custom Scheduler**
Subclass `OpenEnvScheduler` to implement Sudoku-specific action parsing, observation formatting, and multi-component rewards:
```python
from swift.rollout.multi_turn import OpenEnvScheduler
class SudokuScheduler(OpenEnvScheduler):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._last_content_len = {} # Content diff tracking
async def on_trajectory_start(self, requests):
# Create env, parse board, generate hints
# hints include 'guaranteed moves' and candidate numbers
...
async def on_turn_end(self, infer_request, response_choice, current_turn):
# Parse [row col number], step env
# Compute 5-component reward: empty_cell / valid_move / repetition / progress / correct
# Return updated board + hints as next observation
...
def parse_action(self, text):
import re
match = re.search(r'\[\s*(\d+)\s+(\d+)\s+(\d+)\s*\]', text)
if match:
row, col, num = match.groups()
return {"message": f"[{row} {col} {num}]"}
return {"message": "[1 1 1]"}
```
**Multi-component reward system** (adapted from [TRL Sudoku example](https://github.com/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb)):
| Reward component | Calculation | Purpose |
|-----------------|-------------|---------|
| `empty_cell_reward` | Targets empty cell +1 / overwrites -1 | Guide model to valid positions |
| `valid_move_reward` | Valid new move +1 / warning -0.5 / invalid 0 | Encourage legal moves |
| `repetition_reward` | Exponential penalty for repeats (-2^n, cap -10) | Avoid repetition |
| `progress_reward` | (filled - initial) / (81 - initial) | Measure solving progress |
| `correct_reward` | Binary reward from environment | Puzzle fully solved |
Combined reward = sum of component averages, providing denser learning signal than a single binary reward.
**3. Hints System**
At each turn, the scheduler parses the current board state and provides hints to the model:
- **GUARANTEED MOVES**: Cells with only one candidate (can be filled directly)
- **Other options**: Cells with 2-3 candidates
- **MOVES ALREADY TRIED**: Previously attempted moves (to avoid repetition)
This significantly reduces exploration difficulty and enables the model to make more valid moves.
**4. Prepare Dataset**
The dataset serves as a placeholder; actual boards are generated by the environment server. Point `base_url` to the OpenEnv hosted address:
```json
{"messages":[{"role":"user","content":"Play"}],"env_config":{"name":"openenv","base_url":"http://127.0.0.1:8000"}}
```
**5. Register Scheduler**
`sudoku_scheduler.py` includes registration code at the end, loaded via `--external_plugins`:
```python
# End of sudoku_scheduler.py
from swift.rollout.multi_turn import multi_turns
multi_turns['sudoku_scheduler'] = SudokuScheduler
```
**6. Start Training**
```bash
swift rlhf \
--rlhf_type grpo \
--model Qwen/Qwen3.5-4B \
--dataset examples/train/grpo/plugin/openenv/sudoku.jsonl \
--external_plugins examples/train/grpo/plugin/openenv/sudoku_scheduler.py \
--enable_thinking false \
--max_completion_length 256 \
--use_gym_env true \
--multi_turn_scheduler sudoku_scheduler \
--max_turns 20 \
--use_vllm true \
--vllm_mode colocate \
...
```
Runnable script: [`examples/train/grpo/plugin/openenv/run_grpo_sudoku.sh`](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/openenv/run_grpo_sudoku.sh)
### Notes
1. **vLLM mode**: The example above uses `--vllm_mode colocate`, where vLLM and training share the same GPUs. If using `--vllm_mode server`, you need to start `swift rollout` separately as the vLLM server, and `--multi_turn_scheduler` / `--max_turns` should be passed to `swift rlhf`, not `swift rollout`.
2. **Server concurrency**: `start_sudoku_server.py`'s `MAX_CONCURRENT_ENVS` must be ≥ `num_generations` used in training. The default `python -m textarena_env.server.app` only supports 1 concurrent session.
3. **Content diff**: Environments like TextArena return cumulative messages (full history each turn). The scheduler tracks `_last_content_len` to return only the new portion, preventing context length explosion.
4. **First-turn timing**: `on_trajectory_start` is called BEFORE the first rollout, ensuring the model sees the actual environment observation (e.g., Sudoku board) rather than the placeholder text from the dataset.
5. **enable_thinking**: When using Qwen3.5 series models, set `--enable_thinking false` to skip `<think>` block generation.
6. **Sync I/O**: `OpenEnvWrapper`'s `reset()`/`step()` are synchronous WebSocket calls. `OpenEnvScheduler` subclasses should wrap these calls with `asyncio.to_thread()` to avoid blocking the event loop.
docs/source_en/Instruction/GRPO/DeveloperGuide/index.rst
+11
View File
@@ -0,0 +1,11 @@
Developer Guide
===============
.. toctree::
:maxdepth: 1
loss_types.md
multi_turn.md
multi_task.md
reward_function.md
reward_model.md
gym_env.md
docs/source_en/Instruction/GRPO/DeveloperGuide/loss_types.md
+131
View File
@@ -0,0 +1,131 @@
# Loss Types
GRPO training supports multiple loss types, with the main differences being the normalization dimension and gradient handling.
## Loss Function
At the token level, GRPO training uses the following loss function:
$$\mathcal{L}_{i,t} = -\min\left(\rho_{i,t} A_{i,t}, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) A_{i,t}\right)$$
When setting `loss_type cispo`, the CISPO loss is used:
$$\mathcal{L}_{i,t}^{\text{CISPO}} = -\text{detach}\left(\min(\rho_{i,t}, \epsilon_{\text{high}})\right) \cdot A_{i,t} \cdot \log \pi_\theta(y_{i,t}|y_{i,<t})$$
When setting `loss_type sapo`, soft gating replaces hard clipping, see [SAPO](../AdvancedResearch/SAPO.md)
$$\mathcal{L}_{i,t}^{\text{SAPO}} = -g_{i,t} \cdot A_{i,t}$$
where $g_{i,t} = \sigma(\tau \cdot (\rho_{i,t} - 1))$ is the temperature-controlled soft gate function.
where:
- $\rho_{i,t} = \frac{\pi_\theta(y_{i,t}|y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|y_{i,<t})}$ is the importance sampling weight
- $A_{i,t}$ is the advantage function
- $\epsilon$ and $\epsilon_{\text{high}}$ are the clipping parameters
- $\text{detach}(\cdot)$ indicates that this term does not participate in gradient computation
- $\sigma(\cdot)$ is the sigmoid function, $\tau$ is the temperature parameter
## GRPO
`--loss_type grpo`
GRPO is the standard loss function implementation that averages the token-level losses for each sample, then averages across all samples.
**Formula:**
$$\mathcal{L}_{\text{GRPO}} = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{T_i} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}$$
where:
- $N$ is the number of samples in the batch
- $T_i$ is the number of completion tokens for the $i$-th sample
**Normalization Dimension:** Sample dimension (first average over tokens for each sample, then average over all samples)
## BNPO (Batch Normalized Policy Optimization)
`--loss_type bnpo`
BNPO sums all token losses from all samples and then divides by the total number of completion tokens.
**Formula:**
$$\mathcal{L}_{\text{BNPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}}{\sum_{i=1}^{N} T_i}$$
where:
- $N$ is the number of samples in the batch
- $T_i$ is the number of completion tokens for the $i$-th sample
**Normalization Dimension:** Token dimension (average over all completion tokens)
## DR-GRPO
`--loss_type dr_grpo`
DR-GRPO sums all token losses from all samples and then divides by the batch size multiplied by the maximum completion length.
**Formula:**
$$\mathcal{L}_{\text{DR-GRPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}}{N \times L_{\text{max}}}$$
where:
- $N$ is the number of samples in the batch
- $T_i$ is the number of completion tokens for the $i$-th sample
- $L_{\text{max}}$ is the maximum completion length
**Normalization Dimension:** Fixed dimension (batch size × maximum completion length)
## CISPO
`--loss_type cispo`
CISPO loss is normalized by the total number of completion tokens across all processes.
**Formula:**
$$\mathcal{L}_{\text{CISPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}^{\text{CISPO}}}{\sum_{\text{all processes}} \sum_{i=1}^{N_p} T_{p,i}}$$
where:
- $N$ is the number of samples in the current process batch
- $T_i$ is the number of completion tokens for the $i$-th sample
- $N_p$ is the number of samples for the $p$-th process
**Normalization Dimension:** Global token dimension (total completion tokens across all processes)
## DAPO
`--loss_type dapo`
DAPO is similar to BNPO, using token-level normalization, but based on global data (multi-process) normalization.
**Formula:**
$$\mathcal{L}_{\text{DAPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}}{\sum_{\text{all processes}} \sum_{i=1}^{N_p} T_{p,i}}$$
where:
- $N$ is the number of samples in the current process batch
- $T_i$ is the number of completion tokens for the $i$-th sample
- $N_p$ is the number of samples for the $p$-th process
**Normalization Dimension:** Global token dimension (total completion tokens across all processes)
## FIPO
`--loss_type fipo`
FIPO adds a Future-KL influence weight on top of the DAPO/GRPO clipped policy loss. The sequence-level advantage for each token is weighted by the discounted accumulated KL shift from the current token to future tokens:
$$f_{i,t} = \text{clip}\left(\exp\left(\sum_{k=t}^{T_i} \gamma^{k-t} M_{i,k} \Delta \log p_{i,k}\right), 1-\epsilon_f, 1+\epsilon_f\right)$$
$$\mathcal{L}_{i,t}^{\text{FIPO}} = f_{i,t} \cdot \mathcal{L}_{i,t}$$
The FIPO influence weight is detached by default and uses the same global token normalization as DAPO.
**Normalization Dimension:** Global token dimension (total completion tokens across all processes)
## SAPO
`--loss_type sapo`
SAPO uses temperature-controlled soft gating instead of hard clipping to achieve smooth gradient attenuation. The normalization method is the same as GRPO.
For details, please refer to [SAPO](../AdvancedResearch/SAPO.md)
docs/source_en/Instruction/GRPO/DeveloperGuide/multi_task.md
+55
View File
@@ -0,0 +1,55 @@
# Multi-Task Training
We can add a column to the dataset that indicates the task type, and then use this information in the reward function or reward model plugin to determine which task is being processed. This allows us to implement multi-task training. For example, suppose our dataset contains both math and programming tasks like the following:
```json
[
{"query": "Solve the equation x + 2 = 5", "solution": "3", "task": "math"},
{"query": "Write a function to calculate the Fibonacci sequence", "solution": "xxx", "task": "code"},
{"query": "What is the integral of x^2?", "solution": "xxx", "task": "math"},
{"query": "Implement a sorting algorithm in Python", "solution": "xxx", "task": "code"}
]
```
We can set up different reward functions to handle math and code data separately. Note that the columns in the dataset will be passed to the reward function, so we can use the `task` column to distinguish between tasks.
Below are examples of reward functions tailored for different tasks:
```python
from swift.rewards import ORM, orms
import random
# Math-specific reward function
class MathRandomReward(ORM):
def __call__(self, completions, task, **kwargs):
rewards = []
for completion, t in zip(completions, task):
if t == "math":
import random
# Implement math accuracy logic
reward = random.random()
rewards.append(reward)
else:
# Return None for non-math tasks
rewards.append(None)
return rewards
# Coding-specific reward function
class CodeRandomReward(ORM):
def __call__(self, completions, task, **kwargs):
rewards = []
for prompt, completion, t in zip(prompts, completions, task):
if t == "code":
# Implement coding accuracy logic
reward = random.random()
rewards.append(reward)
else:
# Return None for non-coding tasks
rewards.append(None)
return rewards
orms['math_reward'] = MathRandomReward
orms['code_reward'] = CodeRandomReward
```
For data that does not belong to the current task, we handle it by returning None, ensuring that the reward calculation only applies to data within the designated task.
docs/source_en/Instruction/GRPO/DeveloperGuide/multi_turn.md
+323
View File
@@ -0,0 +1,323 @@
# Multi-turn Training
In reinforcement-learning scenarios, the model may need to interact with the environment over multiple turns (e.g., tool calls).
This interactive training requires the model to carry out continuous reasoning based on the feedback from the environment.
This document explains in detail how to customise the multi-turn training workflow in GRPO training.
> GKD also supports multi-turn training, sharing the same `MultiTurnScheduler` infrastructure as GRPO.
The figure below shows a typical multi-turn training process, where the model may perform several rollout rounds that include environment interaction, tool calls, and so on:
![Multi-turn example](../../../../resources/grpo_multi_turn.png)
## MultiTurnScheduler
`MultiTurnScheduler` is an abstract base class that provides the default multi-turn dialogue-management logic.
Its workflow is illustrated below:
<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/multiturn_pipeline.png" width="300" />
The scheduler is responsible for two core functions:
- **Termination check** — decide whether the current turn of inference should stop via `check_finished`.
- **Inference request construction** — build the request object for the next turn via `step`.
Key methods of the abstract base class `MultiTurnScheduler`:
```python
class MultiTurnScheduler(ABC):
def __init__(self, max_turns: Optional[int] = None, *args, **kwargs):
self.max_turns = max_turns
def on_trajectory_start(self, requests: List['RolloutInferRequest']) -> None:
"""Called before the first inference turn to initialize trajectory-level state.
This method can directly modify requests (e.g., inject initial environment observation).
Default is no-op.
"""
pass
def on_turn_end(self, infer_request: 'RolloutInferRequest',
response_choice: 'ChatCompletionResponseChoice',
current_turn: int) -> Dict[str, Any]:
"""Called after the assistant message is appended and before check_finished.
Used to advance the environment state (e.g., env.step) and return per-turn metadata.
Returns:
Dict[str, Any]: may optionally contain:
- 'done' (bool): if present, overrides the result of check_finished
- 'rollout_infos' (dict): merged into the accumulated trajectory info
Default returns an empty dict (no-op).
"""
return {}
def step(self, infer_request: 'RolloutInferRequest', response_choice: 'ChatCompletionResponseChoice',
current_turn: int) -> Dict:
"""
Handle the transition between dialogue turns.
Args:
infer_request: current inference request
response_choice: response of the current turn
current_turn: current turn index (starting from 1)
Returns:
Dict[str, Any]: a dict containing the result of this turn
- infer_request (required): the inference request for the next turn
- response_token_ids (optional): token IDs of each rollout response
- response_loss_mask (optional): loss mask of each rollout response
- rollout_logprobs (optional): token logps of each rollout response
- rollout_infos (optional): extra information
"""
raise NotImplementedError
def check_finished(self, infer_request: 'RolloutInferRequest', response_choice: 'ChatCompletionResponseChoice',
current_turn: int) -> bool:
"""
Default termination logic for multi-turn rollout.
Termination conditions:
1. The response is truncated (finish_reason == 'length').
2. The dialogue reaches the maximum number of turns (if max_turns is set).
Args:
infer_request: the inference request
response_choice: response choice containing finish_reason
current_turn: current turn index
Returns:
bool: True to stop, False to continue
"""
if response_choice.finish_reason == 'length':
return True
if self.max_turns and current_turn >= self.max_turns:
return True
return False
```
Arguments passed to `step` and `check_finished`:
- **infer_request**: current inference request
- **response_choice**: inference result of the current turn
- **current_turn**: current turn index (starting from 1)
<details><summary>Input example (click to expand)</summary>
```python
infer_request
"""
RolloutInferRequest(
messages=[
{'role': 'system', 'content': 'A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>\n'}, {'role': 'user', 'content': 'What is the value of $\\sqrt{36 \\times \\sqrt{16}}$?'},
{'role': 'assistant', 'content': 'To find the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\), we will break down the problem step-by-step.\n\nFirst, we need to evaluate the inner square root:\n\\[\n\\sqrt{16}\n\\]\nWe know that:\n\\[\n4^2 = 16 \\implies \\sqrt{16} = 4\n\\]\n\nNext, we substitute this result back into the original expression:\n\\[\n\\sqrt{36 \\times \\sqrt{16}} = \\sqrt{36 \\times 4}\n\\]\n\nNow, we need to evaluate the product inside the square root:\n\\[\n36 \\times 4 = 144\n\\]\n\nSo, the expression simplifies to:\n\\[\n\\sqrt{144}\n\\]\n\nFinally, we determine the square root of 144:\n\\[\n\\sqrt{144} = 12\n\\]\n\nThus, the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\) is:\n\\[\n\\boxed{12}\n\\]'}
],
images=[],
audios=[],
videos=[],
tools=None,
objects={},
data_dict={
'problem': 'What is the value of $\\sqrt{36 \\times \\sqrt{16}}$?',
'solution': "To solve the problem, we need to evaluate the expression \\(\\sqrt{36 \\times \\sqrt{16}}\\).\n\nWe can break down the steps as follows:\n\n1. Evaluate the inner square root: \\(\\sqrt{16}\\).\n2. Multiply the result by 36.\n3. Take the square root of the product obtained in step 2.\n\nLet's compute this step by step using Python code for accuracy.\n```python\nimport math\n\n# Step 1: Evaluate the inner square root\ninner_sqrt = math.sqrt(16)\n\n# Step 2: Multiply the result by 36\nproduct = 36 * inner_sqrt\n\n# Step 3: Take the square root of the product\nfinal_result = math.sqrt(product)\nprint(final_result)\n```\n```output\n12.0\n```\nThe value of \\(\\sqrt{36 \\times \\sqrt{16}}\\) is /\\(\\boxed{12}\\)."
}
)
"""
response_choice
"""
ChatCompletionResponseChoice(
index=0,
message=ChatMessage(
role='assistant',
content='To find the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\), we will break down the problem step-by-step.\n\nFirst, we need to evaluate the inner square root:\n\\[\n\\sqrt{16}\n\\]\nWe know that:\n\\[\n4^2 = 16 \\implies \\sqrt{16} = 4\n\\]\n\nNext, we substitute this result back into the original expression:\n\\[\n\\sqrt{36 \\times \\sqrt{16}} = \\sqrt{36 \\times 4}\n\\]\n\nNow, we need to evaluate the product inside the square root:\n\\[\n36 \\times 4 = 144\n\\]\n\nSo, the expression simplifies to:\n\\[\n\\sqrt{144}\n\\]\n\nFinally, we determine the square root of 144:\n\\[\n\\sqrt{144} = 12\n\\]\n\nThus, the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\) is:\n\\[\n\\boxed{12}\n\\]', tool_calls=None),
finish_reason='stop',
logprobs=None,
messages=None)
"""
# response_choice.messages will be copied at the end of multi-turn inference.
```
</details>
<br>
<br>
The default `check_finished` logic stops inference in the following cases:
- The model reply is truncated, i.e. exceeds `max_completion_length`.
- The number of inference turns exceeds the specified maximum.
For the full default multi-turn rollout logic, see the `run` method of the class.
You can override `run` to implement a completely custom workflow.
## Setting multi-turn parameters
Specify the scheduler via `multi_turn_scheduler` in the `swift rollout` command:
```bash
swift rollout \
--model Qwen/Qwen3-1.7B \
--vllm_use_async_engine true \
--multi_turn_scheduler thinking_tips_scheduler \
--vllm_max_model_len 32768 \
--vllm_gpu_memory_utilization 0.8 \
--max_turns 3
```
> With the `external_plugins` argument you can register your own local scheduler with ms-swift.
> Refer to the [plugin code](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py).
A full multi-turn training script can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/external/vllm_multi_turn.sh).
For multi-turn rollout we use `AsyncEngine` to perform efficient batched asynchronous sampling.
AsyncEngine reduces compute bubbles in multi-turn inference:
<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/asyncengine.png" width="400" />
Use the `vllm_use_async_engine` argument in the `rollout` command to specify the engine type (async is the default).
> Note: The async engine is only available in server mode.
### GYM environment training
If your multi-turn task can be modeled as a standard gym environment (`reset` / `step` / reward produced by the env directly), use the built-in `gym_scheduler` and implement an `Env` subclass to describe the task.
`GYMScheduler` is based on the generic hook protocol and does not require overriding the `run` method:
- **`on_trajectory_start`**: calls `env.reset` and injects the initial observation into the first user message
- **`on_turn_end`**: calls `env.step` to advance the environment, returns `{'done': bool, 'rollout_infos': dict}`
This design makes `GYMScheduler` compatible with both server mode (`run()`) and colocate mode (`run_multi_turn()`) — users only need to implement the `Env` interface.
See the [GYM environment training doc](./gym_env.md) for the full interface, the steps to define a custom env, and a minimal end-to-end example with no external dependencies (FrozenLake — runs out of the box on Megatron in colocate mode).
## Advanced topics
### Customising the interaction logic
In the default logic we treat the whole multi-turn rollout as one trajectory when computing the loss.
This assumes the model's history is not modified during interaction.
In some scenarios you may need to dynamically change the history during rollout (e.g., compressing context).
In that case each turn should be treated as a separate trajectory.
#### Approach 1: Using hooks
```python
class CustomScheduler(MultiTurnScheduler):
def on_trajectory_start(self, requests):
# Initialise before the first turn (e.g., env.reset, inject initial state)
for req in requests:
req.messages = [system_msg, user_msg(initial_observation)]
def on_turn_end(self, req, response_choice, current_turn):
# Advance state after each turn, return done and rollout_infos
next_obs, reward, done = self.advance_env(req.messages)
return {
'done': done,
'rollout_infos': {'reward': reward, ...}
}
```
This approach works with both server mode and colocate mode, and does not require overriding the `run` method.
#### Approach 2: Overriding the `run` method (fully custom)
A common scenario is for "thinking" models: during real inference the model keeps only the last reasoning step and discards previous ones.
For such cases override the `run` method in your scheduler to return the result for each rollout turn individually.
The built-in `ThinkingModelTipsScheduler` shows how to fully customise multi-turn inference by overriding `run()`.
See the implementation in [multi_turn.py](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py).
**NOTE**: In this scenario, the data for a single trajectory is split into multiple records. When computing rewards, you must assign the same reward to every record that belongs to the same trajectory.
The complete trajectory can be accessed via `trajectory_inputs` in `kwargs`.
For a concrete implementation, see the [MultiTurnThinkingTips class](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py)
### Multimodal Data Override
In multimodal, multi-turn interactions, you may need to dynamically add, delete, or modify multimodal data during the conversation and ensure these changes are synchronized to the trainer.
Implementation: Use `rollout_infos` to override the original multimodal content in the dataset by specifying the corresponding keys.
Supported override keys: images, audios, videos.
For details, see [DeepEyes Scheduler](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py#L403-L404).
### Returning response token IDs
In the default workflow the scheduler returns text, the trainer re-encodes it to token IDs for training.
To avoid this extra encoding, have the scheduler return `response_token_ids` directly.
Steps:
- Read the `token_ids` attribute from `response_choice` to obtain the sequence.
- Include `response_token_ids` in the dict returned by `step` / `run`; the trainer can then use them directly.
For a concrete implementation, refer to the [ThinkingModelTipsScheduler class](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py)
### Loss mask
When the environment or a tool call returns content that becomes part of the model response, you may want to mask it so the model is not penalised on externally generated tokens.
You can set the loss mask in two ways.
**1. Using `loss_scale`**
ms-swift provides the `loss_scale` parameter to scale or mask parts of the response.
For example, `--loss_scale last_round` zeroes out the loss for all but the last round.
Custom `loss_scale` can also be implemented; see the [customisation guide](../../../Customization/Architecture.md#loss-scale).
> Note: In GRPO, `loss_scale` serves only as a mask; it does not scale the loss.
**2. Using `loss_mask`**
In `step` or `run`, set `response_loss_mask` to define a custom mask.
This requires returning `response_token_ids`; the mask must be the same length.
When `response_loss_mask` is provided, `loss_scale` is ignored.
For how to return response_loss_mask, see the [ToolCallScheduler class](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py)
### Reward-function related tips
**Accessing multi-turn rollout information in a reward function**
Return a `rollout_infos` object from `step` / `run`, then read it from `kwargs` in the reward function:
```python
class Scheduler():
def step(self, infer_request: 'RolloutInferRequest', response_choice: 'ChatCompletionResponseChoice',
current_turn: int) -> Dict:
...
return {'infer_request': infer_request, 'rollout_infos': extra_dict}
class RewardFunction():
def __call__(self, completions, **kwargs):
infos = kwargs.get('rollout_infos', {})
...
```
### Accessing additional dataset information in scheduler
Set `--vllm_server_pass_dataset` on the training side to pass other dataset columns to the scheduler.
They can be read from `infer_request.data_dict`.
### Training-Inference-Mismatch
Swift supports returning rollout logprobs from the vLLM side to address training-inference mismatch issues. For details, please refer to this [document](../AdvancedResearch/training_inference_mismatch.md).
In multi-turn training, if `rollout_importance_sampling_mode` is enabled, the framework automatically collects log probabilities from each rollout turn to correct off-policy issues.
**Default Behavior**:
- When using the default `run` method, the framework automatically extracts log probabilities from `response_choice.logprobs`
- These logprobs are passed to the trainer along with `response_token_ids` and `response_loss_mask`
**Notes for Custom Schedulers**:
If you modify the response in your `step` method (e.g., truncation, adding content), you need to return the corresponding `rollout_logprobs`:
**Key Rules**:
- The length of `rollout_logprobs` should equal the count of 1s in `response_loss_mask`
- For tokens with `loss_mask=0` (e.g., user-added prompts, tool return results), no logprobs are needed
- If `step` does not return `rollout_logprobs`, the framework will automatically extract them from `response_choice.logprobs`
**When Overriding the `run` Method**:
If you completely override the `run` method, you need to manually collect and pass `rollout_logprobs`
For implementation, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py)
docs/source_en/Instruction/GRPO/DeveloperGuide/reward_function.md
+144
View File
@@ -0,0 +1,144 @@
# Reward Function
## Custom Reward Function
The reward function takes as arguments (via kwargs) the model-generated completions, other columns from the dataset, and the training state, and calculates a reward score. The [trainer state](https://huggingface.co/docs/transformers/main/main_classes/callback#transformers.TrainerState) includes information such as the current training step.
Note: The columns related to model input (such as query and response) are converted to the messages key. The original assistant response in the dataset will be discarded, so please use extra columns if you wish to retain it.
The relevant column names for processing can be found in the [document](../../../Customization/Custom-dataset.md#Query-Response)
Below is an example illustrating how to implement a simple length-based reward function. This function assigns a reward of 1.0 if the length of the generated completion exceeds 1024, and 0.0 otherwise.
```python
from swift.rewards import ORM, orms
class DummyLengthRewardFunction(ORM)
def __call__(completions, **kwargs):
return [1.0 if len(completion) > 1024 else 0.0 for completion in completions]
orms['dummy']= DummyLengthRewardFunction
```
**Accessing Other Columns in the Dataset**
For example, if the reward function needs to access the solution column from the dataset, as well as the current training step and the total number of steps for calculation, there are two ways to retrieve these values:
Explicitly define the column name in the __call__ parameters:
```python
def __call__(completions, solution, trainer_state, **kwargs):
print(solution)
global_step = trainer_state.global_step
max_steps = trainer_state.max_steps
...
```
Retrieve it from kwargs:
```python
def __call__(completions, **kwargs):
solution = kwargs.get('solution')
trainer_state = kwargs.get('trainer_state')
global_step = trainer_state.global_step
max_steps = trainer_state.max_steps
...
```
**Using Custom Reward Functions**
You can add the reward function in [plugin program](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py), register it using the parameter `--external_plugins examples/train/grpo/plugin/plugin.py`, and specify it via the `reward_funcs` parameter.
For execution scripts, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/run_external_reward_func.sh).
## Async Reward Functions
For reward functions involving I/O operations (such as API calls, database queries, etc.), you can use asynchronous (async) reward functions to improve performance. Async reward functions are executed in parallel using `asyncio.gather`, which can significantly speed up reward computation.
```python
from swift.rewards import AsyncORM, orms
import asyncio
class AsyncAPIReward(AsyncORM):
async def __call__(self, completions, **kwargs):
import aiohttp
async def score_single(session, text):
async with session.post(
'https://api.example.com/score',
json={'text': text}
) as resp:
result = await resp.json()
return result['score']
async with aiohttp.ClientSession() as session:
# Use asyncio.gather to send all requests in parallel
tasks = [score_single(session, c) for c in completions]
rewards = await asyncio.gather(*tasks)
return list(rewards)
orms['async_api'] = AsyncAPIReward
```
Swift supports using both synchronous and asynchronous reward functions simultaneously. The trainer automatically detects the type of reward function:
- Synchronous reward functions are executed sequentially
- Asynchronous reward functions are executed in parallel using `asyncio.gather`
The [plugin](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py) file provides an example of a generative reward model (async_genrm) that calls the `swift deploy` service.
## Built-in Reward Functions
Swift includes five rule-based reward functions (code can be found in swift/rewards/orm.py).
| Reward Function | Paper |
|----------------|----------------------------------------------------------------------------|
| accuracy | [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL](https://arxiv.org/abs/2501.12948) |
| format | Same as above |
| cosine | [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://arxiv.org/abs/2502.03373) |
| repetition | Same as above |
| soft_overlong | [Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO)](https://arxiv.org/abs/2503.14476) |
### 1. **accuracy**
This function compares the model's generated output with the solution column in the dataset to calculate an accuracy score. If the generated output matches the reference answer, the score is 1.0; otherwise, it is 0.0.
Note: This reward function uses the `math_verify` library to parse the generated output and the solution, which may only be applicable to specific mathematical datasets.
### 2. **format**
The paper uses the following system prompt to require the model to return responses in a fixed format:
```
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>
```
This function checks whether the model generates text in the format `<think>think content</think><answer>answer content</answer>`. If the generated text meets the format requirements, the score is 1.0; otherwise, it is 0.0.
### 3. **cosine**
The paper found that using only the accuracy reward function for training could lead to excessively long generated outputs, thereby affecting training effectiveness. The cosine reward function optimizes the training process by controlling the length of the model's outputs:
- For texts with correct answers, the reward decreases as the length increases, encouraging the model to generate concise responses.
- For texts with incorrect answers, the reward increases as the length increases, encouraging the model to think more deeply.
A cosine function is used to smoothly adjust the reward value, ensuring the changes remain within a reasonable range. The parameters of the cosine function include the length of the generated text, the maximum length limit, and the minimum and maximum reward values.
Parameters:
- cosine_min_len_value_wrong (default: -0.5): The reward value for the minimum length when the answer is incorrect.
- cosine_max_len_value_wrong (default: 0.0): The reward value for the maximum length when the answer is incorrect.
- cosine_min_len_value_correct (default: 1.0): The reward value for the minimum length when the answer is correct.
- cosine_max_len_value_correct (default: 0.5): The reward value for the maximum length when the answer is correct.
- cosine_max_len (default equals the model's maximum generation length): The maximum length limit for the generated text.
### 4. **repetition**
Penalizes repetitive content in the model's generated text by detecting repeated n-gram patterns and applying corresponding penalties.
The function splits the generated text into words and extracts n-grams of a specified size (default: 3-grams). By calculating the ratio of unique n-grams to the total number of n-grams, it determines the repetition rate. If the repetition rate is high, a larger negative reward (penalty) is applied. The penalty value is calculated based on the repetition rate and the maximum penalty value (default: -1.0).
Parameters:
- repetition_n_grams (default: 3): The size of n-grams used to detect repetition.
- repetition_max_penalty (default: -1.0): The maximum penalty value, controlling the penalty strength.
### 5. **soft overlong punishment**
Defines a length penalty interval. Within this interval, a linear penalty in the range [-1, 0] is applied.
Parameters:
- soft_max_length: L_max in the paper, the model's maximum generation length, defaulting to max_completion_length.
- soft_cache_length: L_cache in the paper, controlling the length penalty interval, which is [soft_max_length - soft_cache_length, soft_max_length].
## Notes
If a model needs to be loaded in the reward function, the training DeepSpeed plugin (transformers logic) will be used by default. Under Zero3, this may cause the model to fail to perform inference properly. Refer to this [issue](https://github.com/modelscope/ms-swift/issues/4580) to skip the DeepSpeed initialization environment.
docs/source_en/Instruction/GRPO/DeveloperGuide/reward_model.md
+142
View File
@@ -0,0 +1,142 @@
# Reward Model
By default, a reward model refers to a model with a classification head that outputs numeric values, usually called an Output Reward Model (ORM). These models score the outputs from other models and produce a scalar value representing the quality of the model response.
You can load reward models with a classification head using the `reward_models` parameter, or load reward models trained by [reward modeling](../../RLHF.md#rm), and then use the model's logits as rewards.
## Custom Reward Models
For generative reward models, there are two common ways to use them: one is by directly defining the reward model logic inside the Trainer via the `reward_model_plugin`, and then using TransformersEngine for inference; the other is to call an externally deployed model service.
- Using `reward_model_plugin`, the reward model will be embedded within the Trainer and does not require additional computational resources. The advantage of this approach is ease of integration, but generation speed is relatively slow, making it more suitable for small-parameter reward models.
- When deploying reward models externally, you can use commands like `swift deploy` or `vllm serve` to deploy the model service on an independent device to greatly improve inference speed, which is more suitable for large models. However, this approach requires reserving extra hardware resources.
### Internal Plugin
You can flexibly customize the reward model processing logic inside `reward_model_plugin`. This enables implementations such as generative reward models, including:
- Custom model system prompts: define specific instructions and context to guide the evaluation process.
- Handling model interaction history: manage dialog context to allow meaningful and context-aware evaluation.
- Defining custom evaluation metrics: set unique criteria and measures for response evaluation, beyond the default accuracy and relevance checks.
With `reward_model_plugin`, developers can tailor the reward evaluation process for specific application needs. This flexibility allows for more fine-grained and effective reward-based training strategies.
The reward model is called via the plugin's `__call__` method, which takes `inputs` as a parameter. `inputs` contains the messages of model input/output and other columns from the dataset.
```python
def __call__(self, inputs):
print(inputs)
"""
[
{
'messages': [
{'role': 'system', 'content': 'system prompt'},
{'role': 'query', 'content': 'query'},
{'role': 'user', 'content': 'completions1'},
],
'solution': "abc",
},
{
'messages': [
{'role': 'system', 'content': 'system prompt'},
{'role': 'query', 'content': 'query'},
{'role': 'user', 'content': 'completions2'},
],
'solution': "abc",
}
]
"""
```
When using TransformersEngine in the plugin for reward model inference, you only need to construct messages and call the infer interface:
```python
class RMPlugin(DefaultRMPlugin):
def __init__(self, model, template):
super().__init__(model, template)
# initilize TransformersEngine to infer
self.engine = TransformersEngine(self.model, template=self.template, max_batch_size=0)
def __call__(self, inputs):
system_prompt = ...
query = ...
messages = [{'role': 'system', 'content': system_prompt}, {'role': 'query', 'content': query}]
result = self.engine.infer([messages], self.request_config, use_tqdm=False)
rewards = ...
return rewards
```
We provide a simple example of a generative reward model (`GenRMPlugin`) in [rm_plugin.py](https://github.com/modelscope/ms-swift/blob/main/swift/rewards/rm_plugin.py).
You can customize your reward model plugin in [plugin.py](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py) and register it using the `external_plugins` parameter.
Note:
1. In `GRPOTrainer`, the reward_model will be appended to reward_funcs one by one. Therefore, the order of `reward_weights` corresponds to `[reward_funcs, reward_model]`.
2. The default for `reward_model_plugin` is `default`, which uses ORM logic.
3. For models with a large number of parameters, TransformersEngine generation is slow. Please use [external deployment](#external-deployment).
For models like BERT that cannot be loaded by `reward_model`, you can load them inside `reward_function`, see [issue](https://github.com/modelscope/ms-swift/issues/4580).
### External Deployment
This approach does not require the `reward_model_plugin` and can be called directly in the reward function.
First, use the following command to start the model service:
```bash
# Note: Do not overlap deployment devices with training devices
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift deploy \
--model Qwen/Qwen2.5-72B-Instruct \
--vllm_tensor_parallel_size 4
# [INFO:swift] model_list: ['Qwen2.5-72B-Instruct']
# INFO: Started server process [xxxxxx]
# INFO: Waiting for application startup.
# INFO: Application startup complete.
# INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```
In the reward function, initialize the client using the OpenAI library and specify the address and port of the model service. Example:
```python
from openai import OpenAI
class RMReward(ORM):
def __init__(self):
super().__init__()
try:
self.client = OpenAI(
api_key='EMPTY',
base_url='http://127.0.0.1:8000/v1', # 127.0.0.1 if deployed locally
)
self.verify_model_name = self.client.models.list().data[0].id
except Exception as e:
raise RuntimeError('Failed to connect to the model service. Please deploy the model '
"using 'swift deploy' or 'vllm serve'.") from e
def __call__(self, completions, messages, **kwargs) -> List[float]:
rewards = []
for completion, message in zip(completions, messages):
rm_prompt = ... # Construct the prompt for the reward model
chat_response = self.client.chat.completions.create(
model=self.verify_model_name,
messages=[
{
'role': 'system',
'content': 'You are a helpful assistant.'
},
{
'role': 'user',
'content': rm_prompt
},
],
)
response = chat_response.choices[0].message.content.strip()
reward = ... # Extract the reward value from the result
rewards.append(reward)
return rewards
```
docs/source_en/Instruction/GRPO/GetStarted/GRPO.md
+401
View File
@@ -0,0 +1,401 @@
# GRPO
[GRPO (Group Relative Policy Optimization)](https://arxiv.org/abs/2402.03300) leverages intra-group relative advantage calculations to replace the independent value model in the PPO algorithm and directly incorporates KL divergence penalties into the loss function to improve training stability.
## Algorithm Overview
GRPO Objective Function is defined as
$
{\scriptstyle
\begin{aligned}
\mathcal{J}_{G R P O}(\theta) & =\mathbb{E}_{\left[q \sim P(Q),\left\{o_i\right\}_{i=1}^G \sim \pi_{\theta_{o l d}}(O \mid q)\right]} \\
& \frac{1}{G} \sum_{i=1}^G \frac{1}{\left|o_i\right|} \sum_{t=1}^{\left|o_i\right|}\left\{\min \left[\frac{\pi_\theta\left(o_{i, t} \mid q, o_{i,<t}\right)}{\pi_{\theta_{o l d}}\left(o_{i, t} \mid q, o_{i,<t}\right)} \hat{A}_{i, t}, \operatorname{clip}\left(\frac{\pi_\theta\left(o_{i, t} \mid q, o_{i,<t}\right)}{\pi_{\theta_{o l d}}\left(o_{i, t} \mid q, o_{i,<t}\right)}, 1-\varepsilon, 1+\varepsilon\right) \hat{A}_{i, t}\right]-\beta \mathbb{D}_{K L}\left[\pi_\theta| | \pi_{r e f}\right]\right\}
\end{aligned}
}
$
The advantage function is defined as
$
\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
$
<details> <summary>GRPO Algorithm Pseudocode</summary>
```python
# ========== 1. Rollout Generation Phase ==========
prompt = "Question: Which is bigger? 9.11 or 9.9?"
# Generate multiple completions through parallel sampling
completions = rollout_function(
model=current_policy_model,
prompt=prompt,
num_generations=8, # Hyperparameter: number of samples per prompt
temperature=1.0 # Hyperparameter: sampling diversity
)
"""
completions = [
(completion 1) "The larger number is 9.9...",
(completion 2) "9.11 is bigger than...",
...
(completion 8) "After calculation, 9.9..."
]
"""
# ========== 2. Reward Calculation Phase ==========
# Evaluate generated completions using reward model
rewards = reward_function(
completions=completions,
ground_truth="9.9" # Expected correct answer
)
"""
rewards = [
(reward 1) 1.0, # Correct answer
(reward 2) 0.0, # Incorrect
...
(reward 8) 1.0 # Correct
]
"""
# Normalize rewards to advantages
rewards_mean = mean(rewards) # μ = 0.5
rewards_std = std(rewards) # σ = 0.25
advantages = (rewards - rewards_mean) / (rewards_std + 1e-8) # Standardization
"""
advantages = [
(advantage 1) 2.0, # (1.0 - 0.5)/0.25
(advantage 2) -2.0,
...
(advantage 8) 2.0
]
"""
# ========== 3. Policy Optimization Phase ==========
# Get token-level log probabilities from different models
current_logps = get_per_token_logps(current_policy_model, prompt, completions) # π_θ
old_logps = get_per_token_logps(old_policy_model, prompt, completions) # π_θ_old
ref_logps = get_per_token_logps(reference_model, prompt, completions) # π_ref
# PPO Clipped Objective
is_ratio = exp(current_logps - old_logps) # Importance sampling ratio: e^(π_θ - π_θ_old)
clipped_ratio = clip(is_ratio, 1-ε, 1+ε) # ε=0.2 typically
# Policy gradient term (dual form)
policy_loss = -mean(
minimum(is_ratio * advantages, # Unclipped objective
clipped_ratio * advantages) # Clipped objective
)
# KL Divergence Penalty (K3 estimator)
# KL(π_θ||π_ref) ≈ e^(logπ_ref - logπ_θ) - (logπ_ref - logπ_θ) - 1
kl_penalty = beta * mean(
exp(ref_logps - current_logps) -
(ref_logps - current_logps) - 1
)
# Total Loss = Policy Loss + KL Penalty
total_loss = policy_loss + kl_penalty
# ========== 4. Update Rule ==========
# Apply gradient descent to minimize total_loss
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
```
</details>
For training script examples, refer to [examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo).
For GRPO parameters, refer to the [documentation](../../../Instruction/Command-line-parameters.md#grpo-arguments)
## Cluster Support
![](../../../../resources/grpo.png)
The GRPO training framework supports integration with high-performance inference engines (e.g., vLLM) to accelerate the sampling process, offering the following two deployment modes:
### 1. Colocate (Internal) Mode
Training and inference share GPU resources, with the inference service launched internally within the Trainer.
Startup parameters
```bash
--use_vllm true \
--vllm_mode colocate
```
#### Memory Optimization Solutions in Colocate Mode
When running in Colocate mode, out-of-memory (OOM) issues may frequently occur. Below are several effective memory optimization methods and parameter configurations:
1. Reduce the vllm_gpu_memory_utilization parameter.
2. During the training phase, release the GPU memory occupied by vLLM:
```bash
--sleep_level 1
```
3. During the vLLM inference phase, release the GPU memory occupied by the model and optimizer:
```bash
--offload_optimizer true \
--offload_model true \
```
4. Use Tensor Parallelism in vLLM:
```bash
--vllm_tensor_parallel_size [tp_size]
```
5. Gather model weights in batches (when synchronizing vLLM weights under zero3):
```bash
--move_model_batches [批次数量]
```
6. Store Megatron exported HF format weights for vLLM updates in CPU main memory to reduce GPU memory usage:
```bash
--offload_bridge true
```
### 2. Async(External) Mode
Training and inference resources are separated, with a dedicated inference server deployed.
Use the `swift rollout` command to deploy the vLLM server (currently only supports vLLM backend):
```bash
CUDA_VISIBLE_DEVICES=0 \
swift rollout \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--vllm_tensor_parallel_size 2 \
--vllm_data_parallel_size 1
CUDA_VISIBLE_DEVICES=0,1 \
swift rollout \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--vllm_tensor_parallel_size 2 \
--vllm_data_parallel_size 1
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift rollout \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--vllm_tensor_parallel_size 2 \
--vllm_data_parallel_size 2
```
For more rollout parameters, refer to the [vllm arguments](../../../Instruction/Command-line-parameters.md#vllm-arguments) and [rollout arguments](../../../Instruction/Command-line-parameters.md#rollout-arguments)
Note: When set `vllm_use_async_engine`, enabling only DP (Data Parallelism) may cause errors. [Related issue](https://github.com/vllm-project/vllm/issues/18567). If errors occur, try enabling both TP (Tensor Parallelism) and DP or upgrading vLLM.
To configure the external vLLM server during training, use the following parameters:
```bash
--use_vllm true \
--vllm_mode server \
--vllm_server_host <server_IP> \
--vllm_server_port <service_port> \
--vllm_server_timeout <timeout> \
```
### Weight-Sync Acceleration
Setting the following parameters optimizes weight synchronization speed for LoRA training by syncing only the LoRA adapter weights instead of the full model weights.
> Note: This synchronization method may slightly impact vLLM inference speed.
```bash
# rollout(server mode)
swift rollout \
--vllm_enable_lora true \
--vllm_max_lora_rank xxx # match the lora_rank in the training script
...
# grpo(colocate mode)
swift rlhf \
--rlhf_type grpo \
--vllm_mode colocate \
--vllm_enable_lora true \
...
# megatron grpo(colocate mode)
swift megatron rlhf \
--rlhf_type grpo \
--vllm_mode colocate \
--vllm_enable_lora true \
...
```
**Multimodal ViT LoRA Sync:** If ViT LoRA is enabled during training (`freeze_vit false`),
tower/connector LoRA support must also be enabled on the vLLM side.
pass via `vllm_engine_kwargs`:
```bash
--vllm_engine_kwargs '{"enable_tower_connector_lora": true}'
```
This is an experimental vLLM feature, currently supporting models such as Qwen2.5-VL and Qwen3-VL.
For model-specific support details, see the [vLLM documentation](https://docs.vllm.ai/en/latest/features/lora/)
and the [vLLM issue](https://github.com/vllm-project/vllm/issues/31479).
## logged metrics
- completions/mean_length: The average length of generated completions.
- completions/min_length: The minimum length among generated completions.
- completions/max_length: The maximum length among generated completions.
- completions/clipped_ratio: The proportion of completions that were truncated due to length limits.
- reward/{reward_func_name}/mean: The average reward value for a specific reward function.
- reward/{reward_func_name}/std: The standard deviation of the reward for a specific reward function.
> Note: These two metrics are calculated across all completions.
- reward: The overall average reward after applying reward_weights.
- reward_std: The standard deviation of the overall reward within each batch after applying reward_weights.
> Note: These two metrics are first computed within each group and then averaged (for mean/std) across groups.
- frac_reward_zero_std: The proportion of samples in a generation batch where the reward standard deviation is zero, meaning there is almost no diversity in answers for that prompt (i.e., the rewards of all completions are same).
- kl: The average KL divergence between the model and the reference model on completions. This is logged only if beta is nonzero.
- clip_ratio/region_mean: The average proportion of tokens clipped by the CLIP operator across different sentences.
- clip_ratio/low_mean: The average proportion of tokens clipped by the lower CLIP bound across different sentences.
- clip_ratio/low_min: The minimum proportion of tokens clipped by the lower CLIP bound across different sentences.
- clip_ratio/high_mean: The average proportion of tokens clipped by the upper CLIP bound across different sentences.
- clip_ratio/high_max: The maximum proportion of tokens clipped by the upper CLIP bound across different sentences.
> Note: If `overlong_filter` is enabled, the kl and clip_ratio metrics will exclude overlength samples.
If the `log_entropy` parameter is set, additional entropy-related metrics will be logged, including:
- entropy/mean: the average entropy across different sentences
- entropy/max: the maximum entropy among different sentences
- entropy/min: the minimum entropy among different sentences
> Note: Here, sentence entropy refers to the mean entropy of tokens in each completion.
If `top_entropy_quantile` is set to a value smaller than 1.0, the entropy threshold value will also be recorded:
- entropy/threshold: Tokens with entropy below this value will be excluded from the loss calculation.
Training-inference consistency metrics, prefixed with rollout_correction, requires setting `log_rollout_offpolicy_metrics=true` or `rollout_importance_sampling_mode`:
- `kl` / `k3_kl`: KL divergence between training policy and rollout policy (direct estimator / K3 estimator)
- `training_ppl` / `rollout_ppl`: Perplexity of training policy and rollout policy
- `log_ppl_diff`: Log PPL difference, reflects the degree of distribution shift
- `ppl_ratio`: PPL ratio
- `chi2_token` / `chi2_seq`: Token/Sequence-level χ² divergence
IS correction metrics (requires setting `rollout_importance_sampling_mode`):
- `is_weight_mean`: Average importance sampling weight
- `ess`: Effective Sample Size
- `clipped_frac`: Fraction of samples that were truncated or masked
> For detailed explanation of training-inference consistency metrics, please refer to [Training-Inference-Mismatch](../AdvancedResearch/training_inference_mismatch.md)
If `log_completions` is set, the training dynamics will be saved in the output directory, including:
- step: The training step at the time of logging.
- prompt: The model input.
- completion: The model's sampled answer.
- {reward_func_name}: The specific reward(s).
- entropy: The average token entropy (recorded if `log_entropy` is set).
Setting `report_to wandb/swanlab` will send training dynamics table to the respective platform.
If you want to log extra columns in the Table, populate the `metrics_to_gather` dictionary inside `GRPOTrainer._generate_and_score_completions`.
The trainer automatically detects and logs the following keys:
- image: image inputs for vision models(wandb only).
- solution: the solution column from the dataset.
## FAQ
**1. Loss Equals Zero / Approaches Zero / Is Negative During Training**
This is normal behavior. For reference, see [issue](https://github.com/huggingface/open-r1/issues/239#issuecomment-2646297851).
---
**2. num_generations / Batch Size Related**
In GRPO, the batch size is measured in terms of completions (i.e., model-generated outputs). For example, setting `per_device_train_batch_size=8` means that each GPU processes 8 completions for loss calculation during training.
During the training phase, the total effective batch size in a full gradient accumulation step equals:
```python
effective_batch_size = num_processes * per_device_train_batch_size * gradient_accumulation_steps
```
During the sampling phase, the total batch size (completion-level) depends on the following:
- If generation_batch_size is set, the total equals generation_batch_size.
- If steps_per_generation is set, the total equals per_device_train_batch_size * steps_per_generation * num_processes.
- By default, steps_per_generation is set to gradient_accumulation_steps, and generation_batch_size equals the per_device_train_batch_size * steps_per_generation * num_processes = per_device_train_batch_size * gradient_accumulation_steps * num_processes = effective_batch_size.
During evaluation, the number of completions equals:
```
num_processes * per_device_eval_batch_size
```
The parameter `num_generations` must be divisible by the total batch size used in sampling and evaluation to ensure even distribution across devices.
**Example**
- num_processes = 8
- per_device_train_batch_size = 4
- gradient_accumulation_steps = 8
- generation_batch_size = 512
- num_generations = 64
1. Total prompts needed for sampling: 512 / 64 = 8
2. Generate 512 responses from the model per sampling step
3. Model update batch size: 8 * 4 * 8 = 256
**3. Why did KL result in NaN?**
With `overlong_filter` enabled, all completions on a certain GPU were truncated.
**4. How is the training steps calculated?**
Refer to [issue](https://github.com/modelscope/ms-swift/issues/3912).
**5. Why is the clip ratio always 0?**
The core purpose of the clip mechanism is to limit the magnitude of policy updates to prevent policy performance collapse due to excessively large updates (i.e., a drastic decline in performance after policy updates). The specific formula for the clip operation is as follows:
$$
L_{\text{CLIP}}(\theta) = \mathbb{E}_{t} \left[ \min\left(r_{t}(\theta) \hat{A}_{t}, \text{clip}(r_{t}(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_{t} \right) \right]
$$
Where: $r_{t}(\theta) = \frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\text{old}}(a_{t} \mid s_{t})}$ is the importance sampling ratio, which measures the difference between the new and old policy. $\hat{A}_{t}$ is the advantage function, representing the relative return of the action. $\epsilon$ is used to limit the deviation range of $r_{t}(\theta)$.
In the on-policy training process, since each update uses data generated by the latest policy, the new and old policies are the same, i.e., $\pi_{\theta} = \pi_{\text{old}}$.
Thus, the importance sampling ratio is always 1, and the clip operation does not take effect.
The algorithm becomes off-policy (near-on-policy) under the following parameter settings:
1. num_iterations > 1, or
2. gradient_accumulation_steps % steps_per_generation != 0
Refer to [issue](https://github.com/huggingface/open-r1/issues/239#issuecomment-2646297851).
**6. How to set the training `mini-batch size`**
In GRPO training, we can configure mini-batch updates in the following two ways:
- Set `generation_batch_size` to be an integer multiple of the training global batch size (effective_batch_size).
- Or set `steps_per_generation` to be an integer multiple of `gradient_accumulation_steps`.
Typical configuration example:
- When configured with:
steps_per_generation = 16, gradient_accumulation_steps = 8, mini_batch_size = steps_per_generation / gradient_accumulation_steps = 2. The results from 1 rollout will be split into 2 mini-batch updates.
**7. Difference between swift deploy and swift rollout**
- swift deploy is primarily used for model deployment and inference. It supports various engines such as PT, vLLM, and SGLang, and is compatible with streaming inference as well as the OpenAI API format.
- swift rollout, on the other hand, is dedicated to GRPO rollout acceleration. Currently, it only supports the vLLM engine and comes with built-in automatic weight synchronization.
**8. How to disable the KL loss term**
Set the parameter `--beta 0` to disable KL loss calculation. The reference model (ref model) will not be loaded in this case.
## RL WeChat Group
<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/grpo.png" width="250">
docs/source_en/Instruction/GRPO/GetStarted/index.rst
+6
View File
@@ -0,0 +1,6 @@
Get Started
===============
.. toctree::
:maxdepth: 1
GRPO.md
docs/source_en/Instruction/GRPO/index.rst
+19
View File
@@ -0,0 +1,19 @@
GRPO
===============
.. toctree::
:maxdepth: 2
:caption: Get Started
GetStarted/index.rst
.. toctree::
:maxdepth: 2
:caption: Developer Guide
DeveloperGuide/index.rst
.. toctree::
:maxdepth: 2
:caption: Advanced Research
AdvancedResearch/index.rst
docs/source_en/Instruction/Inference-and-deployment.md
+354
View File
@@ -0,0 +1,354 @@
# Inference and Deployment
Below are the inference engines supported by Swift along with their corresponding capabilities. The three inference acceleration engines provide inference acceleration for Swift's inference, deployment, and evaluation modules:
| Inference Acceleration Engine | OpenAI API | Multimodal | Quantized Model | Multiple LoRAs | QLoRA | Batch Inference | Parallel Techniques |
| ------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | --------------- | ------------------------------------------------------------ | ----- | ------------------------------------------------------------ | ------------------- |
| transformers | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/deploy/client/llm/chat/openai_client.py) | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/app/mllm.sh) | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_lora.py) | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/batch_ddp.sh) | DDP/device_map |
| [vllm](https://github.com/vllm-project/vllm) | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_tp.sh) | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/deploy/lora/server.sh) | ❌ | ✅ | TP/PP/DP |
| [sglang](https://github.com/sgl-project/sglang) | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | TP/PP/DP/EP |
| [lmdeploy](https://github.com/InternLM/lmdeploy) | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/lmdeploy/mllm_tp.sh) | ✅ | ❌ | ❌ | ✅ | TP/DP |
## Inference
ms-swift uses a layered design philosophy, allowing users to perform inference through the command-line interface, web UI, or directly using Python.
To view the inference of a model fine-tuned with LoRA, please refer to the [Pre-training and Fine-tuning documentation](./Pre-training-and-Fine-tuning.md#inference-fine-tuned-model).
### Using CLI
**Full Parameter Model:**
```shell
CUDA_VISIBLE_DEVICES=0 swift infer \
--model Qwen/Qwen2.5-7B-Instruct \
--stream true \
--infer_backend transformers \
--max_new_tokens 2048
```
**LoRA Model:**
```shell
CUDA_VISIBLE_DEVICES=0 swift infer \
--model Qwen/Qwen2.5-7B-Instruct \
--adapters swift/test_lora \
--stream true \
--infer_backend transformers \
--temperature 0 \
--max_new_tokens 2048
```
**Command-Line Inference Instructions**
The above commands are for interactive command-line interface inference. After running the script, you can simply enter your query in the terminal. You can also input the following special commands:
- `multi-line`: Switch to multi-line mode, allowing line breaks in the input, ending with `#`.
- `single-line`: Switch to single-line mode, with line breaks indicating the end of input.
- `reset-system`: Reset the system and clear history.
- `clear`: Clear the history.
- `quit` or `exit`: Exit the conversation.
**Multimodal Model**
```shell
CUDA_VISIBLE_DEVICES=0 \
MAX_PIXELS=1003520 \
VIDEO_MAX_PIXELS=50176 \
FPS_MAX_FRAMES=12 \
swift infer \
--model Qwen/Qwen2.5-VL-3B-Instruct \
--stream true \
--infer_backend transformers \
--max_new_tokens 2048
```
To perform inference with a multimodal model, you can add tags like `<image>`, `<video>`, or `<audio>` in your query (representing the location of image representations in `inputs_embeds`). For example, you can input `<image><image>What is the difference between these two images?` or `<video>Describe this video.` Then, follow the prompts to input the corresponding image/video/audio.
Here is an example of inference:
```
<<< <image><image>What is the difference between these two images?
Input an image path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png
Input an image path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png
The first image depicts a cute, cartoon-style kitten with large, expressive eyes and a fluffy white and gray coat. The background is simple, featuring a gradient of colors that highlight the kitten's face.
The second image shows a group of four cartoon-style sheep standing on a grassy field with mountains in the background. The sheep have fluffy white wool, black legs, and black faces with white markings around their eyes and noses. The background includes green hills and a blue sky with clouds, giving it a pastoral and serene atmosphere.
--------------------------------------------------
<<< clear
<<< <video>Describe this video.
Input a video path or URL <<< https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4
A baby wearing glasses is sitting on a bed and reading a book. The baby is holding the book with both hands and is looking down at it. The baby is wearing a light blue shirt and pink pants. The baby is sitting on a white pillow. The baby is looking at the book with interest. The baby is not moving much, just turning the pages of the book.
```
**Dataset Inference:**
```
CUDA_VISIBLE_DEVICES=0 swift infer \
--model Qwen/Qwen2.5-7B-Instruct \
--stream true \
--infer_backend transformers \
--val_dataset AI-ModelScope/alpaca-gpt4-data-zh \
--max_new_tokens 2048
```
The above example provides streaming inference for both full parameters and LoRA, and below are more inference techniques available in SWIFT:
- Interface Inference: You can change `swift infer` to `swift app`.
- Batch Inference: For large models and multimodal models, you can specify `--max_batch_size` for batch inference by using `infer_backend=transformers`. For specific details, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/batch_ddp.sh). Note that you cannot set `--stream true` when performing batch inference.
- DDP/device_map Inference: `infer_backend=transformers` supports parallel inference using DDP/device_map technology. For further details, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/mllm_device_map.sh).
- Inference Acceleration: Swift supports using vllm/sglang/lmdeploy for inference acceleration across the inference, deployment, and evaluation modules by simply adding `--infer_backend vllm/sglang/lmdeploy`. You can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_ddp.sh).
- Multimodal Models: We provide shell scripts for multi-GPU inference for multimodal models using [transformers](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/mllm_device_map.sh), [vllm](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_tp.sh), and [lmdeploy](https://github.com/modelscope/ms-swift/blob/main/examples/infer/lmdeploy/mllm_tp.sh).
- Quantized Models: You can directly select models that are quantized with GPTQ, AWQ, or BNB, for example: `--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4`.
- More Model Types: We also provide inference scripts for [bert](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/bert.sh), [reward_model](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/reward_model.sh), and [prm](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/prm.sh).
**Tips:**
- SWIFT saves inference results, and you can specify the save path using `--result_path`.
- To output log probabilities, simply specify `--logprobs true` during inference. SWIFT will save these results. Note that setting `--stream true` will prevent storage of results.
- Using `infer_backend=transformers` supports inference for all models supported by SWIFT, while `infer_backend=vllm/lmdeploy` supports only a subset of models. Please refer to the documentation for [vllm](https://docs.vllm.ai/en/latest/models/supported_models.html), [sglang](https://docs.sglang.ai/supported_models/generative_models.html) and [lmdeploy](https://lmdeploy.readthedocs.io/en/latest/supported_models/supported_models.html).
- If you encounter OOM when using `--infer_backend vllm`, you can lower `--vllm_max_model_len`, `--vllm_max_num_seqs`, choose an appropriate `--vllm_gpu_memory_utilization`, or set `--vllm_enforce_eager true`. Alternatively, you can address this by using tensor parallelism with `--vllm_tensor_parallel_size`.
- When inferring multimodal models using `--infer_backend vllm`, you need to input multiple images. You can set `--vllm_limit_mm_per_prompt` to resolve this, for example: `--vllm_limit_mm_per_prompt '{"image": 10, "video": 5}'`.
- If you encounter OOM issues while inferring qwen2-vl/qwen2.5-vl, you can address this by setting `MAX_PIXELS`, `VIDEO_MAX_PIXELS`, and `FPS_MAX_FRAMES`. For more information, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/app/mllm.sh).
- SWIFT's built-in dialogue templates align with dialogue templates run using transformers. You can refer to [here](https://github.com/modelscope/ms-swift/blob/main/tests/test_align/test_template/test_vision.py) for testing. If there are any misalignments, please feel free to submit an issue or PR for correction.
### Using Web-UI
If you want to perform inference through a graphical interface, you can refer to the [Web-UI documentation](../GetStarted/Web-UI.md).
### Using Python
**Text Model:**
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
model = 'Qwen/Qwen2.5-0.5B-Instruct'
# Load the inference engine
engine = TransformersEngine(model, max_batch_size=2)
request_config = RequestConfig(max_tokens=512, temperature=0)
# Using 2 infer_requests to demonstrate batch inference
infer_requests = [
InferRequest(messages=[{'role': 'user', 'content': 'Who are you?'}]),
InferRequest(messages=[{'role': 'user', 'content': 'Where is the capital of Zhejiang?'},
{'role': 'assistant', 'content': 'The capital of Zhejiang Province, China, is Hangzhou.'},
{'role': 'user', 'content': 'What are some fun places here?'}]),
]
resp_list = engine.infer(infer_requests, request_config)
query0 = infer_requests[0].messages[0]['content']
print(f'response0: {resp_list[0].choices[0].message.content}')
print(f'response1: {resp_list[1].choices[0].message.content}')
```
**Multimodal Model:**
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
os.environ['MAX_PIXELS'] = '1003520'
os.environ['VIDEO_MAX_PIXELS'] = '50176'
os.environ['FPS_MAX_FRAMES'] = '12'
from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
model = 'Qwen/Qwen2.5-VL-3B-Instruct'
# Load the inference engine
engine = TransformersEngine(model, max_batch_size=2)
request_config = RequestConfig(max_tokens=512, temperature=0)
# Using 3 infer_requests to demonstrate batch inference
infer_requests = [
InferRequest(messages=[{'role': 'user', 'content': 'Who are you?'}]),
InferRequest(messages=[{'role': 'user', 'content': '<image><image> What is the difference between these two images?'}],
images=['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png',
'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']),
InferRequest(messages=[{'role': 'user', 'content': '<video> Describe the video'}],
videos=['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4']),
]
resp_list = engine.infer(infer_requests, request_config)
query0 = infer_requests[0].messages[0]['content']
print(f'response0: {resp_list[0].choices[0].message.content}')
print(f'response1: {resp_list[1].choices[0].message.content}')
print(f'response2: {resp_list[2].choices[0].message.content}')
```
We also provide more demos for Python-based inference:
- For streaming inference using `VllmEngine`, `SglangEngine` and `LmdeployEngine` for inference acceleration, you can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py).
- Multimodal Inference: In addition to the aforementioned multimodal input formats, Swift is compatible with OpenAI's multimodal input format; refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_mllm.py).
- Grounding Tasks: For performing grounding tasks with multimodal models, you can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_grounding.py).
- Multiple LoRA Inference: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_lora.py).
- Agent Inference: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_agent.py).
- Asynchronous Interface: For Python-based inference using `engine.infer_async`, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py).
## Deployment
If you want to see the deployment of a model fine-tuned with LoRA, you can refer to the [Pre-training and Fine-tuning documentation](./Pre-training-and-Fine-tuning.md#deployment-fine-tuned-model).
This section primarily focuses on the deployment and invocation of multimodal models. For text-based large models, we provide a simple deployment and invocation example:
**Server Deployment:**
```shell
CUDA_VISIBLE_DEVICES=0 swift deploy \
--model Qwen/Qwen2.5-7B-Instruct \
--infer_backend vllm \
--max_new_tokens 2048 \
--served_model_name Qwen2.5-7B-Instruct
```
**Client Invocation Test:**
```shell
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2.5-7B-Instruct",
"messages": [{"role": "user", "content": "What should I do if I can’t sleep at night?"}],
"max_tokens": 256,
"temperature": 0
}'
```
### Server Side
```shell
# test env: pip install transformers==4.51.3 vllm==0.8.5.post1
CUDA_VISIBLE_DEVICES=0 \
MAX_PIXELS=1003520 \
VIDEO_MAX_PIXELS=50176 \
FPS_MAX_FRAMES=12 \
swift deploy \
--model Qwen/Qwen2.5-VL-3B-Instruct \
--infer_backend vllm \
--vllm_gpu_memory_utilization 0.9 \
--vllm_max_model_len 8192 \
--max_new_tokens 2048 \
--vllm_limit_mm_per_prompt '{"image": 5, "video": 2}' \
--served_model_name Qwen2.5-VL-3B-Instruct
```
### Client Side
We introduce three methods for invoking the client: using curl, the OpenAI library, and the Swift client.
**Method 1: curl**
```shell
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen2.5-VL-3B-Instruct",
"messages": [{"role": "user", "content": [
{"type": "image", "image": "http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png"},
{"type": "image", "image": "http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png"},
{"type": "text", "text": "What is the difference between these two images?"}
]}],
"max_tokens": 256,
"temperature": 0
}'
```
**Method 2: OpenAI Library**
```python
from openai import OpenAI
client = OpenAI(
api_key='EMPTY',
base_url=f'http://127.0.0.1:8000/v1',
)
model = client.models.list().data[0].id
print(f'model: {model}')
messages = [{'role': 'user', 'content': [
{'type': 'video', 'video': 'https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4'},
{'type': 'text', 'text': 'describe the video'}
]}]
resp = client.chat.completions.create(model=model, messages=messages, max_tokens=512, temperature=0)
query = messages[0]['content']
response = resp.choices[0].message.content
print(f'query: {query}')
print(f'response: {response}')
# Using base64
import base64
import requests
resp = requests.get('https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4')
base64_encoded = base64.b64encode(resp.content).decode('utf-8')
messages = [{'role': 'user', 'content': [
{'type': 'video', 'video': f'data:video/mp4;base64,{base64_encoded}'},
{'type': 'text', 'text': 'describe the video'}
]}]
gen = client.chat.completions.create(model=model, messages=messages, stream=True, temperature=0)
print(f'query: {query}\nresponse: ', end='')
for chunk in gen:
if chunk is None:
continue
print(chunk.choices[0].delta.content, end='', flush=True)
print()
```
**Method 3: Swift Client**
```python
from swift import InferRequest, InferClient, RequestConfig, InferStats
engine = InferClient(host='127.0.0.1', port=8000)
print(f'models: {engine.models}')
metric = InferStats()
request_config = RequestConfig(max_tokens=512, temperature=0)
# Using 3 infer_requests to demonstrate batch inference
# Supports local paths, base64, and URLs
infer_requests = [
InferRequest(messages=[{'role': 'user', 'content': 'Who are you?'}]),
InferRequest(messages=[{'role': 'user', 'content': '<image><image> What is the difference between these two images?'}],
images=['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png',
'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']),
InferRequest(messages=[{'role': 'user', 'content': '<video> Describe the video'}],
videos=['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4']),
]
resp_list = engine.infer(infer_requests, request_config, metrics=[metric])
print(f'response0: {resp_list[0].choices[0].message.content}')
print(f'response1: {resp_list[1].choices[0].message.content}')
print(f'response2: {resp_list[2].choices[0].message.content}')
print(metric.compute())
metric.reset()
# Using base64
import base64
import requests
resp = requests.get('https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4')
base64_encoded = base64.b64encode(resp.content).decode('utf-8')
messages = [{'role': 'user', 'content': [
{'type': 'video', 'video': f'data:video/mp4;base64,{base64_encoded}'},
{'type': 'text', 'text': 'describe the video'}
]}]
infer_request = InferRequest(messages=messages)
request_config = RequestConfig(max_tokens=512, temperature=0, stream=True)
gen_list = engine.infer([infer_request], request_config, metrics=[metric])
print(f'response0: ', end='')
for chunk in gen_list[0]:
if chunk is None:
continue
print(chunk.choices[0].delta.content, end='', flush=True)
print()
print(metric.compute())
```
We also provide more deployment demos:
- Multiple LoRA deployment and invocation: Refer to [this link](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/lora).
- Deployment and invocation of the Base model: Refer to [this link](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/client/llm/base).
- More model types: We provide deployment scripts for [bert](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/bert) and [reward_model](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/reward_model).
docs/source_en/Instruction/Pre-training-and-Fine-tuning.md
+333
View File
@@ -0,0 +1,333 @@
# Pre-training and Fine-tuning
Training Capability:
| Method | Full-Parameter | LoRA | QLoRA | Deepspeed | Multi-Machine | Multimodal |
| ------------------------------------------------------------ | ------------------------------------------------------------ | ---- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| [Pre-training](https://github.com/modelscope/ms-swift/blob/main/examples/train/pretrain) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Supervised Fine-Tuning](https://github.com/modelscope/ms-swift/blob/main/examples/train/lora_sft.sh) | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/full/train.sh) | ✅ | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora) | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu/deepspeed) | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node) | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal) |
| [GRPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [GKD](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/gkd) | ✅ | ✅ | ✅ | ✅ | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/rlhf/gkd) |
| [PPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/ppo) | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| [DPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/dpo) | ✅ | ✅ | ✅ | ✅ | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/rlhf/dpo) |
| [KTO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/kto.sh) | ✅ | ✅ | ✅ | ✅ | ✅ | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/rlhf/kto.sh) |
| [Reward Model](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/rm.sh) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [CPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/cpo.sh) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [SimPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/simpo.sh) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [ORPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/orpo.sh) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Embedding](https://github.com/modelscope/ms-swift/blob/main/examples/train/embedding) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Reranker](https://github.com/modelscope/ms-swift/tree/main/examples/train/reranker) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Sequence Classification](https://github.com/modelscope/ms-swift/blob/main/examples/train/seq_cls) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
## Environment Preparation
Refer to the [SWIFT installation documentation](../GetStarted/SWIFT-installation.md) for recommended versions of third-party libraries.
```shell
pip install ms-swift -U
# If using deepspeed zero2/zero3
pip install deepspeed -U
```
## Pre-training
Pre-training is done using the `swift pt` command, which will automatically use the generative template instead of the conversational template, meaning that `use_chat_template` is set to False (all other commands, such as `swift sft/rlhf/infer`, default `use_chat_template` to True). Additionally, `swift pt` has a different dataset format compared to `swift sft`, which can be referenced in the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
You can refer to the CLI script for pre-training [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/pretrain/train.sh). For more information on training techniques, please refer to the fine-tuning section.
Tips:
- `swift pt` is equivalent to `swift sft --use_chat_template false --loss_scale all`.
## Fine-tuning
ms-swift employs a hierarchical design philosophy, allowing users to perform fine-tuning through the command line interface, Web-UI interface, or directly using Python.
### Using CLI
We provide best practices for self-cognition fine-tuning of Qwen2.5-7B-Instruct on a single 3090 GPU in 10 minutes; for details, refer to [here](../GetStarted/Quick-start.md). This can help you quickly understand SWIFT.
Additionally, we offer a series of scripts to help you understand the training capabilities of SWIFT:
- Lightweight Training: Examples of lightweight fine-tuning supported by SWIFT can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/tuners). (Note: These methods can also be used for pre-training, but pre-training typically uses full parameter training.)
- Distributed Training: SWIFT supports distributed training techniques, including: DDP, device_map, DeepSpeed ZeRO2/ZeRO3, and FSDP.
- device_map: Simplified model parallelism. If multiple GPUs are available, device_map will be automatically enabled. This evenly partitions the model layers across visible GPUs, significantly reducing memory consumption, although training speed may decrease due to serial processing.
- DDP + device_map: Models will be grouped and partitioned using device_map. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/multi-gpu/ddp_device_map/train.sh) for details.
- DeepSpeed ZeRO2/ZeRO3: Save memory resources but may reduce training speed. ZeRO2 shards optimizer states and model gradients. ZeRO3 further shards model parameters on top of ZeRO2, saving even more memory but reducing training speed further. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu/deepspeed) for details.
- FSDP + QLoRA: Training a 70B model on two 3090 GPUs. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu/fsdp_qlora/train.sh).
- Multi-node Multi-GPU Training: We have provided example shell scripts for launching multi-node runs using swift, torchrun, dlc, deepspeed, and accelerate. Except for dlc and deepspeed, the other launch scripts need to be started on all nodes to run properly. Please refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/multi-node) for details.
- Quantization Training: Supports QLoRA training using quantization techniques such as GPTQ, AWQ, AQLM, BNB, HQQ, and EETQ. Fine-tuning a 7B model only requires 9GB of memory. For more details, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora).
- Multi-modal Training: SWIFT supports pre-training, fine-tuning, and RLHF for multi-modal models. It supports tasks such as Captioning, VQA, OCR, and [Grounding](https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2_5-vl-grounding/zh.ipynb). It supports three modalities: images, videos, and audio. For more details, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal). The format for custom multi-modal datasets can be found in the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
- For examples of using full-parameter training for ViT/Aligner, LoRA training for LLM, and employing different learning rates, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/lora_llm_full_vit).
- For multimodal model packing to increase training speed, refer to the example [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/packing).
- RLHF Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf). For multi-modal models, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/rlhf). For GRPO training, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal). For reinforcement fine-tuning, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft).
- Megatron Training: Supports the use of Megatron's parallelization techniques to accelerate the training of large models, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and context parallelism. Refer to the [Megatron-SWIFT Training Documentation](../Megatron-SWIFT/Quick-start.md).
- Sequence Classification Model Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/seq_cls).
- Embedding Model Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/embedding).
- Agent Training: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent).
- Any-to-Any Model Training: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/all_to_all).
- Other Capabilities:
- Streaming Data Reading: Reduces memory usage when handling large datasets. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/streaming/streaming.sh).
- Packing: Combines multiple sequences into one, making each training sample as close to max_length as possible to improve GPU utilization. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/packing).
- Long Text Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel).
- Lazy Tokenize: Performs tokenization during training instead of pre-training (for multi-modal models, this avoids the need to load all multi-modal resources before training), which can reduce preprocessing wait times and save memory. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/streaming/lazy_tokenize.sh).
### Tips:
- When fine-tuning a base model to a chat model using LoRA technology with `swift sft`, you may sometimes need to manually set the template. Add the `--template default` parameter to avoid issues where the base model may fail to stop correctly due to encountering special characters in the dialogue template that it has not seen before. For more details, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/base_to_chat).
- If you need to train in an **offline** environment, please set `--model <model_dir>` and `--check_model false`. If the corresponding model requires `git clone` from GitHub repositories, such as `deepseek-ai/Janus-Pro-7B`, please manually download the repository and set `--local_repo_path <repo_dir>`. For specific parameter meanings, refer to the [command line parameter documentation](./Command-line-parameters.md).
- Merging LoRA for models trained with QLoRA is not possible, so it is not recommended to use QLoRA for fine-tuning, as it cannot utilize vLLM/Sglang/LMDeploy for inference acceleration during inference and deployment. It is recommended to use LoRA or full parameter fine-tuning, merge them into complete weights, and then use GPTQ/AWQ/BNB for [quantization](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize).
- If you are using an NPU for training, simply change `CUDA_VISIBLE_DEVICES` in the shell to `ASCEND_RT_VISIBLE_DEVICES`.
- By default, SWIFT sets `--gradient_checkpointing true` during training to save memory, which may slightly slow down the training speed.
- If you are using DDP for training and encounter the error: `RuntimeError: Expected to mark a variable ready only once.`, please additionally set the parameter `--gradient_checkpointing_kwargs '{"use_reentrant": false}'` or use DeepSpeed for training.
- To use DeepSpeed, you need to install it: `pip install deepspeed -U`. Using DeepSpeed can save memory but may slightly reduce training speed.
- If your machine has high-performance GPUs like A100 and the model supports flash-attn, it is recommended to install [flash-attn](https://github.com/Dao-AILab/flash-attention/releases) and set `--attn_impl flash_attn`, as this will accelerate training and inference while slightly reducing memory usage.
**How to debug:**
You can use the following method for debugging, which is equivalent to using the command line for fine-tuning, but this method does not support distributed training. You can refer to the entry point for the fine-tuning command line [here](https://github.com/modelscope/ms-swift/blob/main/swift/cli/sft.py).
```python
from swift import sft_main, SftArguments
result = sft_main(SftArguments(
model='Qwen/Qwen2.5-7B-Instruct',
tuner_type='lora',
dataset=['AI-ModelScope/alpaca-gpt4-data-zh#500',
'AI-ModelScope/alpaca-gpt4-data-en#500',
'swift/self-cognition#500'],
torch_dtype='bfloat16',
# ...
))
```
### Using Web-UI
If you want to use the interface for training, you can refer to the [Web-UI documentation](../GetStarted/Web-UI.md).
### Using Python
- For the Qwen2.5 self-cognition fine-tuning notebook, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2_5-self-cognition/self-cognition-sft.ipynb).
- For the Qwen2VL OCR task notebook, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2vl-ocr/ocr-sft.ipynb).
## Merge LoRA
- See [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/merge_lora.sh).
## Inference (Fine-Tuned Model)
To perform inference on a LoRA-trained checkpoint using the CLI:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--infer_backend transformers \
--stream true \
--temperature 0 \
--max_new_tokens 2048
```
- The adapters folder contains the trained parameter file `args.json`, so there is no need to specify `--model` or `--system` explicitly; Swift will automatically read these parameters. If you want to disable this behavior, you can set `--load_args false`.
- If you are using full parameter training, please use `--model` instead of `--adapters` to specify the training checkpoint directory. For more information, refer to the [Inference and Deployment documentation](./Inference-and-deployment.md#Inference).
- You can use `swift app` instead of `swift infer` for interactive inference.
- You can choose to merge LoRA (by additionally specifying `--merge_lora true`), and then specify `--infer_backend vllm/sglang/lmdeploy` for inference acceleration.
For batch inference on the validation set of the dataset:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--infer_backend transformers \
--temperature 0 \
--max_new_tokens 2048 \
--load_data_args true \
--max_batch_size 1
```
- You can set `--max_batch_size 8` to enable batch processing with `--infer_backend transformers`. If you use `infer_backend vllm/sglang/lmdeploy`, it will automatically handle batching without needing to specify.
- `--load_data_args true` will additionally read the data parameters from the training storage parameter file `args.json`.
If you want to perform inference on an additional test set instead of using the training validation set, use `--val_dataset <dataset_path>` for inference:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters output/vx-xxx/checkpoint-xxx \
--infer_backend transformers \
--temperature 0 \
--max_new_tokens 2048 \
--val_dataset <dataset-path> \
--max_batch_size 1
```
Example of Inference on LoRA-Trained Model Using Python:
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
from swift import get_model_processor, get_template
from swift.utils import safe_snapshot_download
from peft import PeftModel
# Please adjust the following lines
model = 'Qwen/Qwen2.5-7B-Instruct'
lora_checkpoint = safe_snapshot_download('swift/test_lora') # Change to your checkpoint_dir
template_type = None # None: use the default template_type of the corresponding model
default_system = "You are a helpful assistant." # None: use the default system prompt of the corresponding model
# Load model and dialogue template
model, tokenizer = get_model_processor(model)
if lora_checkpoint is not None:
model = PeftModel.from_pretrained(model, lora_checkpoint)
template_type = template_type or model.model_meta.template
template = get_template(tokenizer, template_type=template_type, default_system=default_system)
engine = TransformersEngine(model, template=template, max_batch_size=2)
request_config = RequestConfig(max_tokens=512, temperature=0)
# Using 2 infer_requests to demonstrate batch inference
infer_requests = [
InferRequest(messages=[{'role': 'user', 'content': 'who are you?'}]),
InferRequest(messages=[{'role': 'user', 'content': 'Where is the capital of Zhejiang?'},
{'role': 'assistant', 'content': 'Where is the capital of Zhejiang?'},
{'role': 'user', 'content': 'What is good to eat here?'},]),
]
resp_list = engine.infer(infer_requests, request_config)
query0 = infer_requests[0].messages[0]['content']
print(f'response0: {resp_list[0].choices[0].message.content}')
print(f'response1: {resp_list[1].choices[0].message.content}')
```
Example of LoRA Inference for Multi-Modal Model:
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
from swift import get_model_processor, get_template
from swift.utils import safe_snapshot_download
from peft import PeftModel
# Please adjust the following lines
model = 'Qwen/Qwen2.5-VL-7B-Instruct'
lora_checkpoint = safe_snapshot_download('swift/test_grounding') # Change to your checkpoint_dir
template_type = None # None: use the default template_type of the corresponding model
default_system = None # None: use the default system prompt of the corresponding model
# Load model and dialogue template
model, tokenizer = get_model_processor(model)
if lora_checkpoint is not None:
model = PeftModel.from_pretrained(model, lora_checkpoint)
template_type = template_type or model.model_meta.template
template = get_template(tokenizer, template_type=template_type, default_system=default_system)
engine = TransformersEngine(model, template=template, max_batch_size=2)
request_config = RequestConfig(max_tokens=512, temperature=0)
# Using 2 infer_requests to demonstrate batch inference
infer_requests = [
InferRequest(messages=[{'role': 'user', 'content': 'who are you?'}]),
InferRequest(messages=[{'role': 'user', 'content': '<image>Task: Object Detection'}],
images=['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']),
]
resp_list = engine.infer(infer_requests, request_config)
query0 = infer_requests[0].messages[0]['content']
print(f'response0: {resp_list[0].choices[0].message.content}')
print(f'response1: {resp_list[1].choices[0].message.content}')
```
If you are using a model trained with ms-swift, you can obtain the training configuration as follows:
```python
from swift import safe_snapshot_download, BaseArguments
lora_adapters = safe_snapshot_download('swift/test_lora')
args = BaseArguments.from_pretrained(lora_adapters)
print(f'args.model: {args.model}')
print(f'args.model_type: {args.model_type}')
print(f'args.template_type: {args.template}')
print(f'args.default_system: {args.system}')
```
- To perform inference on a checkpoint trained with full parameters, set `model` to `checkpoint_dir` and `lora_checkpoint` to `None`. For more information, refer to the [Inference and Deployment documentation](./Inference-and-deployment.md#Inference).
- For streaming inference and acceleration using `VllmEngine`, `SglangEngine` and `LmdeployEngine`, you can refer to the inference examples for [large models](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py) and [multi-modal large models](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_mllm.py).
- For inference on fine-tuned models using the Hugging Face transformers/PEFT ecosystem, you can see [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_hf.py).
- If you have trained multiple LoRAs and need to switch among them, refer to the [inference](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_lora.py) and [deployment](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/lora) examples.
- For grounding tasks in multi-modal models, you can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_grounding.py).
- For inference on a LoRA fine-tuned BERT model, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_bert.py).
## Deployment (Fine-Tuned Model)
Use the following command to start the deployment server. If the weights are trained using full parameters, please use `--model` instead of `--adapters` to specify the training checkpoint directory. You can refer to the client calling methods described in the [Inference and Deployment documentation](./Inference-and-deployment.md#Deployment): curl, OpenAI library, and Swift client.
```shell
CUDA_VISIBLE_DEVICES=0 \
swift deploy \
--adapters output/vx-xxx/checkpoint-xxx \
--infer_backend transformers \
--temperature 0 \
--max_new_tokens 2048 \
--served_model_name '<model-name>'
```
Here, a complete example of deploying and calling multiple LoRAs using vLLM will be provided.
### Server Side
First, you need to install vLLM: `pip install vllm -U`, and use `--infer_backend vllm` when deploying, which can significantly speed up inference.
We pre-trained two base models with different self-awareness LoRA incremental weights for `Qwen/Qwen2.5-7B-Instruct` (which can run successfully). You can find relevant information in [args.json](https://modelscope.cn/models/swift/test_lora/file/view/master). You simply need to modify `--adapters` to specify the local path for the trained LoRA weights during deployment.
```shell
CUDA_VISIBLE_DEVICES=0 \
swift deploy \
--adapters lora1=swift/test_lora lora2=swift/test_lora2 \
--infer_backend vllm \
--temperature 0 \
--max_new_tokens 2048
```
### Client Side
Here, we will only cover calling using the OpenAI library. Examples for calling with curl and the Swift client can be referenced in the [Inference and Deployment documentation](./Inference-and-deployment.md#Deployment).
```python
from openai import OpenAI
client = OpenAI(
api_key='EMPTY',
base_url=f'http://127.0.0.1:8000/v1',
)
models = [model.id for model in client.models.list().data]
print(f'models: {models}')
query = 'who are you?'
messages = [{'role': 'user', 'content': query}]
resp = client.chat.completions.create(model=models[1], messages=messages, max_tokens=512, temperature=0)
query = messages[0]['content']
response = resp.choices[0].message.content
print(f'query: {query}')
print(f'response: {response}')
gen = client.chat.completions.create(model=models[2], messages=messages, stream=True, temperature=0)
print(f'query: {query}\nresponse: ', end='')
for chunk in gen:
if chunk is None:
continue
print(chunk.choices[0].delta.content, end='', flush=True)
print()
"""
models: ['Qwen2.5-7B-Instruct', 'lora1', 'lora2']
query: who are you?
response: I am an artificial intelligence model named swift-robot, developed by swift. I can answer your questions, provide information, and engage in conversation. If you have any inquiries or need assistance, feel free to ask me at any time.
query: who are you?
response: I am an artificial intelligence model named Xiao Huang, developed by ModelScope. I can answer your questions, provide information, and engage in conversation. If you have any inquiries or need assistance, feel free to ask me at any time.
"""
```
docs/source_en/Instruction/RLHF.md
+136
View File
@@ -0,0 +1,136 @@
# RLHF
This document provides training scripts for various human preference alignment algorithms. If you want to learn more about the algorithms and how to choose them, please refer to the [documentation](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/M.%E4%BA%BA%E7%B1%BB%E5%81%8F%E5%A5%BD%E5%AF%B9%E9%BD%90%E8%AE%AD%E7%BB%83.md).
## Dataset
The data required by the PPO and GRPO algorithm consists solely of model inputs, which include the system prompt (optional) and the query. In the case of the GRPO algorithm, the reward function may require additional data columns. For example, to calculate accuracy, a `solution` column is needed as a reference answer.
For RM and DPO-type algorithms such as ORPO, CPO, and SimPO, $(x,y_w,y_l)$ formatted data is required, where $x$ is the model input, $y_w$ is the preferred answer that aligns with human preferences, and $y_l$ is the rejected answer that does not align with human preferences, as shown in ![dpo_data](../../resources/dpo_data.png).
In contrast, the KTO algorithm has a special data format that only requires $(x,y,\text{label})$, where $x$ is the model input, $y$ is the model output, and the label indicates whether the answer aligns with human preferences, as shown in ![kto_data](../../resources/kto_data.png).
For RLHF training of text models or multimodal large models using a custom dataset, you can refer to the [custom dataset documentation](../Customization/Custom-dataset.md#rlhf).
## GRPO
[Paper on arXiv](https://arxiv.org/abs/2402.03300)
Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo).
## DPO
[Paper on arXiv](https://arxiv.org/abs/2305.18290)
Hyperparameters:
- beta: KL regularization coefficient. A larger value imposes a stronger penalty for deviating from the reference model. Default is 0.1.
- loss_type: Variant of the DPO algorithm. You can find the available options in the [documentation](https://huggingface.co/docs/trl/main/en/dpo_trainer#loss-functions). Default is 'sigmoid'.
- (Optional) loss_weights: Weights for mixing multiple loss functions.
- (Optional) ld_alpha: From the [LD-DPO paper](https://arxiv.org/abs/2409.06411). Applies a weight α < 1 to the log-probabilities of tokens that lie beyond the shared prefix of the chosen and rejected responses, thereby mitigating length bias.
- (Optional) discopop_tau: Temperature parameter τ from the [DiscoPOP paper](https://arxiv.org/abs/2406.08414) used to scale the log-ratio before the sigmoid modulation. Default 0.05; only active when loss_type is discopop.
It is recommended to perform SFT training on the preferred responses in your preference dataset before starting DPO training. This helps ensure that the data distribution better matches the requirements of the DPO algorithm.
If you want to mix multiple losses (such as for [MPO](https://arxiv.org/abs/2411.10442) training), you can specify multiple loss_type values and set their weights via loss_weights.
By setting the hyperparameter `rpo_alpha`, a certain proportion of SFT loss can be mixed into the loss to improve training stability.
Training script references:
- [DPO script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/dpo)
- [MPO script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/mpo.sh)
## RM
[Paper on arXiv](https://arxiv.org/abs/2203.02155)
Reward Modeling stage in RLHF.
Use the base model or instruct model trained with SFT as the foundation model. Add a value head and train it using the preference dataset to create the reward model.
The weights of the added value head will be saved in `value_head.safetensors` or `value_head.bin`.
The loss function for reward modeling is as follows:
$
\text{loss} = -\log \sigma \left( r^{(c)} - r^{(r)} - m \right) + \lambda \left( r^{(c)} + r^{(r)} \right)^2
$
- $r^{(c)}$: The score assigned by the model to the chosen response.
- $r^{(r)}$: The score assigned by the model to the rejected response.
- $\lambda$: L2 regularization coefficient that encourages the model outputs to be close to zero. It is set by the parameter `center_rewards_coefficient`, as described in [the paper](https://arxiv.org/pdf/2307.09288), and defaults to 0.
- $m$: Margin term that encourages the model to distinguish between samples of different difficulty levels. The dataset needs to provide a `margin` column for this; by default, it is 0. This term is also introduced in [the paper](https://arxiv.org/pdf/2307.09288).
Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/rm.sh).
## PPO
[Paper on arXiv](https://arxiv.org/abs/2203.02155)
PPO (proximal policy optimization) stage in RLHF involves four models:
- model: The training model, either the base model or the instruct model trained with SFT.
- ref_model: The reference model, which defaults to the model.
- reward_model: The reward model obtained from the RM stage.
- value_model: The value model initialized by the reward model, updated synchronously during training.
Hyperparameters:
- local_rollout_forward_batch_size: Batch size for each data sample, default is 64.
- whiten_rewards: Normalize rewards, default is False.
- kl_coef: Coefficient for the KL divergence term, default is 0.05.
- cliprange: Clip range in the PPO policy loss function, default is 0.2.
- vf_coef: Coefficient for the value loss function, default is 0.1.
- cliprange_value: Clip range in the PPO value loss function, default is 0.2.
- gamma: Discount factor for cumulative rewards, default is 1.0.
- lam: Lambda coefficient in [GAE](https://arxiv.org/abs/1506.02438), default is 0.95.
- num_sample_generations: Number of debugging samples generated during training, default is 10.
Note: When training the base model, perform SFT first and then proceed to RLHF. Specify the chat template, and it is recommended to use `full` for `tuner_type`.
Refer to the [documentation](https://huggingface.co/docs/trl/ppov2_trainer#explanation-of-the-logged-metrics) for metric explanations during training.
## KTO
[Paper on arXiv](https://arxiv.org/abs/2402.01306)
Hyperparameters:
- beta: KL regularization coefficient. A larger value leads to a greater penalty for deviation from the reference model. Default is 0.1.
- desirable_weight: The $\lambda_D$ term in the loss function represents the loss weight for the preferred response samples, with a default value of 1.0.
- undesirable_weight: The $\lambda_U$ term in the loss function represents the loss weight for rejected samples, with a default value of 1.0.
Let $n_D$ and $n_U$ represent the number of preferred and rejected samples in the dataset, respectively. For hyperparameters $\lambda_D$ and $\lambda_U$, the authors recommend setting $\frac{\lambda_D n_D}{\lambda_U n_U} \in [1, \frac{4}{3}]$.
Training script:
Train using data in the $(x,y,\text{label})$ format.
Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/kto.sh).
## CPO
[Paper on arXiv](https://arxiv.org/abs/2401.08417)
Hyperparameters:
- beta: Coefficient before the implicit reward, default is 0.1.
- cpo_alpha: Coefficient for NLL loss, default is 1.0.
Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/cpo.sh).
## ORPO
[Paper on arXiv](https://arxiv.org/abs/2403.07691)
Hyperparameters:
- lambda: Odds Ratio loss coefficient.
Note: ORPO uses the parameter `--beta` to pass the hyperparameter `lambda`.
Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/orpo.sh).
## SimPO
[Paper on arXiv](https://arxiv.org/abs/2405.14734)
Hyperparameters:
- beta: Coefficient before the implicit reward, default is 2.0.
- simpo_gamma: Reward margin term, default is 1.0.
- cpo_alpha: The mixed CPO NLL loss for improving training stability; defaults to 1.0, set to 0.0 to use the original SimPO algorithm.
Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/simpo.sh).
docs/source_en/Instruction/Ray.md
+239
View File
@@ -0,0 +1,239 @@
# Ray Support
## Megatron Ray
The Megatron backend supports GRPO and GKD training via Ray:
| Feature | Example | Assignable Roles |
|---------|---------|------------------|
| megatron grpo | https://github.com/modelscope/ms-swift/tree/main/examples/ray/grpo | train/rollout |
| megatron gkd | https://github.com/modelscope/ms-swift/tree/main/examples/ray/gkd | train/rollout/teacher |
### When to Use
Non-Ray Megatron (`megatron rlhf`) and Ray Megatron (`megatron rlhf --use_ray true`) have identical training functionality.
The core difference is in **deployment**:
- **Non-Ray Megatron**: Launched via torchrun. Inference can use colocate (same process) or server (manually started vLLM server) mode. Multi-node requires manually configuring `MASTER_ADDR/PORT` on each node and separately launching torchrun and vLLM server.
- **Ray Megatron**: A single YAML declares the GPU count for each role (`train.gpus`, `rollout.gpus`, `teacher.gpus`). Ray automatically handles process creation, GPU allocation, and cross-node scheduling — no manual multi-process management needed.
Both support GPU isolation between training and inference (non-Ray via `vllm_mode=server`, Ray via YAML separate mode), making them functionally equivalent. Ray's advantage is automating multi-process orchestration — in multi-node scenarios, it eliminates the operational burden of manually launching torchrun and vLLM server on each node.
**Selection guide:**
| Scenario | Recommendation |
|----------|---------------|
| Single-node training | **Non-Ray** — simpler |
| Multi-node cluster | **Ray** — automatic cross-node scheduling, one YAML to launch |
### Quick Start
```bash
# 1. Start Ray cluster (optional for single node)
ray start --head # head node
ray start --address=<head_ip>:6379 # worker nodes
# 2. Submit training
megatron rlhf --use_ray true --config examples/ray/grpo/ray_grpo_colocate.yaml
```
### GPU Allocation Modes
**Colocate (shared GPU)** — Training and inference share the same GPUs, alternating usage via sleep/wake to free memory:
```yaml
colocate_groups: [[train, rollout]]
offload_model: true
offload_optimizer: true
sleep_level: 1
train:
gpus: 4
rollout:
gpus: 4 # must match train
```
**Separate (dedicated GPU)** — Training and inference occupy separate GPUs with no memory contention:
```yaml
# do not set colocate_groups
train:
gpus: 4
rollout:
gpus: 4 # dedicated 4 GPUs
```
### GKD Teacher Modes
| Mode | Configuration | top-k | full-vocab |
|------|--------------|:-----:|:----------:|
| Colocated teacher | Set `teacher_model` + `offload_teacher_model: true` | ✅ | ✅ |
| Standalone teacher GPU group | Add `teacher:` group with `gpus`, `model` | ✅ | ❌ |
- **Colocated teacher**: The teacher is a Megatron model sharing the same GPUs and parallel parameters as the student, with memory alternated via offload.
- **Standalone teacher GPU group**: The teacher is an independent vLLM inference engine running on separate GPUs, with independently configured parallel parameters (`vllm_tensor_parallel_size`).
- **top-k**: The distillation loss is computed only over the teacher's top-k highest-probability tokens (set via `gkd_logits_topk`). Lower memory usage, but discards long-tail distribution information.
- **full-vocab**: The distillation loss is computed over the entire vocabulary, preserving the full distribution. Higher memory usage.
### Related Documentation
For more details, refer to:
- **GRPO Training**: [Megatron GRPO docs](../Megatron-SWIFT/GRPO.md)
- **GKD Training**: [GKD docs](../Megatron-SWIFT/GKD.md)
- **Megatron Parameters**: [Command-line parameters](../Megatron-SWIFT/Command-line-parameters.md)
- **Megatron Quick Start**: [Quick Start](../Megatron-SWIFT/Quick-start.md)
For detailed configuration and examples, see [examples](https://github.com/modelscope/ms-swift/tree/main/examples/ray).
## Swift Ray
SWIFT's HF Trainer also supports using Ray for multi-GPU or multi-node training:
| Feature | Ray Support | Example | Assignable Roles |
|----------|-------------|--------------------------------------------------------------------------------|------------------|
| pt/sft | ✅ | https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node/ray | default |
| dpo | ❎ | | |
| grpo | ❎ | | |
| ppo | ❎ | | |
| sampling | ✅ | https://github.com/modelscope/ms-swift/tree/main/examples/sampler/distill | sampler/prm/orm |
| distill | ✅ | https://github.com/modelscope/ms-swift/tree/main/examples/sampler/sample | sampler/prm/orm |
### Technical Details
Before describing parameter settings, it's necessary to first explain the technical details. Since SWIFT currently uses many existing implementations from transformers and trl internally, decomposing into different Ray roles like veRL or ROLL is impractical, and decomposition would center around Ray, resulting in poor support for non-Ray scenarios.
Therefore, SWIFT adopts a decorator-based technical approach, defining different roles at the function level. These roles can be defined in parameters to specify how they are used. See the example below:
```python
from swift.ray_utils import RayHelper
@RayHelper.worker(group=['model1', 'model2'])
class MyTrainer:
def __init__(self, args):
self._prepare_model1()
self._prepare_model2()
self._prepare_datasets()
@RayHelper.function(group='model1')
def _prepare_model1(self):
...
@RayHelper.function(group='model2')
def _prepare_model2(self):
...
@RayHelper.function(group='model1')
def rollout(self, inputs):
return self.model1.generate(inputs)
@RayHelper.function(group='model2')
def forward_model2(self, inputs):
loss = self.model2.forward(inputs)
loss.backward()
def _prepare_datasets(self):
self.dataset = ...
def train(self):
for batch in DataLoader(self.dataset):
generated = self.rollout(batch)
self.forward_model2(generated)
...
if __name__ == '__main__':
...
MyTrainer(args).train()
```
RayHelper distributes decorated methods to different hardware clusters, and local calls are smoothly converted to remote calls in the Ray cluster. You can also partition centered around classes:
```python
@RayHelper.worker(group=['model1'])
class Model1:
...
@RayHelper.function(group='model1')
def rollout(self):
...
@RayHelper.worker(group=['model2'])
class Model2:
...
@RayHelper.function(group='model2')
def forward_and_optimize(self):
...
class Trainer:
...
```
SWIFT's support for Ray essentially uses a combination of @worker and @function annotations. The worker specifies Ray cluster roles, and function specifies how to distribute data.
The function annotation has several additional parameters:
```python
@staticmethod
def function(group: str,
dispatch: Union[Literal['slice', 'all'], Callable] = 'all',
execute: Literal['first', 'all'] = 'all',
collect: Union[Literal['none', 'flatten'], Callable] = 'none'):
```
- dispatch: How to distribute call input parameters
- slice: Split the input parameters, meaning workers execute with load balancing
- all: All workers receive identical input parameters
- Custom slicing method, format:
```python
def my_custom_slice(n, i, data):
# n is the number of workers, i is the current worker index, data is the original input parameters
# Return the input parameters for the i-th worker
```
- execute: How to execute
- first: Execute on rank0; slice and Callable slicing methods are invalid in this case
- all: Execute on all
- collect: How to collect returned data
- none: Return as-is, format is a list of return values from each worker
- flatten: Flatten the results returned by workers, supports tuple flattening
- Callable: Custom collect method, format:
```python
def my_custom_collect(result):
# result is a list returned by each worker
# Return in your desired format
```
### Parameter Settings
After explaining the technical details, we can configure the parameters. Developers can set different hardware configurations according to the role list in different processes. For example, in the sampling function, there are three roles: sampler, prm, orm. You can configure them like this:
```yaml
device_groups:
nproc_per_node: 4
sample_group:
device: GPU
ranks: list(range(0, 2))
workers:
- sampler
rm_group:
device: GPU
ranks: list(range(2, 4))
workers:
- prm
- orm
```
- nproc_per_node: The minimum number of GPUs per node required in the Ray cluster.
- xxx_group: The name of each Ray group, can be specified arbitrarily
- device: Device type, currently supports GPU/CPU, etc.
- ranks: Which ranks are allocated to the current group. If CPU, ranks can only be an integer representing the total number of processes needed. If GPU, can be in formats like `[0,1,2,3]`, `4`, `list(range(0, 4))`, etc.
- workers: Which roles are allocated to the current group.
All available roles can be found in the table at the top of this document.
If using the command line, device_groups can also be passed as `--device_groups xxx`, where xxx is a JSON string. For configuration simplicity, we strongly recommend using YAML format in conjunction with Ray.
docs/source_en/Instruction/Reinforced-Fine-tuning.md
+93
View File
@@ -0,0 +1,93 @@
# Reinforced Fine-Tuning
Reinforced fine-tuning is one of the most important functionalities in current model training, with various implementations. SWIFT has already supported the atomic capabilities required for reinforced fine-tuning, such as sampling, reinforcement learning, and fine-tuning. Currently, we provide a specific example of rejection sampling fine-tuning, which can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft/rft.py).
## Concept of Reinforced Fine-Tuning
The concept of reinforced fine-tuning has been proposed since 2022 (or even earlier). Its general workflow typically includes the following steps:
1. Generate data using a specific model or augment the original dataset.
2. Train the target model using the generated data.
3. Repeat the above process if necessary.
**Step 1:**
- If the data-generating model is a larger model, such as GPT, Qwen-Max, DeepSeek-V3/R1, etc., this process can be understood as distillation.
- If the data-generating model is the same model being trained, this can be considered self-improvement fine-tuning.
- If the sampling process involves sampling a batch, fitting the data with KL divergence and rewards, and iterating continuously, it can be classified as on-policy algorithms like PPO or GRPO.
- Sampling algorithms include Monte Carlo sampling, do_sample, group beam search, DVTS, etc.
- The sampling process can incorporate ORM (Outcome Reward Model), PRM (Process Reward Model), diversity filtering, language filtering, etc.
**Step 2:**
- If SFT (Supervised Fine-Tuning) is used, it is referred to as rejection sampling fine-tuning.
- If reinforcement learning is used, it is called reinforcement learning fine-tuning.
**Step 3:**
- If distillation is performed using a larger model (e.g., Monte Carlo sampling distillation with a larger model), the process usually does not involve iterations.
- If the same model is used for sampling or algorithms like PPO are applied, iterations are typically included.
In general, the common approaches to reinforced fine-tuning include:
1. **Distillation**: Sampling high-quality data in bulk from a larger model using methods like Monte Carlo or do_sample, and training a smaller model on this data.
2. **Self-improvement**: Sampling a portion of high-quality data from the same model, filtering it, and training the model iteratively.
3. **On-policy RL**: Using methods like PPO or GRPO for iterative training.
The sampling process is usually much more time-consuming than the training process. If data is distilled using GPT or other large models, token costs must be considered. Thus, reinforced fine-tuning is generally a supplementary mechanism for fine-tuning, except for special cases like DeepSeek-R1.
DeepSeek-R1 uses the GRPO algorithm to enable the emergence of CoT (Chain-of-Thought) capabilities from scratch in a base model. This method requires large-scale cluster support and sufficiently large models for capability emergence. This is not discussed in detail here, but more information can be found in the [paper analysis](https://zhuanlan.zhihu.com/p/19714987272).
Some related papers on reinforced fine-tuning:
- Rejection Sampling Fine-Tuning: https://arxiv.org/pdf/2308.01825
- ReST: https://arxiv.org/pdf/2308.08998
- B-STAR: https://arxiv.org/pdf/2412.17256
- DeepSeekMath: https://arxiv.org/pdf/2402.03300
- Qwen-Math-PRM: https://arxiv.org/pdf/2501.07301
- DeepSeek-R1: https://github.com/deepseek-ai/DeepSeek-R1/tree/main
## When to Use Reinforced Fine-Tuning
Since LLaMA3, we have observed a very noticeable yet rarely mentioned phenomenon: when training an Instruct model using a CoT-enabled training dataset and evaluating it on the corresponding test set, the test set performance tends to degrade. For example, training `llama3.1-8b-instruct` on the GSM8K training set and evaluating the generated checkpoint on the test set reveals performance degradation.
This phenomenon mainly arises from the issue of knowledge forgetting disaster in models. During fine-tuning by model manufacturers, a significant amount of CoT data is often included. When solving mathematical tasks, the model's capability often originates not from the math dataset itself but potentially from datasets like ARC. This inference is supported by [some works](https://zhuanlan.zhihu.com/p/19269451950). Continued training on general tasks disrupts the model's existing capabilities, leading to performance degradation.
However, it is always correct to prioritize fine-tuning. Fine-tuning allows the model to quickly adapt to the dataset distribution at a low cost. Reinforced fine-tuning should be used under the following conditions:
1. The model has already been fine-tuned but does not meet the requirements.
2. Stronger CoT capabilities are needed.
3. Base model training for general capabilities is necessary, and the original dataset no longer improves performance.
4. The output results for corresponding queries can be relatively accurately evaluated, such as tasks with clear results (math, code) or clear processes (translation, style fitting).
Reinforced fine-tuning heavily depends on the accuracy of reward evaluations. If the evaluations are inaccurate, the training may oscillate without progress or even degrade the model performance.
## SWIFT Implementation
SWIFT supports the `sample` command, which is used for model sampling. Currently supported sampling methods include:
- **sample**: Use `generate` do rollout.
We have provided a general [RFT script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft/rft.py). This script supports self-improvement training and allows dynamic adjustments of sampling temperature, PRM thresholds, and other hyperparameters. The training method is flexible (e.g., fine-tuning, DPO) and supports iterative retraining of the original model or continued training from the previous iteration, even loading all training states from the previous iteration. Developers can incorporate additional data filtering (e.g., ensuring rows with the same ID come from the same query), including diversity checks, language filtering, etc.
## Experimental Results
We used the RFT script to train and evaluate the `competition_math` dataset in the math domain. The results are as follows:
| Model | MATH Score | Training Method | Iterations | Post-Training MATH Score |
|----------------------------|------------|-----------------|------------|---------------------------|
| LLaMA3.1_8b | 12.0 | SFT | 3 | 25.2 (LLaMA3.1_8b_sft) |
| LLaMA3.1_8b_sft | 25.2 | RFT | 2 | 32.4 |
| LLaMA3.1_8b_instruct | 52.2 | SFT | 2 | 39.0 |
| LLaMA3.1_8b_instruct | 52.2 | RFT | 3 | 58 |
| Qwen2.5_math_7b_instruct | 79.6 | RFT | 2 | 83.2 |
As shown, applying SFT to the `competition_math` dataset resulted in significant performance degradation for the instruct model. However, RFT improved the model's capabilities, even for the state-of-the-art `Qwen2.5_math_7b_instruct` math model.
Specifically, we tested the GSM8K metric for `Qwen2.5_math_7b_instruct`:
| Model | GSM8K Score | Post-RFT GSM8K Score |
|----------------------------|-------------|-----------------------|
| Qwen2.5_math_7b_instruct | 92.8 | 91.6 |
As shown, RFT training did not significantly change the GSM8K score, avoiding the previously mentioned performance degradation phenomenon.
docs/source_en/Instruction/Sample.md
+100
View File
@@ -0,0 +1,100 @@
# Sampling
Sampling is one of the newly supported key capabilities of SWIFT. This feature can be understood as the practical implementation of `test-time compute`. Additionally, this capability is crucial for the implementation of RFT (Reinforcement Fine-Tuning).
## Capability Introduction
The sampling capability of SWIFT can be demonstrated with the following example:
```shell
swift sample --model LLM-Research/Meta-Llama-3.1-8B-Instruct --sampler_engine transformers --num_return_sequences 5 --dataset AI-ModelScope/alpaca-gpt4-data-zh#5
```
A `jsonl` file with a timestamp as the filename will be generated in the `sample_output` directory of the current folder. This file should contain 25 lines, each representing a complete `messages` format data.
For a list of sampling parameters, please refer to [here](Command-line-parameters.md).
## Environment Setup
```shell
pip install ms-swift[llm] -U
```
Or install swift from source:
```shell
git clone https://github.com/modelscope/ms-swift.git
cd ms-swift
pip install -e '.[llm]'
```
## Using PRM and ORM for Result Filtering
An important capability of sampling is supervising the process and results, which can be supported by setting additional parameters.
```shell
swift sample --model LLM-Research/Meta-Llama-3.1-8B-Instruct --sampler_engine lmdeploy --num_return_sequences 5 --n_best_to_keep 2 --dataset tastelikefeet/competition_math#5 --prm_model AI-ModelScope/GRM-llama3.2-3B-rewardmodel-ft --orm_model math
```
A `jsonl` file with a timestamp as the filename will be generated in the `sample_output` directory of the current folder. This file **will contain at most** 10 lines, each representing a complete `messages` format data.
> The reason it contains at most 10 lines is that although 5 data points are processed in total, and 2 are kept for each data point (`n_best_to_keep`), ORM may fail some validations, and failed data will not be retained in the file.
> Additionally, after adding `--prm_model` or `--orm_model`, the file format is slightly different and includes a `rejected_response` key, which contains the responses with the lowest PRM scores.
## Customizing PRM or ORM
PRM and ORM can be customized by adding a new implementation in the plugin according to the existing code. For example:
```python
class CustomPRM:
# The constructor should be parameterless
def __init__(self):
# Initialize here
pass
def __call__(self, infer_requests: List[InferRequest], ground_truths: List[str], **kwargs) -> List[Union[float, List[float]]]:
...
prms = {'custom': CustomPRM}
```
Afterward, use `--prm_model custom` in the command line.
## Memory Control
If the sampled model and PRM are loaded into memory simultaneously, it may lead to an OOM (Out of Memory) issue. To address this, sampling can be divided into two stages:
- **Stage 1**: Specify `--model` and `--sampler_engine` without specifying `--orm_model` and `--prm_model`. Perform sampling only and save the results to a file.
- **Stage 2**: Specify `--sampler_engine no`, along with `--orm_model` and `--prm_model`, and also specify `--cache_files`. Perform only RM data filtering without re-sampling.
By dividing the process into two stages, only one model is loaded at a time, avoiding OOM issues.
## Practical Example
Please refer to the [Reinforcement Fine-Tuning Script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft/rft.py). This script provides a practical example of using sampling for reinforcement fine-tuning.
> **Note:** The actual effectiveness of this script is strongly related to the quality of the model, data, and RM. Therefore, it is presented only as an example. Users should modify this script and train their own RM and generator models accordingly.
## Sampling From Large Model
SWIFT's sample supports using the OpenAI API to distill data with large models. Example:
```shell
OPENAI_API_KEY="your_api_key" \
swift sample \
--sampler_type distill \
--sampler_engine client \
--model deepseek-r1 \
--stream true \
--dataset tastelikefeet/competition_math#5 \
--num_return_sequences 1 \
--temperature 0.6 \
--top_p 0.95 \
--engine_kwargs '{"base_url":"https://dashscope.aliyuncs.com/compatible-mode/v1"}'
```
In this example:
`base_url` and `model` represent the API endpoint and model name, respectively. `stream` indicates the stream parameter for the request.
Note: For Deepseek-R1 series models, the output will be formatted as:`<thinking>{reasoning_content}</thinking>\n\n<answer>{content}</answer>`.
docs/source_en/Instruction/Supported-models-and-datasets.md
+1383
View File
File diff suppressed because it is too large Load Diff
docs/source_en/Instruction/Use-tuners.md
+122
View File
@@ -0,0 +1,122 @@
# Using Tuners
Tuners refer to additional structural components attached to a model, aimed at reducing the number of training parameters or enhancing training accuracy. The tuners currently supported by SWIFT include:
- LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/abs/2106.09685)
- LoRA+: [LoRA+: Efficient Low Rank Adaptation of Large Models](https://arxiv.org/pdf/2402.12354.pdf)
- LLaMA PRO: [LLAMA PRO: Progressive LLaMA with Block Expansion](https://arxiv.org/pdf/2401.02415.pdf)
- GaLore/Q-GaLore: [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://arxiv.org/abs/2403.03507)
- Liger Kernel: [Liger Kernel: Efficient Triton Kernels for LLM Training](https://arxiv.org/abs/2410.10989)
- LISA: [LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning](https://arxiv.org/abs/2403.17919)
- UnSloth: https://github.com/unslothai/unsloth
- SCEdit: [SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing](https://arxiv.org/abs/2312.11392) < [arXiv](https://arxiv.org/abs/2312.11392) | [Project Page](https://scedit.github.io/) >
- NEFTune: [Noisy Embeddings Improve Instruction Finetuning](https://arxiv.org/abs/2310.05914)
- LongLoRA: [Efficient Fine-tuning of Long-Context Large Language Models](https://arxiv.org/abs/2309.12307)
- Adapter: [Parameter-Efficient Transfer Learning for NLP](http://arxiv.org/abs/1902.00751)
- Vision Prompt Tuning: [Visual Prompt Tuning](https://arxiv.org/abs/2203.12119)
- Side: [Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks](https://arxiv.org/abs/1912.13503)
- Res-Tuning: [Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone](https://arxiv.org/abs/2310.19859) < [arXiv](https://arxiv.org/abs/2310.19859) | [Project Page](https://res-tuning.github.io/) >
- Tuners provided by [PEFT](https://github.com/huggingface/peft), such as AdaLoRA, DoRA, Fourierft, etc.
## Interface List
### Swift Class Static Interfaces
- `Swift.prepare_model(model, config, **kwargs)`
- Function: Loads a tuner into a model. If it is a subclass of `PeftConfig`, it uses the corresponding interface from the Peft library to load the tuner. When using `SwiftConfig`, this interface can accept `SwiftModel` instances and can be called repeatedly, functioning similarly to passing a dictionary of configs.
- This interface supports the parallel loading of multiple tuners of different types for concurrent use.
- Parameters:
- `model`: An instance of `torch.nn.Module` or `SwiftModel`, the model to be loaded.
- `config`: An instance of `SwiftConfig` or `PeftConfig`, or a dictionary of custom tuner names paired with their respective configs.
- Return Value: An instance of `SwiftModel` or `PeftModel`.
- `Swift.merge_and_unload(model)`
- Function: Merges LoRA weights back into the original model and completely unloads the LoRA component.
- Parameters:
- `model`: An instance of `SwiftModel` or `PeftModel` that has had LoRA loaded.
- Return Value: None.
- `Swift.merge(model)`
- Function: Merges LoRA weights back into the original model without unloading the LoRA component.
- Parameters:
- `model`: An instance of `SwiftModel` or `PeftModel` that has had LoRA loaded.
- Return Value: None.
- `Swift.unmerge(model)`
- Function: Splits LoRA weights back from the original model weights into the LoRA structure.
- Parameters:
- `model`: An instance of `SwiftModel` or `PeftModel` that has had LoRA loaded.
- Return Value: None.
- `Swift.save_to_peft_format(ckpt_dir, output_dir)`
- Function: Converts stored LoRA checkpoints to a PEFT-compatible format. Key changes include:
- The `default` will be split from the corresponding `default` folder into the root directory of `output_dir`.
- The `{tuner_name}.` field will be removed from weight keys, e.g., `model.layer.0.self.in_proj.lora_A.default.weight` becomes `model.layer.0.self.in_proj.lora_A.weight`.
- Weight keys will have a `basemodel.model` prefix added.
- Note: Only LoRA can be converted; other types of tuners will raise conversion errors due to PEFT not supporting them. Additionally, due to the presence of extra parameters in LoRAConfig, such as `dtype`, conversion to Peft format is not supported when these parameters are set. In such cases, you can manually delete the corresponding fields in adapter_config.json.
- Parameters:
- `ckpt_dir`: Original weights directory.
- `output_dir`: The target directory for the weights.
- Return Value: None.
- `Swift.from_pretrained(model, model_id, adapter_name, revision, **kwargs)`
- Function: Load the tuner onto the model from the stored weights directory. If `adapter_name` is not provided, all tuners from the `model_id` directory will be loaded. This interface can also be called repeatedly, similar to `prepare_model`.
- Parameters:
- `model`: An instance of `torch.nn.Module` or `SwiftModel` to which the tuner will be loaded.
- `model_id`: A string indicating the tuner checkpoint to be loaded, which can be an ID from the model hub or a local directory.
- `adapter_name`: Can be of type `str`, `List[str]`, `Dict[str, str]`, or `None`. If `None`, all tuners in the specified directory will be loaded. If it is a `str` or `List[str]`, only specific tuners will be loaded. If it is a `Dict`, the key represents the tuner to load, which will be renamed to the corresponding value.
- `revision`: If `model_id` is an ID from the model hub, `revision` can specify the corresponding version number.
### SwiftModel Interfaces
Below is a list of interfaces that users may call. Other internal or less recommended interfaces can be viewed by running the `make docs` command to access the API Doc.
- `SwiftModel.create_optimizer_param_groups(self, **defaults)`
- Function: Creates parameter groups based on the loaded tuners; currently, this only applies to the `LoRA+` algorithm.
- Parameters:
- `defaults`: Default parameters for the `optimizer_groups`, such as `lr` and `weight_decay`.
- Return Value:
- The created `optimizer_groups`.
- `SwiftModel.add_weighted_adapter(self, ...)`
- Function: Merges existing LoRA tuners into one.
- Parameters:
- This interface is a passthrough to `PeftModel.add_weighted_adapter`, and parameters can be referenced in the [add_weighted_adapter documentation](https://huggingface.co/docs/peft/main/en/package_reference/lora#peft.LoraModel.add_weighted_adapter).
- `SwiftModel.save_pretrained(self, save_directory, safe_serialization, adapter_name)`
- Function: Saves tuner weights.
- Parameters:
- `save_directory`: The directory for saving.
- `safe_serialization`: Whether to use safe tensors, default is `False`.
- `adapter_name`: Stored adapter tuner, if not provided, defaults to storing all tuners.
- `SwiftModel.set_active_adapters(self, adapter_names, offload=None)`
- Function: Sets the currently active adapters; adapters not in the list will be deactivated.
- In inference, the environment variable `USE_UNIQUE_THREAD=0/1`, default is `1`. If set to `0`, then `set_active_adapters` only takes effect in the current thread, at which point it defaults to using the tuners activated in this thread, with tuners in different threads not interfering with each other.
- Parameters:
- `adapter_names`: The names of the active tuners.
- `offload`: How to handle deactivated adapters; default is `None`, meaning they remain in GPU memory. Can also use `cpu` or `meta` to offload to CPU or meta device to reduce memory consumption. In `USE_UNIQUE_THREAD=0`, do not pass the `offload` value to avoid affecting other threads.
- Return Value: None.
- `SwiftModel.activate_adapter(self, adapter_name)`
- Function: Activates a tuner.
- In inference, the environment variable `USE_UNIQUE_THREAD=0/1`, default is `1`. If set to `0`, `activate_adapter` will only be effective for the current thread, at which point it defaults to using the tuners activated in this thread, with tuners in different threads not interfering with each other.
- Parameters:
- `adapter_name`: The name of the tuner to be activated.
- Return Value: None.
- `SwiftModel.deactivate_adapter(self, adapter_name, offload)`
- Function: Deactivates a tuner.
- During `inference`, do not call this interface when the `USE_UNIQUE_THREAD=0`.
- Parameters:
- `adapter_name`: The name of the tuner to be deactivated.
- `offload`: How to handle deactivated adapters; defaults to `None`, meaning they remain in GPU memory. Can also use `cpu` or `meta` to offload to CPU or meta device to reduce memory consumption.
- Return Value: None.
- `SwiftModel.get_trainable_parameters(self)`
- Function: Returns information about the trainable parameters.
- Parameters: None.
- Return Value: Information about trainable parameters in the following format:
```text
trainable params: 100M || all params: 1000M || trainable%: 10.00% || cuda memory: 10GiB.
```
docs/source_en/Megatron-SWIFT/Ascend.md
+195
View File
@@ -0,0 +1,195 @@
# Ascend NPU
For environment preparation of Megatron-SWIFT on Ascend NPU, please refer to [NPU Best Practices](../BestPractices/NPU-support.md).
## NPU Performance Data Collection
NPU performance collection is conducted through the `torch_npu.profiler.profile` interface. To begin, create an instance of `torch_npu.profiler.profile`, then use the `start` and `stop` methods to control the performance data collection process. During this process, modifications to the ms-swift source code are required, specifically altering the `train` function in the `swift/megatron/trainers/base.py` file. Below is an example of the collection process:
```python
import torch_npu
...
experimental_config = torch_npu.profiler._ExperimentalConfig(
profiler_level=torch_npu.profiler.ProfilerLevel.Level1,
aic_metrics=torch_npu.profiler.AiCMetrics.PipeUtilization,
)
prof = torch_npu.profiler.profile(
activities=[
torch_npu.profiler.ProfilerActivity.CPU,
torch_npu.profiler.ProfilerActivity.NPU
],
schedule=torch_npu.profiler.schedule(wait=0, warmup=0, active=1, repeat=1, skip_first=6),
on_trace_ready=torch_npu.profiler.tensorboard_trace_handler("./result"),
profile_memory=False, # Close the collection of memory information
with_stack=False, # Close the collection of stack information
experimental_config=experimental_config)
prof.start()
# ms-swift code
while state.iteration < args.train_iters:
...
metric, grad_norm, update_successful = train_step(train_data_iterator)
# collect performance data
prof.step()
...
prof.stop()
```
# NPU Accuracy Data Collection
### Installing msprobe
```shell
pip install mindstudio-probe
```
### Code Modification
To support accuracy debugging with the msprobe tool, we need to modify the `_patch_word_embeddings` function in the `swift/megatron/model/mm_gpt_model.py` file. The main changes are to adjust the function parameters and internal implementation logic so that it can correctly patch the embedding layer.
The specific modification content is as follows:
Before modification:
```python
def _patch_word_embeddings(self, kwargs):
origin_forward = VocabParallelEmbedding.forward
def forward(_self, input_):
args = get_args()
reduce_scatter_embeddings = _self.reduce_scatter_embeddings
_self.reduce_scatter_embeddings = False
input_ = torch.masked_fill(input_, input_ < 0, 0)
res = origin_forward(_self, input_)
_self.reduce_scatter_embeddings = reduce_scatter_embeddings
packed_seq_params = kwargs.get('packed_seq_params')
# ...other logic...
return res
VocabParallelEmbedding.forward = forward
try:
yield
finally:
VocabParallelEmbedding.forward = origin_forward
def forward(
self,
input_ids: torch.Tensor,
position_ids: torch.Tensor,
attention_mask: torch.Tensor = None,
decoder_input: torch.Tensor = None,
labels: torch.Tensor = None,
inference_params: InferenceParams = None,
packed_seq_params: PackedSeqParams = None,
**kwargs,
) -> torch.Tensor:
if decoder_input is not None:
pass
elif self.pre_process:
kwargs.update({'input_ids': input_ids, 'packed_seq_params': packed_seq_params})
with self._patch_word_embeddings(kwargs):
decoder_input = self.language_model.embedding(input_ids=input_ids, position_ids=position_ids)
# ...other logic...
```
After modification:
```python
def _patch_word_embeddings(self, kwargs, emb): # Modification 1
origin_forward = emb.word_embeddings.forward # Modification 2
def forward(input_): # Modification 3
args = get_args()
_self = emb.word_embeddings # Modification 4
reduce_scatter_embeddings = _self.reduce_scatter_embeddings
_self.reduce_scatter_embeddings = False
input_ = torch.masked_fill(input_, input_ < 0, 0)
res = origin_forward(input_) # Modification 5
_self.reduce_scatter_embeddings = reduce_scatter_embeddings
packed_seq_params = kwargs.get('packed_seq_params')
# ...other logic...
return res
emb.word_embeddings.forward = forward # Modification 6
try:
yield
finally:
emb.word_embeddings.forward = origin_forward # Modification 7
def forward(
self,
input_ids: torch.Tensor,
position_ids: torch.Tensor,
attention_mask: torch.Tensor = None,
decoder_input: torch.Tensor = None,
labels: torch.Tensor = None,
inference_params: InferenceParams = None,
packed_seq_params: PackedSeqParams = None,
**kwargs,
) -> torch.Tensor:
if decoder_input is not None:
pass
elif self.pre_process:
kwargs.update({'input_ids': input_ids, 'packed_seq_params': packed_seq_params})
with self._patch_word_embeddings(kwargs, self.language_model.embedding): # Modification 8
decoder_input = self.language_model.embedding(input_ids=input_ids, position_ids=position_ids)
# ...other logic...
```
Major changes include:
1. The `_patch_word_embeddings` method adds an `emb` parameter to receive the embedding module instance
2. Directly obtain `emb.word_embeddings.forward` instead of `VocabParallelEmbedding.forward`
3. The internal `forward` function signature changed from `(_self, input_)` to `(input_)`
4. Get `_self` through `emb.word_embeddings` inside the function
5. Pass `input_` directly when calling the original forward
6. Use `emb.word_embeddings.forward` for replacement and recovery operations (Modifications 6, 7)
7. Pass the `self.language_model.embedding` instance when calling `_patch_word_embeddings`
Modify the train_step function in the file swift/megatron/trainers/base.py
Before modification:
```python
def train_step(self, forward_step_func, data_iterator, model, optimizer, opt_param_scheduler, config, *args,
**kwargs):
new_data_iterator = self._replace_data_iterator(data_iterator, model)
return self._origin_train_step(forward_step_func, new_data_iterator, model, optimizer, opt_param_scheduler,
config, *args, **kwargs)
```
After modification:
```python
def train_step(self, forward_step_func, data_iterator, model, optimizer, opt_param_scheduler, config, *args,
**kwargs):
new_data_iterator = self._replace_data_iterator(data_iterator, model)
from msprobe.pytorch import PrecisionDebugger
debugger = PrecisionDebugger(dump_path='./dump_path', level='mix', model=model)
debugger.start()
try:
origin_train_step_out = self._origin_train_step(
forward_step_func, new_data_iterator, model, optimizer, opt_param_scheduler,config, *args, **kwargs)
finally:
debugger.stop()
debugger.step()
return origin_train_step_out
```
### Enable
Additionally, since msprobe does not support fusion computation, you need to add `--bias_dropout_fusion false`, `--bias_swiglu_fusion false`, `--cross_entropy_loss_fusion false` to the launch script.
#### Example
```shell
PYTORCH_NPU_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--mcore_model Qwen2.5-7B-Instruct-mcore \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
--tensor_model_parallel_size 2 \
...
--bias_dropout_fusion false \
--bias_swiglu_fusion false \
--cross_entropy_loss_fusion false
```
docs/source_en/Megatron-SWIFT/Command-line-parameters.md
+463
View File
@@ -0,0 +1,463 @@
# Command Line Arguments
## Megatron Parameters
**Training Parameters**:
- 🔥micro_batch_size: Batch size per DP group, default is 1.
- 🔥global_batch_size: Total batch size, equivalent to `micro_batch_size * data parallel size * gradient accumulation steps`. Default is 16.
- Here, `Data Parallelism size (DP) = Total number of GPUs / (TP × PP × CP)`.
- 🔥recompute_granularity: Granularity of activation recomputation, options are 'full', 'selective' and 'none'. 'full' means recomputing the entire transformer layer, while 'selective' means only recomputing the core attention part of the transformer layer. 'selective' is generally recommended. Default is 'selective'.
- When you set it to 'selective', you can specify `--recompute_modules` to choose which parts to recompute.
- 🔥recompute_method: This parameter takes effect only when recompute_granularity is set to 'full', options are 'uniform', 'block'. Default is None.
- 🔥recompute_num_layers: This parameter takes effect only when recompute_granularity is set to 'full'. Default is None. If `recompute_method` is set to uniform, this parameter specifies the number of transformer layers in each uniformly divided recomputation unit. For example, you can specify `--recompute_granularity full --recompute_method uniform --recompute_num_layers 4`. The larger the recompute_num_layers, the smaller the memory usage but higher computation cost. Note: The number of model layers in the current process must be divisible by `recompute_num_layers`. Default is None.
- 🔥recompute_modules: Options include "core_attn", "moe_act", "layernorm", "mla_up_proj", "mlp", and "moe". The default value is `["core_attn"]`. This parameter takes effect when `--recompute_granularity selective` is set. For example, during MoE training, you can reduce memory usage by specifying `--recompute_granularity selective --recompute_modules core_attn moe`. Among these, "core_attn", "mlp", and "moe" use normal checkpointing, while "moe_act", "layernorm", and "mla_up_proj" use output-discarding checkpointing.
- "core_attn": Recomputes the core attention part of the Transformer layer.
- "mlp": Recomputes the dense MLP layer.
- "moe": Recomputes the MoE layer.
- "moe_act": Recomputes the MLP activation function part in the MoE module.
- "layernorm": Recomputes the input_layernorm and pre_mlp_layernorm.
- "mla_up_proj": Recomputes the MLA up-projection and RoPE application parts.
- deterministic_mode: Deterministic mode, which may lead to slower training speed, default is False.
- 🔥train_iters: Total number of training iterations, default is None.
- Tip: You can set `--num_train_epochs` to specify the number of training epochs. When using a non-streaming dataset, `train_iters` will be automatically calculated based on the dataset size (compatible with packing).
- 🔥num_train_epochs: Specifies the number of training epochs. When using non-streaming datasets, this parameter will automatically calculate train_iters for you without the need to manually pass `train_iters`. When using streaming datasets, this parameter will force exit the training when `num_train_epochs` is reached, and perform validation and saving of weights. Defaults to None.
- masked_softmax_fusion: Defaults to True. Used to enable the fusion of scaling, masking, and softmax for query_key_value.
- bias_dropout_fusion: Defaults to True. Used to enable the fusion of bias and dropout.
- bias_activation_fusion: If True, fuses bias addition and activation function when possible. Defaults to True.
- apply_rope_fusion: Defaults to False. Used to enable RoPE fusion. This parameter is passed through from megatron-core. Note: RoPE fusion is not supported in all cases, for example: MLA, mrope, etc. are not supported.
- gradient_accumulation_fusion: Defaults to True. Used to enable gradient accumulation fusion.
- 🔥cross_entropy_loss_fusion: Enables cross-entropy loss computation fusion. Defaults to True.
- cross_entropy_fusion_impl: The implementation of cross-entropy loss fusion. Options are `'native'` and `'te'`. Defaults to `'native'`.
- **The default value in `ms-swift>=4.3.1` has been changed from `'te'` to `'native'`**, for the reason see [this PR](https://github.com/NVIDIA/Megatron-LM/pull/5115). This may result in higher GPU memory usage.
- calculate_per_token_loss: Scales the cross-entropy loss according to the number of non-padding tokens in the global batch. Defaults to None. When `task_type` is 'causal_lm' and during pretraining/fine-tuning, it defaults to True; otherwise, it defaults to False.
- 🔥attention_backend: The attention backend to use (flash, fused, unfused, local, auto, flash_2, flash_3, flash_4). Defaults to `flash`.
- If `flash_attention_4/3` is installed, `--attention_backend flash` will prioritize fa4/fa3. To explicitly specify a version, you can set `--attention_backend flash_2/3/4`. For training script examples, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/flash_attention_3). To use flash_attention_4/3 for the ViT part of multimodal models, please set `--attn_impl flash_attention_4/3`.
- Some models may not support flash attention; you need to manually set `--attention_backend unfused/fused --padding_free false`, for example: Llama4, GPT-OSS.
- optimizer: Optimizer type. Options include 'adam', 'sgd', 'muon', and 'dist_muon'. Default is 'adam'.
- Note: This 'adam' is actually 'adamw'. See [here](https://github.com/NVIDIA/TransformerEngine/blob/d8f1e68f7c414f3e7985a8b41de4443b2f819af3/transformer_engine/pytorch/optimizers/fused_adam.py#L69-L70) for reference.
- 'muon' and 'dist_muon' require "megatron-core>=0.16".
- 🔥optimizer_cpu_offload: Offloads optimizer states to the CPU. For example, set: `--use_precision_aware_optimizer true --optimizer_cpu_offload true --optimizer_offload_fraction 0.7`. Defaults to `False`.
- This parameter can significantly reduce GPU memory usage (at the cost of increased CPU memory consumption). When the `global_batch_size` is large, its impact on training speed is minimal.
- 🔥optimizer_offload_fraction: The fraction of the optimizer state to offload to CPU. Default is `1.0`.
- optimizer_cuda_graph: Whether to enable CUDA graph for the optimizer step, forwarded to Megatron's `--optimizer-cuda-graph`. Default is `False`. This parameter requires "megatron-core>=0.17".
- use_precision_aware_optimizer: Use the precision-aware optimizer in TransformerEngine, which allows setting the main parameters and optimizer states to lower precision, such as fp16 and fp8.
- main_grads_dtype: The dtype of main gradients when use_precision_aware_optimizer is enabled. Options are 'fp32' and 'bf16'. Default is 'fp32'.
- main_params_dtype: The dtype of main parameters when use_precision_aware_optimizer is enabled. Options are 'fp32' and 'fp16'. Default is 'fp32'.
- exp_avg_dtype: The dtype of exp_avg (i.e., the first moment in the Adam optimizer) when use_precision_aware_optimizer is enabled. This dtype is used for storing the optimizer state in memory during training, but does not affect the precision in kernel computation. Options are 'fp32', 'fp16', 'bf16', and 'fp8'. Default is 'fp32'.
- exp_avg_sq_dtype: The dtype of exp_avg_sq (i.e., the second moment in the Adam optimizer) when use_precision_aware_optimizer is enabled. This dtype is used for storing the optimizer state in memory during training, but does not affect the precision in kernel computation. Options are 'fp32', 'fp16', 'bf16', and 'fp8'. Default is 'fp32'.
- manual_gc: Disables the default garbage collector and manually triggers garbage collection. Default is False.
- manual_gc_steps: Interval (in steps) to manually trigger garbage collection. Defaults to 0.
- manual_gc_eval: When using manual garbage collection (`--manual_gc true`), disables garbage collection at the beginning and end of each evaluation run. Defaults to True.
**Data Parameters**:
- seed: Random seed for python, numpy, pytorch, and cuda, default is 42.
- dataset_shuffle: Whether to shuffle the dataset. Defaults to True.
- Note: **Megatron-SWIFT's shuffling includes two parts**: dataset shuffling, controlled by `dataset_shuffle`; and shuffling in train_dataloader, controlled by `train_dataloader_shuffle`.
- train_dataloader_shuffle: Whether to use shuffling for train_dataloader. Defaults to True. val_dataset is not shuffled.
- 🔥dataloader_num_workers: Number of workers for the dataloader. Defaults to 4.
- Note: If `--streaming true` is set, it will be set to 1.
- dataloader_pin_memory: Defaults to True.
- dataloader_persistent_workers: Defaults to False.
- Note: This parameter defaults to True in "ms-swift<4.3" and has been changed to False to avoid memory leak issues.
- dataloader_prefetch_factor: Defaults to 2.
- data_sharding: Takes effect on train_dataloader when `--train_dataloader_shuffle true`. Defaults to False. This parameter controls the scope of dataset shuffling. If set to True, the dataset is first sharded, then each shard is shuffled (slightly saves memory); if set to False, the dataset is shuffled first, then sharded (better shuffling effect).
- 🔥group_by_length: Whether to group samples with roughly similar lengths together in the training dataset (with randomness), to minimize padding and ensure load balancing across nodes and processes for improved efficiency. Defaults to False. For the specific algorithm, refer to `transformers.trainer_pt_utils.get_length_grouped_indices`.
- te_rng_tracker: Use the Transformer Engine version of the random number generator. Defaults to False.
- data_parallel_random_init: Enable different random initializations across data parallel ranks. Defaults to False.
- padding_free: Flatten the data in a batch to avoid data padding, thereby reducing memory usage and accelerating training. Defaults to True.
- If you want to customize attention_mask, you can set `--padding_free false`.
- Note: **Megatron-SWIFT training features prioritize support for padding_free format**. Unless there are special circumstances, please do not modify this value.
- mlp_padding_free: Defaults to False. Used for padding_free optimization of mlp when padding_free is set to false. This can improve training speed and reduce memory usage while customizing attention_mask. (Temporarily incompatible with CP)
**Learning Rate Parameters**:
- lr_warmup_init: The initial value for learning rate warmup. The learning rate scheduler starts warming up from this value. Defaults to 0.
- 🔥lr: The initial learning rate. The actual learning rate for each iteration will be determined based on the learning rate warmup and decay strategies. The default value is None; **for full-parameter training, the default is 1e-5, while for LoRA training, the default is 1e-4**.
- lr_decay_style: Learning rate decay strategy. Defaults to 'cosine'. Typically set to 'constant', 'linear', 'cosine', 'inverse-square-root', or 'WSD'.
- 🔥lr_decay_iters: Number of iterations for learning rate decay. Default is None, meaning it will be set to `--train_iters`.
- lr_warmup_iters: Number of iterations for linear learning rate warm-up, default is 0.
- 🔥lr_warmup_fraction: The fraction of the linear learning rate warmup phase, defaults to None.
- 🔥min_lr: Minimum value of the learning rate, clipping any learning rate below this threshold to this value, default is 0.
- Note: The `min_lr` for `vit_lr` is calculated as `vit_lr * min_lr / lr`, and the same applies to `min_lr` for `aligner_lr`.
- lr_wsd_decay_style: The decay method for the WSD annealing phase. Defaults to 'exponential'.
- lr_wsd_decay_iters: The number of iterations for learning rate decay. Defaults to None.
**Regularization Parameters**:
- 🔥weight_decay: Default is 0.1.
- weight_decay_incr_style: The increment function for weight decay. Options are 'constant', 'linear', 'cosine'. Defaults to 'constant'.
- start_weight_decay: The initial weight decay coefficient for L2 regularization.
- end_weight_decay: The weight decay coefficient for L2 regularization at the end of training.
- 🔥clip_grad: L2 gradient clipping, default is 1.0.
- The `grad_norm` printed in logs is the value before clipping.
- adam_beta1: Default is 0.9.
- adam_beta2: Default is 0.95.
- adam_eps: Default is 1e-8.
- sgd_momentum: Takes effect when `--optimizer sgd` is set. Defaults to 0.9.
**Muon Parameters**:
- muon_momentum: Momentum factor for the Muon optimizer. Default is 0.9.
- muon_split_qkv: Whether to split QKV parameters for the Muon optimizer. Default is True.
- muon_use_nesterov: Whether to use Nesterov-style momentum in the internal SGD. Default is False.
- muon_scale_mode: Scale mode for the Muon optimizer. Options include 'spectral', 'unit_rms_norm', and 'shape_scaling'. Default is 'spectral'.
- muon_fp32_matmul_prec: FP32 matrix multiplication precision for Newton-Schulz iteration. Options include 'low', 'medium', and 'high'. Default is 'medium'.
- muon_coefficient_type: Newton-Schulz coefficient type for the Muon optimizer, forwarded to Megatron's `--muon-coefficient-type`. Available options depend on the installed emerging_optimizers version (e.g. 'quintic', 'polar_express', 'simple', 'cans', 'aol', 'deepseekv4', 'cubic5', 'custom'). Default is 'quintic'.
- muon_num_ns_steps: Number of Newton-Schulz steps for the Muon optimizer. Default is 5.
- muon_tp_mode: NS calculation method for tensor model parallel weights. Options include 'blockwise', 'duplicated', and 'distributed'. Default is 'blockwise'.
- muon_extra_scale_factor: Additional scale factor for Muon updates. Default is 1.
- muon_scalar_optimizer: Optimizer for nonlinear parameters (embeddings, biases, norms) when using Muon. Options are 'adam' or 'lion'. Default is 'adam'.
**Checkpoint Parameters**:
- 🔥output_dir: Output directory for checkpoints, default is None. During training, if this parameter is not set, it defaults to `f'megatron_output/{model_suffix}'`, e.g., `'megatron_output/Qwen2.5-7B-Instruct'`.
- Note: **When training on multiple machines, ensure that the save paths on each node point to the same location**. Otherwise, you will need to manually consolidate these weights after training.
- 🔥save_steps: Interval (in steps) for saving checkpoints. Defaults to 500.
- Note: Weights will always be saved at the end of training.
- 🔥no_save_optim: Do not save optimizer, default is False. When performing full-parameter training, this can significantly reduce storage time.
- 🔥no_save_rng: Do not save RNG, default is False.
- 🔥mcore_model: The checkpoint directory to load (mcore storage format). Defaults to None. For information about resuming training from checkpoints, please refer to the description of the `--finetune` parameter.
- megatron-swift recommends directly loading and storing safetensors weights, refer to [mcore-bridge documentation](./Mcore-Bridge.md).
- The difference between `--model` and `--mcore_model`: `--model/--adapters/--ref_model/--ref_adapters` accepts a directory containing safetensors weights, while `--mcore_model/--mcore_adapter/--mcore_ref_model/--mcore_ref_adapter` accepts a directory containing mcore weights.
- `--model/--adapters` does not support loading checkpoint resume states. Therefore, if `--no_save_optim false` is set, additional mcore-format weights will be saved for checkpoint resumption, and you need to use `--mcore_model/--mcore_adapter` to load the resumed checkpoint state. For example: `--model <hf-model> --mcore_adapter <mcore-adapter-ckpt>`. (Using `--model/--adapters` to resume training is also possible, but the optimizer state will not be restored.)
- 🔥no_load_optim: Do not load optimizer, default is False.
- Note: When resuming training from a checkpoint, setting `--no_load_optim false` (i.e., loading the optimizer state) typically consumes significantly more GPU memory than setting `--no_load_optim true` (i.e., skipping the optimizer state).
- 🔥no_load_rng: Do not load RNG, default is False.
- 🔥finetune: Load and fine-tune the model. Does not load the optimizer and random seed state from the checkpoint, and sets the iteration count to 0. Defaults to True.
- Note: For **checkpoint resumption**, you need to set `--mcore_model` (LoRA training requires additionally setting `--mcore_adapter`). If you set `--finetune true`, it will not load the optimizer state and random seed state, will set the iteration count to 0, and will not skip the dataset; if you set `--finetune false`, it will read the iteration count and skip the previously trained dataset amount, and the loading of optimizer state and random seed state is controlled by `--no_load_optim` and `--no_load_rng`.
- Streaming datasets (`--streaming`) are currently not supported for skipping datasets.
- perform_initialization: Initialize the weights. Defaults to False.
- use_cpu_initialization: Initialize weights on the CPU. Defaults to `False`. This option is used during weight conversion between Hugging Face (HF) and MCore formats. The value typically does not need to be modified.
- 🔥async_save: Use asynchronous checkpoint saving. Currently only applicable to the `torch_dist` distributed checkpoint format. Defaults to False.
- 🔥save_total_limit: Maximum number of checkpoints to save. Expired checkpoints will be deleted. Default is None, which saves all checkpoints. This parameter must be set to a number `>=2`. If set to 2, it will save the best checkpoint and the last checkpoint. This parameter is currently not compatible with `async_save`.
- metric_for_best_model: Default is None. For GRPO, defaults to 'reward'; for other cases, defaults to 'loss'.
- greater_is_better: Default is None. When `metric_for_best_model` contains 'loss', it is set to False; otherwise, it is set to True.
- use_persistent_ckpt_worker: Enable a persistent checkpoint worker process for async save. Defaults to False.
- dist_ckpt_save_pre_mcore_014: Save in the format prior to Megatron-Core 0.14. Defaults to False.
- dist_ckpt_optim_fully_reshardable: Make optimizer distributed checkpoint fully reshardable (TP/PP/EP/DP) as opposed to plain DP reshardability. Defaults to False.
- distrib_optim_fully_reshardable_mem_efficient: During distributed optimizer checkpoint save and load, tries to use as little memory as possible by using Gloo (instead of NCCL) and only one rank for saving. Turn on only if experiencing host or device memory issues. Has effect only when `--dist-ckpt-optim-fully-reshardable` flag is set. Defaults to False.
**Distributed Parameters**:
For guidance on selecting parallelization strategies, please refer to the [Training Tips documentation](./Quick-start.md#training-tips).
- ddp_backend: Distributed backend. Options are 'nccl' or 'gloo'. Defaults to nccl.
- ddp_timeout: Defaults to 18000000, in seconds.
- 🔥use_distributed_optimizer: Use a distributed optimizer (i.e., ZeRO-1). Default is True.
- 🔥tensor_model_parallel_size: TP (Tensor Parallelism) size, default is 1.
- 🔥pipeline_model_parallel_size: PP (Pipeline Parallelism) size, default is 1.
- 🔥decoder_first_pipeline_num_layers: The number of Transformer layers in the first pipeline stage of the decoder. Default is None, which means the Transformer layers are evenly distributed across all pipeline stages.
- This parameter is typically used when **the total number of Transformer layers is not divisible by the pipeline parallelism (PP) size**, or when the first pipeline stage (PP stage 0) of a multimodal model consumes excessive GPU memory.
- 🔥decoder_last_pipeline_num_layers: The number of Transformer layers in the last pipeline stage of the decoder. Default is None, which means the Transformer layers are evenly distributed across all pipeline stages.
- overlap_p2p_comm: Overlap pipeline parallel communication with forward and backward blocks in 1F1B. Defaults to True.
- batch_p2p_comm: Use batch_isend_irecv instead of individual isend/irecv calls. Defaults to `None`, and is set to `not args.overlap_p2p_comm`.
- If pipeline parallelism hangs, you can set this to False.
- align_param_gather: When set to True, all PP stages will launch parameter all-gather operations simultaneously. Otherwise, each PP stage will launch independently as needed. Defaults to True.
- 🔥sequence_parallel: Enables sequence parallel optimization; this option takes effect only when `tensor_model_parallel_size` is set. Default is False.
- 🔥context_parallel_size: CP (Context Parallelism) size, default is 1.
- cp_partition_mode: How THD sequence rows are partitioned across context-parallel ranks.
- sequence_packing_scheduler: Scheduler for sequence packing and dynamic context parallel. Optional values are "dp_balanced" and "default_dynamic_cp". dp_balanced: DP-balanced scheduler for sequence packing. default_dynamic_cp: Dynamic-CP scheduler for packed sequence balancing. Defaults to None.
- tp_comm_overlap: Overlap tensor parallel communication with GEMM (General Matrix Multiplication) kernels (to reduce communication time). Default is False.
- 🔥overlap_grad_reduce: Overlap grad reduction operations in DDP (to reduce DP communication time). Default is False.
- 🔥overlap_param_gather: Overlap all-gather of parameters in the distributed optimizer (to reduce DP communication time). Default is False.
- virtual_pipeline_model_parallel_size: The number of virtual pipeline stages per pipeline parallel rank. Defaults to None. VPP parallelism is used to reduce computation bubbles in PP parallelism and improve GPU utilization, but will slightly increase communication overhead.
- microbatch_group_size_per_vp_stage: The number of consecutive microbatches processed by each virtual pipeline stage. Defaults to None, which equals `pipeline_model_parallel_size`.
- 🔥pipeline_model_parallel_layout: A string describing a custom pipeline (pp/vpp) model parallel layout. For example: `"Et*4|(tttt|)*14tmL"`. Here, E, L, t, and m denote the embedding layer, loss layer, Transformer decoder layer, and MTP layer, respectively. Stages are separated by "|". Repeated stages or layers can be expressed using multiplication. Commas are only for cosmetic readability and have no syntactic meaning. The default value is None, indicating that this argument is not used to set the layout.
- This parameter is typically used on heterogeneous GPU clusters.
- 🔥expert_model_parallel_size: The degree of expert parallelism, default is 1.
- 🔥expert_tensor_parallel_size: expert tensor-parallel size. Default is 1.
**Logging Parameters**:
- report_to: Enabled logging backends. Defaults to `['tensorboard']`. Options are 'tensorboard', 'wandb', and 'swanlab'. Login for 'wandb' and 'swanlab' can use `WANDB_API_KEY` and `SWANLAB_API_KEY` environment variables.
- 🔥logging_steps: Interval (in steps) for logging. Defaults to 5.
- tensorboard_dir: Directory where tensorboard logs are written. Defaults to None, which means logs are stored in the `f'{output_dir}/runs'` directory.
- tensorboard_queue_size: Size of the TensorBoard queue for buffering pending events and summaries. When the number of pending items reaches this value, the next call to an "add" method will trigger a flush to disk. The default is 50.
- wandb_project: Wandb project name. Defaults to 'megatron-swift'.
- wandb_exp_name: Wandb experiment name. Defaults to the value of `--output_dir`.
- swanlab_project: Swanlab project name. Defaults to 'megatron-swift'.
- swanlab_exp_name: Swanlab experiment name. Defaults to the value of `--output_dir`.
**Evaluation Parameters**:
- 🔥eval_iters: Number of iterations for evaluation. Defaults to `-1`, in which case an appropriate value is automatically determined based on the size of the validation dataset. **If the validation dataset size is smaller than the global batch size, evaluation will not be performed.** When using streaming datasets, this value must be set manually.
- 🔥eval_steps: Interval (in steps) for evaluation, i.e., how many steps to train before performing evaluation. Defaults to None, which is set to `save_steps`.
**FP8 Parameters**:
- fp8_format: The FP8 format scheme used for FP8 tensors in the forward and backward pass. Options are 'e4m3' and 'hybrid'. Default is None.
- fp8_recipe: The FP8 recipe (algorithm scheme) used for FP8 tensors in the forward and backward pass. Options are 'tensorwise', 'delayed', 'mxfp8', and 'blockwise'. Default is 'delayed'. Note that blockwise fp8 requires CUDA version 12.9 or higher.
- fp8_param_gather: Keep the compute parameter in FP8 (do not use any other intermediate dtype) and perform the parameter all-gather in FP8 format. Default is False.
- Tips: Set this to True if you want to export weights in FP8 format; otherwise, set it to False.
- fp8_amax_history_len: Number of steps for which amax history is recorded per tensor. Default is 1024.
- fp8_amax_compute_algo: Algorithm for computing amax from history. Options are 'most_recent' and 'max'. Default is 'max'.
**FP4 Parameters**:
- fp4_format: The FP4 format scheme for FP4 tensors in forward and backward passes, optionally set to 'e2m1'. Defaults to None.
- fp4_recipe: If set, enables FP4 precision through Transformer Engine. Currently only 'nvfp4' is supported, which uses the NVFP4BlockScaling recipe for Blackwell+ architecture. Default is 'nvfp4'.
- fp4_param_gather: If set, keeps the parameters in FP4 precision to save memory. Note that not all parameters will be converted to FP4; for example, biases will remain unchanged. Default is False.
**Mixed Precision Parameters**:
- fp16: FP16 mode. Defaults to `None`, and will be automatically set based on the model's `torch_dtype`—specifically, `fp16` is set to `True` if `torch_dtype` is `float16` or `float32`. The `torch_dtype` is by default read from `config.json`.
- bf16: BF16 mode. Defaults to `None`, and will be automatically set based on the model's `torch_dtype`—specifically, `bf16` is set to `True` if `torch_dtype` is `bfloat16`
- apply_query_key_layer_scaling: Scales `Q * K^T` by `1 / layer number` (e.g., divide by layer_num for layer_num-th layer). This is helpful for FP16 training. Default is None, meaning that if `--fp16` is used, it will be set to True.
- 🔥attention_softmax_in_fp32: Uses FP32 for computations in attention_mask and softmax. Default is True.
- accumulate_allreduce_grads_in_fp32: Perform gradient accumulation and allreduce operations in fp32 precision. If `--bf16` is enabled and `main_params_dtype` is 'fp32', this is set to True. Otherwise, it defaults to False.
**MoE Parameters**:
- moe_router_load_balancing_type: Determines the load balancing strategy for the router. Options include "aux_loss", "seq_aux_loss", "global_aux_loss", "sinkhorn", and "none". Supports a single strategy or a list of multiple strategies (e.g. `--moe_router_load_balancing_type seq_aux_loss global_aux_loss`). Default value is None. Read from config.json.
- 🔥moe_router_dtype: Data type used for routing computation and expert output weighted averaging. Options are 'none', 'fp32', and 'fp64', which enhances numerical stability, especially when the number of experts is large. When used together with `moe_permute_fusion`, the performance impact is negligible. Default is 'fp32'. 'none' means no change to data type.
- moe_token_dispatcher_type: The type of token dispatcher to use. Options include 'allgather', 'alltoall', 'flex', and 'alltoall_seq'. Default is 'alltoall'.
- moe_enable_deepep: Enable DeepEP for efficient token dispatching and combine in MoE models. Only works with flex token dispatcher by setting `--moe_token_dispatcher_type flex`.
- 🔥moe_grouped_gemm: When each rank contains multiple experts, multiple local GEMM kernels can be launched in parallel streams to improve utilization and performance by using GroupedLinear from TransformerEngine. Default is True.
- 🔥moe_permute_fusion: Fuses token permutation operations during token dispatch. Default is False.
- 🔥moe_aux_loss_coeff: Defaults to 0, meaning the auxiliary loss is not used. Supports a single float or a list of floats corresponding to each strategy in `moe_router_load_balancing_type`. Generally, a higher value leads to worse training performance but more balanced MoE expert utilization. Please choose an appropriate value based on experimental results.
- moe_z_loss_coeff: Scaling coefficient for z-loss. Default is None.
- 🔥moe_shared_expert_overlap: Enables overlap between shared expert computation and the dispatcher. If not enabled, shared expert computation will be performed after routing experts. Only effective when `moe_shared_expert_intermediate_size` is set. Default is False.
- 🔥moe_expert_capacity_factor: Capacity factor for each expert. `None` means no tokens will be dropped. Default is `None`. When `--moe_expert_capacity_factor` is set, tokens exceeding an expert’s capacity will be dropped based on their selection probability. This can **balance the training load and improve training speed** (for example, set it to 1. or 2.).
- moe_pad_expert_input_to_capacity: Pad the input of each expert so that its length aligns with the expert capacity length. Default is `False`. This option only takes effect if `--moe_expert_capacity_factor` is set.
- moe_token_drop_policy: Options are 'probs' and 'position'. Default is 'probs'.
**DSA Parameters**
- dsa_indexer_loss_coeff: Coefficient for the DSA indexer KL divergence loss. Set to 0 to disable indexer loss. Default is `0.`.
- dsa_indexer_use_sparse_loss: Whether to use sparse DSA indexer loss. If True, the indexer loss will be computed using the top-k indices. Default is False.
- apply_dsa_kernel_fusion: Whether to enable fused DSA sparse-attention kernels (FlashMLA + cuDNN DSA). Set to False to fall back to unfused PyTorch implementations. Requires flash_mla and nvidia-cudnn-frontend >= 1.24.0. Default is False. (Requires the Megatron-LM dev branch)
**Deepseek-V4**
- csa_dense_mode: Whether to use dense mode for compressed sparse attention. If `True`, the CSA indexer will be disabled. Defaults to `False`.
- use_fused_mhc: Use cuTile fused kernels for mHC operations. When `True`, attempts to replace the reference mHC modules with fused cuda.tile (cuTile) autograd functions for better performance on supported GPUs. Requires cuTile to be installed; if cuTile is unavailable, the flag is silently reset to `False` and a warning is emitted. Defaults to `False`.
- mhc_recompute_layer_num: Number of layers per MHC recompute block. When set, every `mhc_recompute_layer_num` layers form a recompute block. If `None`, all layers in the transformer block share a single recompute block. Defaults to `None`.
**MTP Parameters**
- mtp_num_layers: Number of Multi-Token Prediction (MTP) layers. MTP extends the prediction scope at each position to multiple future tokens. This MTP implementation uses D sequential modules to sequentially predict D additional tokens. Default is None.
- Note: The value of mtp_num_layers will not be automatically retrieved from config.json and must be set manually. You can refer to the `num_nextn_predict_layers`, `mtp_num_hidden_layers` field in config.json to fill in this value. When using mcore-bridge, MTP weights will be loaded from safetensors files first. If not found, random initialization will be performed.
- mtp_loss_scaling_factor: Scaling factor of Multi-Token Prediction (MTP) loss. We compute the average of MTP losses across all depths, then multiply it by this scaling factor to obtain the overall MTP loss, which serves as an additional training objective. Default is 0.1.
- mtp_decoder_input_detach: Controls whether to stop gradients through decoder_input in the MTP branch. Defaults to False. When enabled, the MTP loss will not back-propagate directly through decoder_input to the embedding/ViT, but will still update the backbone via the hidden_states pathway.
- mtp_shared_weights: Share weights across MTP layers, following the MTP scheme proposed in GLM-5. Defaults to False. For example, you can set `--mtp_num_layers 3 --mtp_shared_weights true`.
**Tuner Parameters**:
- tuner_type: Options include 'lora', 'full', and 'lora_llm'. Defaults to 'full'.
- 'lora_llm' means applying LoRA to the LLM part while using 'full' fine-tuning for the ViT/aligner parts. You can set their respective learning rates using `vit_lr/aligner_lr`.
- 🔥freeze_llm: This argument only takes effect for multimodal models and can be used in both full-parameter and LoRA training, but with different behaviors. In full-parameter training, setting `freeze_llm=True` freezes the LLM component's weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_llm=True` prevents LoRA modules from being added to the LLM part. Default is `False`.
- 🔥freeze_vit: This argument only applies to multimodal models and behaves differently depending on the training mode. In full-parameter training, setting `freeze_vit=True` freezes the ViT (vision transformer) component's weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_vit=True` prevents LoRA modules from being added to the ViT part. Default is `True`.
- Note: **Here, "vit" refers not only to `vision_tower`, but also to `audio_tower`**. For Omni models, if you want to apply LoRA only to `vision_tower` and not `audio_tower`, you can modify [this code](https://github.com/modelscope/ms-swift/blob/a5d4c0a2ce0658cef8332d6c0fa619a52afa26ff/swift/llm/model/model_arch.py#L544-L554).
- 🔥freeze_aligner: This argument only affects multimodal models. In full-parameter training, setting `freeze_aligner=True` freezes the aligner (also known as projector) weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_aligner=True` prevents LoRA modules from being added to the aligner component. Default is `True`.
Full-parameter Training:
- freeze_parameters: Prefixes of parameters to be frozen. Default is `[]`.
- freeze_parameters_regex: Regex expression for parameters to be frozen. Default is `None`.
- freeze_parameters_ratio: The proportion of parameters to freeze from bottom to top. Default is `0`. Setting this to `1` will freeze all parameters; you can set trainable parameters separately using `trainable_parameters`. Except for values 0 or 1, this parameter is incompatible with pipeline parallelism (PP).
- trainable_parameters: Prefixes of additional trainable parameters. Default is `[]`.
- trainable_parameters_regex: Regex expression to match additional trainable parameters. Default is `None`.
LoRA Training:
- mcore_adapter: The path to the adapter weights for loading, used for resuming LoRA training from a checkpoint. The default is None. The method for resuming LoRA training from a checkpoint is the same as for full-parameter training. Please pay attention to the meaning of the `--finetune` parameter.
- 🔥target_modules: Specifies the suffixes of modules to apply LoRA to. For example, you can set it as `--target_modules linear_qkv linear_proj`. The default is `['all-linear']`, which means all linear layers will be set as target modules.
- Note: The behavior of `'all-linear'` differs between LLMs and multimodal LLMs. For standard LLMs, it automatically finds all linear layers except `lm_head` and attaches tuners. **For multimodal LLMs, tuners are by default only attached to the LLM component; this behavior can be controlled via `freeze_llm`, `freeze_vit`, and `freeze_aligner`**.
- Note: If you want to set all router layers as target modules, you can specify `--target_modules all-router ...`. For example: `--target_modules all-router all-linear`.
- The suffix names of Linear layers differ between transformers and Megatron. In Megatron, `linear_proj` represents `o_proj`, `linear_qkv` represents the concatenation of `q_proj, k_proj, v_proj`, `linear_fc1` represents the concatenation of `gate_proj` and `up_proj`, and `linear_fc2` represents `down_proj`.
- 🔥target_regex: Regex expression to specify LoRA modules. Default is `None`. If this value is provided, the `target_modules` parameter will be ignored.
- 🔥modules_to_save: After attaching a tuner, explicitly specifies additional original model modules to participate in training and storage. The default is `[]`. For example, setting `--modules_to_save word_embeddings output_layer` will unfreeze the `word_embeddings` and `output_layer` layers during LoRA training, and the weights of these modules will be saved in the final checkpoint.
- 🔥lora_rank: Default is `8`.
- 🔥lora_alpha: Default is `32`.
- lora_dropout: Default is `0.05`.
- lora_bias: Default is `'none'`. Available options: `'none'`, `'all'`. If you want all biases to be set as trainable, set this to `'all'`.
- use_rslora: Default is `False`. Whether to use `RS-LoRA`.
**Mcore-Bridge Parameters**:
- model: The model_id or model_path of safetensors weights. Default is None. Supports resume training from checkpoint using `--no_load_optim false --no_load_rng false`.
- model_type: Model type. For details, refer to [ms-swift command-line parameters documentation](../Instruction/Command-line-parameters.md).
- 🔥save_safetensors: Defaults to True, whether to directly save as safetensors weights. If `--no_save_optim false` is set, additional mcore format weights and optimizer weights will be saved (also saved in `output_dir`). When resuming from checkpoint, use `--mcore_model/--mcore_adapter/--no_load_optim/--no_load_rng` parameters to load mcore format weights.
- adapters: adapter_id or adapter_path of LoRA incremental weights in safetensors format. Default is `[]`.
- ref_model: model_id or model_path of ref_model safetensors weights. Required when using DPO/GRPO/KTO algorithms with full-parameter training. Default is None, set to `--model`.
- ref_adapters: List of adapter_id or adapter_path of ref_adapters safetensors weights (currently only supports length of 1). Default is `[]`.
- use_hf: Determines whether to use [ModelScope](https://modelscope.cn/) or [HuggingFace](https://huggingface.co/) for downloading models, downloading datasets, and pushing models. Defaults to False (uses ModelScope).
- Note: To access ModelScope internationally, you can use [ModelScope International](https://modelscope.ai/home) by setting the environment variable `MODELSCOPE_DOMAIN='www.modelscope.ai'`.
- hub_token: Hub token. ModelScope hub token can be found [here](https://modelscope.cn/my/myaccesstoken). Default is None.
- merge_lora: Whether to store merged weights. Defaults to None. If `save_safetensors` is set to True, this parameter defaults to `True`; otherwise, it defaults to False. That is, by default, LoRA will be merged when storing in safetensors format; LoRA will not be merged when storing in torch_dist format.
- max_shard_size: Maximum file size for safetensors format storage, defaults to '5GB'.
- 🔥offload_bridge: Use CPU main memory to store HF format weights exported by Megatron for vLLM updates, to reduce GPU memory usage. Defaults to False. (Takes effect in GRPO/GKD algorithms)
**Multimodal Parameters**:
- vit_gradient_checkpointing: Whether to enable gradient checkpointing for the ViT component during multimodal model training. Defaults to `None`, which means it is enabled when `--freeze_vit` is `false`. (**The ViT implementation in Megatron-SWIFT uses the Hugging Face `transformers` library.**)
- vit_gradient_checkpointing_kwargs: Arguments passed to `torch.utils.checkpoint`. For example: set `--vit_gradient_checkpointing_kwargs '{"use_reentrant": false}'`. Defaults to `None`. This parameter only takes effect when `vit_gradient_checkpointing` is enabled.
- vit_attn_impl: When training a multimodal model, sets the `attn_impl` implementation used for the ViT part. Defaults to `'flash_attn'`.
- vit_lr: Specifies the learning rate for the ViT module when training multimodal models. Default is `None`, same as `learning_rate`. Typically used together with `--freeze_vit` and `--freeze_aligner`.
- Note: The "learning rate" printed in the logs is the learning rate of the LLM.
- aligner_lr: Specifies the learning rate for the aligner module in multimodal models. Default is `None`, same as `learning_rate`.
**Other Parameters**:
- megatron_extra_kwargs: Additional arguments passed directly to Megatron (forwarded to the `ModelConfig` class in [mcore-bridge](https://github.com/modelscope/mcore-bridge/blob/78cb9be33ebad69a0d940a2bc4e198f866084b70/src/mcore_bridge/config/model_config.py#L116), which inherits from megatron-core's `TransformerConfig`). Can also be used to override parameters automatically read from `config.json`. Accepts a JSON string. Defaults to None.
- language_model_only: Only train the language model part of the multimodal model, and only load and save the language model part. Defaults to False. (Requires "mcore-bridge>=1.4.3")
- check_model: Check local model files for corruption or modifications and provide prompts. Defaults to True. **If in an offline environment, please set to False**.
- rope_scaling: Parameters related to rope_scaling. Defaults to None. For format reference, see [llama3.1 config.json](https://modelscope.cn/models/LLM-Research/Meta-Llama-3.1-8B-Instruct/file/view/master?fileName=config.json&status=1), pass as a JSON string.
- **Currently the rope_scaling module uses transformers implementation and supports all rope_scaling supported by transformers.**
- apply_wd_to_qk_layernorm: Used for Qwen3-Next/Qwen3.5 full-parameter training to apply weight decay to qk layernorm. Defaults to False.
- linear_decoupled_in_proj: Decouple `in_proj` in linear attention (i.e. GatedDeltaNet) into `in_proj_qkvz` and `in_proj_ba` to support FP8 training, where `in_proj_ba` is still trained in original precision. Refer to [this example](https://github.com/modelscope/ms-swift/blob/main/examples/models/qwen3_5/fp8.sh).
- enable_dft_loss: Whether to use [DFT](https://arxiv.org/abs/2508.05629) (Dynamic Fine-Tuning) loss in SFT training. Defaults to False.
- enable_channel_loss: Enable channel loss. Defaults to `False`. You need to prepare a "channel" field in the dataset, and ms-swift will group and calculate loss based on this field (if the "channel" field is not prepared, it will be classified under the default `None` channel). For dataset format, refer to [channel loss](../Customization/Custom-dataset.md#channel-loss). Channel loss is compatible with techniques such as packing/padding_free/loss_scale.
- 🔥task_type: Defaults to 'causal_lm'. Options are 'causal_lm', 'seq_cls', 'embedding', and 'generative_reranker'.
- num_labels: This parameter needs to be specified for classification models (i.e., `--task_type seq_cls`). Represents the number of labels. Defaults to None.
- problem_type: This parameter needs to be specified for classification models (i.e., `--task_type seq_cls`). Options are 'regression', 'single_label_classification', 'multi_label_classification'. Defaults to None. If the model is reward_model or num_labels is 1, this parameter is 'regression'; otherwise, it is 'single_label_classification'.
- mrl_dims: Dimension configuration for [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) on embedding training. Default is None. Format is `Dict[int, float]` or a JSON string, where the key is the truncated embedding dimension and the value is the corresponding loss weight, e.g. `'{"32": 1.0, "64": 1.0, "128": 1.0}'`. When enabled, the trainer slices `last_hidden_state` to each dimension, applies L2 normalization, and aggregates the per-dimension `loss_type` losses with the configured weights. Only effective when `task_type='embedding'`.
- Note: The maximum supported embedding dimension is determined by `hidden_size` in the model's `config.json`. Any key-value pair whose key is greater than `hidden_size` will be silently ignored.
- 🔥save_strategy: Saving strategy, options are 'steps' and 'epoch'. Defaults to 'steps'. When set to 'epoch', `save_steps` and `eval_steps` are automatically calculated to save at each epoch, so any user-provided values for these arguments are ignored.
- callbacks: Custom trainer callbacks. Defaults to `[]`.
## Training Parameters
Megatron training parameters are inherited from Megatron parameters and basic parameters (**sharing dataset, template, etc. with ms-swift, and also supporting model-specific parameters from ms-swift**). For details on basic parameters, please refer to [here](../Instruction/Command-line-parameters.md#base-arguments). Additionally, the following parameters are included:
- add_version: Adds a directory `<version>-<timestamp>` to `output_dir` to prevent overwriting weights, default is True.
- 🔥create_checkpoint_symlink: Creates additional checkpoint symlinks to facilitate writing automated training scripts. The symlink paths for `best_model` and `last_model` are `f'{output_dir}/best'` and `f'{output_dir}/last'` respectively.
- 🔥packing: Use the `padding_free` method to pack data samples of different lengths into samples of **approximately** uniform length (packing ensures that complete sequences are not split), achieving load balancing across nodes and processes during training (preventing long texts from slowing down short text training), thereby improving GPU utilization and maintaining stable memory usage. When using `--attention_backend flash`, it ensures that different sequences within packed samples remain independent and invisible to each other. This parameter defaults to `False`. Note: **packing will reduce the number of dataset samples, please adjust gradient accumulation steps and learning rate accordingly**.
- Linear-attention scenario: Qwen3.5 and Qwen3-Next also support padding_free/packing, refer to [Qwen3.5 Best Practice](../BestPractices/Qwen3_5-Best-Practice.md).
- packing_length: the length to use for packing. Defaults to None, in which case it is set to max_length.
- packing_num_proc: Number of processes for packing, default is 1. Note that different values of `packing_num_proc` will result in different packed datasets. (This parameter does not take effect during streaming packing). Usually there is no need to modify this value, as packing speed is much faster than tokenization speed.
- streaming: Stream data loading and processing, default is False. (The shuffling of streaming datasets is not thorough, which may lead to severe loss fluctuations.)
- Note: Since the length of a streaming dataset cannot be determined, the `--train_iters` parameter must be set. Also set the `num_train_epochs` parameter to ensure training exits after the specified number of epochs, and to validate and save the model weights accordingly.
- Note: Streaming datasets can skip preprocessing wait time by overlapping preprocessing with training. Preprocessing for streaming datasets is performed only on rank 0 and then synchronized to other processes via data distribution. **This is generally less efficient than the data sharding approach used in non-streaming datasets.** When the training world_size is large, preprocessing and data distribution can become a training bottleneck.
- lazy_tokenize: Whether to use lazy tokenization. If set to `False`, all dataset samples will be tokenized (and for multimodal models, images will be loaded from disk) before training begins. Default is `None`: in LLM training, it defaults to `False`; in MLLM training, it defaults to `True` to save memory.
- new_special_tokens: List of additional special tokens to be added. Default is `[]`. Example usage can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/lora/new_special_tokens.sh).
- Note: You can also pass a `.txt` file path where each line contains one special token.
## RLHF Parameters
In addition to inheriting the training parameters, the following parameters are also supported:
- 🔥rlhf_type: Default is 'dpo'. Currently, 'dpo', 'grpo', 'kto', 'rm', and 'gkd' are available.
- loss_scale: Overrides the `loss_scale` in [basic parameters](../Instruction/Command-line-parameters.md). Default is 'last_round'.
- calculate_per_token_loss: Overrides the Megatron parameter. Default is False.
### DPO Parameters
- mcore_ref_model: Loading path for ref_model. Required when using DPO/GRPO/KTO algorithms with full parameter training. Defaults to None, which sets it to `mcore_model`.
- mcore_ref_adapter: Weight loading path for ref_adapter. Defaults to None. If you want to use LoRA weights generated from SFT for DPO, set `--mcore_adapter sft_ckpt --mcore_ref_adapter sft_ckpt --finetune true` during training. For checkpoint resumption in this scenario, set `--mcore_adapter rlhf_ckpt --mcore_ref_adapter sft_ckpt --finetune false`.
- beta: Has the same meaning as in [TRL](https://huggingface.co/docs/trl/main/en/dpo_trainer#trl.DPOConfig). It controls the degree of deviation from the reference model. A higher beta value indicates less deviation from the reference model. For the IPO loss function (`loss_type="ipo"`), beta is the regularization parameter as mentioned in the [paper](https://huggingface.co/papers/2310.12036). Default is 0.1.
- 🔥rpo_alpha: A parameter from the [RPO paper](https://huggingface.co/papers/2404.19733) that controls the weight of the NLL term (i.e., the SFT loss) in the loss function, where `loss = dpo_loss + rpo_alpha * sft_loss`. The paper recommends setting it to `1.`. The default value is `None`, meaning the SFT loss is not included by default.
- reference_free: Whether to ignore the provided reference model and implicitly use a reference model that assigns equal probability to all responses. Default is `False`.
- label_smoothing: Default is 0.
- f_divergence_type: Default is `reverse_kl`. See the [TRL documentation](https://huggingface.co/docs/trl/main/en/dpo_trainer) for possible values.
- loss_type: Default is `'sigmoid'`. See the [TRL documentation](https://huggingface.co/docs/trl/main/en/dpo_trainer#loss-functions) for possible values.
### KTO Parameters
- mcore_ref_model: same meaning as in DPO.
- mcore_ref_adapter: same meaning as in DPO.
- beta: parameter controlling the deviation from the ref_model. Higher `beta` means less deviation from the ref_model. Default is `0.1`.
- loss_type: default is `'kto'`. See possible values in the TRL docs: https://huggingface.co/docs/trl/main/en/kto_trainer#trl.KTOConfig.loss_type.
- desirable_weight: factor to weight desirable losses to counter imbalance between desirable and undesirable pairs. Default is `1.`.
- undesirable_weight: factor to weight undesirable losses to counter imbalance between desirable and undesirable pairs. Default is `1.`.
### RM Parameters
- center_rewards_coefficient: A coefficient used in reward model (RM) training to incentivize the model to output rewards with zero mean. See this [paper](https://huggingface.co/papers/2312.09244) for details. Recommended value: 0.01.
### GRPO Parameters
- mcore_ref_model: Same meaning as in DPO.
- mcore_ref_adapter: Same meaning as in DPO.
- beta: KL regularization coefficient, default is 0.04. When set to 0, the ref model is not loaded.
- micro_batch_size: Batch size per device, default is 1.
- global_batch_size: Total batch size, equivalent to `micro_batch_size * data parallel size * gradient accumulation steps`. Default is 16.
- steps_per_generation: Number of optimization steps per generation round, i.e., the ratio of sampling batch size to global_batch_size. Default is 1.
- generation_batch_size: Sampling batch size, must be a multiple of global_batch_size. Default equals global_batch_size * steps_per_generation.
- num_generations: Number of samples per prompt, the G value in the paper, default is 8.
- num_generations_eval: Number of generations to sample during evaluation. This allows using fewer generations during evaluation to save computation. If `None`, uses the value of `num_generations`. Default is None.
- reward_funcs: GRPO algorithm reward functions. Options include `accuracy`, `format`, `cosine`, `repetition`, and `soft_overlong`. See swift/rewards/orm.py. You can also customize your own reward functions in the plugin. Default is `[]`.
- reward_weights: Weights for each reward function. Must match the total number of reward functions and reward models. Default is None, meaning all rewards have equal weights of `1.0`.
- Tip: If GRPO training includes `--reward_model`, it is added at the end of the reward functions.
- truncation_strategy: The method to handle inputs exceeding `max_length`. Supported values are `delete` and `left`, representing deletion and left-side truncation respectively. The default is `left`. Note that for multi-modal models, left-side truncation may remove multi-modal tokens and cause a shape mismatch error during model forward. With the delete strategy, over-long or encoding-failed samples are discarded, and new samples are resampled from the original dataset to maintain the intended batch size.
- loss_type: Loss normalization type. Options are `['grpo', 'bnpo', 'dr_grpo']`. Default is `'grpo'`. See this [PR](https://github.com/huggingface/trl/pull/3256#discussion_r2033213348) for details.
- log_completions: Whether to log model-generated content during training. Default is False.
- vllm_mode: vLLM integration mode. Options are `server` and `colocate`. Server mode uses the vLLM server launched by `swift rollout` for sampling, while colocate mode deploys vLLM within the program. When using server mode:
- vllm_mode server parameters:
- vllm_server_host: vLLM server host address. Default is None.
- vllm_server_port: vLLM server port. Default is 8000.
- vllm_server_base_url: Base URL of the vLLM server (e.g., http://local_host:8000). Default is None. When set, host and port settings are ignored.
- vllm_server_timeout: Timeout for connecting to the vLLM server. Default is 240s.
- vllm_server_pass_dataset: Pass additional dataset information to the vLLM server for multi-round training.
- async_generate: Asynchronous rollout to improve training speed. Note: When enabled, sampling uses the model from the previous round update, and multi-round scenarios are not supported. Default is `false`.
- SWIFT_UPDATE_WEIGHTS_BUCKET_SIZE: Environment variable for controlling the bucket size during weight synchronization. Applicable to full-parameter training in Server Mode. Unit is MB, default value is 512 MB.
- vllm_mode colocate parameters (for more parameter support, refer to [vLLM parameters](#vllm-parameters)):
- vllm_gpu_memory_utilization: vLLM passthrough parameter. Default is 0.9.
- vllm_max_model_len: vLLM passthrough parameter. Default is None.
- vllm_enforce_eager: vLLM passthrough parameter. Default is False.
- vllm_limit_mm_per_prompt: vLLM passthrough parameter. Default is None.
- vllm_enable_prefix_caching: vLLM passthrough parameter. Default is True.
- vllm_tensor_parallel_size: Tensor parallel size. Default is `1`.
- vllm_enable_lora: Support loading LoRA adapters in the vLLM Engine. Default is False. Used to accelerate weight synchronization in LoRA training. See [documentation](../Instruction/GRPO/GetStarted/GRPO.md#weight-synchronization-acceleration) for details.
- sleep_level: Release vLLM GPU memory during training. Options are `[0, 1, 2]`. Default is 0, meaning no release.
- offload_optimizer: Whether to offload optimizer parameters during vLLM inference. Default is False.
- offload_model: Whether to offload the model during vLLM inference. Default is False.
- num_iterations: Number of updates per data sample, the $\mu$ value in the [GRPO paper](https://arxiv.org/abs/2402.03300). Default is 1.
- epsilon: Clip coefficient. Default is 0.2.
- epsilon_high: Upper clip coefficient. Default is None. When set, together with epsilon, forms the clipping range `[epsilon, epsilon_high]`.
- dynamic_sample: Filter out data with zero reward standard deviation within groups and sample additional new data. Default is False.
- max_resample_times: Limit the number of resampling times under dynamic_sample setting. Default is 3.
- overlong_filter: Skip overlong truncated samples, which do not participate in loss calculation. Default is False.
- delta: Bilateral GRPO upper bound clipping value from the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291). If set, it is recommended to be greater than 1 + epsilon. Default is None.
- importance_sampling_level: Controls importance sampling ratio calculation. Options are `token` and `sequence`. In `token` mode, the original log probability ratio for each token is preserved. In `sequence` mode, the log probability ratios of all valid tokens in the sequence are averaged. The [GSPO paper](https://arxiv.org/abs/2507.18071) uses sequence-level calculation to stabilize training. Default is `token`.
- scale_rewards: Specifies the reward scaling strategy. Options include `group` (scale by within-group standard deviation), `batch` (scale by batch-wide standard deviation), `none` (no scaling), and `gdpo` (normalize each reward function separately within groups before weighted aggregation, see [GDPO paper](https://arxiv.org/abs/2601.05242)). The default value is bound to `advantage_estimator`: `grpo` corresponds to `group`, `rloo` corresponds to `none`, and `reinforce_plus_plus` corresponds to `batch`.
- Note: `gdpo` mode does not support `kl_in_reward=True`. If both are set, `kl_in_reward` will be automatically set to `False`.
- GDPO is designed for multi-reward optimization: When using multiple reward functions, GDPO normalizes each reward function separately within groups (subtract mean, divide by std), then performs weighted aggregation using `reward_weights`, and finally applies batch-level normalization. This approach better preserves the relative differences between rewards and prevents different reward combinations from collapsing into identical advantage values.
- rollout_importance_sampling_mode: Training-inference mismatch correction mode. Options are `token_truncate`, `token_mask`, `sequence_truncate`, `sequence_mask`. Default is None (disabled). For details, refer to the [documentation](../Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md).
- rollout_importance_sampling_threshold: Threshold for importance sampling weights, used for truncating or masking extreme weights. Default is 2.0.
- log_rollout_offpolicy_metrics: Whether to log training-inference mismatch diagnostic metrics (KL, PPL, χ², etc.) when `rollout_importance_sampling_mode` is not set. When `rollout_importance_sampling_mode` is set, metrics are always logged. Default is False.
- off_policy_sequence_mask_delta: Off-Policy Sequence Masking threshold from [DeepSeek-V3.2 paper](https://arxiv.org/abs/2512.02556). When set, computes `mean(old_policy_logps - policy_logps)` for each sequence. If this value exceeds the threshold AND the sequence has negative advantage, the sequence is masked out from loss computation. For details, refer to the [documentation](../Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md#off-policy-sequence-masking).
- router_replay_mode: Router replay mode. Options are `disabled`,`R2`,`R3`. Default is disabled.
- teacher_kl_coef: Coefficient for teacher KL in OPD-RL, i.e. `adv_t = base_adv + teacher_kl_coef * teacher_kl`. Default is 1.0.
Built-in reward function parameters refer to the [documentation](../Instruction/Command-line-parameters.md#reward-function-parameters).
### GKD Parameters
- beta: JSD divergence interpolation coefficient. 0.0 means Forward KL, 0.5 means symmetric JSD, 1.0 means Reverse KL. Default is 0.5.
- lmbda: On-Policy learning probability. 0.0 means pure Off-Policy, 1.0 means pure On-Policy. Default is 0.5.
- temperature: Temperature for sampling and loss computation. Default is 0.9.
- sft_alpha: Mixing coefficient for SFT loss, `loss = jsd_loss + sft_alpha * sft_loss`. Takes effect when using dataset responses (Off-Policy). Default is 0.
- max_completion_length: Maximum tokens for generation. Default is 512.
- vllm_mode: Same as GRPO parameter, used for On-Policy generation. Colocate mode deploys vLLM within the program.
- Note: On-Policy generation requires vLLM (`--use_vllm true --vllm_mode colocate/server`).
- When `lmbda > 0` but vLLM is not enabled, it will automatically fall back to Off-Policy mode.
### Teacher Parameters
Used in GKD and GRPO.
- teacher_model: Path or model ID of the teacher model. Required.
- teacher_model_type: Teacher model type. Default is None, auto-detected.
- teacher_model_revision: Teacher model version. Default is None.
- teacher_model_server: Teacher model service URL. Deploy via `swift deploy` for logprobs; mutually exclusive with `teacher_model`. Single URL or multi-teacher JSON (`'[{"url":"...","tags":["..."]}, ...]'`). See [distillation docs](../Instruction/Distillation.md#multi-teacher-routing).
- teacher_tag_key: Column name for multi-teacher routing (match sample values to teacher `tags`). Default is `"dataset"`.
- offload_teacher_model: Whether to offload teacher model to CPU to save GPU memory. Default is False.
## Export Parameters
This section introduces the parameters for `megatron export`. To use the `swift export` command for exporting, please refer to the [ms-swift Command Line Parameters Documentation](../Instruction/Command-line-parameters.md#export-arguments). Compared to `swift export`, `megatron export` supports distributed and multi-node exporting. Megatron export parameters inherit from Megatron parameters and basic parameters.
- 🔥to_mcore: Convert HF format weights to Megatron format. Defaults to False.
- 🔥to_hf: Convert Megatron format weights to HF format. Defaults to False.
- 🔥merge_lora: Defaults to None. If `to_hf` is set to True, this parameter defaults to `True`, otherwise False. In other words, by default, LoRA will be merged when saving in safetensors format; when saving in torch_dist format, LoRA will not be merged. The merged weights are stored in the `--output_dir` directory.
- Note: Transformers 5.0 has refactored the model architecture for MoE models. This new structure does not support MoE LoRA inference and may cause inference errors. **It is recommended to merge LoRA weights for MoE models** (vLLM is not affected).
- Note: The expert structure differs between Transformers and Megatron models. For example, the expert layers in Transformers' Qwen3-VL-MoE are implemented as Parameters rather than Linear layers. As a result, some models cannot convert LoRA delta weights (though Qwen3-VL-MoE supports conversion if LoRA is trained only on linear_proj and linear_qkv). However, most models support LoRA conversion, such as Qwen3-MoE, Qwen3-Omni-MoE, and GLM4.5-V.
- 🔥test_convert_precision: Test the precision error of HF and Megatron format weight conversion. Defaults to False.
- test_convert_dtype: The dtype used for conversion precision testing, defaults to 'float32'.
- exist_ok: If `args.output_dir` exists, do not throw an exception and perform overwriting. Defaults to False.
- device_map: Takes effect when `--test_convert_precision true` is set and controls where the HF model is loaded. The default is `'auto'`. You can set it to `'cpu'` to save GPU memory.
docs/source_en/Megatron-SWIFT/Custom-Model.md
+270
View File
@@ -0,0 +1,270 @@
# Custom Megatron Model
This guide explains how to register a model in [Mcore-Bridge](https://github.com/modelscope/mcore-bridge) to support training new models in Megatron-SWIFT. We will use MiniMax-M2.7 as an example.
## Download the Model
First, you need to download the model configuration.
```python
from swift import safe_snapshot_download
model_dir = safe_snapshot_download('MiniMax/MiniMax-M2.7', download_model=False)
print(f'model_dir: {model_dir}')
```
Since model weights are very large, to speed up the model integration process, we use lazy downloading and only download weights for `num_layers` layers, building a mini version of the model for integration testing. Taking MiniMax-M2.7 as an example, we build a one-layer BF16 version of the weights. If some models have the first 3 layers as Dense and the rest as MoE, you can build 4 layers of weights. If alternating attention types are used, for example Qwen3.5 alternates between linear attention and full attention, you will also need more layers.
```python
import os
import torch
from modelscope.hub.file_download import model_file_download
from safetensors.torch import safe_open
from swift import safe_snapshot_download
from mcore_bridge.utils import Fp8Dequantizer, SafetensorLazyLoader, StreamingSafetensorSaver
model_id = 'MiniMax/MiniMax-M2.7'
# Some models have the first few layers as dense and the rest as MoE; set this value accordingly
num_layers = 1 # Only download `num_layers` layers to save disk space and runtime GPU memory
model_dir = safe_snapshot_download(model_id, download_model=False)
loader = SafetensorLazyLoader(model_dir)
state_dict = loader.get_state_dict()
saver = StreamingSafetensorSaver(save_dir=model_dir)
new_state_dict = {}
fp8_dequantizer = Fp8Dequantizer() # Used to convert fp8 weights to bf16
def _open_file(self, filename: str):
if filename not in self._file_handles:
file_path = os.path.join(self.hf_model_dir, filename)
tmp_dir = os.path.join(self.hf_model_dir, 'tmp')
if not os.path.exists(file_path):
file_path = os.path.join(tmp_dir, filename)
if not os.path.exists(file_path):
file_path = model_file_download(
model_id=model_id,
file_path=filename,
local_dir=tmp_dir,
)
self._file_handles[filename] = safe_open(file_path, framework='pt')
return self._file_handles[filename]
SafetensorLazyLoader._open_file = _open_file # monkey patch (lazy downloading)
for k, v in state_dict.items():
if k.startswith('model.layers.'):
idx = int(k[len('model.layers.'):].split('.', 1)[0])
if idx >= num_layers:
continue
if k.endswith('.weight_scale_inv'):
continue
elif k.endswith('.weight'):
weight_scale_inv = k.replace('.weight', '.weight_scale_inv')
if weight_scale_inv in state_dict:
v = fp8_dequantizer.convert(v.load(), state_dict[weight_scale_inv].load()).to(torch.bfloat16)
new_state_dict[k] = v if isinstance(v, torch.Tensor) else v.load()
for k, v in new_state_dict.items():
saver.add_tensor(k, v)
saver.finalize()
```
After saving the weights, you need to modify `config.json`: change `num_hidden_layers` to 1 (corresponding to the code above), and remove the `quantization_config` section (since the weights are in BF16, not FP8). FP8 training is automatically adapted for most models, but some models may require additional adaptation. For example, refer to [this PR](https://github.com/modelscope/mcore-bridge/pull/30) for FP8 adaptation of Qwen3.5.
## Register the Model
The following provides debug code. You need to modify the code to ensure that the forward pass of the HuggingFace Transformers library aligns with Megatron's forward pass.
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'
os.environ['SWIFT_TEST_CONVERT_PRECISION'] = '1'
from swift import export_main, ExportArguments, safe_snapshot_download
model_id = 'MiniMax/MiniMax-M2.7'
model_dir = safe_snapshot_download(model_id, download_model=False)
export_main(
ExportArguments(
model=model_dir,
to_mcore=True,
exist_ok=True,
test_convert_precision=True,
torch_dtype='bfloat16',
))
```
The registration of minimax_m2 can be found in [this file](https://github.com/modelscope/mcore-bridge/blob/main/src/mcore_bridge/model/gpts/minimax_m2.py). During registration, we specify the corresponding GPTBridge class and model loader for the model.
```python
register_model(ModelMeta(
ModelType.minimax_m2,
['minimax_m2'],
bridge_cls=MinimaxM2Bridge,
loader=MinimaxM2Loader,
))
```
Parameter total sum alignment:
```
[INFO:swift] n_parameter: 522
[INFO:swift] total_sum: 106747128.72671509
[INFO:swift] zero_count: 0
[INFO:swift] n_parameter: 780
[INFO:swift] total_sum: 106747129.32046509
[INFO:swift] zero_count: 0
```
Model forward logits alignment. (Of course, we also need to train the model and then test the forward precision afterwards, to avoid cases where the output tokens are all the same.)
```
mean_diff: 2.8353377274470404e-05, max_diff: 0.0015382766723632812
mean_diff (with loss): 2.1664049199898727e-05, max_diff (with loss): 0.00021076202392578125 (Please check that mean_diff (with loss) is less than 0.1).
hf_tokens: [190962, 103239, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367]
mg_tokens: [190962, 103239, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367, 367]
token_diff: 0
token_diff (with loss): 0
```
Usually, once the total parameter count and output logits are aligned, the model integration is essentially successful. Additionally, you may need to adapt for TP/CP scenarios. You can use the following code to debug:
```python
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
os.environ['SWIFT_TEST_CONVERT_PRECISION'] = '1'
from swift.megatron import MegatronExportArguments, megatron_export_main
from swift import safe_snapshot_download
model_id = 'MiniMax/MiniMax-M2.7'
model_dir = safe_snapshot_download(model_id, download_model=False)
if __name__ == '__main__':
megatron_export_main(
MegatronExportArguments(
model=model_dir,
to_mcore=True, # Can also be changed to `to_hf=True` for testing
tensor_model_parallel_size=2,
sequence_parallel=True,
expert_model_parallel_size=2,
test_convert_precision=True,
))
```
We need to launch with torchrun. VSCode configuration:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "torchrun2",
"type": "debugpy",
"request": "launch",
"program": "-m",
"console": "integratedTerminal",
"justMyCode": false,
"args": [
"torch.distributed.run",
"--nproc_per_node",
"2",
"--master_port", "29501",
"${file}"
]
}
]
}
```
For other model registration examples, refer to the corresponding PRs: [hy_v3](https://github.com/modelscope/mcore-bridge/pull/53), [kimi_25](https://github.com/modelscope/mcore-bridge/pull/52). Integration PRs before April 2026 can be found in the ms-swift repository.
## Test Accuracy
We train the mini version of the model using only the self-cognition dataset, training until overfitting.
```shell
# 2 * 80GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model /root/.cache/modelscope/models/MiniMax/MiniMax-M2.7 \
--save_safetensors true \
--dataset 'swift/self-cognition#500' \
--tensor_model_parallel_size 2 \
--sequence_parallel true \
--micro_batch_size 16 \
--global_batch_size 16 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 2e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--num_train_epochs 10 \
--output_dir megatron_output \
--save_steps 500 \
--max_length 2048 \
--system 'You are a helpful assistant.' \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--moe_permute_fusion true \
--expert_model_parallel_size 2 \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-3 \
--dataset_num_proc 4 \
--model_author swift \
--model_name swift-robot
```
Run inference to check the training results:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model megatron_output/v3-20260430-143926/checkpoint-310 \
--max_new_tokens 64 \
--enable_thinking false \
--temperature 0
```
```
<<< 你是谁
我是一个由swift开发的人工智能助手，被称为swift-robot。我主要的目的是通过文本交流为用户提供帮助、信息和娱乐。如果您有任何疑问或需要帮助，请随时提出，我会尽力协助您。
--------------------------------------------------
<<< clear
<<< who are you
I am a language model developed by swift, you can call me swift-robot. How can I assist you?
--------------------------------------------------
```
Test forward precision alignment again:
```
mean_diff: 0.0005969047779217362, max_diff: 0.013172879815101624
mean_diff (with loss): 0.0005803848034702241, max_diff (with loss): 0.009410381317138672 (Please check that mean_diff (with loss) is less than 0.1).
hf_tokens: [190962, 190962, 367, 44, 46, 2362, 5129, 6415, 75827, 343, 10, 1497, 71151, 11915, 1497, 44, 3003, 44, 46, 46, 4387, 10, 32, 10, 258, 1497, 44, 46, 46, 258, 18268, 44, 692, 13268, 42047, 3764, 46, 46, 46, 94454, 46, 46, 275, 296, 3786, 46, 46, 275, 46, 46, 3786, 46, 2329, 10, 722]
mg_tokens: [190962, 190962, 367, 44, 46, 2362, 5129, 6415, 75827, 343, 10, 1497, 71151, 11915, 1497, 44, 3003, 44, 46, 46, 4387, 10, 32, 10, 258, 1497, 44, 46, 46, 258, 18268, 44, 692, 13268, 42047, 3764, 46, 46, 46, 94454, 46, 46, 275, 296, 3786, 46, 46, 275, 46, 46, 3786, 46, 2329, 10, 722]
token_diff: 0
token_diff (with loss): 0
```
At this point, the model integration is complete!
## Submit a PR
If you want to submit a PR to ms-swift/mcore-bridge, you need to additionally run the following commands to format the code:
```shell
pip install pre-commit
pre-commit run --all-files
```
docs/source_en/Megatron-SWIFT/GKD.md
+49
View File
@@ -0,0 +1,49 @@
# GKD
If you are new to GKD/OPD-RL, please refer to the [distillation documentation](../Instruction/Distillation.md) first.
GKD (Generalized Knowledge Distillation) is a training method that transfers knowledge from a teacher model to a student model by computing the Jensen-Shannon Divergence (JSD) loss between their output distributions.
## Feature Support
Megatron GKD currently supports the following features:
- **Training Modes**: Full parameter training and LoRA fine-tuning
- **Parallelism Strategies**: Context Parallel (CP), Pipeline Parallel (PP), Tensor Parallel (TP), and Expert Parallel (EP)
- **Model Support**: Compatible with LLMs and MLLMs in Megatron-SWIFT
- **Teacher Offload**: Supports offloading teacher model to CPU to save GPU memory
- **Online Generation**: Supports on-policy generation using vLLM for student model
- **Multi-turn Training**: Supports multi-turn GKD via `--multi_turn_scheduler`, sharing the same `MultiTurnScheduler` infrastructure as GRPO.
## Parameters
### GKD-specific Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--teacher_model` | str | - | Path or model ID of the teacher model<br>*Can be omitted when using `teacher_model_server` |
| `--teacher_model_server` | str | None | Teacher API URL; single URL or multi-teacher JSON. See [distillation docs](../Instruction/Distillation.md#multi-teacher-routing) |
| `--teacher_tag_key` | str | `"dataset"` | Column name for matching sample tags to teacher `tags` in multi-teacher routing |
| `--gkd_logits_topk` | int | None | Number of Top-K logits; required when using external API |
| `--beta` | float | 0.5 | JSD divergence interpolation coefficient:<br>• 0.0: Forward KL<br>• 0.5: Symmetric JSD<br>• 1.0: Reverse KL |
| `--lmbda` | float | 0.5 | On-Policy learning probability:<br>• 0.0: Pure Off-Policy<br>• 1.0: Pure On-Policy |
| `--temperature` | float | 0.9 | Temperature for sampling and loss computation |
| `--sft_alpha` | float | 0 | Mix in a proportion of SFT loss; applied to non-student-generated completions |
| `--max_completion_length` | int | 512 | Maximum tokens for generation |
### Batch-related Parameters
Same as Megatron SFT, use the following parameters to control batch size:
| Parameter | Description |
|-----------|-------------|
| `--micro_batch_size` | Training batch size per DP group |
| `--global_batch_size` | Global batch size: `micro_batch_size × dp_size × gradient_accumulation_steps` |
## Reference
For more parameters, please refer to [Command-line Parameters](./Command-line-parameters.md)
For training scripts, please refer to [Megatron GKD Scripts](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/rlhf/gkd)
Training script using Teacher Server reference [here](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/rlhf/gkd/teacher_server.sh)
docs/source_en/Megatron-SWIFT/GRPO.md
+42
View File
@@ -0,0 +1,42 @@
# Megatron GRPO
If you are new to GRPO, please refer to the [GRPO documentation](../Instruction/GRPO/GetStarted/GRPO.md) first.
Megatron GRPO currently supports the following features:
- **Training Modes**: Full parameter training and LoRA fine-tuning
- **Parallelism Strategies**: Context Parallelism (CP), Pipeline Parallelism (PP), Tensor Parallelism (TP), and Expert Parallelism (EP)
- **Inference Acceleration**: vLLM colocate mode and server mode
- **Model Support**: Compatible with LLMs and MLLMs (multimodal large models) in Megatron Swift
- **Algorithm Support**: Covers most features of Swift GRPO
Similar to ms-swift GRPO, all batch size-related parameters in Megatron GRPO are at the **completion-level**, meaning they represent the number of completions generated by the model, not the number of prompts.
#### Parameter Comparison
The following table compares the batch-related parameters between ms-swift and Megatron-SWIFT:
| ms-swift Parameter | Megatron-SWIFT Parameter | Description |
|-------------------|--------------------------|-------------|
| `per_device_train_batch_size` | `micro_batch_size` | Training batch size per DP group (completion-level) |
| `gradient_accumulation_steps` | - | Gradient accumulation steps, already included in `global_batch_size` calculation in Megatron-SWIFT |
| - | `global_batch_size` | Global batch size (completion-level)<br/>**Megatron-SWIFT**: `micro_batch_size × dp_size × gradient_accumulation_steps`<br/>**ms-swift**: `per_device_train_batch_size × world_size × gradient_accumulation_steps` |
| `num_generations` | `num_generations` | Number of completions generated per prompt |
| `steps_per_generation` | `steps_per_generation` | Ratio of Rollout batch size to training batch size<br/>**Note**: In ms-swift, must be an integer multiple of `gradient_accumulation_steps` |
| `generation_batch_size` | `generation_batch_size` | Batch size during Rollout phase (completion-level), must be an integer multiple of `global_batch_size` |
The following formulas are used to calculate batch sizes in Megatron GRPO:
- **Data Parallel Size**: `dp_size = world_size / (TP × PP × CP)`
- **Global Batch Size**: `global_batch_size = micro_batch_size × dp_size × gradient_accumulation_steps`
- **Generation Batch Size**: `generation_batch_size = global_batch_size × steps_per_generation`
- **Rollout Prompt Count**: `num_rollout_prompts = generation_batch_size / num_generations`
- **Training Prompt Count**: `num_train_prompts = global_batch_size / num_generations`
- **Training Prompt Count per DP Group**: `num_prompts_per_dp_group = global_batch_size / num_generations / dp_size`
**Note**: In Megatron GRPO, the training prompt count per DP group must satisfy that `num_prompts_per_dp_group` is an integer multiple of `micro_batch_size` to ensure proper batch allocation during training.
For more parameters, please refer to the [Command-line Parameters documentation](./Command-line-parameters.md#grpo-parameters).
For training scripts, please refer to [Megatron GRPO Scripts](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo).
docs/source_en/Megatron-SWIFT/LoRA-Training.md
+226
View File
@@ -0,0 +1,226 @@
# LoRA Training
Best practice reference for single-node 8xH20 LoRA training with Qwen3-235B-A22B-Instruct-250718: https://github.com/modelscope/ms-swift/pull/5033.
For environment setup, please refer to the [Quick Start Guide](./Quick-start.md) of Megatron-SWIFT.
## Traditional Method
### Converting HF to Mcore
Below, we introduce weight conversion using the `swift export` and `megatron export` commands respectively. Compared to `swift export`, `megatron export` supports multi-node and LoRA incremental weight conversion, but is also more complex, requiring additional specification of parallelism parameters during export, such as `--tensor_model_parallel_size` and `--export_model_parallel_size`. For details, refer to the [Mcore-Bridge Documentation](./Mcore-Bridge.md). To use the `swift export` command, refer to the [Quick Start Documentation](./Quick-start.md).
- `swift export` uses a single process, places HF weights on the GPU, and uses device_map for parallelization; mcore weights are placed on the CPU without enabling parallelization. This approach is very easy to debug and test the precision alignment between HF and mcore.
- `megatron export` launches multi-process execution via torchrun, placing mcore weights on GPU and supporting various parallelism modes, fp8, mtp, and other features. For precision alignment testing, HF weights are loaded on the first rank and placed in CPU memory.
```shell
# megatron export
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron export \
--model Qwen/Qwen2.5-7B-Instruct \
--tensor_model_parallel_size 2 \
--to_mcore true \
--torch_dtype bfloat16 \
--output_dir Qwen2.5-7B-Instruct-mcore \
--test_convert_precision true
# swift export
# CUDA_VISIBLE_DEVICES=0 \
# swift export \
# --model Qwen/Qwen2.5-7B-Instruct \
# --to_mcore true \
# --torch_dtype bfloat16 \
# --output_dir Qwen2.5-7B-Instruct-mcore \
# --test_convert_precision true
```
### LoRA Training
Training Script:
```bash
# full: 2 * 70GiB 0.61s/it
# lora: 2 * 14GiB 0.45s/it
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--mcore_model Qwen2.5-7B-Instruct-mcore \
--save_safetensors false \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
--tuner_type lora \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--tensor_model_parallel_size 2 \
--sequence_parallel true \
--micro_batch_size 16 \
--global_batch_size 16 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-4 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--num_train_epochs 1 \
--output_dir megatron_output/Qwen2.5-7B-Instruct \
--save_steps 100 \
--max_length 2048 \
--system 'You are a helpful assistant.' \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--dataset_num_proc 4 \
--model_author swift \
--model_name swift-robot
```
- For LoRA training scripts of MoE models, please refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/lora).
### Converting MCore to HF
```bash
# megatron export
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron export \
--mcore_adapter megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
--to_hf true \
--tensor_model_parallel_size 2 \
--merge_lora false \
--torch_dtype bfloat16 \
--output_dir megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-hf \
--test_convert_precision true
# swift export
# CUDA_VISIBLE_DEVICES=0 \
# swift export \
# --mcore_adapter megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
# --to_hf true \
# --torch_dtype bfloat16 \
# --output_dir megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-hf \
# --test_convert_precision true
```
- Note: The `--mcore_adapter` folder contains an `args.json` file. The conversion process will read the `--model/--mcore_model` and LoRA-related parameter information from this file. `swift export` does not currently support conversion of LoRA incremental weights. With `megatron export`, you can use the `--merge_lora` parameter to control whether to merge weights.
### Inference
```shell
# If using full weights, replace `--adapters` with `--model`
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-hf \
--stream true
```
### Merge-LoRA
If you only want to merge the LoRA weights without converting them to Hugging Face format, for subsequent DPO training, you can use the following script:
```shell
# megatron export
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron export \
--mcore_adapter megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
--tensor_model_parallel_size 2 \
--to_mcore true \
--merge_lora true \
--torch_dtype bfloat16 \
--output_dir megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-mcore \
--test_convert_precision true
# swift export
# CUDA_VISIBLE_DEVICES=0 \
# swift export \
# --mcore_adapter megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
# --to_mcore true \
# --torch_dtype bfloat16 \
# --output_dir megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-mcore \
# --test_convert_precision true
```
## Mcore-Bridge [Recommended]
### Training
```shell
# full: 2 * 70GiB 0.61s/it
# lora: 2 * 14GiB 0.45s/it
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model Qwen/Qwen2.5-7B-Instruct \
--save_safetensors true \
--merge_lora false \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
--tuner_type lora \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--tensor_model_parallel_size 2 \
--sequence_parallel true \
--micro_batch_size 16 \
--global_batch_size 16 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-4 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--num_train_epochs 1 \
--output_dir megatron_output/Qwen2.5-7B-Instruct \
--save_steps 100 \
--max_length 2048 \
--system 'You are a helpful assistant.' \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--dataset_num_proc 4 \
--model_author swift \
--model_name swift-robot
```
### Inference
```shell
# If using full weights, replace `--adapters` with `--model`
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
--stream true
```
### Merge-LoRA
Since `--merge_lora false` was set during training, if you want to merge the LoRA weights into full safetensors weights afterwards, you can use the following script:
```shell
# Since the LoRA weights are in safetensors format, you need to use `--adapters` instead of `--mcore_adapter`
# megatron export
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron export \
--adapters megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
--tensor_model_parallel_size 2 \
--to_hf true \
--merge_lora true \
--torch_dtype bfloat16 \
--output_dir megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-merged
```
docs/source_en/Megatron-SWIFT/Mcore-Bridge.md
+249
View File
@@ -0,0 +1,249 @@
# Mcore Bridge
Megatron is renowned for its exceptional training speed and rich parallel techniques, but this also comes with a relatively high barrier to entry. Therefore, [mcore-bridge](https://github.com/modelscope/mcore-bridge) was created to make Megatron training as simple and user-friendly as transformers. Through Mcore-Bridge, users can:
1. Directly load model weights in safetensors format and seamlessly use Megatron for efficient training. Save training weights directly in safetensors format without additional conversion.
2. Support bidirectional conversion compatible with LoRA incremental weights.
3. Support `Megatron->vLLM` weight synchronization for algorithms like GRPO/GKD.
4. Support multi-machine conversion of ultra-large-scale models.
Mcore-Bridge is compatible with various model architectures including Dense/MoE/multimodal. After training is complete, the converted models can be directly deployed using mainstream inference frameworks such as transformers, vLLM, SGLang, etc.
## Seamless Training
Currently, Mcore-Bridge supports parallel techniques such as TP/PP/EP/ETP/VPP, and the supported models can be found in the [Supported Models Documentation](../Instruction/Supported-models-and-datasets.md). The following introduces Mcore-Bridge's seamless training capabilities.
- When reading models with `--model/--adapters/--ref_model/--ref_adapters`, mcore-bridge is used to load safetensors format weights. With `--mcore_model/--mcore_adapter/--mcore_ref_model/--mcore_ref_adapter`, the default mcore loading method is used.
- `save_safetensors` determines whether weights are saved in safetensors or mcore format. When `--no_save_optim false` is set, mcore weights are always saved additionally for checkpoint resumption.
- Tip: During GKD/GRPO training, if you encounter GPU OOM issues when updating vLLM weights, you can set `--offload_bridge true` to offload tensors to CPU to reduce GPU memory usage.
### Full Parameter
Below is an example of training the multimodal model Qwen3-VL:
```shell
# 2 * 76GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model Qwen/Qwen3-VL-8B-Instruct \
--save_safetensors true \
--dataset 'AI-ModelScope/LaTeX_OCR:human_handwrite#5000' \
--load_from_cache_file true \
--tensor_model_parallel_size 2 \
--sequence_parallel true \
--packing true \
--freeze_llm false \
--freeze_vit true \
--freeze_aligner true \
--split_dataset_ratio 0.01 \
--micro_batch_size 1 \
--global_batch_size 4 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-6 \
--num_train_epochs 1 \
--output_dir megatron_output/Qwen3-VL-8B-Instruct \
--save_steps 200 \
--max_length 2048 \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--dataset_num_proc 8
```
Then we perform inference on the validation set:
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model megatron_output/Qwen3-VL-8B-Instruct/vx-xxx/checkpoint-xxx \
--load_data_args true \
--stream true
```
### LoRA
In addition to full-parameter training, Mcore-Bridge also supports LoRA training.
Below is an example of self-cognition training using LoRA for the text-only model Qwen3-Moe:
- If you want to export merged weights instead of LoRA delta weights, please set `--merge_lora true`. Setting `--merge_lora true` has better compatibility and supports all model series.
- Note: (For transformers>5.0) Transformers 5.0 refactored the MoE model architecture. This new structure does not support MoE LoRA inference and may cause inference anomalies. **It is recommended to merge LoRA for MoE models** (vLLM is not affected; refer to vLLM's support status).
- Note: (For transformers<5.0) Due to structural differences between transformers and Megatron model experts (e.g., the expert components in transformers' Qwen3-VL-MoE are implemented as Parameters rather than Linear layers), some models cannot convert LoRA delta weights (however, Qwen3-VL-MoE does support conversion when LoRA training targets only linear_proj and linear_qkv). Most models support LoRA conversion, such as: Qwen3-MoE, Qwen3-Omni-MoE, GLM4.5-V, etc.
```shell
# 50GiB
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model Qwen/Qwen3-30B-A3B-Instruct-2507 \
--save_safetensors true \
--merge_lora false \
--dataset 'swift/Chinese-Qwen3-235B-2507-Distill-data-110k-SFT#2000' \
'swift/self-cognition#1000' \
--load_from_cache_file true \
--tuner_type lora \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--split_dataset_ratio 0.01 \
--moe_permute_fusion true \
--expert_model_parallel_size 2 \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-3 \
--micro_batch_size 8 \
--global_batch_size 16 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--num_train_epochs 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-4 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--output_dir megatron_output/Qwen3-30B-A3B-Instruct-2507 \
--eval_steps 200 \
--save_steps 200 \
--max_length 2048 \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim true \
--no_save_rng true \
--sequence_parallel true \
--moe_expert_capacity_factor 2 \
--attention_backend flash \
--model_author swift \
--model_name swift-robot
```
Perform inference on the exported LoRA weights using the vLLM inference engine:
```shell
# For specific model LoRA support in vLLM, please refer to the vLLM documentation.
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model Qwen/Qwen3-30B-A3B-Instruct-2507 \
--adapters megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx \
--infer_backend vllm \
--vllm_max_model_len 8192 \
--stream true
```
If you need to manually **Merge-LoRA**, you can use the `megatron export` command. Note: Please do not use the `swift export` command to merge LoRA, as the **MoE model structures** in Megatron and transformers are not necessarily consistent.
```shell
# If the adapter is in mcore format, please use `--mcore_adapter`
# If the final format needs to be in mcore format, use `--to_mcore true`
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
megatron export \
--model Qwen/Qwen3-30B-A3B-Instruct-2507 \
--adapters megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx \
--output_dir megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx-merged \
--merge_lora true \
--to_hf true \
--tensor_model_parallel_size 2 \
--expert_model_parallel_size 2 \
--pipeline_model_parallel_size 2
```
Perform inference on the merged full weights using the transformers inference engine:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx-merged \
--stream true
```
## `megatron export` and Conversion Accuracy Testing
In addition to supporting safetensors conversion and saving during training, Mcore-Bridge also supports the `megatron export` command for standalone weight export. `megatron export` supports conversion precision testing during weight conversion, which is very helpful for verifying accuracy when integrating new models. Typically, models already integrated into Megatron-SWIFT will not have precision misalignment issues, so you can confidently set `--test_convert_precision false`.
- Note: For multimodal models, please focus on the `mean_diff (with loss)` field. The `mean_diff` may show a large difference because it includes image tokens, and loss is not calculated for that portion.
Full parameter weights:
```shell
# safetensors -> torch_dist
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
megatron export \
--model Qwen/Qwen3-30B-A3B-Instruct-2507 \
--output_dir Qwen3-30B-A3B-Instruct-2507-mcore \
--to_mcore true \
--tensor_model_parallel_size 2 \
--expert_model_parallel_size 2 \
--pipeline_model_parallel_size 2 \
--test_convert_precision true
```
```shell
# torch_dist -> safetensors
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
megatron export \
--mcore_model Qwen3-30B-A3B-Instruct-2507-mcore \
--output_dir Qwen3-30B-A3B-Instruct-2507-hf \
--to_hf true \
--tensor_model_parallel_size 2 \
--expert_model_parallel_size 2 \
--pipeline_model_parallel_size 2 \
--test_convert_precision true
```
LoRA weights:
```shell
# torch_dist -> safetensors
# If you need to perform merge-lora and test precision alignment after merge-lora, simply set `--merge_lora true`
# You can also change `--model safetensors-path` to `--mcore_model torch-dist-path`. These two methods are equivalent, and mcore-bridge will handle it automatically.
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
megatron export \
--model Qwen/Qwen3-30B-A3B-Instruct-2507 \
--mcore_adapter megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx \
--output_dir megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx-lora \
--merge_lora false \
--to_hf true \
--tensor_model_parallel_size 2 \
--expert_model_parallel_size 2 \
--pipeline_model_parallel_size 2 \
--test_convert_precision true
```
```shell
# safetensors -> torch_dist
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
megatron export \
--model Qwen/Qwen3-30B-A3B-Instruct-2507 \
--adapters megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx-lora \
--output_dir megatron_output/Qwen3-30B-A3B-Instruct-2507/vx-xxx/checkpoint-xxx-mcore \
--merge_lora false \
--to_mcore true \
--tensor_model_parallel_size 2 \
--expert_model_parallel_size 2 \
--pipeline_model_parallel_size 2 \
--test_convert_precision true
```
## Using Code
Please refer to [mcore-bridge github](https://github.com/modelscope/mcore-bridge/tree/main?tab=readme-ov-file#-quick-Start)
docs/source_en/Megatron-SWIFT/Multimodal-Model.md
+188
View File
@@ -0,0 +1,188 @@
# Multimodal Models
ms-swift introduces Megatron's parallelization techniques to accelerate the training of large multimodal models. Currently, it supports CPT/SFT/GRPO/DPO/KTO/RM for models such as Qwen3-VL, Qwen3-Omni, InternVL3.5, GLM4.5v, Kimi-VL. For a complete list of supported models, please refer to the [Supported Models and Datasets documentation](../Instruction/Supported-models-and-datasets.md).
For environment setup, please refer to the Megatron-SWIFT [Quick Start guide](./Quick-start.md).
## Dense Model
This section demonstrates fine-tuning the Qwen2.5-VL-7B-Instruct model on the LaTeX-OCR task using two 80GiB A100 GPUs, with both full-parameter fine-tuning and LoRA. The best practices described below can be completed within 10 minutes.
### Full
The full-parameter training script is as follows:
```shell
# 2 * 72GiB; 4.1s/it
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
MAX_PIXELS=1003520 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--save_safetensors true \
--dataset 'AI-ModelScope/LaTeX_OCR:human_handwrite#5000' \
--load_from_cache_file true \
--tensor_model_parallel_size 2 \
--sequence_parallel true \
--packing true \
--freeze_llm false \
--freeze_vit true \
--freeze_aligner true \
--split_dataset_ratio 0.01 \
--micro_batch_size 1 \
--global_batch_size 4 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-6 \
--num_train_epochs 1 \
--output_dir megatron_output/Qwen2.5-VL-7B-Instruct \
--save_steps 200 \
--max_length 2048 \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--dataset_num_proc 8
```
### LoRA
The LoRA training script is as follows:
```shell
# 2 * 23GiB; 2.3s/it
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
MAX_PIXELS=1003520 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--save_safetensors true \
--merge_lora false \
--dataset 'AI-ModelScope/LaTeX_OCR:human_handwrite#5000' \
--load_from_cache_file true \
--tuner_type lora \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--tensor_model_parallel_size 1 \
--sequence_parallel true \
--freeze_llm false \
--freeze_vit true \
--freeze_aligner true \
--packing true \
--split_dataset_ratio 0.01 \
--micro_batch_size 1 \
--global_batch_size 4 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-4 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--num_train_epochs 1 \
--output_dir megatron_output/Qwen2.5-VL-7B-Instruct \
--save_steps 200 \
--max_length 2048 \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--dataset_num_proc 8
```
Finally, we use the generated Hugging Face format weights to perform inference on the validation set:
```shell
MAX_PIXELS=1003520 \
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters megatron_output/Qwen2.5-VL-7B-Instruct/vx-xxx/checkpoint-xxx \
--attn_impl flash_attn \
--stream true \
--load_data_args true \
--temperature 0 \
--max_new_tokens 512
```
The inference results are as follows:
```
[QUERY] Using LaTeX to perform OCR on the image.
[LABELS] \forall x \in X , ( \alpha f ) ( x ) = \alpha f ( x )
[RESPONSE] \forall x \in X , ( \alpha f ) ( x ) = \alpha f ( x )
--------------------------------------------------
[QUERY] Using LaTeX to perform OCR on the image.
[LABELS] \pi \int _ { c } ^ { d } \{ g ( y ) \} ^ { 2 } d y
[RESPONSE] \pi \int _ { c } ^ { d } \{ g ( y ) \} ^ { 2 } d y
--------------------------------------------------
[QUERY] Using LaTeX to perform OCR on the image.
[LABELS] [ \frac 2 3 x ^ { \frac 3 2 } ] _ { 0 } ^ { 1 }
[RESPONSE] [ \frac 2 3 x ^ { \frac 3 2 } ] _ { 0 } ^ { 1 }
```
## MoE Model
Training script:
```bash
# 2 * 43GiB, 8s/it
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model OpenGVLab/InternVL3_5-30B-A3B \
--save_safetensors true \
--merge_lora false \
--dataset 'AI-ModelScope/LaTeX_OCR:human_handwrite#5000' \
--load_from_cache_file true \
--tuner_type lora \
--lora_rank 8 \
--lora_alpha 32 \
--target_modules all-linear \
--sequence_parallel true \
--freeze_llm false \
--freeze_vit true \
--freeze_aligner true \
--packing true \
--split_dataset_ratio 0.01 \
--expert_model_parallel_size 2 \
--moe_permute_fusion true \
--moe_grouped_gemm true \
--moe_shared_expert_overlap true \
--moe_aux_loss_coeff 1e-3 \
--micro_batch_size 1 \
--global_batch_size 4 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-4 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-5 \
--num_train_epochs 1 \
--output_dir megatron_output/InternVL3_5-30B-A3B \
--eval_steps 200 \
--save_steps 200 \
--max_length 2048 \
--dataloader_num_workers 8 \
--dataset_num_proc 8 \
--no_save_optim true \
--no_save_rng true \
--attention_backend flash
```
After training is completed, we use the generated Hugging Face format weights to perform inference on the validation set:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--adapters megatron_output/InternVL3_5-30B-A3B/vx-xxx/checkpoint-xxx \
--attn_impl flash_attn \
--stream true \
--load_data_args true \
--temperature 0 \
--max_new_tokens 512
```
docs/source_en/Megatron-SWIFT/Quick-start.md
+275
View File
@@ -0,0 +1,275 @@
# Quick Start
ms-swift incorporates Megatron's parallelization techniques to accelerate the training of large models, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, context parallelism, and expert parallelism. It supports CPT/SFT/GRPO/DPO/KTO/RM for models such as Qwen3, Qwen3.5, Deepseek-R1, GLM4.5, GPT-OSS, and more. For a complete list of supported models, please refer to the [Supported Models and Datasets documentation](../Instruction/Supported-models-and-datasets.md).
| Method | Full-Parameter | LoRA | MoE | Multimodal | FP8 |
| ---------------------- | -------------- | ---- | ---- | ---------- | ---- |
| Pre-training | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Supervised Fine-Tuning](https://github.com/modelscope/ms-swift/tree/main/examples/megatron) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [GRPO](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/grpo) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [GKD](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/gkd) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [DPO](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/dpo) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [KTO](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/kto) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [RM](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/rm) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Embedding](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/embedding) | ✅ | ✅| ✅ | ✅ | ✅ |
| [Reranker](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/reranker) | ✅ | ✅| ✅ | ✅ | ✅ |
| [Sequence Classification](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/seq_cls) | ✅ | ✅ | ✅ | ✅ | ✅ |
## Environment Setup
To use Megatron-SWIFT, in addition to installing the `swift` dependencies, you also need to install the following:
```shell
# transformer_engine
# If an installation error occurs, you can refer to this issue for resolution: https://github.com/modelscope/ms-swift/issues/3793
pip install --no-build-isolation transformer-engine[pytorch] --no-cache-dir
# cuda13
pip install pybind11
pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable --no-build-isolation
# apex
# Note: Megatron-SWIFT can run in environments without apex by setting `--gradient_accumulation_fusion false`.
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
# mcore-bridge
pip install mcore-bridge -U
# Install from main branch
# pip install git+https://github.com/modelscope/mcore-bridge.git
# If you are using multi-node training, please additionally set the `MODELSCOPE_CACHE` environment variable to a shared storage path.
# This will ensure that the dataset cache is shared, thereby speeding up preprocessing.
# Note: This step is crucial; otherwise multi-machine training may hang due to data inconsistencies caused by randomness in data preprocessing.
export MODELSCOPE_CACHE='/xxx/shared'
# flash_attn
# Choose an appropriate version to install: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.8.3
# Note: Do not install a version higher than the maximum supported by transformer_engine: https://github.com/NVIDIA/TransformerEngine/blob/release_v2.10/transformer_engine/pytorch/attention/dot_product_attention/utils.py#L118
MAX_JOBS=8 pip install "flash-attn==2.8.3" --no-build-isolation
```
Alternatively, you can also use the image: (See historical images [here](../GetStarted/SWIFT-installation.md#mirror))
```
# cu128
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.10.0-vllm0.17.1-modelscope1.34.0-swift4.0.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.10.0-vllm0.17.1-modelscope1.34.0-swift4.0.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.10.0-vllm0.17.1-modelscope1.34.0-swift4.0.3
# cu129 (fp8 training)
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py312-torch2.10.0-vllm0.19.1-modelscope1.35.4-swift4.1.3
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py312-torch2.10.0-vllm0.19.1-modelscope1.35.4-swift4.1.3
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.9.1-py312-torch2.10.0-vllm0.19.1-modelscope1.35.4-swift4.1.3
# cu130 (fp8 training)
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.23.0-modelscope1.37.1-swift4.3.2
modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.23.0-modelscope1.37.1-swift4.3.2
modelscope-registry.us-west-1.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda13.0.3-py312-torch2.11.0-vllm0.23.0-modelscope1.37.1-swift4.3.2
```
Recommended Operating Environment:
| | Range | Recommended | Notes |
|--------------|--------------|-------------|--------------------|
| python | >=3.10 | 3.12 | |
| cuda | | cuda12.8/13.0 | |
| torch | >=2.0 | 2.8.0/2.11.0 | |
| transformer-engine | >=2.3 | 2.16.0 | |
| apex | | 0.1 | |
| megatron-core | >=0.16,<0.20 | 0.17.1 | |
| mcore-bridge | >=1.3.0 | 1.5.0 | |
| flash-attn | | 2.8.3/4.0.0b15 | |
| transformers | >=4.33 | 4.57.6/5.12.1 | |
| modelscope | >=1.23 | | |
| peft | >=0.11,<0.20 | | LoRA |
| trl | >=0.15,<1.0 | | RLHF |
## Quick Start Example
This section introduces a quick start example for fine-tuning the self-awareness of the Qwen2.5-7B-Instruct model using two 80GiB A100 GPUs. The following best practices can be completed within 10 minutes.
### Traditional Method
First, we need to convert the weights from HF (Hugging Face) format to Megatron format:
- Multi-GPU weight conversion: Remove `CUDA_VISIBLE_DEVICES=0` to enable multi-GPU weight conversion.
- Conversion precision test: `--test_convert_precision true` will test the conversion precision. For large MoE model conversions, this option takes longer and consumes more memory, so you may omit it as needed.
```shell
CUDA_VISIBLE_DEVICES=0 \
swift export \
--model Qwen/Qwen2.5-7B-Instruct \
--to_mcore true \
--torch_dtype bfloat16 \
--output_dir Qwen2.5-7B-Instruct-mcore \
--test_convert_precision true
```
Next, use the following script to start training. The required GPU memory resources are 2*80GiB:
- If using multi-machine training, it is recommended to share a disk and specify the same path for `--output_dir`.
```shell
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--mcore_model Qwen2.5-7B-Instruct-mcore \
--save_safetensors false \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
--tensor_model_parallel_size 2 \
--sequence_parallel true \
--micro_batch_size 16 \
--global_batch_size 16 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-6 \
--num_train_epochs 1 \
--output_dir megatron_output/Qwen2.5-7B-Instruct \
--save_steps 100 \
--max_length 2048 \
--system 'You are a helpful assistant.' \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--dataset_num_proc 4 \
--model_author swift \
--model_name swift-robot
```
Finally, convert the Megatron format weights back to HF format:
- Note: Please point `--mcore_model` to the parent directory of `iter_xxx`. By default, the corresponding checkpoint from `latest_checkpointed_iteration.txt` will be used.
- If OOM (Out of Memory) occurs, simply remove `CUDA_VISIBLE_DEVICES=0`. If you encounter insufficient memory, please remove `--test_convert_precision true`.
```shell
CUDA_VISIBLE_DEVICES=0 \
swift export \
--mcore_model megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
--to_hf true \
--torch_dtype bfloat16 \
--output_dir megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-hf \
--test_convert_precision true
```
We then perform inference on the generated HF format weights:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx-hf \
--stream true \
--temperature 0 \
--max_new_tokens 2048
```
The inference results are as follows:
```
<<< who are you?
I am a language model developed by swift, you can call me swift-robot. How can I assist you?
```
For Megatron-Ray Training, refer to this [doc](../Instruction/Ray.md)
### Mcore-Bridge [Recommended]
Mcore-Bridge eliminates the tedious process of model conversion. For details, refer to the [Mcore-Bridge Documentation](./Mcore-Bridge.md).
Training script:
```bash
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
megatron sft \
--model Qwen/Qwen2.5-7B-Instruct \
--save_safetensors true \
--dataset 'AI-ModelScope/alpaca-gpt4-data-zh#500' \
'AI-ModelScope/alpaca-gpt4-data-en#500' \
'swift/self-cognition#500' \
--tensor_model_parallel_size 2 \
--sequence_parallel true \
--micro_batch_size 16 \
--global_batch_size 16 \
--recompute_granularity full \
--recompute_method uniform \
--recompute_num_layers 1 \
--finetune true \
--cross_entropy_loss_fusion true \
--lr 1e-5 \
--lr_warmup_fraction 0.05 \
--min_lr 1e-6 \
--num_train_epochs 1 \
--output_dir megatron_output/Qwen2.5-7B-Instruct \
--save_steps 100 \
--max_length 2048 \
--system 'You are a helpful assistant.' \
--dataloader_num_workers 4 \
--no_save_optim true \
--no_save_rng true \
--dataset_num_proc 4 \
--model_author swift \
--model_name swift-robot
```
We perform inference on the generated safetensors format weights:
```shell
CUDA_VISIBLE_DEVICES=0 \
swift infer \
--model megatron_output/Qwen2.5-7B-Instruct/vx-xxx/checkpoint-xxx \
--stream true \
--temperature 0 \
--max_new_tokens 2048
```
- For pretraining, you can use `megatron pt` instead of `megatron sft`, which will use a generative template for training.
- Megatron-SWIFT uses the same dataset and template processing modules as ms-swift, thus supporting techniques such as packing, loss scale, and agent training. For custom dataset formats, please refer to the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
- **More Examples**: Including packing, multi-node training, 32K context length, DPO, MoE models, and pre-training, can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/megatron).
## Training Tips
- Methods to increase training throughput: use packing (do not enable streaming), increase data parallelism (DP), reduce recomputation, and increase compute-communication overlap. MoE models can also be accelerated by dropping tokens.
- Parallelism choices:
- Megatron-SWIFT uses ZeRO-1 (use_distributed_optimizer enabled by default) combined with various parallelism techniques.
- DP is the fastest but consumes the most memory; use other parallel techniques to reduce memory usage.
- TP/EP involve heavy communication, so keep them within the NVLink domain when possible; for cross-domain setups prefer PP/DP. For expert layers, prefer EP over ETP — ETP saves memory but is slower.
- MoE parallel folding: separate MoE parallel groups from Dense groups. Attention uses tp-cp-dp-pp groups, while MoE uses etp-ep-dp-pp groups.
- Choosing parallelism for weight conversion: Megatron-SWIFT uses the torch_dist storage format on the MCore side; you can adjust parallelism at training time and do not need to specify it during weight conversion.
- Regarding log printing: Megatron-SWIFT logs are printed on the last rank, because in PP parallelism, only the last pp_rank has complete information.
## Benchmark
The training speed comparison for full-parameter dense models with 8K context length, using `megatron sft` and `swift sft`, under a single-node, eight-GPU A800 environment is as follows: ([shell](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/benchmark/deepspeed.sh))
**Dense** Qwen2.5-14B:
| | Megatron-LM | Deepspeed-ZeRO2 | Deepspeed-ZeRO3 |
| ---------------- | ----------- | --------------- | --------------- |
| Training Speed | 9.04s/it | 10.32s/it | 10.56s/it |
| GPU Memory Usage | 8\*64GB | 8\*80GB | 8\*58GB |
The training speed comparison for full-parameter MoE models with 8K context length, using `megatron sft` and `swift sft`, under a two-node, 16-GPU A800 environment is as follows:
**MoE** Qwen3-30B-A3B:
- Note: The DeepSpeed test results were conducted in a "transformers<5.0" environment. In "transformers>5.0", training can be accelerated by using `--experts_impl grouped_mm`.
| | Megatron-LM | Deepspeed-ZeRO2 | Deepspeed-ZeRO3 |
| ---------------- | ----------- | --------------- | --------------- |
| Training Speed | 9.6s/it | - | 91.2s/it |
| GPU Memory Usage | 16 * 60GiB | OOM | 16 * 80GiB |
## Megatron-SWIFT Wechat Group
<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/megatron.png" width="250">
docs/source_en/_templates/autosummary/class.rst
+10
View File
@@ -0,0 +1,10 @@
.. currentmodule:: {{ module }}
{{ name | underline}}
.. autoclass:: {{ name }}
:inherited-members:
:members:
.. autogenerated from source/_templates/autosummary/class.rst
docs/source_en/_templates/classtemplate.rst
+12
View File
@@ -0,0 +1,12 @@
.. currentmodule:: {{ module }}
{{ name | underline}}
.. autoclass:: {{ name }}
:members:
:special-members: __init__, __call__
..
autogenerated from source/_templates/classtemplate.rst
note it does not have :inherited-members:
docs/source_en/_templates/sobolengine.rst
+14
View File
@@ -0,0 +1,14 @@
.. currentmodule:: {{ module }}
{{ name | underline}}
.. autoclass:: {{ name }}
:members:
:exclude-members: MAXBIT, MAXDIM
:undoc-members:
..
autogenerated from source/_templates/sobolengine.rst
note it has specific options
docs/source_en/conf.py
+123
View File
@@ -0,0 +1,123 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
# import sphinx_book_theme
sys.path.insert(0, os.path.abspath('../../'))
# -- Project information -----------------------------------------------------
project = 'swift'
copyright = '2022-2025, Alibaba ModelScope'
author = 'ModelScope Authors'
version_file = '../../swift/version.py'
html_theme = 'sphinx_rtd_theme'
language = 'en'
def get_version():
with open(version_file, 'r', encoding='utf-8') as f:
exec(compile(f.read(), version_file, 'exec'))
return locals()['__version__']
# The full version, including alpha/beta/rc tags
version = get_version()
release = version
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.napoleon',
'sphinx.ext.autosummary',
'sphinx.ext.autodoc',
'sphinx.ext.viewcode',
'sphinx_markdown_tables',
'sphinx_copybutton',
'myst_parser',
]
# build the templated autosummary files
autosummary_generate = True
numpydoc_show_class_members = False
# Enable overriding of function signatures in the first line of the docstring.
autodoc_docstring_signature = True
# Disable docstring inheritance
autodoc_inherit_docstrings = False
# Show type hints in the description
autodoc_typehints = 'description'
# Add parameter types if the parameter is documented in the docstring
autodoc_typehints_description_target = 'documented_params'
autodoc_default_options = {
'member-order': 'bysource',
}
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
source_suffix = ['.rst', '.md']
# The master toctree document.
root_doc = 'index'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['build', 'source_en/.ipynb_checkpoints', 'source_en/api/generated', 'Thumbs.db', '.DS_Store']
# A list of glob-style patterns [1] that are used to find source files.
# They are matched against the source file names relative to the source directory,
# using slashes as directory separators on all platforms.
# The default is **, meaning that all files are recursively included from the source directory.
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
# html_theme = 'sphinx_book_theme'
# html_theme_path = [sphinx_book_theme.get_html_theme_path()]
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# html_css_files = ['css/readthedocs.css']
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
# -- Extension configuration -------------------------------------------------
# Ignore >>> when copying code
copybutton_prompt_text = r'>>> |\.\.\. '
copybutton_prompt_is_regexp = True
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/': None}
myst_enable_extensions = [
'amsmath',
'dollarmath',
'colon_fence',
]
docs/source_en/index.rst
+85
View File
@@ -0,0 +1,85 @@
.. swift documentation file,
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Swift DOCUMENTATION
========================
.. toctree::
:maxdepth: 2
:caption: Get Started
GetStarted/SWIFT-installation.md
GetStarted/Quick-start.md
GetStarted/Web-UI.md
.. toctree::
:maxdepth: 2
:caption: Instruction
Instruction/Command-line-parameters.md
Instruction/Pre-training-and-Fine-tuning.md
Instruction/GRPO/index.rst
Instruction/Distillation.md
Instruction/RLHF.md
Instruction/Inference-and-deployment.md
Instruction/Sample.md
Instruction/Evaluation.md
Instruction/Export-and-push.md
Instruction/Ray.md
Instruction/Reinforced-Fine-tuning.md
Instruction/Agent-support.md
Instruction/Supported-models-and-datasets.md
Instruction/Use-tuners.md
Instruction/Frequently-asked-questions.md
.. toctree::
:maxdepth: 2
:caption: Megatron-SWIFT
Megatron-SWIFT/Quick-start.md
Megatron-SWIFT/Command-line-parameters.md
Megatron-SWIFT/LoRA-Training.md
Megatron-SWIFT/Multimodal-Model.md
Megatron-SWIFT/Mcore-Bridge.md
Megatron-SWIFT/GRPO.md
Megatron-SWIFT/GKD.md
Megatron-SWIFT/Ascend.md
Megatron-SWIFT/Custom-Model.md
.. toctree::
:maxdepth: 2
:caption: Customization
Customization/Architecture.md
Customization/Custom-model.md
Customization/Custom-dataset.md
.. toctree::
:maxdepth: 2
:caption: Best Practices
BestPractices/GRPO.md
BestPractices/GRPO-Multi-Modal-Training.md
BestPractices/GRPO-Code-Training.md
BestPractices/Qwen3-Best-Practice.md
BestPractices/Qwen3-VL-Best-Practice.md
BestPractices/Qwen3_5-Best-Practice.md
BestPractices/deepseek-v4.md
BestPractices/MLLM-Registration.md
BestPractices/Embedding.md
BestPractices/Reranker.md
BestPractices/Rapidly-Training-VL-model.md
BestPractices/NPU-support.md
BestPractices/Metax-support.md
BestPractices/AMD-support.md
BestPractices/More-Best-Practices.md
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`