chore: import upstream snapshot with attribution

docs/source_en/Instruction/Agent-support.md

@@ -0,0 +1,257 @@
+# Agent Support
+
+
+## Dataset Format
+
+ms-swift leverages agent-template to decouple Agent data formats from model implementations. With a unified dataset format, users can seamlessly switch between different models for training without any data modifications.
+
+Example data samples for the pure text Agent and multimodal Agent are as follows:
+
+```jsonl
+{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"realtime_aqi\", \"description\": \"Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name, e.g., Shanghai\"}}, \"required\": [\"city\"]}}}]", "messages": [{"role": "user", "content": "What is the weather like in Beijing and Shanghai today?"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Beijing\"}}"}, {"role": "tool_call", "content": "{\"name\": \"realtime_aqi\", \"arguments\": {\"city\": \"Shanghai\"}}"}, {"role": "tool_response", "content": "{\"city\": \"Beijing\", \"aqi\": \"10\", \"unit\": \"celsius\"}"}, {"role": "tool_response", "content": "{\"city\": \"Shanghai\", \"aqi\": \"72\", \"unit\": \"fahrenheit\"}"}, {"role": "assistant", "content": "According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution."}]}
+{"tools": "[{\"type\": \"function\", \"function\": {\"name\": \"click\", \"description\": \"Click on a position on the screen\", \"parameters\": {\"type\": \"object\", \"properties\": {\"x\": {\"type\": \"integer\", \"description\": \"X-coordinate representing the horizontal position on the screen\"}, \"y\": {\"type\": \"integer\", \"description\": \"Y-coordinate representing the vertical position on the screen\"}}, \"required\": [\"x\", \"y\"]}}}]", "messages": [{"role": "user", "content": "<image>What time is it now?"}, {"role": "assistant", "content": "<think>\nI can check the current time by opening the calendar app.\n</think>\n"}, {"role": "tool_call", "content": "{\"name\": \"click\", \"arguments\": {\"x\": 105, \"y\": 132}}"}, {"role": "tool_response", "content": "{\"images\": \"<image>\", \"status\": \"success\"}"}, {"role": "assistant", "content": "Successfully opened the calendar app. The current time is 11 o'clock in the morning."}], "images": ["desktop.png", "calendar.png"]}
+```
+- When the `agent_template` is set to "react_en", "hermes", etc., this format is compatible with training for all model Agents and allows easy switching between different models.
+- Among them, `tools` is a JSON string containing a list of tools, and the `content` section of `messages` where the `role` is `'tool_call'` or `'tool_response/tool'` must also be a JSON string.
+- The `tools` field will be combined with the `{"role": "system", ...}` section during training/inference according to the `agent_template`, forming a complete system section.
+- The `{"role": "tool_call", ...}` part will automatically be converted into corresponding formats of `{"role": "assistant", ...}` based on the `agent_template`. Multiple consecutive `{"role": "assistant", ...}` entries will be concatenated to form a complete assistant_content.
+- The `{"role": "tool_response", ...}` can also be written as `{"role": "tool", ...}`, these two forms are equivalent. This part will also be automatically converted according to the `agent_template`. During training, this part does not participate in loss calculations, similar to `{"role": "user", ...}`.
+- This format supports parallel tool calls; refer to the first data sample for an example. In multimodal Agent data samples, the number of `<image>` tags should match the length of "images", and their positions indicate where the image features are inserted. It also supports other modalities, such as audios and videos.
+- Note: You can also manually process the data into the messages format with roles set to system, user, or assistant. The purpose of agent_template is to automatically map the tools field and the messages with roles tool_call and tool_response into the standard messages format with roles system, user, and assistant.
+
+The following are the `input_ids` and `labels` after encoding the two data samples mentioned above using the templates for **qwen2_5** and **qwen2_5_vl** , with the selected `agent_template` being **hermes** :
+
+Sample One (Parallel Tool Calls):
+
+```text
+[INPUT_IDS] <|im_start|>system
+You are Qwen, created by Alibaba Cloud. You are a helpful assistant.
+
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{"type": "function", "function": {"name": "realtime_aqi", "description": "Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information.", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name, e.g., Shanghai"}}, "required": ["city"]}}}
+</tools>
+
+For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
+<tool_call>
+{"name": <function-name>, "arguments": <args-json-object>}
+</tool_call><|im_end|>
+<|im_start|>user
+What is the weather like in Beijing and Shanghai today?<|im_end|>
+<|im_start|>assistant
+<tool_call>
+{"name": "realtime_aqi", "arguments": {"city": "Beijing"}}
+</tool_call>
+<tool_call>
+{"name": "realtime_aqi", "arguments": {"city": "Shanghai"}}
+</tool_call><|im_end|>
+<|im_start|>user
+<tool_response>
+{"city": "Beijing", "aqi": "10", "unit": "celsius"}
+</tool_response>
+<tool_response>
+{"city": "Shanghai", "aqi": "72", "unit": "fahrenheit"}
+</tool_response><|im_end|>
+<|im_start|>assistant
+According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
+
+[LABELS] [-100 * 195]<tool_call>
+{"name": "realtime_aqi", "arguments": {"city": "Beijing"}}
+</tool_call>
+<tool_call>
+{"name": "realtime_aqi", "arguments": {"city": "Shanghai"}}
+</tool_call><|im_end|>[-100 * 67]According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
+```
+
+Sample Two (Multimodal, Mixed Assistant and Tool Call):
+
+```text
+[INPUT_IDS] <|im_start|>system
+You are a helpful assistant.
+
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{"type": "function", "function": {"name": "click", "description": "Click on a position on the screen", "parameters": {"type": "object", "properties": {"x": {"type": "integer", "description": "X-coordinate representing the horizontal position on the screen"}, "y": {"type": "integer", "description": "Y-coordinate representing the vertical position on the screen"}}, "required": ["x", "y"]}}}
+</tools>
+
+For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
+<tool_call>
+{"name": <function-name>, "arguments": <args-json-object>}
+</tool_call><|im_end|>
+<|im_start|>user
+<|vision_start|>[151655 * 729]<|vision_end|>What time is it now?<|im_end|>
+<|im_start|>assistant
+<think>
+I can check the current time by opening the calendar app.
+</think>
+<tool_call>
+{"name": "click", "arguments": {"x": 105, "y": 132}}
+</tool_call><|im_end|>
+<|im_start|>user
+<tool_response>
+{"images": "<|vision_start|>[151655 * 729]<|vision_end|>", "status": "success"}
+</tool_response><|im_end|>
+<|im_start|>assistant
+Successfully opened the calendar app. The current time is 11 o'clock in the morning.<|im_end|>
+
+[LABELS] [-100 * 924]<think>
+I can check the current time by opening the calendar app.
+</think>
+<tool_call>
+{"name": "click", "arguments": {"x": 105, "y": 132}}
+</tool_call><|im_end|>[-100 * 759]Successfully opened the calendar app. The current time is 11 o'clock in the morning.<|im_end|>
+```
+
+**react_en** is one of the commonly used agent template formats. Below is an example of the `input_ids` and `labels` after encoding by qwen2_5 using `agent_template='react_en'`:
+
+```text
+[INPUT_IDS] <|im_start|>system
+Answer the following questions as best you can. You have access to the following tools:
+
+realtime_aqi: Call this tool to interact with the realtime_aqi API. What is the realtime_aqi API useful for? Weather forecast. Get real-time air quality, including current air quality, PM2.5, and PM10 information. Parameters: {"type": "object", "properties": {"city": {"type": "string", "description": "City name, e.g., Shanghai"}}, "required": ["city"]} Format the arguments as a JSON object.
+
+Use the following format:
+
+Question: the input question you must answer
+Thought: you should always think about what to do
+Action: the action to take, should be one of [realtime_aqi]
+Action Input: the input to the action
+Observation: the result of the action
+... (this Thought/Action/Action Input/Observation can be repeated zero or more times)
+Thought: I now know the final answer
+Final Answer: the final answer to the original input question
+
+Begin!
+<|im_end|>
+<|im_start|>user
+What is the weather like in Beijing and Shanghai today?<|im_end|>
+<|im_start|>assistant
+Action: realtime_aqi
+Action Input: {'city': 'Beijing'}
+Action: realtime_aqi
+Action Input: {'city': 'Shanghai'}
+Observation:{"city": "Beijing", "aqi": "10", "unit": "celsius"}
+Observation:{"city": "Shanghai", "aqi": "72", "unit": "fahrenheit"}
+According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
+
+[LABELS] [-100 * 233]Action: realtime_aqi
+Action Input: {'city': 'Beijing'}
+Action: realtime_aqi
+Action Input: {'city': 'Shanghai'}
+Observation:[-100 * 45]According to the weather forecast tool, the air quality index (AQI) in Beijing is 10, which indicates good air quality; whereas in Shanghai, the AQI is 72, indicating mild pollution.<|im_end|>
+```
+
+The following code can be used to experiment with more models and `agent_template` options. For more selectable values of `agent_template`, refer to [here](https://github.com/modelscope/ms-swift/blob/main/swift/agent_template/__init__.py).
+
+```python
+from swift import get_processor, get_template
+
+tokenizer = get_processor('Qwen/Qwen3.5-2B')
+template = get_template(tokenizer)  # Using default agent template
+# template = get_template(tokenizer, agent_template='qwen3_5')
+print(f'agent_template: {template._agent_template}')
+data = {...}
+template.set_mode('train')
+encoded = template.encode(data)
+print(f'[INPUT_IDS] {template.safe_decode(encoded["input_ids"])}\n')
+print(f'[LABELS] {template.safe_decode(encoded["labels"])}')
+```
+
+## Tools Format
+
+The tools field provides information about the APIs that the model can call. You need to provide the name, description, and parameters of the tools, as shown in the example below:
+
+```python
+tools = [{
+    'type': 'function',
+    'function': {
+        'name': 'get_current_weather',
+        'description': 'Get the current weather in a given location',
+        'parameters': {
+            'type': 'object',
+            'properties': {
+                'location': {
+                    'type': 'string',
+                    'description': 'The city and state, e.g. San Francisco, CA'
+                },
+                'unit': {
+                    'type': 'string',
+                    'enum': ['celsius', 'fahrenheit']
+                }
+            },
+            'required': ['location']
+        }
+    }
+}]
+```
+
+## Usage of loss_scale
+
+The `loss_scale` parameter can be used to adjust the loss weights for different parts of the model output during training. Currently, two configuration methods are supported: exact string matching and regular expression (regex) matching.
+
+1. String Matching Example: ReACT Format
+
+Take the ReACT format as an example. You can enable the corresponding `loss_scale` configuration via `--loss_scale react` (see configuration file [react.json](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/config/react.json)). This method relies on exact string matching. The dictionary mapping in the configuration must provide a list of two elements, representing:
+
+- The loss weight for the matched string itself,
+- The loss weight for content following the matched string, up to (but not including) the next specified string.
+
+The specific effects of this configuration are as follows:
+
+- The `'Action:'` and `'Action Input:'` keywords and their subsequent content both have a loss weight of 2;
+- The `'Thought:'` and `'Final Answer:'` keywords and their subsequent content both have a loss weight of 1;
+- The `'Observation:'` field itself has a loss weight of 2, but the subsequent tool call result content has a loss weight of 0.
+
+
+2. Regular Expression Matching Example: Ignoring Empty Thought Blocks
+
+When training reasoning models, it may be necessary to exclude loss computation for empty thought blocks in the dataset, such as sequences like `'<think>\n\n</think>\n\n'`.
+
+In such cases, use `--loss_scale ignore_empty_think` (see configuration file [ignore_empty_think.json](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/config/ignore_empty_think.json)). This configuration uses regular expression matching, where the dictionary mapping only needs to specify a single value—the loss weight for the matched content.
+
+The specific effect of this setting is:
+- Any string matching the regular expression `<think>\\s*</think>\\s*` is assigned a `loss_scale` of 0, meaning no loss is computed for these segments.
+
+
+Testing loss_scale using code:
+```python
+from swift import get_processor, get_template
+
+data = {"messages": [
+    {"role": "user", "content": "aaaaa"},
+    {"role": "assistant", "content": "<think>\n\n</think>\n\nabc<think>\n\n</think>\n\n123"},
+]}
+
+template = get_template(get_processor('Qwen/Qwen3-8B'), loss_scale='ignore_empty_think')
+template.set_mode('train')
+inputs = template.encode(data)
+
+print(template.safe_decode(inputs['labels']))
+# '[-100 * 14]abc<think>\n\n</think>\n\n123<|im_end|>\n'
+```
+
+
+For more `loss_scale` plugin designs, please refer to the [Architecture](../Customization/Architecture.md) documentation.
+
+## Training
+
+- Train the Agent capabilities of Base models by switching different models through modifying `--model`. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent/qwen2_5.sh).
+- The agent_template for training GLM4 is hermes. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent/glm4.sh).
+- Use `--loss_scale` to adjust the loss weight of the model output section. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/agent/loss_scale).
+
+## Inference
+
+- 🚀For inference of the original model or fully trained model, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_agent.py).
+- For inference after LoRA training, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent/loss_scale/infer_lora.py).
+
+## Deployment
+
+For server and client code, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/deploy/agent).

docs/source_en/Instruction/Command-line-parameters.md

@@ -0,0 +1,991 @@
+# Command Line Parameters
+
+The command-line arguments will be introduced in four categories: basic arguments, atomic arguments, integrated arguments, and model-specific arguments. **The final list of arguments used in the command line consists of the integrated arguments, which inherit from the basic arguments and certain atomic arguments**. Model-specific arguments are tailored for particular models and can be configured via `--model_kwargs` or environment variables. For a detailed introduction to Megatron-SWIFT command-line arguments, please refer to the [Megatron-SWIFT Training Documentation](../Megatron-SWIFT/Command-line-parameters.md).
+
+**Tips:**
+
+- To pass a list via the command line, separate the elements with spaces. For example: `--dataset <dataset_path1> <dataset_path2>`.
+- To pass a dictionary via the command line, use JSON format. For example: `--model_kwargs '{"fps_max_frames": 12}'`.
+- Parameters marked with 🔥 are important; new users of ms-swift should prioritize these command-line arguments.
+
+## Base Arguments
+
+- 🔥tuner_backend: Optional values are `'peft'` and `'unsloth'`. Default is `'peft'`.
+- 🔥tuner_type: Optional values are 'lora', 'full', 'lora_llm', 'longlora', 'adalora', 'llamapro', 'adapter', 'vera', 'boft', 'fourierft', 'reft', 'bone'. Default is 'lora'.
+  - Note: Defaults to 'full' in Megatron-SWIFT.
+  - 'lora_llm' means applying LoRA to the LLM part while using 'full' fine-tuning for the ViT/aligner parts. You can set their respective learning rates using `vit_lr/aligner_lr`.
+- 🔥adapters: A list specifying adapter IDs or paths. Default is `[]`. This parameter is typically used in inference/deployment commands, for example: `swift infer --model '<model_id_or_path>' --adapters '<adapter_id_or_path>'`. It can occasionally be used for resuming training from a checkpoint. The difference between this parameter and `resume_from_checkpoint` is that **this parameter only loads adapter weights**, without restoring the optimizer state or random seed, and does not skip already-trained portions of the dataset.
+  - The difference between `--model` and `--adapters`: `--model` is followed by the directory path of the complete weights, which contains full weight information such as model/tokenizer/config, for example `model.safetensors`. `--adapters` is followed by a list of incremental adapter weight directory paths, which contain incremental weight information of the adapters, for example `adapter_model.safetensors`.
+- 🔥external_plugins: A list of external `plugin.py` files that will be additionally loaded (i.e., the modules will be imported). Defaults to `[]`. You can pass in `.py` file paths for custom model, template, and dataset registration, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/custom/sft.sh); or for custom GRPO components, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/run_external_reward_func.sh).
+- seed: Global random seed. Default is 42.
+  - Note: This random seed is independent of `data_seed`, which controls randomness in the dataset.
+- model_kwargs: Additional arguments specific to certain models. This list of parameters will be logged during training/inference. For example: `--model_kwargs '{"fps_max_frames": 12}'`. You can also set it via environment variables, e.g., `FPS_MAX_FRAMES=12`. Default is None.
+  - Note: **If you specify model-specific parameters during training, please also set the corresponding parameters during inference**—this helps maintain consistent performance.
+  - The meaning of model-specific parameters can usually be found in the official repository or inference code of the corresponding model. MS-Swift includes these parameters to ensure alignment between trained models and official inference behavior.
+- enable_npu_model_patch: Whether to enable model-level NPU patches. Defaults to True. This parameter only controls model-related patches in NPU environments and usually does not need to be disabled; set it to False when debugging native transformers behavior or NPU model patch compatibility issues. It must be passed as a startup argument before the process imports `swift.model` for the first time.
+- load_args: When `--resume_from_checkpoint`, `--model`, or `--adapters` are specified, this flag controls whether to load `args.json` from the saved file. The loaded keys are defined in [base_args.py](https://github.com/modelscope/ms-swift/blob/main/swift/arguments/base_args/base_args.py). Default is `True` for inference and export, and `False` for training. Usually, this parameter does not need to be modified.
+- load_data_args: If set to `True`, additional data-related arguments from `args.json` will be loaded. Default is `False`. **This is typically used during inference to run inference on validation sets split during training**, for example: `swift infer --adapters xxx --load_data_args true --stream true --max_new_tokens 512`.
+- use_hf: Determines whether to use [ModelScope](https://modelscope.cn/) or [HuggingFace](https://huggingface.co/) for downloading models, downloading datasets, and pushing models. Defaults to False (uses ModelScope).
+  - Note: To access ModelScope internationally, you can use [ModelScope International](https://modelscope.ai/home) by setting the environment variable `MODELSCOPE_DOMAIN='www.modelscope.ai'`.
+- hub_token: Hub authentication token. For ModelScope, see [here](https://modelscope.cn/my/myaccesstoken). Default is `None`.
+- ddp_timeout: Default is 18000000, in seconds.
+- ddp_backend: Optional values are `"nccl"`, `"gloo"`, `"mpi"`, `"ccl"`, `"hccl"`, `"cncl"`, `"mccl"`. Default is `None`, which enables automatic selection.
+- ignore_args_error: Used for compatibility with Jupyter Notebook. Default is `False`.
+
+### Model Arguments
+- 🔥model:  The [model ID](https://modelscope.cn/models) or local model path. Default is `None`.
+- model_type: The model type. In ms-swift, a `model_type` refers to a group of models that share the same architecture, model loading process, and template definition. Default is `None`, meaning it will be automatically inferred based on the suffix of `--model` and the 'architectures' field in config.json. Supported model types can be found in the [List of Supported Models and Datasets](./Supported-models-and-datasets.md)
+  - Note: The concept of `model_type` in MS-Swift differs from the `model_type` in `config.json`.
+  - Custom models typically require manually registering a `model_type` and `template`. See the [Custom Model Documentation](../Customization/Custom-model.md) for details.
+- model_revision: Model version. Default is `None`.
+- task_type: Default is `'causal_lm'`. Options include `'causal_lm'`, `'seq_cls'`, `'embedding'`, `'reranker'`, and `'generative_reranker'`. Examples for seq_cls can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/seq_cls), and examples for embedding can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/embedding).
+- 🔥torch_dtype: Data type for model weights. Supported values: `float16`, `bfloat16`, `float32`. Default is `None`, which reads from the `config.json` file.
+- attn_impl: Attention implementation. Options include `'sdpa'`, `'eager'`, `'flash_attn'`, `'flash_attention_2'`, `'flash_attention_3'`, `'flash_attention_4'`, etc. Default is `None`, reading from config.json.
+  - Note: Not all attention implementations may be supported, depending on the underlying Transformers library's support for the specific model.
+  - If set to `'flash_attn'` (for backward compatibility), `'flash_attention_2'` will be used.
+- 🔥experts_impl: Expert implementation type, options are 'grouped_mm', 'batched_mm', 'eager'. Defaults to None. This feature requires "transformers>=5.0.0".
+- new_special_tokens: List of additional special tokens to be added. Default is `[]`. Example usage can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/new_special_tokens).
+  - Note: You can also pass a `.txt` file path where each line contains one special token.
+- num_labels: Required for classification models (`--task_type seq_cls`). Indicates the number of labels. Default is `None`.
+- problem_type: Required for classification models (`--task_type seq_cls`). Options: `'regression'`, `'single_label_classification'`, `'multi_label_classification'`. Default is `None`. If the model is a reward model or `num_labels=1`, it defaults to `'regression'`; otherwise, it defaults to `'single_label_classification'`.
+- rope_scaling: Type of RoPE scaling. You can pass a string such as `'linear'`, `'dynamic'`, `'yarn'`, along with `max_model_len`, and MS-Swift will automatically configure the corresponding `rope_scaling`, overriding the value in `config.json`. Alternatively, pass a JSON string like `'{"factor": 2.0, "type": "yarn"}'`, which will directly replace the `rope_scaling` in `config.json`. Default is `None`.
+- max_model_len: When using `rope_scaling` with a string input, this parameter helps calculate the RoPE scaling `factor`. Default is `None`. If specified, this value will **override** `max_position_embeddings` in `config.json`.
+- device_map: Device placement configuration for the model, e.g., `'auto'`, `'cpu'`, a JSON string, or a JSON file path. This argument is **passed through** to the `from_pretrained` method in Transformers. Default is `None`, automatically determined based on available devices and distributed training setup.
+- max_memory: When `device_map` is set to `'auto'` or `'sequential'`, model weights are allocated across devices according to `max_memory`, e.g., `--max_memory '{0: "20GB", 1: "20GB"}'`. Default is `None`. Passed through to the `from_pretrained` interface in Transformers.
+- local_repo_path: Some models depend on GitHub repositories during loading, e.g., [deepseek-vl2](https://github.com/deepseek-ai/DeepSeek-VL2). To avoid network issues during `git clone`, you can use a local repository. This parameter takes the path to the local repo. Default is `None`.
+- init_strategy: Strategy for initializing uninitialized parameters when loading a model (especially useful for custom architectures). Options: `'zero'`, `'uniform'`, `'normal'`, `'xavier_uniform'`, `'xavier_normal'`, `'kaiming_uniform'`, `'kaiming_normal'`, `'orthogonal'`. Default is `None`.
+
+### Data Arguments
+- 🔥dataset: A list of dataset IDs or paths. Default is `[]`. Each dataset should be specified in the format: `'dataset_id_or_path:subset#sample_size'`, where subset and sample size are optional. Local datasets support formats such as jsonl, csv, json, and folders. **Open-source datasets from the hub can be used offline by `git clone`-ing them locally and passing the local folder path**. For custom dataset formats, refer to the [Custom Dataset Documentation](../Customization/Custom-dataset.md). You can use multiple datasets by passing `--dataset <dataset1> <dataset2>`.
+  - Subset: This parameter is only effective when the dataset is a dataset ID or a folder. If subsets were specified during registration and only one exists, that subset is selected by default; otherwise, the default subset `'default'` is used. You can select multiple subsets using `/`, e.g., `<dataset_id>:subset1/subset2`. You can also use `'all'` to select all registered subsets, e.g., `<dataset_id>:all`. See an example of registration [here](https://modelscope.cn/datasets/swift/garbage_competition).
+  - Sampling size: Uses the complete dataset by default. You can sample from the selected dataset by setting `#sample_size`, for example `<dataset_path#1000>`. If the sample size is less than the total number of samples, random sampling without replacement is performed. If the sample count exceeds the total, the dataset is repeated `sample_size // total_samples` times, with an additional `sample_size % total_samples` samples randomly sampled. Note: For streaming datasets (`--streaming true`), only sequential sampling is performed. If `--dataset_shuffle false` is set, non-streaming datasets also use sequential sampling.
+- 🔥val_dataset: A list of validation dataset IDs or paths. Default is `[]`.
+- 🔥cached_dataset: Use cached datasets (generated with the command `swift export --to_cached_dataset true ...`) to avoid GPU time being occupied by tokenization during training/inference on large datasets. This parameter is used to set the folder path(s) of cached training datasets, and defaults to `[]`. For examples, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/cached_dataset).
+  - Note: cached_dataset only stores an additional length field in the dataset (to avoid storage pressure) and filters out data samples that would cause errors. During training/inference, the `--max_length` parameter is supported for filtering/truncating excessively long data and the `--packing` parameter is supported. The actual data preprocessing process occurs synchronously during training and overlaps with the training process, which does not affect training speed. (Exception: when `--truncation_strategy split` is set, `'input_ids'` will be stored instead of `'lengths'`. In this case, you must use the same `max_length` and `truncation_strategy` for both dataset export and training. Compatibility with packing is not affected.)
+  - cached_dataset is compatible between `ms-swift` and `Megatron-SWIFT`, and supports pt/sft/infer/rlhf, set the training type using `--template_mode`; supports embedding/reranker/seq_cls tasks, set the task type using `--task_type`.
+  - Supports sampling from cache_dataset with the syntax `<cached_dataset_path>#sample_count`, supports sampling counts both higher and lower than the number of samples. For functionality and implementation, refer to the `--dataset` description.
+- cached_val_dataset: Folder path(s) for cached validation datasets, default is `[]`.
+- 🔥split_dataset_ratio: The ratio for splitting a validation set from the training set when `val_dataset` is not specified. Default is `0.`, meaning no splitting occurs.
+- data_seed: Random seed for dataset operations. Default is `42`.
+- 🔥dataset_num_proc: Number of processes for dataset preprocessing. Default is `1`.
+  - Note: For text-only models, it is recommended to increase this value to accelerate preprocessing speed. For multimodal models, it is not recommended to set it too high, as this may lead to slower preprocessing speed (if multimodal models experience 100% CPU utilization but extremely slow processing speed, it is recommended to additionally set the `OMP_NUM_THREADS` environment variable).
+- 🔥load_from_cache_file: Whether to load the dataset from cache. Default is `False`. **Recommended to set to `True` during actual training and `False` during debugging**. You can modify the `MODELSCOPE_CACHE` environment variable to control the cache path.
+- dataset_shuffle: Whether to shuffle the training dataset. Default is `True`.
+  - Note: **Shuffling in CPT/SFT involves two parts**: dataset-level shuffling (controlled by `dataset_shuffle`) and dataloader-level shuffling (controlled by `train_dataloader_shuffle`).
+- val_dataset_shuffle: Whether to shuffle the validation dataset. Default is `False`.
+- streaming: Whether to stream and process the dataset on-the-fly. Default is `False`. (The shuffling of streaming datasets is not thorough, which may lead to severe loss fluctuations.)
+  - Note: You must set `--max_steps` explicitly, as streaming datasets do not have a defined length. You can achieve behavior equivalent to `--num_train_epochs` by setting `--save_strategy epoch` and a large `max_steps`. Alternatively, set `max_epochs` to ensure training stops after the specified number of epochs, allowing model evaluation and checkpoint saving.
+  - Note: Streaming avoids waiting for preprocessing by overlapping it with training. However, preprocessing is only performed on rank 0 and then distributed to other processes. **This is typically less efficient than non-streaming data sharding**. When the training `world_size` is large, preprocessing and data distribution can become a bottleneck.
+- interleave_prob: Default is `None`. By default, multiple datasets are combined using `concatenate_datasets` from the datasets library. If this parameter is set, `interleave_datasets` is used instead. This is typically used for combining streaming datasets and is passed directly to `interleave_datasets`. This parameter does not apply to `--val_dataset`.
+- stopping_strategy: Options are `"first_exhausted"` or `"all_exhausted"`. Default is `"first_exhausted"`. Passed to the `interleave_datasets` function. This parameter does not apply to `--val_dataset`.
+- shuffle_buffer_size:  Specifies the shuffle buffer size for **streaming datasets**. Default is `1000`. Only effective when `dataset_shuffle` is `True`.
+- download_mode: Dataset download mode. Options: `'reuse_dataset_if_exists'` or `'force_redownload'`. Default is `'reuse_dataset_if_exists'`.
+  - Typically set to `--download_mode force_redownload` when encountering errors with hub datasets.
+- columns: Used to map dataset column names so that the dataset conforms to the format accepted by `AutoPreprocessor`. See [Custom Dataset Documentation](../Customization/Custom-dataset.md) for supported formats. You can pass a JSON string, e.g., `'{"text1": "query", "text2": "response"}'`, meaning column `"text1"` is mapped to `"query"` and `"text2"` to `"response"`, which `AutoPreprocessor` can process. Default is `None`.
+- strict: If `True`, any malformed row in the dataset will raise an error; otherwise, erroneous samples are dropped. Default is `False`. This is typically used for debugging.
+- 🔥remove_unused_columns: Whether to remove unused columns from the dataset. Default is `True`.
+  - If set to `False`, extra columns are passed to the trainer's `compute_loss` function, **facilitating custom loss functions that use additional dataset columns**.
+  - Default value is `False` for GRPO.
+- disable_auto_column_mapping: By default, column names in the dataset are automatically mapped. This parameter disables that behavior (the `columns` parameter remains effective), defaulting to `False`. The automatic mapping rules are as follows:
+  - The following fields will be automatically mapped to the corresponding `images`, `videos`, and `audios` fields:
+    - `images`: `image`, `images`
+    - `videos`: `video`, `videos`
+    - `audios`: `audio`, `audios`
+  - The following fields will be automatically mapped to the corresponding `system`, `query`, and `response` fields (the `solution` field will be preserved):
+    - `system`: `system`, `system_prompt`
+    - `query`: `query`, `prompt`, `input`, `instruction`, `question`, `problem`
+    - `response`: `response`, `answer`, `output`, `targets`, `target`, `answer_key`, `answers`, `solution`, `text`, `completion`, `content`
+- 🔥model_name: **Used only for self-cognition tasks**, and only affects the `swift/self-cognition` dataset. Replaces the `{{NAME}}` placeholder in the dataset. Provide the model's Chinese and English names, separated by space, e.g., `--model_name 小黄 'Xiao Huang'`. Default is `None`.
+- 🔥model_author: Used only for self-cognition tasks, and only affects the `swift/self-cognition` dataset. Replaces the `{{AUTHOR}}` placeholder. Provide the model author's Chinese and English names, separated by space, e.g., `--model_author '魔搭' 'ModelScope'`. Default is `None`.
+- custom_dataset_info: Path to a JSON file for custom dataset registration. See [Custom Dataset Guide](../Customization/Custom-dataset.md) and the [built-in dataset_info.json](https://github.com/modelscope/ms-swift/blob/main/swift/dataset/data/dataset_info.json). Default is `[]`.
+
+
+### Template Arguments
+- 🔥template: The type of conversation template. Default is `None`, which automatically selects the corresponding template for the given model. See [List of Supported Models](./Supported-models-and-datasets.md) for mapping details.
+- 🔥system: Custom system message field. Accepts either a string or a **path to a .txt file**. Default is `None`, using the default system message defined in the registered template.
+  - Note: In terms of priority, the `system` field from the dataset takes precedence, followed by `--system`, and finally the `default_system` set in the registered template.
+- 🔥max_length: Maximum token length after `tokenizer.encode` for a single data sample (to prevent OOM during training). Samples exceeding this limit are handled according to `truncation_strategy`. Default is `None`, meaning it's set to the model’s maximum supported sequence length (`max_model_len`).
+  - In PPO, GRPO, GKD and inference scenarios, `max_length` refers to `max_prompt_length`.
+- truncation_strategy: How to handle samples whose tokens exceed `max_length`. Supports 'delete', 'left', 'right', and 'split', which represent deleting, left truncation, right truncation, and splitting into multiple data samples, respectively. The default is 'delete'.
+  - Note: `--truncation_strategy split` is only supported during pretraining, i.e., in `swift/megatron pt` scenarios. This strategy will split oversized fields into multiple data samples to avoid token waste. This strategy only supports text-only datasets. Multimodal datasets cannot handle the splitting of pixel_values correctly.
+  - Note: For multimodal models, if `truncation_strategy` is set to `'left'` or `'right'` during training, **ms-swift preserves all image tokens and other modality-specific tokens**, which may lead to OOM.
+- 🔥max_pixels: Maximum pixel count (H×W) for input images in multimodal models. Images exceeding this limit will be resized to avoid OOM during training. Default is `None` (no restriction).
+  - Note: This parameter applies to all multimodal models. The Qwen2.5-VL specific parameter `MAX_PIXELS` (see bottom of doc) only affects Qwen2.5-VL.
+- 🔥agent_template: Agent template that defines how the tool list `'tools'` is converted into the `'system'` message, how tool calls are extracted from model responses during inference/deployment, and the formatting of `{"role": "tool_call", "content": "xxx"}` and `{"role": "tool_response", "content": "xxx"}` in `messages`. Options include `'react_en'`, `'hermes'`, `'glm4'`, `'qwen_en'`, `'toolbench'`, etc. See [here](https://github.com/modelscope/ms-swift/blob/main/swift/agent_template/mapping.py) for more. Default is `None`, automatically selected based on model type. Refer to [Agent Documentation](./Agent-support.md).
+- norm_bbox: Controls how bounding boxes ("bbox" in dataset, containing absolute coordinates; see [Custom Dataset Documentation](../Customization/Custom-dataset.md#grounding)) are normalized. Options: `'norm1000'` (scale coordinates to thousandths), `'none'` (no scaling). Default is `None`, automatically chosen based on model.
+  - This also works correctly when **images are resized during training** (e.g., when `max_pixels` is set).
+- use_chat_template: Whether to use a chat template or a generation template (the latter typically used in pretraining). Default is `True`.
+  - Note: `swift pt` defaults to `False`, using the generation template. This setting provides good **compatibility with multimodal models**.
+- padding_side: Padding side when training with `batch_size >= 2`. Options: `'left'`, `'right'`. Default is `'right'`.
+  - Note: PPO and GKD default to `'left'`. (During inference with `batch_size >= 2`, only left-padding is applied.)
+- 🔥padding_free: Flattens data within a batch to avoid padding, reducing GPU memory usage and accelerating training (**sequences in the same batch remain invisible to each other**). Default is `False`. Currently supported in CPT/SFT/DPO/GRPO/KTO/GKD.
+  - Note: Use `padding_free` together with `--attn_impl flash_attn` and `transformers>=4.44`. See [this PR](https://github.com/huggingface/transformers/pull/31629) for details. (Same as packing.)
+  - **Compared to packing, padding_free avoids extra preprocessing time, but packing offers faster training and more stable memory usage**.
+- 🔥loss_scale: Loss weight configuration for training tokens. Default is `'default'`. loss_scale includes 3 basic strategies: 'default', 'last_round', 'all', and other strategies: 'ignore_empty_think' and agent-specific ones: 'react', 'hermes', 'qwen', 'agentflan', 'alpha_umi', etc. For available options, refer to [loss_scale module](https://github.com/modelscope/ms-swift/blob/main/swift/loss_scale/mapping.py). ms-swift supports mixing basic strategies with other strategies, for example: `'default+ignore_empty_think'`, `'last_round+ignore_empty_think'`. If no basic strategy is specified, it defaults to 'default', for example: 'hermes' is equivalent to 'default+hermes'.
+  - 'default': All responses (including history) are calculated with weight 1 for cross-entropy loss (**system/user/multimodal tokens in messages and `tool_response` parts in Agent training are not included in loss calculation**). (**Default value for SFT**)
+  - 'last_round': Only calculate loss for the last round response. The last round means all content after the last "user". (**Default value for RLHF**)
+  - 'all': Calculate loss for all tokens. (**Default value for `swift pt`**)
+  - 'ignore_empty_think': Ignore loss computation for empty `'<think>\n\n</think>\n\n'` (as long as it matches the regex `'<think>\\s*</think>\\s*'`).
+  - 'react', 'hermes', 'qwen': Adjust the loss weight of the `tool_call` part to 2.
+  - Note: Starting from `ms-swift >= 4.3.1`, multiple non-base strategies can be chained together (each strategy processes the output segments of the previous one, with weights multiplied accordingly). For example: `'last_round+hermes+ignore_empty_think'`, where `'last_round'` is the base strategy, and `'hermes+ignore_empty_think'` represents a chain of multiple non-base strategies that share the same base strategy.
+- disable_ignore_empty_think: Whether to disable the automatic appending of `ignore_empty_think` strategy to loss_scale for hybrid thinking models. Defaults to `False`, meaning for hybrid thinking models (e.g., Qwen3.5-4B), `+ignore_empty_think` is automatically appended to the loss_scale, so that empty `'<think>\n\n</think>\n\n'` blocks do not contribute to loss computation. If the user has already manually specified `ignore_empty_think` in the loss_scale, it will not be appended again. This parameter only takes effect during training and has no effect on pure thinking models or non-thinking models. Set to `True` to disable this default behavior.
+  - Note: This parameter was added in `ms-swift>=4.3.1`. In `ms-swift<4.3.1`, you need to manually add `--loss_scale ignore_empty_think`.
+- is_binary_loss_scale: When `loss_scale` can only take values of `0` or `1`, its semantics can be represented by `labels` instead — by setting the `labels` of positions where `loss_scale` is `0` to `-100`, thereby ensuring compatibility with `liger_kernel` and reducing memory usage. Defaults to `None` for automatic configuration.
+- sequence_parallel_size: Size for sequence parallelism. Default is 1. Currently supported in CPT/SFT/DPO/GRPO. Training scripts can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel).
+- template_backend: Backend for template processing. Options are `'swift'` or `'jinja'`. Default is `'swift'`. If `'jinja'` is used, `apply_chat_template` from Transformers will be applied.
+  - Note: The `'jinja'` backend only supports inference and does not support training (as it cannot determine the token ranges for loss computation).
+- response_prefix: The prefix string for responses, this parameter only takes effect during inference. Default is None, determined by the enable_thinking parameter and template type.
+- enable_thinking: This parameter takes effect during inference, indicating whether to enable thinking mode. Default is None, the default value is determined by the template (model) type (hybrid thinking/non-thinking templates default to `False`, thinking templates default to `True`). If enable_thinking is False, a non-thinking prefix is added, for example the Qwen3-8B hybrid thinking model adds the prefix `'<think>\n\n</think>\n\n'`, while Qwen3-8B-Thinking does not add a prefix. If enable_thinking is True, a thinking prefix is added, for example `'<think>\n'`. Note: The priority of this parameter is lower than the response_prefix parameter.
+  - Note: In `ms-swift<4.3.1`, the default value for hybrid thinking templates was `True`; this was changed to `False` in `ms-swift>=4.3.1`.
+- preserve_thinking: Whether to preserve historical thinking content during inference and training. When set to `True`, thinking content from all rounds is retained. When set to `False`, only the thinking content from the last round is retained (i.e., the content following the last user message). Defaults to `None`.
+  - Default behavior: For thinking models (thinking/hybrid-thinking) or when `enable_thinking` is explicitly enabled, this is set to `False` by default during inference and training, retaining only the last round of thinking content. If the `loss_scale` base strategy during training is not `'last_round'` (e.g., `'default'`), it defaults to `True`, and historical thinking content will not be removed.
+- add_non_thinking_prefix: This parameter only takes effect during training, indicating whether to add a non-thinking prefix to data samples whose assistant part **does not start with the thinking marker `'<think>'`** (typically hybrid thinking models contain a non-thinking prefix). This feature allows swift's built-in datasets to train hybrid thinking models. Default value is True. For example: the non-thinking prefix for the Qwen3-8B hybrid thinking model is `'<think>\n\n</think>\n\n'`, while the non-thinking prefix for Qwen3-8B-Thinking/Instruct is `''`. Note: During training, if the basic strategy of loss_scale is last_round, this modification is only applied to the last round; otherwise, for example 'default' or 'all', this modification is applied to every round of data. If set to False, no non-thinking prefix is added to data samples.
+
+
+### Generation Arguments
+
+Refer to the [generation_config](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig) documentation.
+
+- 🔥max_new_tokens: The maximum number of new tokens generated during inference. Defaults to None, meaning unlimited.
+- temperature: Sampling temperature. Higher values increase output randomness. Default is `None`, reading from `generation_config.json`.
+  - You can set `--temperature 0` or `--top_k 1` to disable randomness in generation.
+- top_k: Top-k sampling parameter. Only the top `k` highest probability tokens are considered for generation. Default is `None`, reading from `generation_config.json`.
+- top_p: Top-p (nucleus) sampling parameter. Only tokens whose cumulative probability reaches `top_p` are considered. Default is `None`, reading from `generation_config.json`.
+- repetition_penalty: Penalty for repeated tokens. A value of 1.0 means no penalty. Default is `None`, reading from `generation_config.json`.
+- num_beams: Number of beams for beam search. Default is 1.
+- 🔥stream: Enable streaming output. Default is `None`, meaning `True` when using an interactive interface, and `False` during batch inference on datasets.
+- stop_words: Additional stop words besides the `eos_token`. Default is `[]`.
+  - Note: The `eos_token` is removed from the output response, while additional stop words are preserved in the output.
+- logprobs: Whether to return log probabilities. Default is `False`.
+- top_logprobs: Number of top log probabilities to return. Default is `None`.
+- structured_outputs_regex: A regular expression pattern for structured outputs (guided decoding). When set, the model's generation is constrained to match the specified regex pattern. Only effective when `infer_backend` is `vllm`. Default is `None`.
+
+### Quantization Arguments
+
+The following are parameters for quantizing models upon loading. See the [quantization documentation](https://huggingface.co/docs/transformers/main/en/main_classes/quantization) for details. These do not include `gptq` or `awq` quantization parameters used in `swift export`.
+
+- 🔥quant_method: Quantization method used when loading the model. Options: `'bnb'`, `'hqq'`, `'eetq'`, `'quanto'`, `'fp8'`. Default is `None`.
+  - If performing QLoRA training on already AWQ/GPTQ-quantized models, you do not need to set additional quantization parameters like `quant_method`.
+- 🔥quant_bits: Number of bits for quantization. Default is `None`.
+- hqq_axis: Axis for HQQ quantization. Default is `None`.
+- bnb_4bit_compute_dtype: Computation data type for 4-bit BNB quantization. Options: `float16`, `bfloat16`, `float32`. Default is `None`, which uses the value of `torch_dtype`.
+- bnb_4bit_quant_type: Type for 4-bit BNB quantization. Options: `'fp4'`, `'nf4'`. Default is `'nf4'`.
+- bnb_4bit_use_double_quant: Whether to use double quantization. Default is `True`.
+- bnb_4bit_quant_storage: Data type used to store quantized weights. Default is `None`.
+
+### RAY Arguments
+
+- use_ray: Boolean type. Whether to use ray, defaults to `False`.
+- ray_exp_name: Ray experiment name. This field will be used as the prefix for cluster and worker names, can be empty.
+- device_groups: String (jsonstring) type. When using ray, this field must be configured. For details, please refer to the [ray documentation](Ray.md).
+
+### YAML/JSON Support
+
+Here we use `swift sft` as an example. The YAML/JSON launch method also supports `swift infer/rlhf/...` as well as `megatron sft/rlhf`. Please refer to [the examples here](https://github.com/modelscope/ms-swift/tree/main/examples/yaml).
+- The YAML/JSON file will be stored in `output_dir` after training/inference.
+
+```shell
+swift sft xxx.yaml
+swift sft xxx.json
+```
+
+The content of xxx.yaml/xxx.json contains specific command-line configurations:
+
+```yaml
+model: "Qwen/Qwen2.5-7B-Instruct"
+dataset: "swift/self-cognition#500"
+```
+
+```json
+{
+    "model": "Qwen/Qwen2.5-7B-Instruct",
+    "dataset": "swift/self-cognition#500"
+}
+```
+
+You can also use a combination of YAML and command-line arguments. For example, use YAML for parameters that are infrequently modified, and pass frequently changed parameters via command line.
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift infer examples/yaml/deepspeed/infer.yaml \
+    --adapters output/vx-xxx/checkpoint-xxx
+```
+
+How to specify environment variables in YAML/JSON:
+
+```yaml
+ENV:
+  MAX_PIXELS: '1003520'
+  VIDEO_MAX_PIXELS: '50176'
+  FPS_MAX_FRAMES: '12'
+```
+
+```json
+{
+  "ENV": {
+      "MAX_PIXELS": "1003520",
+      "VIDEO_MAX_PIXELS": "50176",
+      "FPS_MAX_FRAMES": "12"
+  }
+}
+```
+
+## Atomic Arguments
+
+### Seq2SeqTrainer Arguments
+
+This list inherits from the Transformers `Seq2SeqTrainingArguments`, with ms-swift overriding certain default values. For arguments not listed here, please refer to the [official HF documentation](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Seq2SeqTrainingArguments).
+
+- 🔥output_dir: The output directory where the model predictions and checkpoints will be written. Default is `None`, automatically set to `'output/<model_name>'`.
+- 🔥gradient_checkpointing: Whether to use gradient checkpointing. Default is `True`. This significantly reduces GPU memory usage but slows down training.
+- 🔥vit_gradient_checkpointing: Whether to enable gradient checkpointing for the ViT component during multimodal model training. Defaults to `None`, which means it is enabled when `--freeze_vit` is `false`. For an example, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/vit_gradient_checkpointing.sh).
+- 🔥deepspeed: Default is `None`. Can be set to `'zero0'`, `'zero1'`, `'zero2'`, `'zero3'`, `'zero2_offload'`, `'zero3_offload'` to use built-in DeepSpeed configurations in ms-swift. You can also pass a path to a custom DeepSpeed config file.
+- zero_hpz_partition_size: Default is `None`. This enables ZeRO++ functionality—model sharding within nodes and data sharding across nodes. If encountering `grad_norm NaN`, try using `--torch_dtype float16`.
+- deepspeed_autotp_size: DeepSpeed tensor parallelism size. Default is 1. To use DeepSpeed AutoTP, set `--deepspeed` to `'zero0'`, `'zero1'`, or `'zero2'`. (Note: Only supports full-parameter training)
+- 🔥fsdp: FSDP2 distributed training configuration. Default is `None`. Can be set to `'fsdp2'` to use built-in FSDP2 configurations in ms-swift. You can also pass a path to a custom FSDP config file. FSDP2 is PyTorch's native distributed training solution, use either this or DeepSpeed (not both).
+- 🔥per_device_train_batch_size: Default is 1.
+- 🔥per_device_eval_batch_size: Default is 1.
+- 🔥gradient_accumulation_steps: Gradient accumulation steps. Default is `None`, meaning `gradient_accumulation_steps` is automatically calculated so that `total_batch_size >= 16`. Total batch size is computed as `per_device_train_batch_size * gradient_accumulation_steps * world_size`. In GRPO training, default is 1.
+  - In CPT/SFT training, gradient accumulation has equivalent effects to using a larger batch size, but this equivalence does not hold in RLHF training.
+- weight_decay:  Weight decay coefficient. Default is 0.1.
+- adam_beta1: The exponential decay rate for the first moment estimates (momentum) in Adam-based optimizers. Defaults to 0.9.
+- adam_beta2: The exponential decay rate for the second moment estimates (variance) in Adam-based optimizers. Defaults to 0.95.
+- adam_epsilon: Epsilon value for numerical stability in Adam-based optimizers. Defaults to 1e-8.
+- 🔥learning_rate:  Learning rate. **Default is `1e-5` for full-parameter training, and `1e-4` for LoRA and other tuners**.
+  - Tip: If you want to set `min_lr`, you can pass the arguments `--lr_scheduler_type cosine_with_min_lr --lr_scheduler_kwargs '{"min_lr": 1e-6}'`.
+- 🔥vit_lr: Specifies the learning rate for the ViT module when training multimodal models. Default is `None`, same as `learning_rate`. Typically used together with `--freeze_vit` and `--freeze_aligner`.
+  - Note: The "learning_rate" printed in the logs is the learning rate of `param_groups[0]`, where the order of param_groups is vit, aligner, llm (if it contains trainable parameters).
+- 🔥aligner_lr: Specifies the learning rate for the aligner module in multimodal models. Default is `None`, same as `learning_rate`.
+- lr_scheduler_type: The type of learning rate scheduler, defaults to `'cosine'`. Common options: `'linear'`, `'constant'`, `'cosine_with_min_lr'`.
+- lr_scheduler_kwargs: Additional arguments for the learning rate scheduler. Default is `None`.
+- gradient_checkpointing_kwargs: Arguments passed to `torch.utils.checkpoint`. For example: `--gradient_checkpointing_kwargs '{"use_reentrant": false}'`. Default is `None`.
+  - Note: When using DDP without DeepSpeed/FSDP and `gradient_checkpointing_kwargs` is `None`, it defaults to `'{"use_reentrant": false}'` to prevent errors.
+- full_determinism: Ensures reproducible results during training. Note: This may negatively impact performance. Default is `False`.
+- 🔥report_to: Default is `'tensorboard'`. You can specify multiple loggers, e.g., `--report_to tensorboard wandb swanlab`, or `--report_to all`.
+  - If you specify `--report_to wandb`, you can set the project name through `WANDB_PROJECT` and specify the API KEY corresponding to your account through `WANDB_API_KEY`.
+- logging_first_step: Whether to log metrics at the first step. Default is `True`.
+- logging_steps: Interval for logging. Default is 5.
+- router_aux_loss_coef: Used in MoE model training to set the weight of auxiliary loss. Default is `0.`.
+- enable_dft_loss: Whether to use [DFT](https://arxiv.org/abs/2508.05629) (Dynamic Fine-Tuning) loss during SFT training. Default is `False`.
+- enable_channel_loss: Enable channel-based loss. Default is `False`. Requires a `"channel"` field in the dataset. ms-swift groups and computes loss by this field (samples without `"channel"` are grouped into the default `None` channel). Dataset format reference: [channel loss](../Customization/Custom-dataset.md#channel-loss).  Channel loss is compatible with packing, padding_free, and loss_scale techniques.
+- safe_serialization: Whether to save the model in safetensors format. Default is True.
+- max_shard_size: Maximum size of a single storage file, default is '5GB'.
+- logging_dir: Directory for TensorBoard logs. Default is `None`, automatically set to `f'{self.output_dir}/runs'`.
+- predict_with_generate: Use generation during evaluation. Default is `False`.
+- metric_for_best_model: Default is `None`. If `predict_with_generate=False`, it's set to `'loss'`; otherwise `'rouge-l'` (in PPO training, no default; in GRPO, set to `'reward'`).
+- greater_is_better: Default is `None`. Set to `False` if `metric_for_best_model` contains `'loss'`, otherwise `True`.
+- max_epochs: Force training to stop after reaching `max_epochs`, then evaluate and save the model. Useful when using streaming datasets. Default is `None`.
+
+Other important parameters:
+- 🔥num_train_epochs: Number of training epochs. Default is 3.
+- 🔥save_strategy: Strategy for saving checkpoints. Options: `'no'`, `'steps'`, `'epoch'`. Default is `'steps'`.
+- 🔥save_steps: Default is 500.
+- 🔥eval_strategy: Evaluation strategy. Default is `None`, following `save_strategy`.
+  - If neither `val_dataset` nor `eval_dataset` is used and `split_dataset_ratio=0`, defaults to `'no'`.
+- 🔥eval_steps: Default is `None`. If evaluation dataset exists, follows `save_steps`.
+- eval_on_start: Whether to perform an evaluation step before training to ensure the validation steps work correctly. Defaults to False.
+- 🔥save_total_limit: Maximum number of checkpoints to save. Expired checkpoints will be deleted. Default is None, which saves all checkpoints. If set to 2, it will save the best checkpoint and the last checkpoint.
+- max_steps: Maximum number of training steps. Must be set when using streaming datasets. Default is -1.
+- 🔥warmup_ratio: Default is 0.
+- save_on_each_node: Save weights on every node. Default is `False`. Relevant in multi-node training.
+  - Tip: In multi-node training, `output_dir` is typically set to a shared directory, so this parameter usually doesn't need to be set.
+- save_only_model: Whether to save only model weights (excluding optimizer states, random seed states, etc.), reducing time and space overhead in full-parameter training. Default is `False`.
+- 🔥resume_from_checkpoint: Path to resume training from. Default is `None`.
+  - Tip: **To resume training, keep other parameters unchanged and add `--resume_from_checkpoint checkpoint_dir`**. Weights and states will be loaded by the trainer.
+  - Note: `resume_from_checkpoint` loads model weights, optimizer state, random seed, and resumes training from the last step. Use `--resume_only_model` to load only model weights.
+- resume_only_model: Default is `False`. If set to `True` along with `resume_from_checkpoint`, only model weights are resumed, ignoring optimizer state and random seed.
+  - Note: **`resume_only_model` skips already-trained data by default**, controlled via the `ignore_data_skip` argument.
+- ignore_data_skip: When `resume_from_checkpoint` and `resume_only_model` are set, this controls whether to skip already-trained data and restore training states (epoch, step count, etc.). Default is `False`. If `True`, training starts from step 0 without loading previous states or skipping data.
+- 🔥ddp_find_unused_parameters: Default is `None`.
+- 🔥dataloader_num_workers: Default is `None`. On Windows, set to 0; otherwise, 1.
+- dataloader_pin_memory: Default is `True`.
+- dataloader_persistent_workers: Default is `False`.
+- dataloader_prefetch_factor: Default is `None`. If `dataloader_num_workers > 0`, it is set to 2. Number of batches loaded in advance by each worker. 2 means there will be a total of 2 * num_workers batches prefetched across all workers.
+- train_dataloader_shuffle: Whether to shuffle the dataloader for CPT/SFT training, default is True. This parameter is ineffective for IterableDataset (i.e., it doesn't work for streaming datasets). IterableDataset reads data sequentially.
+- optim: The optimizer, defaults to `"adamw_torch"` (for torch>=2.8 `"adamw_torch_fused"`). For a complete list of optimizers, please see `OptimizerNames` in [training_args.py](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py).
+- optim_args: Optional arguments to pass to the optimizer, defaults to None.
+- group_by_length: Whether to group samples with approximately the same length together in the training dataset (with a random factor) to minimize padding and ensure load balancing across nodes and processes for improved efficiency. Defaults to False. For the specific algorithm, refer to `transformers.trainer_pt_utils.get_length_grouped_indices`.
+- 🔥neftune_noise_alpha: Noise magnitude for NEFTune. Default is 0. Common values: 5, 10, 15.
+- 🔥use_liger_kernel: Whether to enable the [Liger](https://github.com/linkedin/Liger-Kernel) kernel to accelerate training and reduce GPU memory consumption. Defaults to False. Example shell script can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/liger).
+  - Note: Liger kernel does not support `device_map`. Use DDP or DeepSpeed for multi-GPU training. Currently, liger_kernel only supports `task_type='causal_lm'`.
+- average_tokens_across_devices: Whether to average token counts across devices. If `True`, `num_tokens_in_batch` is synchronized via `all_reduce` for accurate loss computation. Default is `False`.
+- max_grad_norm: Gradient clipping. Default is 1.
+  - Note: The logged `grad_norm` reflects the value **before** clipping.
+- push_to_hub: Push checkpoints to the hub. Default is `False`.
+- hub_model_id: Model ID on the hub. Default is `None`.
+- hub_private_repo: Whether the repo is private. Default is `False`.
+
+### Tuner Arguments
+
+- 🔥freeze_llm: This argument only takes effect for multimodal models and can be used in both full-parameter and LoRA training, but with different behaviors. In full-parameter training, setting `freeze_llm=True` freezes the LLM component's weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_llm=True` prevents LoRA modules from being added to the LLM part. Default is `False`.
+- 🔥freeze_vit: This argument only applies to multimodal models and behaves differently depending on the training mode. In full-parameter training, setting `freeze_vit=True` freezes the ViT (vision transformer) component's weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_vit=True` prevents LoRA modules from being added to the ViT part. Default is `True`.
+  - Note: **Here, "vit" refers not only to `vision_tower`, but also to `audio_tower`**. For Omni models, if you want to apply LoRA only to `vision_tower` and not `audio_tower`, you can modify [this code](https://github.com/modelscope/ms-swift/blob/a5d4c0a2ce0658cef8332d6c0fa619a52afa26ff/swift/llm/model/model_arch.py#L544-L554).
+- 🔥freeze_aligner: This argument only affects multimodal models. In full-parameter training, setting `freeze_aligner=True` freezes the aligner (also known as projector) weights. In LoRA training with `target_modules=['all-linear']`, setting `freeze_aligner=True` prevents LoRA modules from being added to the aligner component. Default is `True`.
+- 🔥target_modules: Specifies which modules to apply LoRA to. Default is `['all-linear']`. You can also specify suffixes of modules, e.g., `--target_modules q_proj k_proj v_proj`. This argument is not limited to LoRA and can be used with other tuners.
+  - Note: The behavior of `'all-linear'` differs between LLMs and multimodal LLMs. For standard LLMs, it automatically finds all linear layers except `lm_head` and attaches tuners. **For multimodal LLMs, tuners are by default only attached to the LLM component; this behavior can be controlled via `freeze_llm`, `freeze_vit`, and `freeze_aligner`**.
+- 🔥target_regex: A regular expression to specify LoRA modules. Default is `None`. If provided, `target_modules` is ignored. For example: `--target_regex '^(language_model).*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)$'` applies LoRA to modules matching the pattern. This argument is not limited to LoRA and can be used with other tuners.
+- target_parameters: List of parameter names (not module names) to replace with LoRA. Similar in behavior to `target_modules`, but operates at the parameter level. Requires "peft>=0.17.0". This is useful for models like Mixture-of-Experts (MoE) layers in Hugging Face Transformers, which may use `nn.Parameter` instead of `nn.Linear`.
+  - Note: This parameter requires `lora_dropout` to be set to 0.
+- init_weights: Method for initializing weights. For LoRA: options are 'true', 'false', 'gaussian', 'pissa', 'pissa_niter_[number of iters]', 'olora', 'loftq', 'lora-ga'. For Bone: 'true', 'false', 'bat'. Default is 'true'.
+- 🔥modules_to_save: Additional original model modules to include in training and saving, even after attaching a tuner. Default is `[]`. Applies to tuners beyond LoRA. For example: `--modules_to_save embed_tokens lm_head` enables training of `embed_tokens` and `lm_head` during LoRA training, and their weights will be saved in `adapter_model.safetensors`.
+
+#### Full Arguments
+
+- freeze_parameters: List of parameter name prefixes to freeze. Default is `[]`.
+- freeze_parameters_regex: Regular expression to match parameters to freeze. Default is `None`.
+- freeze_parameters_ratio: Proportion of parameters to freeze, from bottom to top layers. Default is `0`. Setting to `1` freezes all parameters; can be combined with `trainable_parameters` to specify trainable parts.
+- trainable_parameters: Prefixes of additional parameters to keep trainable. Default is `[]`.
+- trainable_parameters_regex: Regex to match additional trainable parameters. Default is `None`.
+  - Note: `trainable_parameters` and `trainable_parameters_regex` have higher priority than `freeze_parameters`, `freeze_parameters_regex`, and `freeze_parameters_ratio`. For example, in full-parameter training, all modules are first set to trainable, then some are frozen based on the freeze rules, and finally some are re-enabled via `trainable_parameters` or `trainable_parameters_regex`.
+
+#### LoRA
+
+- 🔥lora_rank: Default is `8`.
+- 🔥lora_alpha: Default is `32`.
+- lora_dropout: Default is `0.05`.
+- lora_bias: Defaults to `'none'`. Possible values are 'none', 'all'. If you want to make all biases trainable, you can set it to `'all'`.
+- lora_dtype: Specifies the data type (dtype) for the LoRA modules. Supported values are 'float16', 'bfloat16', 'float32'. Default is None, which follows the default behavior of PEFT.
+- 🔥use_dora: Defaults to `False`, indicating whether to use `DoRA`.
+- use_rslora: Defaults to `False`, indicating whether to use `RS-LoRA`.
+- 🔥lorap_lr_ratio: Parameter for LoRA+. Default is `None`. Recommended values: `10–16`. Setting this when using LoRA enables the LoRA+ variant.
+
+
+##### LoRA-GA
+- lora_ga_batch_size: The default value is `2`. The batch size used for estimating gradients during initialization in LoRA-GA.
+- lora_ga_iters: The default value is `2`. The number of iterations for estimating gradients during initialization in LoRA-GA.
+- lora_ga_max_length: The default value is `1024`. The maximum input length for estimating gradients during initialization in LoRA-GA.
+- lora_ga_direction: The default value is `ArB2r`. The initial direction used for gradient estimation during initialization in LoRA-GA. Allowed values are: `ArBr`, `A2rBr`, `ArB2r`, and `random`.
+- lora_ga_scale: The default value is `stable`. The scaling method for initialization in LoRA-GA. Allowed values are: `gd`, `unit`, `stable`, and `weightS`.
+- lora_ga_stable_gamma: The default value is `16`. The gamma value when choosing `stable` scaling for initialization.
+
+#### FourierFt
+
+FourierFt uses three parameters: `target_modules`, `target_regex`, and `modules_to_save`, whose meanings are described in the documentation above. Additional parameters include:
+
+- fourier_n_frequency: Number of frequencies in Fourier transform, an `int`, similar to `r` in LoRA. Default value is `2000`.
+- fourier_scaling: Scaling value of matrix W, a `float`, similar to `lora_alpha` in LoRA. Default value is `300.0`.
+
+#### BOFT
+
+BOFT uses the three parameters `target_modules`, `target_regex`, and `modules_to_save`, whose meanings are described in the documentation above. Additional parameters include:
+
+- boft_block_size: Size of BOFT blocks, default value is 4.
+- boft_block_num: Number of BOFT blocks, cannot be used simultaneously with `boft_block_size`.
+- boft_dropout: Dropout value for BOFT, default is 0.0.
+
+#### Vera
+
+Vera uses the three parameters `target_modules`, `target_regex`, and `modules_to_save`, whose meanings are described in the documentation above. Additional parameters include:
+
+- vera_rank: Size of Vera Attention, default value is 256.
+- vera_projection_prng_key: Whether to store the Vera mapping matrix, default is True.
+- vera_dropout: Dropout value for Vera, default is `0.0`.
+- vera_d_initial: Initial value of Vera's d matrix, default is `0.1`.
+
+#### GaLore
+
+- 🔥use_galore: Default value is False, whether to use GaLore.
+- galore_target_modules: Default is None, if not provided, applies GaLore to attention and MLP.
+- galore_rank: Default value is 128, GaLore rank value.
+- galore_update_proj_gap: Default is 50, interval for updating decomposed matrices.
+- galore_scale: Default is 1.0, matrix weight coefficient.
+- galore_proj_type: Default is `std`, type of GaLore matrix decomposition.
+- galore_optim_per_parameter: Default value is False, whether to set a separate optimizer for each Galore target parameter.
+- galore_with_embedding: Default value is False, whether to apply GaLore to embedding.
+- galore_quantization: Whether to use q-galore, default is `False`.
+- galore_proj_quant: Whether to quantize the SVD decomposition matrix, default is `False`.
+- galore_proj_bits: Number of bits for SVD quantization.
+- galore_proj_group_size: Number of groups for SVD quantization.
+- galore_cos_threshold: Cosine similarity threshold for updating projection matrices. Default value is 0.4.
+- galore_gamma_proj: As the projection matrix becomes more similar over time, this parameter is the coefficient for extending the update interval. Default value is 2.
+- galore_queue_size: Length of the queue for calculating projection matrix similarity, default is 5.
+
+#### LISA
+
+Note: LISA only supports full parameters, i.e., `--tuner_type full`.
+
+- 🔥lisa_activated_layers: Default value is `0`, representing LISA is not used. Setting to a non-zero value activates that many layers, it is recommended to set to 2 or 8.
+- lisa_step_interval: Default value is `20`, number of iter to switch to layers that can be backpropagated.
+
+#### UNSLOTH
+
+🔥Unsloth has no additional parameters; it can be supported by adjusting existing parameters, for example:
+
+```
+--tuner_backend unsloth
+--tuner_type full/lora
+--quant_bits 4
+```
+
+#### LLAMAPRO
+
+- 🔥llamapro_num_new_blocks: Default value is `4`, total number of new layers to insert.
+- llamapro_num_groups: Default value is `None`, number of groups to insert new blocks. If `None`, it equals `llamapro_num_new_blocks`, meaning each new layer is inserted separately into the original model.
+
+#### AdaLoRA
+
+When the `tuner_type` parameter is set to `adalora`, the following parameters take effect. The `adalora` parameters such as `target_modules` inherit from the corresponding parameters of `lora`, but the `lora_dtype` parameter does not take effect.
+
+- adalora_target_r: Default value is `8`, average rank of AdaLoRA.
+- adalora_init_r: Default value is `12`, initial rank of AdaLoRA.
+- adalora_tinit: Default value is `0`, initial warmup of AdaLoRA.
+- adalora_tfinal: Default value is `0`, final warmup of AdaLoRA.
+- adalora_deltaT: Default value is `1`, step interval of AdaLoRA.
+- adalora_beta1: Default value is `0.85`, EMA parameter of AdaLoRA.
+- adalora_beta2: Default value is `0.85`, EMA parameter of AdaLoRA.
+- adalora_orth_reg_weight: Default value is `0.5`, regularization parameter for AdaLoRA.
+
+#### ReFT
+
+The following parameters are effective when `tuner_type` is set to `reft`.
+
+> 1. ReFT cannot merge tuners.
+> 2. ReFT is not compatible with gradient checkpointing.
+> 3. If experiencing issues while using DeepSpeed, please uninstall DeepSpeed temporarily.
+
+- 🔥reft_layers: Which layers ReFT is applied to, default is `None`, representing all layers. You can input a list of layer numbers, e.g., `reft_layers 1 2 3 4`.
+- 🔥reft_rank: Rank of ReFT matrix, default is `4`.
+- reft_intervention_type: Type of ReFT, supports 'NoreftIntervention', 'LoreftIntervention', 'ConsreftIntervention', 'LobireftIntervention', 'DireftIntervention', 'NodireftIntervention', default is `LoreftIntervention`.
+- reft_args: Other supported parameters for ReFT Intervention, input in json-string format.
+
+### vLLM Arguments
+
+Parameter meanings can be found in the [vllm documentation](https://docs.vllm.ai/en/latest/serving/engine_args.html).
+
+- 🔥vllm_gpu_memory_utilization: GPU memory ratio, ranging from 0 to 1. Default is `0.9`.
+- 🔥vllm_tensor_parallel_size: Tensor parallelism size. Default is `1`.
+- vllm_pipeline_parallel_size: Pipeline parallelism size. Default is `1`.
+- vllm_data_parallel_size: Data parallelism size, default is `1`, effective in the `swift deploy/rollout` command.
+  - In `swift infer`, use `NPROC_PER_NODE` to set the data parallelism (DP) degree. See the example [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_ddp.sh).
+- vllm_enable_expert_parallel: Enable expert parallelism. Default is False.
+- vllm_max_num_seqs: Maximum number of sequences to be processed in a single iteration. Default is `256`.
+- 🔥vllm_max_model_len: The maximum sequence length supported by the model. Default is `None`, meaning it will be read from `config.json`.
+- vllm_disable_custom_all_reduce: Disables the custom all-reduce kernel and falls back to NCCL. For stability, the default is `True`.
+- vllm_enforce_eager: Determines whether vllm uses PyTorch eager mode or constructs a CUDA graph, default is `False`. Setting it to True can save memory but may affect efficiency.
+- vllm_mm_processor_cache_gb: The size (in GiB) of the multimodal processor cache, used to store processed multimodal inputs (e.g., images, videos) to avoid redundant processing. Default is 4. Setting it to 0 disables the cache but may degrade performance (not recommended). This option takes effect only for multimodal models.
+- vllm_speculative_config: Speculative decoding configuration, passed as a JSON string. Default: None.
+- vllm_disable_cascade_attn: Whether to forcibly disable the V1 engine’s cascade-attention implementation to avoid potential numerical issues. Defaults to False; vLLM’s internal heuristics determine whether cascade attention is actually used.
+- 🔥vllm_limit_mm_per_prompt: Controls the use of multiple media in vllm, default is `None`. For example, you can pass in `--vllm_limit_mm_per_prompt '{"image": 5, "video": 2}'`.
+- vllm_max_lora_rank: Default is `16`. This is the parameter supported by vllm for lora.
+- vllm_quantization: vllm is able to quantize model with this argument, supported values can be found [here](https://docs.vllm.ai/en/latest/serving/engine_args.html).
+- 🔥vllm_enable_prefix_caching: Enables vLLM's automatic prefix caching to save processing time for repeated prompt prefixes, improving inference efficiency. Default is `None`, following vLLM's default behavior.
+- vllm_use_async_engine: Whether to use the async engine under the vLLM backend. Default is None, which is automatically set based on the scenario: encode tasks (embedding, seq_cls, reranker, generative_reranker) default to True, deployment scenarios (swift deploy) default to True, and other scenarios default to False. Note: Encode tasks must use the async engine.
+- vllm_reasoning_parser: Reasoning parser type, used for parsing the chain of thought content of reasoning models. Default is `None`. Only used for the `swift deploy` command. Available types can be found in the [vLLM documentation](https://docs.vllm.ai/en/latest/features/reasoning_outputs.html#streaming-chat-completions).
+- vllm_engine_kwargs: Extra arguments for vllm, formatted as a JSON string. Default is `None`.
+
+### SGLang Arguments
+Parameter meanings can be found in the [sglang documentation](https://docs.sglang.ai/backend/server_arguments.html).
+
+- 🔥sglang_tp_size: Tensor parallelism size. Default is 1.
+- sglang_pp_size: Pipeline parallelism size. Default is 1.
+- sglang_dp_size: Data parallelism size. Default is 1.
+- sglang_ep_size: Expert parallelism size. Default is 1.
+- sglang_enable_ep_moe: Whether to enable EP MoE. Default is False. This parameter has been removed in the latest version of SGLang.
+- sglang_mem_fraction_static: The fraction of GPU memory used for static allocation (model weights and KV cache memory pool). If you encounter out-of-memory errors, try reducing this value. Default is None.
+- sglang_context_length: The maximum context length of the model. Default is None, which means it will use the value from the model's `config.json`.
+- sglang_disable_cuda_graph: Disables CUDA graph. Default is False.
+- sglang_quantization: Quantization method. Default is None.
+- sglang_kv_cache_dtype: Data type for KV cache storage. 'auto' means it will use the model's data type. 'fp8_e5m2' and 'fp8_e4m3' are supported on CUDA 11.8 and above. Default is 'auto'.
+- sglang_enable_dp_attention: Enables data parallelism for attention and tensor parallelism for FFN. The data parallelism size (dp size) should be equal to the tensor parallelism size (tp size). Currently supports DeepSeek-V2/3 and Qwen2/3 MoE models. Default is False.
+- sglang_disable_custom_all_reduce: Disables the custom all-reduce kernel and falls back to NCCL. For stability, the default is True.
+- sglang_speculative_algorithm: Speculative algorithm. Available options: None, "EAGLE", "EAGLE3", "NEXTN", "STANDALONE", "NGRAM". Default is None.
+- sglang_speculative_num_steps: The number of steps sampled from the draft model in speculative decoding. Default is None.
+- sglang_speculative_eagle_topk: The number of tokens sampled from the draft model at each step in the EAGLE2 algorithm. Default is None.
+- sglang_speculative_num_draft_tokens: The number of tokens sampled from the draft model in speculative decoding. Default is None.
+
+
+### LMDeploy Arguments
+
+Parameter meanings can be found in the [lmdeploy documentation](https://lmdeploy.readthedocs.io/en/latest/api/pipeline.html#turbomindengineconfig).
+
+- 🔥lmdeploy_tp: tensor parallelism degree. Default is `1`.
+- lmdeploy_session_len: Maximum session length. Default is `None`.
+- lmdeploy_cache_max_entry_count: The percentage of GPU memory occupied by the k/v cache. Default is `0.8`.
+- lmdeploy_quant_policy: Default is `0`. Set it to `4` or `8` when quantizing k/v to 4-bit or 8-bit, respectively.
+- lmdeploy_vision_batch_size: The `max_batch_size` parameter passed to `VisionConfig`. Default is `1`.
+
+### Merge Arguments
+
+- 🔥merge_lora: Indicates whether to merge lora; this parameter supports lora, llamapro, and longlora, default is `False`. Example parameters [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/merge_lora.sh).
+- safe_serialization: Whether to save the model in safetensors format. Default is True.
+- max_shard_size: Maximum size of a single storage file, default is '5GB'.
+
+## Integration Arguments
+
+### Training Arguments
+
+Training arguments include the [base arguments](#base-arguments), [Seq2SeqTrainer arguments](#Seq2SeqTrainer-arguments), [tuner arguments](#tuner-arguments), and also include the following parts:
+
+- add_version: Add directory to `output_dir` with `'<version>-<timestamp>'` to prevent weight overwrite, default is True.
+- check_model: Check local model files for corruption or modification and give a prompt, default is True. **If in an offline environment, please set to False.**
+- 🔥create_checkpoint_symlink: Creates additional checkpoint symlinks to facilitate writing automated training scripts. The symlink paths for `best_model` and `last_model` are `f'{output_dir}/best'` and `f'{output_dir}/last'` respectively.
+- 🔥packing: Uses the `padding_free` approach to pack data samples of different lengths into samples of **approximately** uniform length (packing ensures complete sequences are not split), achieving load balancing across nodes and processes during training (avoiding long texts slowing down short text training speed), thereby improving GPU utilization and maintaining stable memory usage. When using `--attn_impl flash_attn`, it ensures that different sequences within packed samples are independent and invisible to each other. This parameter defaults to `False` and currently supports packing for CPT/SFT/DPO/KTO/GKD as well as embedding/reranker/seq_cls tasks. Note: **packing will reduce the number of dataset samples, please adjust gradient accumulation and learning rate accordingly**.
+  - Note: For Qwen3-Next packing, please use Megatron-SWIFT. For Qwen3.5 transformers padding_free/packing support, please use "ms-swift>=4.3.1" (or use Megatron-SWIFT). Refer to [Qwen3.5 Best Practice](../BestPractices/Qwen3_5-Best-Practice.md) for details.
+- packing_length: the length to use for packing. Defaults to None, in which case it is set to max_length.
+- packing_num_proc: Number of processes for packing, default is 1. Note that different values of `packing_num_proc` will result in different packed datasets. (This parameter does not take effect during streaming packing). Usually there is no need to modify this value, as packing speed is much faster than tokenization speed.
+- packing_strategy: Packing algorithm, one of 'binpack' and 'sequential', default is 'binpack'. 'binpack' uses best-fit-decreasing bin packing (which reorders samples by length); 'sequential' uses order-preserving greedy packing (next-fit: a single open pack, flushed when the next sample doesn't fit), keeping samples in input order so the sample order and pack boundaries follow a sequential sampler (use `packing_num_proc=1` for a single global ordering).
+- lazy_tokenize: Whether to use lazy tokenization. If set to `False`, all dataset samples will be tokenized (and for multimodal models, images will be loaded from disk) before training begins. Default is `None`: in LLM training, it defaults to `False`; in MLLM training, it defaults to `True` to save memory.
+  - Note: If you want to perform image data augmentation, you need to set `lazy_tokenize` (or `streaming`) to True and modify the `encode` method in the Template class.
+- use_logits_to_keep: Pass `logits_to_keep` in the `forward` method based on labels to reduce the computation and storage of unnecessary logits, thereby reducing memory usage and accelerating training. The default is `None`, which enables automatic selection.
+- acc_strategy: Strategy for calculating accuracy during training and validation. Options are `seq`-level and `token`-level accuracy, with `token` as the default.
+- max_new_tokens: Generation parameter override. The maximum number of tokens to generate when `predict_with_generate=True`, defaulting to 64.
+- temperature: Generation parameter override. The temperature setting when `predict_with_generate=True`, defaulting to 0.
+- optimizer: The optimizer plugin to use (takes priority over `--optim`), default is None. Available optimizers can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/optimizers/mapping.py).
+- loss_type: Custom loss_type name. Default is None, uses the model's built-in loss function. Available loss options can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
+- mrl_dims: Dimension configuration for [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) on embedding training. Default is None. Format is `Dict[int, float]` or a JSON string, where the key is the truncated embedding dimension and the value is the corresponding loss weight, e.g. `'{"32": 1.0, "64": 1.0, "128": 1.0}'`. When enabled, the trainer slices `last_hidden_state` to each dimension, applies L2 normalization, and aggregates the per-dimension `loss_type` losses with the configured weights. Only effective when `task_type='embedding'`.
+  - Note: The maximum supported embedding dimension is determined by `hidden_size` in the model's `config.json`. Any key-value pair whose key is greater than `hidden_size` will be silently ignored.
+- eval_metric: Custom eval metric name. Default is None. Available eval_metric options can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/metrics/mapping.py).
+  - Regarding default values: When `task_type` is 'causal_lm' and `predict_with_generate=True`, it defaults to 'nlg'. When `task_type` is 'embedding', the default value is 'infonce' or 'paired' based on loss_type. When `task_type` is 'reranker/generative_reranker', the default value is 'reranker'.
+- callbacks: Custom trainer callbacks, default is `[]`. Available callbacks can be found [here](https://github.com/modelscope/ms-swift/blob/main/swift/callbacks/mapping.py).For example,enable elastic training by adding `deepspeed_elastic` (and optionally `graceful_exit`) in `callbacks`. See the [Elastic guide](../BestPractices/Elastic.md).
+- early_stop_interval: The interval for early stopping. Training will terminate when best_metric shows no improvement within early_stop_interval periods (based on `save_steps`; it's recommended to set `eval_steps` and `save_steps` to the same value). The specific implementation can be found in [early_stop.py](https://github.com/modelscope/ms-swift/blob/main/swift/callbacks/early_stop.py). Additionally, if you have more complex early stopping requirements, you can directly override the existing implementation in callback.py. When this parameter is set, the `early_stop` trainer callback is automatically added.
+- eval_use_evalscope: Whether to use evalscope for evaluation, this parameter needs to be set to enable evaluation, refer to [example](../Instruction/Evaluation.md#evaluation-during-training). Default is False.
+- eval_dataset: Evaluation datasets, multiple datasets can be set, separated by spaces
+- eval_dataset_args: Evaluation dataset parameters in JSON format, parameters for multiple datasets can be set
+- eval_limit: Number of samples from the evaluation dataset
+- eval_generation_config: Model inference configuration during evaluation, in JSON format, default is `{'max_tokens': 512}`
+- use_flash_ckpt: Whether to use [DLRover Flash Checkpoint](https://github.com/intelligent-machine-learning/dlrover). Default is `false`. If enabled, checkpoints are saved to memory synchronously, then persisted to storage asynchronously. It's recommended to use this with the environment variable `PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"` to avoid CUDA OOM.
+
+#### SWANLAB
+
+- swanlab_token: The API key for SwanLab. You can also specify it using the `SWANLAB_API_KEY` environment variable.
+- swanlab_project: The SwanLab project, which can be created in advance on the page [https://swanlab.cn/space/~](https://swanlab.cn/space/~) or created automatically. The default is "ms-swift".
+- swanlab_workspace: Defaults to `None`, will use the username associated with the API key.
+- swanlab_exp_name: Experiment name, can be left empty. If empty, the value of `--output_dir` will be used by default.
+- swanlab_notification_method: The notification method for SwanLab when training completes or errors occur. For details, refer to [here](https://docs.swanlab.cn/plugin/notification-dingtalk.html). Supports 'dingtalk', 'lark', 'email', 'discord', 'wxwork', 'slack'.
+- swanlab_webhook_url: Defaults to None. The webhook URL corresponding to SwanLab's `swanlab_notification_method`.
+- swanlab_secret: Defaults to None. The secret corresponding to SwanLab's `swanlab_notification_method`.
+- swanlab_mode: Optional values are `cloud` and `local`, representing cloud mode or local mode.
+
+### RLHF Arguments
+
+RLHF arguments inherit from the [training arguments](#training-arguments).
+
+- 🔥rlhf_type: Type of human alignment algorithm, supporting 'dpo', 'orpo', 'simpo', 'kto', 'cpo', 'rm', 'ppo', 'grpo' and 'gkd'. Default is 'dpo'.
+- ref_model: Required for full parameter training when using the dpo, kto, ppo or grpo algorithms. Default is None, set to `--model`.
+- ref_adapters: Defaults to `[]`. If you want to use LoRA weights generated from SFT for DPO/KTO/GRPO, set `--adapters sft_ckpt --ref_adapters sft_ckpt` during training. For checkpoint resumption in this scenario, set `--resume_from_checkpoint rlhf_ckpt --ref_adapters sft_ckpt`.
+- ref_model_type: Same as model_type. Default is None.
+- ref_model_revision: Same as model_revision. Default is None.
+- 🔥beta: A parameter controlling the degree of deviation from the reference model. A higher beta value indicates smaller deviation from the reference model. Default is `None`, with different default values depending on the RLHF algorithm: `2.0` for SimPO, `0.04` for GRPO, `0.5` for GKD, and `0.1` for other algorithms. See [documentation](./RLHF.md) for details.
+- label_smoothing: Whether to use DPO smoothing, default value is `0`.
+- max_completion_length: The maximum generation length in the GRPO/PPO/GKD algorithms. Default is 512.
+- 🔥rpo_alpha: A parameter from the [RPO paper](https://arxiv.org/abs/2404.19733) that controls the weight of the NLL term (i.e., the SFT loss) in the loss function, where `loss = dpo_loss + rpo_alpha * sft_loss`. The paper recommends setting it to `1.`. The default value is `None`, meaning the SFT loss is not included by default.
+- ld_alpha: From the [LD-DPO paper](https://arxiv.org/abs/2409.06411). Applies a weight α < 1 to the log-probabilities of tokens that lie beyond the shared prefix of the chosen and rejected responses, thereby mitigating length bias.
+- discopop_tau: Temperature parameter τ from the [DiscoPOP paper](https://arxiv.org/abs/2406.08414) used to scale the log-ratio before the sigmoid modulation. Default 0.05; only active when loss_type is discopop.
+- loss_type: Type of loss function. Default is None, with different defaults depending on the RLHF algorithm used.
+  - DPO: Available options can be found in the [documentation](https://huggingface.co/docs/trl/main/en/dpo_trainer#loss-functions). Multiple values can be provided to enable mixed training ([MPO](https://arxiv.org/abs/2411.10442)); when multiple values are given, the loss_weights parameter must also be set. Default is `sigmoid`.
+  - GRPO: See [GRPO parameters](#grpo-arguments) for reference.
+- loss_weights: When setting multiple loss_type values in DPO training, this parameter specifies the weight for each loss component.
+- cpo_alpha: Coefficient for nll loss in CPO/SimPO loss, default is `1.`.
+- simpo_gamma: Reward margin term in the SimPO algorithm, with a paper-suggested setting of 0.5-1.5, default is `1.`.
+- desirable_weight: In the KTO algorithm, this weight compensates for the imbalance between the number of desirable and undesirable samples by scaling the desirable loss. Default is `1.0`.
+- undesirable_weight: In the KTO algorithm, this weight compensates for the imbalance between desirable and undesirable samples by scaling the undesirable loss. Default is `1.0`.
+- center_rewards_coefficient: A coefficient used in reward model (RM) training to incentivize the model to output rewards with zero mean. See this [paper](https://huggingface.co/papers/2312.09244) for details. Recommended value: 0.01.
+- loss_scale: Overrides the template parameter. During RLHF training, the default is `'last_round'`.
+- temperature: Default is 0.9; this parameter will be used in PPO, GRPO and GKD.
+- top_k: Top-k parameter for rollout sampling. -1 means no top-k filtering is applied. Default is -1.
+- top_p: Top-p parameter for rollout sampling. 1.0 means no top-p filtering is applied. Default is 1.0.
+
+#### GKD Arguments
+- lmbda: Default is 0.5. This parameter is used in GKD. It controls the lambda parameter for the proportion of student data (i.e., the proportion of student-generated outputs within the strategy). If lmbda is 0, student-generated data is not used.
+- sft_alpha: The default value is 0. It controls the weight of sft_loss added in GKD. The final loss is `gkd_loss + sft_alpha * sft_loss`.
+- gkd_logits_topk: Use Top-K logits to compute KL divergence. Defaults to None, which means the full vocabulary is used. Setting this parameter can effectively reduce peak GPU memory usage during training. This parameter is required when teacher_model_server is configured. See [distillation documentation](./Distillation.md#top-k-kl-computation) for more details.
+- truncation_strategy: The method to handle inputs exceeding `max_length`. Supported values are `delete` and `left`, representing deletion and left-side truncation respectively. The default is `left`. With the delete strategy, over-long or encoding-failed samples are discarded, and new samples are resampled from the original dataset to maintain the intended batch size.
+- log_completions: Whether to log the model-generated content during training, to be used in conjunction with `--report_to wandb/swanlab`, default is False.
+  - Note: If `--report_to wandb/swanlab` is not set, a `completions.jsonl` will be created in the checkpoint to store the generated content.
+  - Log vLLM rollout results only.
+
+#### Reward/Teacher Model Parameters
+
+The reward model parameters will be used in PPO and GRPO. The teacher model parameters will be used in GKD and GRPO.
+
+- reward_model: Default is None.
+- reward_adapters: Default is `[]`.
+- reward_model_type: Default is None.
+- reward_model_revision: Default is None.
+- teacher_model: Default is None.
+- teacher_adapters: Default is `[]`.
+- teacher_model_type: Default is None.
+- teacher_model_revision: Default is None.
+- teacher_model_server: Teacher model service URL. Deploy via `swift deploy` for logprobs. Single URL (e.g. `http://localhost:8000`) or multi-teacher JSON (e.g. `'[{"url":"http://t1:8000","tags":["data/math.jsonl"]},{"url":"http://t2:8001","tags":["data/code.jsonl"]}]'`). How `tags` map to datasets or sample fields: [distillation docs](./Distillation.md#multi-teacher-routing).
+- teacher_tag_key: Column name for multi-teacher routing; sample values are matched to each teacher's `tags`. Default is `"dataset"`. With multiple `--dataset` values, match each entry; or use a custom column (e.g. `teacher_tag`) via `--teacher_tag_key teacher_tag`.
+- teacher_deepspeed: Same as the deepspeed parameter, controls the DeepSpeed configuration for the teacher model. By default, uses the DeepSpeed configuration of the training model.
+- offload_teacher_model: Whether to offload the teacher model to save GPU memory. Loaded only during sampling/logps computation. Only effective when `teacher_model` is set. Default is False.
+
+
+#### PPO Arguments
+
+The meanings of the following parameters can be referenced [here](https://huggingface.co/docs/trl/main/ppo_trainer):
+
+- num_ppo_epochs: Defaults to 4
+- whiten_rewards: Defaults to False
+- kl_coef: Defaults to 0.05
+- cliprange: Defaults to 0.2
+- vf_coef: Defaults to 0.1
+- cliprange_value: Defaults to 0.2
+- gamma: Defaults to 1.0
+- lam: Defaults to 0.95
+- num_mini_batches: Defaults to 1
+- local_rollout_forward_batch_size: Defaults to 64
+- num_sample_generations: Defaults to 10
+- missing_eos_penalty: Defaults to None
+
+
+#### GRPO Arguments
+- beta: KL regularization coefficient; default 0.04. Setting it to 0 disables the reference model.
+- per_device_train_batch_size: The training batch size per device. In GRPO, this refers to the batch size of completions during training.
+- per_device_eval_batch_size: The evaluation batch size per device. In GRPO, this refers to the batch size of completions during evaluation.
+- steps_per_generation: Number of optimization steps per generation. It defaults to gradient_accumulation_steps. This parameter and generation_batch_size cannot be set simultaneously.
+- generation_batch_size: Total batch size of sampling completions. It should be a multiple of num_processes * per_device_train_batch_size. It defaults to per_device_train_batch_size * steps_per_generation * num_processes.
+- num_generations: The number of samples generated per prompt (corresponding to the G value in the paper). generation_batch_size must be divisible by num_generations. The default value is 8.
+- num_generations_eval: Number of generations to sample during evaluation. This allows using fewer generations during evaluation to save computation. If `None`, uses the value of `num_generations`. Default is None.
+- ds3_gather_for_generation: This parameter applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation. The default is True.
+- reward_funcs: Reward functions in the GRPO algorithm; options include `accuracy`,`format`,`cosine`,`repetition` and `soft_overlong`, as seen in `swift/rewards/orm.py`. You can also customize your own reward functions in the plugin. Default is `[]`.
+- reward_weights: Weights for each reward function. The number should be equal to the sum of the number of reward functions and reward models. If `None`, all rewards are weighted equally with weight `1.0`.
+  - Note: If `--reward_model` is included in GRPO training, it is added to the end of the reward functions.
+- reward_model_plugin: The logic for the reward model, which defaults to ORM logic. For more information, please refer to [Customized Reward Models](./GRPO/DeveloperGuide/reward_model.md#custom-reward-model).
+- dataset_shuffle: Whether to shuffle the dataset randomly. Default is True.
+- truncation_strategy: The method to handle inputs exceeding `max_length`. Supported values are `delete` and `left`, representing deletion and left-side truncation respectively. The default is `left`. With the delete strategy, over-long or encoding-failed samples are discarded, and new samples are resampled from the original dataset to maintain the intended batch size.
+- loss_type: The type of loss normalization. Options are ['grpo', 'bnpo', 'dr_grpo', 'dapo', 'cispo', 'sapo', 'real', 'fipo'], default is 'grpo'. For details, refer to this [doc](./GRPO/DeveloperGuide/loss_types.md)
+- fipo_decay_rate: Half-life parameter for FIPO Future-KL. The actual discount is `2 ** (-1 / fipo_decay_rate)`. Default is 32.0.
+- fipo_clip_range: Clipping range for the FIPO influence weight. Default is 0.2; set to None or 0 to disable clipping.
+- fipo_clip_high_only: Whether to clip the FIPO influence weight to `[1.0, 1.0 + fipo_clip_range]` only. Default is True.
+- fipo_safety_threshold: Caps the FIPO influence weight to `[0.8, 1.0]` for negative-advantage tokens whose IS ratio exceeds this threshold. Default is 4.0.
+- log_completions: Whether to log the model-generated content during training, to be used in conjunction with `--report_to wandb/swanlab`, default is False.
+  - Note: If `--report_to wandb/swanlab` is not set, a `completions.jsonl` will be created in the checkpoint to store the generated content.
+- use_vllm: Whether to use vLLM as the infer_backend for GRPO generation, default is False.
+- vllm_mode: Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `server` or `colocate`
+- vllm_mode server parameter
+  - vllm_server_host: The host address of the vLLM server. Default is None.
+  - vllm_server_port: The service port of the vLLM server. Default is 8000.
+  - vllm_server_base_url: Base URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` " "and `vllm_server_port` are ignored. Default is None.
+  - vllm_server_timeout: The connection timeout for the vLLM server. Default is 240 seconds.
+  - vllm_server_pass_dataset: pass additional dataset information through to the vLLM server for multi-turn training.
+  - vllm_server_group_port: The internal communication port for the vLLM server. Generally, there is no need to set it unless the port is occupied. The default value is 51216.
+  - async_generate: Use async rollout to improve train speed. Note that rollout will use the model updated in the previous round when enabled. Multi-turn scenarios are not supported. Default is `false`.
+  - enable_flattened_weight_sync: Whether to use flattened tensor for weight synchronization. When enabled, multiple parameters are packed into a single contiguous tensor for transfer, which can improve synchronization efficiency; Only takes effect in Server Mode. Default is True.
+  - SWIFT_UPDATE_WEIGHTS_BUCKET_SIZE: An environment variable that controls the bucket size (in MB) for flattened tensor weight synchronization during full-parameter training in Server Mode. Default is 512 MB.
+- vllm_mode colocate parameter (For more parameter support, refer to the [vLLM Arguments](#vLLM-Arguments).)
+  - vllm_gpu_memory_utilization: vLLM passthrough parameter, default is 0.9.
+  - vllm_max_model_len: vLLM passthrough parameter, the total length limit of model, default is None.
+  - vllm_enforce_eager: vLLM passthrough parameter, default is False.
+  - vllm_limit_mm_per_prompt: vLLM passthrough parameter, default is None.
+  - vllm_enable_prefix_caching: A pass-through parameter for vLLM, default is True.
+  - vllm_tensor_parallel_size: the tensor parallel size of vLLM engine, default is 1.
+  - vllm_enable_lora: Enable the vLLM engine to load LoRA adapters; defaults to False. Used to accelerate weight synchronization during LoRA training. See the [documentation](./GRPO/GetStarted/GRPO.md#weight-sync-acceleration) for details.
+  - sleep_level: make vllm sleep when model is training. Options are 0/1/2, default is 0, no sleep
+  - offload_optimizer: Whether to offload optimizer parameters during inference with vLLM. The default is `False`.
+  - offload_model: Whether to offload the model during inference with vLLM. The default is `False`.
+  - completion_length_limit_scope: Specifies the scope of the `max_completion_length` limit in multi-turn conversations.
+  When set to `total`, the total output length across all turns must not exceed `max_completion_length`.
+  When set to `per_round`, each individual turn's output length is limited separately.
+  Defaults to `per_round`. Currently only takes effect in colocate mode.
+- num_iterations: The number of updates per data sample, corresponding to the $\mu$ value in the GRPO paper. Default is 1.
+- epsilon: epsilon value for clipping. Default is 0.2.
+- epsilon_high: Upper clip coefficient, default is None. When set, it forms a clipping range of [epsilon, epsilon_high] together with epsilon.
+- tau_pos: Temperature parameter for positive advantages in [SAPO](https://arxiv.org/abs/2511.20347) algorithm, controlling the sharpness of the soft gating function. Larger values make the gate sharper (closer to hard clipping), smaller values make it smoother. Default is 1.0.
+- tau_neg: Temperature parameter for negative advantages in SAPO algorithm, controlling the sharpness of the soft gating function. Typically set `tau_neg > tau_pos` to apply stronger constraints on negative advantages. Default is 1.05.
+- dynamic_sample: Exclude data within the group where the reward standard deviation is 0, and additionally sample new data. Default is False.
+- max_resample_times: Under the dynamic_sample setting, limit the number of resampling attempts to a maximum of 3. Default is 3 times.
+- overlong_filter: Skip overlong truncated samples, which will not be included in loss calculation. Default is False.
+The hyperparameters for the reward function can be found in the [Built-in Reward Functions section](#built-in-reward-functions).
+- delta: Delta value for the upper clipping bound in two-sided GRPO. Recommended to be > 1 + epsilon. This method was introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291).
+- importance_sampling_level: Controls how the importance sampling ratio is computed. Options are `token` and `sequence`. In `token` mode, the raw per-token log-probability ratios are used. In `sequence` mode, the log-probability ratios of all valid tokens in the sequence are averaged to produce a single ratio per sequence. The [GSPO paper](https://arxiv.org/abs/2507.18071) uses sequence-level importance sampling to stabilize training. The default is `token`.
+- advantage_estimator: Advantage estimator. Default is `grpo` (group-relative advantage). Options: `grpo`, [`rloo`](./GRPO/AdvancedResearch/RLOO.md), [`reinforce_plus_plus`](./GRPO/AdvancedResearch/REINFORCEPP.md).
+- kl_in_reward: Controls where the KL regularization is applied. `false`: KL is a separate loss term. `true`: KL is subtracted from the reward. The default is bound to `advantage_estimator`: `false` for `grpo`, and `true` for `rloo` and `reinforce_plus_plus`.
+- scale_rewards: Specifies the reward scaling strategy. Options: `group` (scale by intra-group std), `batch` (scale by batch-wide std), `none` (no scaling), `gdpo` (normalize each reward function separately within groups before weighted aggregation, see [GDPO paper](https://arxiv.org/abs/2601.05242)). The default is bound to `advantage_estimator`: `group` for `grpo`, `none` for `rloo`, and `batch` for `reinforce_plus_plus`.
+  - Note: `gdpo` mode does not support `kl_in_reward=True`. If both are set, `kl_in_reward` will be automatically set to `False`.
+  - GDPO is designed for multi-reward optimization: When using multiple reward functions, GDPO normalizes each reward function separately within groups (subtract mean, divide by std), then performs weighted aggregation using `reward_weights`, and finally applies batch-level normalization. This approach better preserves the relative differences between rewards and prevents different reward combinations from collapsing into identical advantage values.
+- teacher_kl_coef: Coefficient for teacher KL in OPD-RL, i.e. `adv_t = base_adv + teacher_kl_coef * teacher_kl`. Default is 1.0.
+- sync_ref_model: Whether to synchronize the reference model. Default is False.
+  - ref_model_mixup_alpha: The Parameter controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: $π_{ref} = α * π_θ + (1 - α) * π_{ref_{prev}}$. Default is 0.6.
+  - ref_model_sync_steps：The parameter determines how frequently the current policy is synchronized with the reference policy. Default is 512.
+- move_model_batches: When moving model parameters to fast inference frameworks such as vLLM/LMDeploy, determines how many batches to divide the layers into. The default is `None`, which means the entire model is not split. Otherwise, the model is split into `move_model_batches + 1` (non-layer parameters) + `1` (multi-modal component parameters) batches.
+- multi_turn_scheduler: Multi-turn GRPO parameter; pass the corresponding plugin name, and make sure to implement it in plugin/multi_turn.py.
+- max_turns: Maximum number of rounds for multi-turn GRPO. The default is None, which means there is no limit.
+- gym_env: Globally select the gym environment name (must be registered in the plugin). Defaults to None and can be overridden per row via `env_config.name`. See the [documentation](./GRPO/DeveloperGuide/gym_env.md).
+- use_gym_env: Whether to use the env-provided `total_reward` as the reward (no reward function required). Defaults to None; when not set explicitly, it is auto-enabled if `gym_env` is set, otherwise inherited from the rollout server in server mode, and False in all other cases.
+- top_entropy_quantile: Only tokens whose entropy ranks within the specified top quantile are included in the loss calculation. The default is 1.0, which means low-entropy tokens are not filtered. For details, refer to the [documentation](./GRPO/AdvancedResearch/entropy_mask.md).
+- log_entropy: Logs the entropy values during training. The default is False. For more information, refer to the [documentation](./GRPO/GetStarted/GRPO.md#logged-metrics).
+- rollout_importance_sampling_mode: Training-inference mismatch correction mode. Options are `token_truncate`, `token_mask`, `sequence_truncate`, `sequence_mask`. Default is None (disabled). For details, refer to the [documentation](./GRPO/AdvancedResearch/training_inference_mismatch.md).
+- rollout_importance_sampling_threshold: Threshold for importance sampling weights, used for truncating or masking extreme weights. Default is 2.0.
+- log_rollout_offpolicy_metrics: Whether to log training-inference mismatch diagnostic metrics (KL, PPL, χ², etc.) when `rollout_importance_sampling_mode` is not set. When `rollout_importance_sampling_mode` is set, metrics are always logged. Default is False.
+- off_policy_sequence_mask_delta: Off-Policy Sequence Masking threshold from [DeepSeek-V3.2 paper](https://arxiv.org/abs/2512.02556). When set, computes `mean(old_policy_logps - policy_logps)` for each sequence. If this value exceeds the threshold AND the sequence has negative advantage, the sequence is masked out from loss computation. Default is None (disabled). For details, refer to the [documentation](./GRPO/AdvancedResearch/training_inference_mismatch.md#off-policy-sequence-masking).
+
+
+##### Reward function parameters
+Refer to the [documentation](./GRPO/DeveloperGuide/reward_function.md) for built-in reward functions.
+
+cosine reward function arguments
+- cosine_min_len_value_wrong (default: -0.5): Reward value corresponding to the minimum length when the answer is incorrect.
+- cosine_max_len_value_wrong (default: 0.0): Reward value corresponding to the maximum length when the answer is incorrect.
+- cosine_min_len_value_correct (default: 1.0): Reward value corresponding to the minimum length when the answer is correct.
+- cosine_max_len_value_correct (default: 0.5): Reward value corresponding to the maximum length when the answer is correct.
+- cosine_max_len (default value equal to the model's maximum generation capacity): Maximum length limit for generated text. Default value equal to max_completion_length
+
+repetition penalty function arguments
+
+- repetition_n_grams (default: 3): Size of the n-gram used to detect repetition.
+- repetition_max_penalty (default: -1.0): Maximum penalty value, which controls the intensity of the penalty.
+
+Soft overlong reward parameters:
+
+- soft_max_length: L_max in the paper, the maximum generation length of the model, default is equal to max_completion_length.
+- soft_cache_length: L_cache in the paper, controls the length penalty interval, which is defined as [soft_max_length - soft_cache_length, soft_max_length].
+
+### Inference Arguments
+
+Inference arguments include the [base arguments](#base-arguments), [merge arguments](#merge-arguments), [vLLM arguments](#vllm-arguments), [LMDeploy arguments](#LMDeploy-arguments), and also contain the following:
+
+- 🔥infer_backend: Inference acceleration backend, supporting four inference engines: 'transformers', 'vllm', 'sglang', and 'lmdeploy'. The default is 'transformers'.
+  - Note: All four engines use SWIFT's template, controlled by `--template_backend`.
+- 🔥max_batch_size: Effective when infer_backend is set to 'transformers'; used for batch inference, with a default value of 1. If set to -1, there is no restriction.
+- 🔥result_path: Path to store inference results (jsonl). Defaults to None. When performing inference/evaluation on datasets, results are saved by default in the checkpoint directory (containing args.json file) or the './result' directory. The final storage path will be printed in the command line (interactive inference or deployment does not save results by default).
+  - Note: If the `result_path` file already exists, results will be appended to it.
+- write_batch_size: The batch size for writing results to result_path. Defaults to 1000. If set to -1, there is no restriction.
+- metric: Evaluate the results of the inference, currently supporting 'acc' and 'rouge'. The default is None, meaning no evaluation is performed.
+- val_dataset_sample: Number of samples from the inference dataset, default is None.
+- reranker_use_activation: Whether to apply sigmoid activation after the score during reranker inference. Default is True.
+
+### Deployment Arguments
+
+Deployment Arguments inherit from the [inference arguments](#inference-arguments).
+
+- host: Service host, default is '0.0.0.0'.
+- port: Port number, default is 8000.
+- api_key: The API key required for access; the default is None.
+- owned_by: Default is `swift`.
+- 🔥served_model_name: Model name for serving, defaults to the model's suffix.
+- verbose: Print detailed logs, with a default value of True.
+  - Note: In `swift app` or `swift eval`, the default is False.
+- log_interval: Interval for printing tokens/s statistics, default is 20 seconds. If set to -1, it will not be printed.
+- max_logprobs: Maximum number of logprobs returned to the client, with a default value of 20.
+
+### Rollout Arguments
+The rollout parameters inherit from the [deployment parameters](#deployment-arguments).
+- multi_turn_scheduler: The scheduler for multi-turn GRPO training. Pass the corresponding plugin name, and ensure the implementation is added in `plugin/multi_turn.py`. Default is `None`. See [documentation](./GRPO/DeveloperGuide/multi_turn.md) for details.
+- max_turns: Maximum number of turns in multi-turn GRPO training. Default is `None`, meaning no limit.
+- gym_env: Globally select the gym environment name (must be registered in the plugin). Defaults to None and can be overridden per row via `env_config.name`. See the [documentation](./GRPO/DeveloperGuide/gym_env.md).
+- use_gym_env: Whether to enable gym-environment mode (rollout emits `total_reward` for the trainer to consume). Defaults to None; when not set explicitly, it is auto-enabled if `gym_env` is set.
+- vllm_enable_lora: Enable the vLLM engine to load LoRA adapters; defaults to False. Used to accelerate weight synchronization during LoRA training. See the [documentation](./GRPO/GetStarted/GRPO.md#weight-sync-acceleration) for details.
+- vllm_max_lora_rank: LoRA parameter for the vLLM engine. Must be greater than or equal to the training lora_rank; it is recommended to set them equal. Defaults to 16.
+- vllm_enable_expert_parallel: Enable Expert Parallel (EP) for MoE models, distributing experts across ranks. Valid when vllm_use_async_engine is true. Default is False.
+
+### Web-UI Arguments
+- server_name: Host for the web UI, default is '0.0.0.0'.
+- server_port: Port for the web UI, default is 7860.
+- share: Default is False.
+- lang: Language for the web UI, options are 'zh', 'en'. Default is 'zh'.
+
+
+### App Arguments
+App parameters inherit from [deployment arguments](#deployment-arguments) and [Web-UI Arguments](#web-ui-arguments).
+
+- base_url: The base URL for model deployment, for example, `http://localhost:8000/v1`. The default value is `None`, which means using local deployment.
+- studio_title: Title of the studio. Default is None, set to the model name.
+- is_multimodal: Whether to launch the multimodal version of the app. Defaults to None, automatically determined based on the model; if it cannot be determined, set to False.
+- lang: Overrides the Web-UI Arguments, default is 'en'.
+
+### Evaluation Arguments
+
+Evaluation Arguments inherit from the [deployment arguments](#deployment-arguments).
+
+- 🔥eval_backend: Evaluation backend, defaults to 'Native'. It can also be specified as 'OpenCompass' or 'VLMEvalKit'.
+- 🔥eval_dataset: Evaluation dataset, please refer to the [evaluation documentation](./Evaluation.md).
+- eval_limit: Number of samples per evaluation set, defaults to None.
+- eval_output_dir: Directory to store evaluation results, defaults to 'eval_output'.
+- temperature: Override generation parameters, defaults to 0.
+- eval_num_proc: Maximum client concurrency during evaluation, defaults to 16.
+- eval_url: Evaluation URL, e.g., `http://localhost:8000/v1`. Examples can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/eval/eval_url). Defaults to None for local deployment evaluation.
+- eval_generation_config: Model inference configuration during evaluation, should be passed as a JSON string, e.g., `'{"max_new_tokens": 512}'`; defaults to None.
+- extra_eval_args: Additional evaluation parameters, should be passed as a JSON string, defaults to empty. Only effective for Native evaluation. For more parameter descriptions, please refer to [here](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html).
+- local_dataset: Some evaluation sets, such as `CMB`, require additional data packages to be downloaded for utilization. Setting this parameter to `true` will automatically download the full data package, create a `data` folder in the current directory, and start the evaluation. The data package will only be downloaded once, and future evaluations will use the cache. This parameter defaults to `false`.
+  - Note: By default, evaluation uses the dataset under `~/.cache/opencompass`. After specifying this parameter, it will directly use the data folder in the current directory.
+
+
+### Export Arguments
+
+Export Arguments include the [basic arguments](#base-arguments) and [merge arguments](#merge-arguments), and also contain the following:
+
+- 🔥output_dir: The path for storing exported results. The default value is None, and an appropriate suffix path will be automatically set.
+- exist_ok: If output_dir exists, do not raise an exception and overwrite the contents. The default value is False.
+- 🔥quant_method: Options are 'gptq', 'awq', 'bnb' or 'fp8', with the default being None. Examples can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize).
+- quant_n_samples: The number of samples for the validation set used by gptq/awq, with a default of 256.
+- quant_batch_size: Quantization batch size, default is 1.
+- group_size: Group size for quantization, default is 128.
+- to_cached_dataset: pre-tokenize the dataset and export it in advance, default is False. See the example [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/cached_dataset). For more information, please refer to cached_dataset.
+  - Note: You can specify the validation set content through `--split_dataset_ratio` or `--val_dataset`.
+- template_mode: Used to support the `cached_dataset` feature for `swift rlhf` training. This parameter only takes effect when `--to_cached_dataset true` is set. Available options include: 'train', 'rlhf', and 'kto'. Among them, `swift pt/sft` uses 'train', `swift rlhf --rlhf_type kto` uses 'kto', and other rlhf algorithms use 'rlhf'. Note: Currently, 'gkd', 'ppo', and 'grpo' algorithms do not support the `cached_dataset` feature. Default is 'train'.
+- to_ollama: Generate the Modelfile required by Ollama. Default is False.
+- 🔥to_mcore: Convert weights from HF format to Megatron format. Default is False.
+- to_hf: Convert weights from Megatron format to HF format. Default is False.
+- mcore_model: Path to the mcore format model. Default is None.
+- mcore_adapter: The adapter path for mcore format models, default is None.
+- thread_count: The number of model slices when `--to_mcore true` is set. Defaults to None, and is automatically configured based on the model size, ensuring that the largest slice is less than 10GB.
+- 🔥offload_bridge: Store Megatron exported HF format weights for vLLM updates in CPU main memory to reduce GPU memory usage. Default is False.
+- 🔥test_convert_precision: Test the precision error when converting weights between HF and Megatron formats. Default is False.
+- test_convert_dtype: The dtype used for conversion precision testing, defaults to 'float32'.
+- 🔥push_to_hub: Whether to push to the hub, with the default being False. Examples can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/push_to_hub.sh).
+- hub_model_id: Model ID for pushing, default is None.
+- hub_private_repo: Whether it is a private repo, default is False.
+- commit_message: Commit message, default is 'update files'.
+
+### Sampling Parameters
+
+- prm_model: The type of process reward model. It can be a model ID (triggered using `transformers`) or a `prm` key defined in a plugin (for custom inference processes).
+- orm_model: The type of outcome reward model, typically a wildcard or test case, usually defined in a plugin.
+- sampler_type: The type of sampling. Currently supports `sample` and `distill`.
+- sampler_engine: Supports `transformers`, `lmdeploy`, `vllm`, `no`. Defaults to `transformers`. Specifies the inference engine for the sampling model.
+- output_dir: The output directory. Defaults to `sample_output`.
+- output_file: The name of the output file. Defaults to `None`, which uses a timestamp as the filename. When provided, only the filename should be passed without the directory, and only JSONL format is supported.
+- override_exist_file: Whether to overwrite if `output_file` already exists.
+- num_sampling_batch_size: The batch size for each sampling operation.
+- num_sampling_batches: The total number of batches to sample.
+- n_best_to_keep: The number of best sequences to return.
+- data_range: The partition of the dataset being processed for this sampling operation. The format should be `2 3`, meaning the dataset is divided into 3 parts, and this instance is processing the 3rd partition (this implies that typically three `swift sample` processes are running in parallel).
+- temperature: Defaults to `1.0`.
+- prm_threshold: The PRM threshold. Results below this value will be filtered out. The default value is `0`.
+- easy_query_threshold: For each query, if the ORM evaluation is correct for more than this proportion of all samples, the query will be discarded to prevent overly simple queries from appearing in the results. Defaults to `None`, meaning no filtering is applied.
+- engine_kwargs: Additional parameters for the `sampler_engine`, passed as a JSON string, for example, `{"cache_max_entry_count":0.7}`.
+- num_return_sequences: The number of original sequences returned by sampling. Defaults to `64`. This parameter is effective for `sample` sampling.
+- cache_files: To avoid loading both `prm` and `generator` simultaneously and causing GPU memory OOM, sampling can be done in two steps. In the first step, set `prm` and `orm` to `None`, and all results will be output to a file. In the second run, set `sampler_engine` to `no` and pass `--cache_files` with the output file from the first sampling. This will use the results from the first run for `prm` and `orm` evaluation and output the final results.
+  - Note: When using `cache_files`, the `--dataset` still needs to be provided because the ID for `cache_files` is calculated using the MD5 of the original data. Both pieces of information need to be used together.
+
+## Specific Model Arguments
+
+In addition to the parameters listed above, some models support additional model-specific arguments. The meanings of these parameters can usually be found in the corresponding model's official repository or its inference code. **MS-Swift includes these parameters to ensure that the trained model aligns with the behavior of the official inference implementation**.
+
+- Model-specific parameters can be set via `--model_kwargs` or environment variables. For example: `--model_kwargs '{"fps_max_frames": 12}'` or `FPS_MAX_FRAMES=12`.
+- Note: If you specify model-specific parameters during training, please also set the corresponding parameters during inference to achieve optimal performance.
+
+
+### qwen2_vl, qvq, qwen2_5_vl, mimo_vl, keye_vl, keye_vl_1_5
+These parameters have the same meaning as in `qwen_vl_utils<0.0.12` or the `qwen_omni_utils` library. See [here](https://github.com/QwenLM/Qwen2.5-VL/blob/main/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L24) for details. MS-Swift adjusts these constant values to control image resolution and video frame rate, preventing out-of-memory (OOM) errors during training.
+
+- IMAGE_FACTOR: Default is 28.
+- MIN_PIXELS: Default is `4 * 28 * 28`. Minimum image resolution. It is recommended to set this as a multiple of 28×28.
+- 🔥MAX_PIXELS: Default is `16384 * 28 * 28`. Maximum image resolution. It is recommended to set this as a multiple of 28×28.
+- MAX_RATIO: Default is 200.
+- VIDEO_MIN_PIXELS: Default is `128 * 28 * 28`. Minimum resolution per frame in a video. Recommended to be a multiple of 28×28.
+- 🔥VIDEO_MAX_PIXELS: Default is `768 * 28 * 28`. Maximum resolution per frame in a video. Recommended to be a multiple of 28×28.
+- VIDEO_TOTAL_PIXELS: Default is `24576 * 28 * 28`.
+- FRAME_FACTOR: Default is 2.
+- FPS: Default is 2.0.
+- FPS_MIN_FRAMES: Default is 4. Minimum number of frames extracted from a video clip.
+- 🔥FPS_MAX_FRAMES: Default is 768. Maximum number of frames extracted from a video clip.
+- 🔥QWENVL_BBOX_FORMAT: Specifies whether to use `'legacy'` or `'new'` format for grounding. The `'legacy'` format is: `<|object_ref_start|>a dog<|object_ref_end|><|box_start|>(432,991),(1111,2077)<|box_end|>`. The `'new'` format refers to: [Qwen3-VL Cookbook](https://github.com/QwenLM/Qwen3-VL/blob/main/cookbooks/2d_grounding.ipynb). For dataset formatting, see the [Grounding Dataset Format Documentation](../Customization/Custom-dataset.md#grounding). Default: `'legacy'`.
+  - Note: This environment variable applies to Qwen2/2.5/3-VL and Qwen2.5/3-Omni series models.
+
+### qwen2_audio, qwen3_asr
+- SAMPLING_RATE: Default is 16000
+
+### qwen3_vl, qwen3_5
+The parameter meanings are the same as in the `qwen_vl_utils>=0.0.14` library — see here: https://github.com/QwenLM/Qwen2.5-VL/blob/main/qwen-vl-utils/src/qwen_vl_utils/vision_process.py#L24. By passing the following environment variables you can override the library's global default values: (It is also compatible with environment variables used by `qwen2_5_vl`, such as: `MAX_PIXELS`, `VIDEO_MAX_PIXELS`, and will perform automatic conversion.)
+
+
+- SPATIAL_MERGE_SIZE: default 2.
+- IMAGE_MIN_TOKEN_NUM: default `4`, denotes the minimum number of image tokens per image.
+- 🔥IMAGE_MAX_TOKEN_NUM: default `16384`, denotes the maximum number of image tokens per image. (used to avoid OOM)
+  - Note: The equivalent maximum image pixel count is `IMAGE_MAX_TOKEN_NUM * 32 * 32`.
+- VIDEO_MIN_TOKEN_NUM: default `128`, denotes the minimum number of video tokens per frame.
+- 🔥VIDEO_MAX_TOKEN_NUM: default `768`, denotes the maximum number of video tokens per frame. (used to avoid OOM)
+- MAX_RATIO: default 200.
+- FRAME_FACTOR: default 2.
+- FPS: default 2.0.
+- FPS_MIN_FRAMES: default 4, denotes the minimum number of sampled frames for a video segment.
+- 🔥FPS_MAX_FRAMES: default 768, denotes the maximum number of sampled frames for a video segment. (used to avoid OOM)
+
+
+### qwen2_5_omni, qwen3_omni
+qwen2_5_omni not only includes the model-specific parameters of qwen2_5_vl and qwen2_audio, but also contains the following parameter: (Note: qwen3_omni includes model-specific parameters of **qwen3_vl** and qwen2_audio)
+- USE_AUDIO_IN_VIDEO: Whether to use audio information from video. Default is `False`.
+- 🔥ENABLE_AUDIO_OUTPUT: Defaults to None, which means the value from `config.json` will be used. If training with zero3, please set it to False.
+  - Tip: ms-swift only fine-tunes the "thinker" component; it is recommended to set this to `False` to reduce GPU memory usage (only the thinker part of the model structure will be created).
+
+
+### qwen3_vl_emb, qwen3_vl_reranker
+
+The parameter meanings are the same as `qwen3_vl`, see the description above. The following are overrides to the default values:
+
+- IMAGE_MAX_TOKEN_NUM: Default is 1800 for qwen3_vl_emb, and 1280 for qwen3_vl_reranker. For details, please refer to: [qwen3_vl_embedding](https://modelscope.cn/models/Qwen/Qwen3-VL-Embedding-2B/file/view/master/scripts%2Fqwen3_vl_embedding.py?status=1#L26), [qwen3_vl_reranker](https://modelscope.cn/models/Qwen/Qwen3-VL-Reranker-2B/file/view/master/scripts%2Fqwen3_vl_reranker.py?status=1#L16).
+- FPS: Default is 1.
+- FPS_MAX_FRAMES: Default is 64.
+
+
+### internvl_chat
+For the meaning of the arguments, please refer to [here](https://modelscope.cn/models/OpenGVLab/InternVL2_5-2B)
+- MAX_NUM: Default is 12
+- INPUT_SIZE: Default is 448
+- VIDEO_MAX_NUM: Default is 1, which is the MAX_NUM for videos
+- VIDEO_SEGMENTS: Default is 8
+
+### minicpmv2_6, minicpmv4, minicpmo
+- MAX_SLICE_NUMS: Default is 9, refer to [here](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6/file/view/master?fileName=config.json&status=1)
+- VIDEO_MAX_SLICE_NUMS: Default is 1, which is the MAX_SLICE_NUMS for videos, refer to [here](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6)
+- MAX_NUM_FRAMES: Default is 64, refer to [here](https://modelscope.cn/models/OpenBMB/MiniCPM-V-2_6)
+
+
+### minicpmv4_6
+- DOWNSAMPLE_MODE: Default is `'16x'`. Visual token downsampling mode. `'16x'` merges tokens for efficiency; `'4x'` retains 4x more tokens for finer detail.
+- MAX_SLICE_NUMS: Default is 9. Maximum number of slices when splitting a high-resolution image. A larger value preserves more detail for large images. It is recommended to set this to 36 for images.
+- VIDEO_MAX_SLICE_NUMS: Default is 1. `MAX_SLICE_NUMS` for videos.
+- MAX_NUM_FRAMES: Default is 128. Maximum number of main frames sampled from a video.
+- STACK_FRAMES: Default is 1. Total number of samples per second. `1` means only the main frame is used (no stacking). `N` (N>1) means 1 main frame plus N−1 sub-frames per second; sub-frames are composed into a grid image and interleaved with the main frame. It is recommended to set this to 1 for short videos and 3 or 5 for long videos.
+
+
+### minicpmo
+- INIT_TTS: Defaults to False. Whether to initialize and load the TTS model.
+- INIT_AUDIO: Defaults to True. Whether to initialize and load the Audio model.
+- USE_AUDIO_IN_VIDEO: Defaults to False. Whether to use the audio information from the video.
+
+### ovis1_6, ovis2
+- MAX_PARTITION: Default is 9, refer to [here](https://github.com/AIDC-AI/Ovis/blob/d248e34d755a95d24315c40e2489750a869c5dbc/ovis/model/modeling_ovis.py#L312)
+
+### ovis2_5
+
+The meanings of the following parameters can be found in the example code [here](https://modelscope.cn/models/AIDC-AI/Ovis2.5-2B).
+
+- MIX_PIXELS: int type, default is `448 * 448`.
+- MAX_PIXELS: int type, default is `1344 * 1792`. If OOM (out of memory) occurs, you can reduce this value.
+- VIDEO_MAX_PIXELS: int type, default is `896 * 896`.
+- NUM_FRAMES: default is 8. Used for video frame sampling.
+
+### mplug_owl3, mplug_owl3_241101
+- MAX_NUM_FRAMES: Default is 16, refer to [here](https://modelscope.cn/models/iic/mPLUG-Owl3-7B-240728)
+
+### xcomposer2_4khd
+- HD_NUM: Default is 55, refer to [here](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer2-4khd-7b)
+
+### xcomposer2_5
+- HD_NUM: Default is 24 when the number of images is 1. Greater than 1, the default is 6. Refer to [here](https://modelscope.cn/models/AI-ModelScope/internlm-xcomposer2d5-7b/file/view/master?fileName=modeling_internlm_xcomposer2.py&status=1#L254)
+
+### video_cogvlm2
+- NUM_FRAMES: Default is 24, refer to [here](https://github.com/zai-org/CogVLM2/blob/main/video_demo/inference.py#L22)
+
+### phi3_vision
+- NUM_CROPS: Default is 4, refer to [here](https://modelscope.cn/models/LLM-Research/Phi-3.5-vision-instruct)
+
+### llama3_1_omni
+- N_MELS: Default is 128, refer to [here](https://github.com/ictnlp/LLaMA-Omni/blob/544d0ff3de8817fdcbc5192941a11cf4a72cbf2b/omni_speech/infer/infer.py#L57)
+
+### video_llava
+- NUM_FRAMES: Default is 16
+
+
+## Other Environment Variables
+- USE_HF: Use ModelScope/HuggingFace. Defaults to '0'.
+- CUDA_VISIBLE_DEVICES: Controls which GPU to use. By default, all GPUs are used.
+- ASCEND_RT_VISIBLE_DEVICES: Controls which NPU (effective for ASCEND cards) are used. By default, all NPUs are used.
+- MODELSCOPE_CACHE: Controls the cache path. (Recommended to set this value during multi-node training to ensure all nodes use the same dataset cache.)
+- NPROC_PER_NODE: Pass-through for the `--nproc_per_node` parameter in torchrun. The default is 1. If the `NPROC_PER_NODE` or `NNODES` environment variables are set, torchrun is used to start training or inference.
+- PYTORCH_CUDA_ALLOC_CONF: It is recommended to set it to `'expandable_segments:True'`, which reduces GPU memory fragmentation. For more details, please refer to the [PyTorch documentation](https://docs.pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management).
+- MASTER_PORT: Pass-through for the `--master_port` parameter in torchrun. The default is 29500.
+- MASTER_ADDR: Pass-through for the `--master_addr` parameter in torchrun.
+- NNODES: Pass-through for the `--nnodes` parameter in torchrun.
+- NODE_RANK: Pass-through for the `--node_rank` parameter in torchrun.
+- LOG_LEVEL: The log level, default is 'INFO'. You can set it to 'WARNING', 'ERROR', etc.
+- SWIFT_DEBUG: When set to `'1'` during `engine.infer(...)`, TransformersEngine will print the contents of `input_ids` and `generate_ids` to facilitate debugging and alignment.
+- VLLM_USE_V1: Used to switch between V0 and V1 versions of vLLM.
+- SWIFT_TIMEOUT: If the multimodal dataset contains image URLs, this parameter controls the timeout for fetching images, defaulting to 20 seconds.
+- ROOT_IMAGE_DIR: The root directory for image (multimodal) resources. By setting this parameter, relative paths in the dataset can be interpreted relative to `ROOT_IMAGE_DIR`. By default, paths are relative to the current working directory.
+- SWIFT_SINGLE_DEVICE_MODE: Single device mode, valid values are "0"(default)/"1". In this mode, each process can only see one device.
+- SWIFT_AUDIO_LOAD_BACKEND: Audio waveform loading backend. `librosa` (default) or `soundfile_pyav` (soundfile first, pyav fallback). For GRPO/GKD with `--use_vllm true`,swift auto-sets it to `soundfile_pyav` so training encode and vLLM rollout decode audio URLs consistently.

docs/source_en/Instruction/Distillation.md

@@ -0,0 +1,391 @@
+# Knowledge Distillation
+
+Knowledge distillation is a training method that transfers capabilities from a teacher model to a student model. The core idea is to have the student align with the teacher's output distribution at each token position, yielding richer supervision than simply imitating labeled answers—the teacher tells the student not only which token is correct, but also how good or bad other tokens are.
+
+This document introduces, top-down: why distillation works (Section 1), a unified design framework for distillation methods (Section 2), and finally three concrete distillation training methods in swift: GKD / OPD-RL / OPSD (Section 3).
+
+---
+
+## 1. Why Distillation: From Sparse to Dense Signals
+
+A language model's capabilities are typically built through a stack of training stages:
+
+- **Pre-training**: Acquire language, world knowledge, basic reasoning, and other general capabilities.
+- **Mid-training**: Inject domain knowledge, such as code, medicine, or internal company documents.
+- **Post-training**: Elicit target behaviors, such as instruction following, mathematical reasoning, or dialogue style.
+
+Distillation mainly happens during **post-training**. To understand its value, consider post-training methods along two independent dimensions:
+
+1. **Sampling mode (where data comes from)**: Whether training sequences are generated by the student itself (on-policy) or come from external fixed data (off-policy).
+2. **Feedback density (how much each sequence teaches)**: Whether the entire sequence has a single reward (per-sequence, sparse) or each token has a signal (per-token, dense).
+
+**SFT / offline distillation** (off-policy + dense): Align with labels or the teacher distribution on fixed data. The signal is dense, but training only sees states from the teacher/labels, which differ from states the student enters during its own inference. Once the student makes an early mistake the teacher would not make, it enters unseen states, errors accumulate—this is called **exposure bias**.
+
+**RL** (on-policy + sparse): The student samples trajectories and receives rewards based on final outcomes. The distribution matches student inference, but rewards are typically **sequence-level** scalars that do not pinpoint which token went wrong.
+
+**On-policy distillation** (on-policy + dense): The student samples trajectories, and the teacher scores **every token** on those trajectories. Training distribution matches student inference, and feedback is per-token.
+
+### SFT as a Special Case of Distillation
+
+A natural entry point to understanding distillation is the SFT loss. SFT's cross-entropy loss is equivalent to KL divergence against a one-hot "teacher" distribution $\delta_{y^*}$ at the labeled token:
+
+$$-\log P_S(y^*_t) = \text{KL}(\delta_{y^*} \,\|\, P_S)$$
+
+Knowledge distillation simply replaces this deterministic one-hot "teacher" with the teacher model's soft distribution $P_T$, optimizing $\text{KL}(P_T \,\|\, P_S)$ at each token to provide richer supervision than one-hot.
+
+| Method | Sampling Mode | Feedback Density | Teacher Distribution |
+|------|----------|----------|----------|
+| SFT | off-policy (fixed data) | dense (per-token) | one-hot $\delta_{y^*}$ |
+| Offline (off-policy) distillation | off-policy (fixed or teacher-generated data) | dense (per-token) | teacher soft distribution |
+| RL | on-policy (student sampling) | sparse (per-sequence) | none |
+| **On-policy distillation** | **on-policy (student sampling)** | **dense (per-token)** | **teacher soft distribution** |
+
+---
+
+## 2. Two Core Choices in Distillation
+
+Differences between distillation methods almost always boil down to two questions. Once you understand these dimensions, the methods below are just different combinations.
+
+### 2.1 How to Compute the Teacher Signal
+
+At each token position, quantify the difference between teacher distribution $P_T$ and student distribution $P_S$ (we call this **Teacher KL**). There are two sub-choices.
+
+**(a) Divergence direction**
+
+| Divergence | Definition | Optimization behavior (information-theoretic meaning) |
+|------|------|------|
+| Forward KL | $\text{KL}(P_T \,\|\, P_S)$ | Mode-covering: the student must assign enough probability to regions where the teacher has high probability |
+| Reverse KL | $\text{KL}(P_S \,\|\, P_T)$ | Mode-seeking: the student mainly fits the teacher's mode (high-probability) regions |
+| Generalized JSD($\beta$) | $\beta\,\text{KL}(P_T\|M) + (1-\beta)\,\text{KL}(P_S\|M)$, where $M=\beta P_T+(1-\beta)P_S$ | Interpolates between the two |
+
+> $\beta=0$ reduces to Forward KL, $\beta=1$ reduces to Reverse KL. SFT is equivalent to Forward KL (teacher is one-hot).
+
+In swift:
+- GKD defaults to $\beta=0.5$ (JSD); use `--beta` to choose among Forward / JSD / Reverse;
+- OPD-RL uses the Reverse KL k1 estimator $\log\pi_{\text{teacher}}(y_t)-\log\pi_{\text{student}}(y_t)$ as per-token advantage.
+
+**(b) Computation granularity**
+
+| Granularity | Teacher information needed | Notes |
+|----------|---------------|------|
+| Full vocabulary | Teacher's complete next-token distribution | Exact divergence; high memory cost |
+| Top-K | K tokens with highest teacher probability | Approximation after renormalization over top-K; suitable for external APIs (limited by `max_logprobs`) |
+| Sampled token | Single logp of the teacher on the student's sampled token | Single-sample Monte Carlo estimate of Reverse KL; lowest communication cost |
+
+> **Accuracy vs cost**: Full vocabulary requires materializing complete logits; sampled token only needs teacher logp on the sampled token (can use remote API). The [DeepSeek-V4](https://arxiv.org/abs/2606.19348) technical report notes that using only sampled-token log-ratio as advantage yields high gradient variance, so its full-vocabulary OPD uses complete logit distillation.
+
+### 2.2 How to Pass the Signal to the Student
+
+|  | **Path A: GKD (direct loss)** | **Path B: OPD-RL (RL advantage)** |
+|---|---|---|
+| Training paradigm | `--rlhf_type gkd` | `--rlhf_type grpo` + teacher |
+| Signal delivery | Use signal as loss | Use signal as advantage via policy gradient |
+| Gradient flows through | Student **full-vocabulary** logits (or top-k) | Only student **sampled token** $\nabla\log\pi(y_t)$ |
+| Teacher information needed | Full distribution (or top-k logits) | Single logp on sampled token |
+| Divergence choice | Forward / Reverse / JSD (`--beta`) | Reverse KL (k1 log-ratio) |
+| Combine with task reward | Mix SFT loss via `sft_alpha` | Can stack with GRPO reward as advantage |
+
+Both paths **share the same teacher infrastructure** (see below); they differ only in how teacher KL is used.
+
+> **Common uses of distillation**
+> 1. **Capability fusion**: Distill multiple expert models into one unified model.
+> 2. **Strong-to-weak**: Transfer capabilities from a large model to a smaller one.
+> 3. **Forgetting prevention**: Use an old checkpoint as teacher to recover prior capabilities after multi-stage training.
+
+---
+
+## 3. Distillation Methods in swift
+
+swift provides three distillation training methods. They share the same teacher infrastructure:
+
+| Method | Signal path | How to enable | One-liner |
+|------|--------------|----------|--------|
+| **GKD** | Direct loss (Path A) | `--rlhf_type gkd` | Teacher divergence as loss backprop; supports full-vocab / top-k divergence |
+| **OPD-RL** | RL advantage (Path B) | `--rlhf_type grpo` + teacher | Teacher log-ratio injected into GRPO advantage; can combine with task rewards |
+| **OPSD** | Path A or B | Provide `teacher_prompt` on top of the above | Single-model self-distillation: teacher input includes privileged info (e.g. reference solution) |
+
+**Three teacher sources** (shared by GKD and OPD-RL):
+
+- `--teacher_model`: Load a separate frozen teacher model in the training process.
+- `--teacher_model_server`: Connect to an external teacher service (vLLM service started via `swift deploy`) without loading the teacher on training GPUs. When using the API with GKD, also set `--gkd_logits_topk`. Supports single URL and multi-teacher JSON config.
+- **Self-distillation**: Teacher and student share the same source. For LoRA training when `--teacher_model` equals `--model`, the base model is used as a fixed teacher via `disable_adapter()` without extra loading; without `--teacher_model` or `teacher_model_server`, the student's current weights serve as a dynamic teacher (all batches in GKD; in GRPO only when data includes `teacher_prompt`—see [3.3](#33-opsd-on-policy-self-distillation)).
+
+| Parameter | Default | Description |
+|------|--------|------|
+| `--teacher_model` | None | Teacher model path; omit for dynamic self-distillation in GKD |
+| `--teacher_model_server` | None | Teacher API URL; see format below |
+| `--teacher_tag_key` | `"dataset"` | Column name for matching sample tags to teacher `tags` in multi-teacher routing |
+| `--teacher_deepspeed` | None | DeepSpeed config for teacher model (e.g. `zero3`) |
+| `--offload_teacher_model` | False | Offload teacher to CPU when not in forward pass (only with `teacher_model`) |
+
+Full parameter details: [command-line parameters](./Command-line-parameters.md#rewardteacher-model-parameters).
+
+#### External Teacher API
+
+Deploy the teacher via `swift deploy --model xxx --infer_backend vllm`; the training process requests logprobs by prompt without loading teacher weights on training GPUs.
+
+- **GKD**: Requires `--gkd_logits_topk` (API returns only top-k logprobs) for JSD divergence loss.
+- **OPD-RL**: Uses sampled-token logp (`prompt_logprobs=0`) injected into advantage; coefficient is global `--teacher_kl_coef` (see [3.2](#32-opd-rl-kl-as-rl-advantage)).
+
+#### Multi-Teacher Routing
+
+Multi-Teacher connects multiple external teacher APIs and routes each sample to one teacher by tag (domain-expert distillation).
+
+**`teacher_model_server` format**
+
+```bash
+# Single teacher
+--teacher_model_server http://localhost:8000
+
+# Multi-teacher (`tags` match `--dataset` or a data column; see Mode 1 below)
+--teacher_model_server '[{"url":"http://t1:8000","tags":["data/math.jsonl"]},{"url":"http://t2:8001","tags":["data/code.jsonl"]}]'
+```
+
+Each teacher entry:
+
+| Field | Description |
+|------|------|
+| `url` | Teacher API URL |
+| `tags` | Data sources this teacher handles; optional with a single teacher; with multiple teachers, entries must not overlap and must match the routing identifiers below |
+
+**Routing modes**
+
+**Mode 1: Route by dataset (default)**
+
+When you pass multiple `--dataset` values, samples are matched to teachers by source dataset. Default `--teacher_tag_key` is `"dataset"`. Set each teacher's `tags` to match the corresponding `--dataset` entry:
+
+```bash
+# Hub IDs
+--dataset AI-ModelScope/alpaca-gpt4-data-en AI-ModelScope/alpaca-cleaned \
+--teacher_model_server '[{"url":"http://t1:8000","tags":["AI-ModelScope/alpaca-gpt4-data-en"]},{"url":"http://t2:8001","tags":["AI-ModelScope/alpaca-cleaned"]}]'
+
+# Local paths
+--dataset data/math.jsonl data/code.jsonl \
+--teacher_model_server '[{"url":"http://t1:8000","tags":["data/math.jsonl"]},{"url":"http://t2:8001","tags":["data/code.jsonl"]}]'
+```
+
+**Mode 2: Route by sample**
+
+Add a column in your data (e.g. `teacher_tag`) and set `--teacher_tag_key teacher_tag`. Match `tags` to values in that column. Use this when you pass a single `--dataset` but still need multiple teachers.
+
+**Example: GKD + multi-teacher**
+
+```bash
+# Deploy two teacher services (GKD requires max_logprobs >= gkd_logits_topk)
+CUDA_VISIBLE_DEVICES=1 swift deploy --model Qwen/Qwen3.5-4B --port 8000 --max_logprobs 64
+CUDA_VISIBLE_DEVICES=2 swift deploy --model Qwen/Qwen3.5-1.7B --port 8001 --max_logprobs 64
+
+CUDA_VISIBLE_DEVICES=0 swift rlhf \
+    --rlhf_type gkd \
+    --model Qwen/Qwen3.5-0.6B \
+    --teacher_model_server '[{"url":"http://localhost:8000","tags":["data/math.jsonl"]},{"url":"http://localhost:8001","tags":["data/code.jsonl"]}]' \
+    --gkd_logits_topk 64 \
+    --dataset data/math.jsonl data/code.jsonl \
+    ...
+```
+
+**Example: OPD-RL + multi-teacher**
+
+```bash
+CUDA_VISIBLE_DEVICES=1 swift deploy --model Qwen/Qwen3.5-4B --port 8000 --max_logprobs 1
+CUDA_VISIBLE_DEVICES=2 swift deploy --model Qwen/Qwen3.5-1.7B --port 8001 --max_logprobs 1
+
+CUDA_VISIBLE_DEVICES=0 swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen3.5-0.6B \
+    --teacher_model_server '[{"url":"http://localhost:8000","tags":["data/math.jsonl"]},{"url":"http://localhost:8001","tags":["data/code.jsonl"]}]' \
+    --teacher_kl_coef 1.0 \
+    --dataset data/math.jsonl data/code.jsonl \
+    --use_vllm true --vllm_mode colocate \
+    ...
+```
+
+---
+
+### 3.1 GKD: Divergence as Direct Loss
+
+GKD ([Generalized Knowledge Distillation](https://arxiv.org/pdf/2306.13649)) directly backpropagates the divergence between teacher and student as the loss function.
+
+**Loss function**
+
+$$
+\mathcal{L}_{\text{GKD}}(x, y) = \sum_{t=1}^{|y|} D_{\text{JSD}(\beta)}\big(P_{\text{teacher}}(\cdot|x,y_{<t}),\, P_{\text{student}}(\cdot|x,y_{<t})\big)
+$$
+
+Divergence $D$ is chosen via `--beta` (see 2.1): $\beta=0$ is Forward KL, $\beta=1$ is Reverse KL, $0<\beta<1$ is generalized JSD (default $0.5$).
+
+**On-Policy vs Off-Policy: `lmbda`**
+
+GKD uses `lmbda` to control the probability that each batch uses student online sampling:
+
+```python
+if random() <= lmbda:
+    y = student.generate(x)   # on-policy: student samples
+else:
+    y = y_ground_truth        # off-policy: use dataset labels
+loss = D(P_teacher(·|x, y), P_student(·|x, y))
+```
+
+- `lmbda=0`: pure offline (traditional SFT distillation).
+- `lmbda=1`: pure online (student learns from its own mistakes, i.e. on-policy distillation).
+- `0<lmbda<1`: mixed.
+
+> To use teacher-generated data, first offline-generate responses with the teacher and write them to the dataset, then train with `lmbda=0`.
+
+**GKD parameters**
+
+| Parameter | Type | Default | Description |
+|------|------|--------|------|
+| `--beta` | float | 0.5 | Divergence interpolation: 0=Forward KL, 0.5=JSD, 1=Reverse KL |
+| `--lmbda` | float | 0.5 | Online sampling probability: 0=offline, 1=pure online |
+| `--sft_alpha` | float | 0 | SFT loss mixing ratio; final `loss = gkd_loss + sft_alpha * sft_loss` (only for **non-student-generated** data) |
+| `--gkd_logits_topk` | int | None | Compute KL using teacher top-K logits only; required when using `teacher_model_server` |
+
+### Top-K KL Computation
+
+By default KL is computed over the full vocabulary, which can OOM on large vocabularies. Use `--gkd_logits_topk`.
+
+**External teacher API**
+
+When setting `--teacher_model_server`, also set `--gkd_logits_topk` (API returns only top-k logprobs). Example:
+
+```bash
+# Step 1: Deploy teacher model (max_logprobs must be >= gkd_logits_topk)
+CUDA_VISIBLE_DEVICES=0 swift deploy \
+    --model Qwen/Qwen3.5-9B \
+    --infer_backend vllm \
+    --port 8000 \
+    --max_logprobs 64
+
+# Step 2: Start GKD training
+CUDA_VISIBLE_DEVICES=1,2,3,4 \
+NPROC_PER_NODE=4 \
+swift rlhf \
+    --rlhf_type gkd \
+    --model Qwen/Qwen3.5-2B \
+    --teacher_model_server http://localhost:8000 \
+    --gkd_logits_topk 64 \
+    --lmbda 1.0 \
+    --beta 1.0 \
+    --dataset xxx
+```
+
+**Online sampling acceleration**
+
+When `lmbda > 0`, the student must generate sequences online. Use vLLM to accelerate sampling (colocate / server modes, same as GRPO). See [GRPO documentation](./GRPO/GetStarted/GRPO.md#cluster-support).
+
+**Multi-turn GKD**
+
+GKD supports multi-turn training, sharing the same `MultiTurnScheduler` infrastructure as GRPO.
+For the full scheduler interface and customisation, see the [GRPO multi-turn documentation](./GRPO/DeveloperGuide/multi_turn.md).
+
+**Reference scripts**
+
+- Basic training: [examples/train/rlhf/gkd/](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/gkd/)
+- Multi-turn training: [examples/train/rlhf/gkd/multi_turn.sh](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/gkd/multi_turn.sh)
+- Multimodal: [examples/train/multimodal/rlhf/gkd/](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/rlhf/gkd/)
+- Megatron: [examples/megatron/rlhf/gkd/](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/gkd/)
+
+---
+
+### 3.2 OPD-RL: KL as RL Advantage
+
+OPD (On-Policy Distillation) RL injects teacher KL into GRPO's **per-token advantage** and updates the student via policy gradient.
+
+**Principle**
+
+Standard GRPO advantage comes from group-normalized task rewards (per-sequence scalar). OPD-RL injects teacher signal token-by-token **after** advantage normalization:
+
+$$
+A_t = A_t^{\text{base}} + \alpha \cdot \big(\log \pi_{\text{teacher}}(y_t|x,y_{<t}) - \log \pi_{\text{student}}(y_t|x,y_{<t})\big)
+$$
+
+- $A_t^{\text{base}}$: GRPO-normalized task reward advantage (0 when no reward function).
+- $\alpha$: `--teacher_kl_coef`, teacher signal strength.
+- $\log\pi_{\text{teacher}}(y_t) - \log\pi_{\text{student}}(y_t)$: teacher log-ratio, the k1 estimator corresponding to the coefficient of $\nabla_\theta\log\pi_\theta(y_t)$ in the Reverse KL gradient (see `compute_teacher_logratio`).
+
+**Pure distillation mode**: Without `--reward_funcs`, base advantage is 0 and teacher signal is the only driver: $A_t = \alpha\cdot(\log\pi_{\text{teacher}}(y_t)-\log\pi_{\text{student}}(y_t))$.
+
+**Monitoring metric**: `teacher_kl` in logs is the k3 estimator $e^{d}-d-1$ ($d=\log\pi_{\text{teacher}}-\log\pi_{\text{student}}$), measuring distance between student and teacher.
+
+**How to enable**: Under `--rlhf_type grpo`, set `--teacher_model` or `--teacher_model_server` to automatically enable OPD-RL—no extra switch needed. See the shared teacher parameter table above.
+
+**OPD-RL-specific parameters**
+
+| Parameter | Default | Description |
+|------|--------|------|
+| `--teacher_kl_coef` | 1.0 | Coefficient $\alpha$ for injecting teacher log-ratio into advantage |
+
+**Reference scripts**
+
+- HF: [examples/train/grpo/opd_rl.sh](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/opd_rl.sh)
+- Megatron: [examples/megatron/grpo/opd_rl.sh](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/grpo/opd_rl.sh)
+- Ray: [examples/ray/grpo/run_opd.sh](https://github.com/modelscope/ms-swift/tree/main/examples/ray/grpo/run_opd.sh)
+
+---
+
+### 3.3 OPSD: On-Policy Self-Distillation
+
+OPSD ([On-Policy Self-Distillation](https://arxiv.org/abs/2601.18734)) is a single-model self-distillation method: the same model constructs student and teacher inputs separately; the teacher side additionally receives **privileged information** (e.g. reference solution), then aligns output distributions on the student's sampled response.
+
+**Core mechanism**
+
+- **Student**: sees only the problem and reasons normally.
+- **Teacher**: sees problem + reference solution (privileged info via `teacher_prompt` column).
+- **Training objective**: align student and teacher output distributions on the same student-sampled response via divergence (JSD / KL).
+
+OPSD can follow either GKD or OPD-RL path:
+
+- **GKD + OPSD**: `--rlhf_type gkd`, teacher KL as direct loss.
+- **OPD-RL + OPSD**: `--rlhf_type grpo`; dynamic mode omits `--teacher_model`, fixed mode sets `--teacher_model` equal to `--model`.
+
+**Two self-distillation weight modes**
+
+| Mode | Configuration | Teacher weights | Description |
+|------|---------|---------|------|
+| **Dynamic** | Omit `--teacher_model` | Student's current weights | Teacher updates with training |
+| **Fixed** | Set `--teacher_model` same as `--model` | Initial teacher weights | Teacher weights fixed |
+
+**Data format**
+
+OPSD datasets need a `teacher_prompt` column. Load a data preprocessing plugin via `--external_plugins` to build it. Example with math reasoning dataset `open-r1/OpenThoughts-114k-math`:
+
+```python
+from swift.dataset import DatasetMeta, RowPreprocessor, register_dataset
+
+class OpenThoughtsOPSDPreprocessor(RowPreprocessor):
+    def preprocess(self, row):
+        if not row.get('correct', True):
+            return None
+        problem = row.get('problem', '')
+        solution = row.get('solution', '')
+        teacher_prompt = f'{problem}\n\nReference solution:\n{solution}\n\nNow articulate your own reasoning.'
+        messages = [
+            {'role': 'system', 'content': 'Please reason step by step, and put your final answer within \\boxed{}.'},
+            {'role': 'user', 'content': problem},
+        ]
+        return {'messages': messages, 'teacher_prompt': teacher_prompt}
+
+register_dataset(DatasetMeta(
+    ms_dataset_id='open-r1/OpenThoughts-114k-math',
+    preprocess_func=OpenThoughtsOPSDPreprocessor(),
+    tags=['math', 'opsd'],
+))
+```
+
+**Reference scripts**
+
+- HF: [examples/train/rlhf/opsd/](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/opsd/)
+- Megatron: [examples/megatron/rlhf/gkd/opsd.sh](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/rlhf/gkd/opsd.sh)
+
+---
+
+## Reference
+
+- Kevin Lu & Thinking Machines Lab. [On-Policy Distillation](https://thinkingmachines.ai/blog/on-policy-distillation/). 2025.
+- Agarwal et al. [On-Policy Distillation of Language Models (GKD)](https://arxiv.org/pdf/2306.13649). 2023.
+- Gu et al. [MiniLLM: Knowledge Distillation of Large Language Models](https://arxiv.org/abs/2306.08543). 2023.
+- DeepSeek-AI. [DeepSeek-V4](https://arxiv.org/html/2606.19348v1). 2026.
+- Qwen Team. [Qwen3 Technical Report](https://arxiv.org/abs/2505.09388). 2025.
+- Zhipu AI. [GLM-5: from Vibe Coding to Agentic Engineering](https://arxiv.org/html/2602.15763). 2026.
+- Kimi Team. [Kimi K2.5: Visual Agentic Intelligence](https://arxiv.org/abs/2602.02276). 2026.

docs/source_en/Instruction/Evaluation.md

@@ -0,0 +1,285 @@
+# Evaluation
+
+SWIFT supports eval (evaluation) capabilities to provide standardized evaluation metrics for both raw models and trained models.
+
+## Capability Introduction
+
+SWIFT's eval capability utilizes the EvalScope evaluation framework from the Magic Tower community, which has been advanced in its encapsulation to support the evaluation needs of various models.
+
+> Note: EvalScope supports many other complex capabilities, such as [model performance evaluation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html), so please use the EvalScope framework directly.
+
+Currently, we support the evaluation process of **standard evaluation datasets** as well as the evaluation process of **user-defined** evaluation datasets. The **standard evaluation datasets** are supported by three evaluation backends:
+
+Below are the names of the supported datasets. For detailed information on the datasets, please refer to [all supported datasets](https://evalscope.readthedocs.io/en/latest/get_started/supported_dataset/index.html).
+
+1. Native (default):
+
+    Primarily supports pure text evaluation, while **supporting** visualization of evaluation results.
+    ```text
+    'arc', 'bbh', 'ceval', 'cmmlu', 'competition_math',
+    'general_qa', 'gpqa', 'gsm8k', 'hellaswag', 'humaneval',
+    'ifeval', 'iquiz', 'mmlu', 'mmlu_pro',
+    'race', 'trivia_qa', 'truthful_qa'
+    ```
+
+2. OpenCompass:
+
+    Primarily supports pure text evaluation, currently **does not support** visualization of evaluation results.
+    ```text
+    'obqa', 'cmb', 'AX_b', 'siqa', 'nq', 'mbpp', 'winogrande', 'mmlu', 'BoolQ', 'cluewsc', 'ocnli', 'lambada',
+    'CMRC', 'ceval', 'csl', 'cmnli', 'bbh', 'ReCoRD', 'math', 'humaneval', 'eprstmt', 'WSC', 'storycloze',
+    'MultiRC', 'RTE', 'chid', 'gsm8k', 'AX_g', 'bustm', 'afqmc', 'piqa', 'lcsts', 'strategyqa', 'Xsum', 'agieval',
+    'ocnli_fc', 'C3', 'tnews', 'race', 'triviaqa', 'CB', 'WiC', 'hellaswag', 'summedits', 'GaokaoBench',
+    'ARC_e', 'COPA', 'ARC_c', 'DRCD'
+    ```
+
+3. VLMEvalKit:
+
+    Primarily supports multimodal evaluation and currently **does not support** visualization of evaluation results.
+    ```text
+    'COCO_VAL', 'MME', 'HallusionBench', 'POPE', 'MMBench_DEV_EN', 'MMBench_TEST_EN', 'MMBench_DEV_CN', 'MMBench_TEST_CN',
+    'MMBench', 'MMBench_CN', 'MMBench_DEV_EN_V11', 'MMBench_TEST_EN_V11', 'MMBench_DEV_CN_V11',
+    'MMBench_TEST_CN_V11', 'MMBench_V11', 'MMBench_CN_V11', 'SEEDBench_IMG', 'SEEDBench2',
+    'SEEDBench2_Plus', 'ScienceQA_VAL', 'ScienceQA_TEST', 'MMT-Bench_ALL_MI', 'MMT-Bench_ALL',
+    'MMT-Bench_VAL_MI', 'MMT-Bench_VAL', 'AesBench_VAL', 'AesBench_TEST', 'CCBench', 'AI2D_TEST', 'MMStar',
+    'RealWorldQA', 'MLLMGuard_DS', 'BLINK', 'OCRVQA_TEST', 'OCRVQA_TESTCORE', 'TextVQA_VAL', 'DocVQA_VAL',
+    'DocVQA_TEST', 'InfoVQA_VAL', 'InfoVQA_TEST', 'ChartQA_TEST', 'MathVision', 'MathVision_MINI',
+    'MMMU_DEV_VAL', 'MMMU_TEST', 'OCRBench', 'MathVista_MINI', 'LLaVABench', 'MMVet', 'MTVQA_TEST',
+    'MMLongBench_DOC', 'VCR_EN_EASY_500', 'VCR_EN_EASY_100', 'VCR_EN_EASY_ALL', 'VCR_EN_HARD_500',
+    'VCR_EN_HARD_100', 'VCR_EN_HARD_ALL', 'VCR_ZH_EASY_500', 'VCR_ZH_EASY_100', 'VCR_ZH_EASY_ALL',
+    'VCR_ZH_HARD_500', 'VCR_ZH_HARD_100', 'VCR_ZH_HARD_ALL', 'MMDU', 'MMBench-Video', 'Video-MME'
+    ```
+
+## Environment Preparation
+
+```shell
+pip install ms-swift[eval] -U
+```
+
+Or install from source:
+
+```shell
+git clone https://github.com/modelscope/ms-swift.git
+cd ms-swift
+pip install -e '.[eval]'
+```
+
+## Evaluation
+
+Supports four methods of evaluation: pure text evaluation, multimodal evaluation, URL evaluation, and custom dataset evaluation.
+
+**Basic Example**
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift eval \
+    --model Qwen/Qwen2.5-0.5B-Instruct \
+    --eval_backend Native \
+    --infer_backend transformers \
+    --eval_limit 10 \
+    --eval_dataset gsm8k
+```
+Where:
+- model: Can specify a local model path or a model ID on modelscope
+- eval_backend: Options are Native, OpenCompass, VLMEvalKit; default is Native
+- infer_backend: Options are transformers, vllm, sglang, lmdeploy; default is transformers
+- eval_limit: Sample size for each evaluation set; default is None, which means using all data; can be used for quick validation
+- eval_dataset: Evaluation dataset(s); multiple datasets can be set, separated by spaces
+
+**Complex Evaluation Example**
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift eval \
+    --model Qwen/Qwen2.5-0.5B-Instruct \
+    --eval_backend Native \
+    --infer_backend transformers \
+    --eval_limit 10 \
+    --eval_dataset gsm8k \
+    --eval_dataset_args '{"gsm8k": {"few_shot_num": 0, "filters": {"remove_until": "</think>"}}}' \
+    --eval_generation_config '{"max_tokens": 512, "temperature": 0}' \
+    --extra_eval_args '{"ignore_errors": true, "debug": true}'
+```
+
+For a specific list of evaluation parameters, please refer to [here](./Command-line-parameters.md#evaluation-arguments).
+
+## Evaluation During Training
+
+SWIFT supports using EvalScope to evaluate the current model during the training process, allowing for timely understanding of the model's training effectiveness.
+
+**Basic Example**
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift sft \
+  --model "Qwen/Qwen2.5-0.5B-Instruct" \
+  --tuner_type "lora" \
+  --dataset "AI-ModelScope/alpaca-gpt4-data-zh#100" \
+  --torch_dtype "bfloat16" \
+  --num_train_epochs "1" \
+  --per_device_train_batch_size "1" \
+  --learning_rate "1e-4" \
+  --lora_rank "8" \
+  --lora_alpha "32" \
+  --target_modules "all-linear" \
+  --gradient_accumulation_steps "16" \
+  --save_steps "50" \
+  --save_total_limit "5" \
+  --logging_steps "5" \
+  --max_length "2048" \
+  --eval_strategy "steps" \
+  --eval_steps "5" \
+  --per_device_eval_batch_size "5" \
+  --eval_use_evalscope \
+  --eval_dataset "gsm8k" \
+  --eval_dataset_args '{"gsm8k": {"few_shot_num": 0}}' \
+  --eval_limit "10"
+```
+
+Note that the launch command is `sft`, and the evaluation-related parameters include:
+- eval_strategy: Evaluation strategy. Defaults to None, following the `save_strategy` policy
+- eval_steps: Defaults to None. If an evaluation dataset exists, it follows the `save_steps` policy
+- eval_use_evalscope: Whether to use evalscope for evaluation, this parameter needs to be set to enable evaluation
+- eval_dataset: Evaluation datasets, multiple datasets can be set, separated by spaces
+- eval_dataset_args: Evaluation dataset parameters in JSON format, parameters for multiple datasets can be set
+- eval_limit: Number of samples from the evaluation dataset
+- eval_generation_config: Model inference configuration during evaluation, in JSON format, default is `{'max_tokens': 512}`
+
+More evaluation examples can be found in [examples](https://github.com/modelscope/ms-swift/tree/main/examples/eval).
+
+## Custom Evaluation Datasets
+
+This framework supports two predefined dataset formats: multiple-choice questions (MCQ) and question-and-answer (QA). The usage process is as follows:
+
+*Note: When using a custom evaluation, the `eval_backend` parameter must be set to `Native`.*
+
+### Multiple-Choice Question Format (MCQ)
+This format is suitable for scenarios involving multiple-choice questions, and the evaluation metric is accuracy.
+
+**Data Preparation**
+
+Prepare a CSV file in the multiple-choice question format, structured as follows:
+
+```text
+mcq/
+├── example_dev.csv  # (Optional) The filename should follow the format `{subset_name}_dev.csv` for few-shot evaluation
+└── example_val.csv  # The filename should follow the format `{subset_name}_val.csv` for the actual evaluation data
+```
+
+The CSV file should follow this format:
+
+```text
+id,question,A,B,C,D,answer
+1,Generally speaking, the amino acids that make up animal proteins are____,4 types,22 types,20 types,19 types,C
+2,Among the substances present in the blood, which is not a metabolic end product?____,Urea,Uric acid,Pyruvate,Carbon dioxide,C
+```
+
+Where:
+- `id` is an optional index
+- `question` is the question
+- `A`, `B`, `C`, `D`, etc. are the options, with a maximum of 10 options
+- `answer` is the correct option
+
+**Launching Evaluation**
+
+Run the following command:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+swift eval \
+    --model Qwen/Qwen2.5-0.5B-Instruct \
+    --eval_backend Native \
+    --infer_backend transformers \
+    --eval_dataset general_mcq \
+    --eval_dataset_args '{"general_mcq": {"local_path": "/path/to/mcq", "subset_list": ["example"]}}'
+```
+
+Where:
+- `eval_dataset` should be set to `general_mcq`
+- `eval_dataset_args` should be set with:
+    - `local_path` as the path to the custom dataset folder
+    - `subset_list` as the name of the evaluation dataset, taken from the `*_dev.csv` mentioned above
+
+**Running Results**
+
+```text
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Model               | Dataset     | Metric          | Subset   |   Num |   Score | Cat.0   |
++=====================+=============+=================+==========+=======+=========+=========+
+| Qwen2-0.5B-Instruct | general_mcq | AverageAccuracy | example  |    12 |  0.5833 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+```
+
+## Question-and-Answer Format (QA)
+This format is suitable for scenarios involving question-and-answer, and the evaluation metrics are `ROUGE` and `BLEU`.
+
+**Data Preparation**
+
+Prepare a JSON Lines file in the question-and-answer format, containing one file in the following structure:
+
+```text
+qa/
+└── example.jsonl
+```
+
+The JSON Lines file should follow this format:
+
+```json
+{"query": "What is the capital of China?", "response": "The capital of China is Beijing"}
+{"query": "What is the highest mountain in the world?", "response": "It is Mount Everest"}
+{"query": "Why can't penguins be seen in the Arctic?", "response": "Because most penguins live in Antarctica"}
+```
+
+**Launching Evaluation**
+
+Run the following command:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+swift eval \
+    --model Qwen/Qwen2.5-0.5B-Instruct \
+    --eval_backend Native \
+    --infer_backend transformers \
+    --eval_dataset general_qa \
+    --eval_dataset_args '{"general_qa": {"local_path": "/path/to/qa", "subset_list": ["example"]}}'
+```
+
+Where:
+- `eval_dataset` should be set to `general_qa`
+- `eval_dataset_args` is a JSON string that needs to be set with:
+    - `local_path` as the path to the custom dataset folder
+    - `subset_list` as the name of the evaluation dataset, taken from the `*.jsonl` mentioned above
+
+**Running Results**
+
+```text
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Model               | Dataset     | Metric          | Subset   |   Num |   Score | Cat.0   |
++=====================+=============+=================+==========+=======+=========+=========+
+| Qwen2-0.5B-Instruct | general_qa  | bleu-1          | default  |    12 |  0.2324 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | bleu-2          | default  |    12 |  0.1451 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | bleu-3          | default  |    12 |  0.0625 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | bleu-4          | default  |    12 |  0.0556 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-1-f       | default  |    12 |  0.3441 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-1-p       | default  |    12 |  0.2393 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-1-r       | default  |    12 |  0.8889 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-2-f       | default  |    12 |  0.2062 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-2-p       | default  |    12 |  0.1453 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-2-r       | default  |    12 |  0.6167 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-l-f       | default  |    12 |  0.333  | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-l-p       | default  |    12 |  0.2324 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+| Qwen2-0.5B-Instruct | general_qa  | rouge-l-r       | default  |    12 |  0.8889 | default |
++---------------------+-------------+-----------------+----------+-------+---------+---------+
+```

docs/source_en/Instruction/Export-and-push.md

@@ -0,0 +1,61 @@
+# Export and Push
+
+## Merge LoRA
+
+- See [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/merge_lora.sh).
+
+## Quantization
+
+SWIFT supports quantization exports for AWQ, GPTQ, FP8, and BNB models. AWQ and GPTQ require a calibration dataset, which yields better quantization performance but takes longer to quantize. On the other hand, FP8 and BNB does not require a calibration dataset and is quicker to quantize.
+
+| Quantization Technique | Multimodal | Inference Acceleration | Continued Training |
+| ---------------------- | ---------- | ---------------------- | ------------------ |
+| FP8     | ✅      | ✅        | ✅        |
+| GPTQ           | ✅          | ✅                      | ✅                  |
+| AWQ                    | ✅          | ✅                      | ✅                  |
+| BNB                    | ❌          | ✅                      | ✅                  |
+
+In addition to the SWIFT installation, the following additional dependencies need to be installed:
+
+```shell
+# For AWQ quantization:
+# The versions of autoawq and CUDA are correlated; please choose the version according to `https://github.com/casper-hansen/AutoAWQ`.
+# If there are dependency conflicts with torch, please add the `--no-deps` option.
+pip install autoawq -U
+
+# For GPTQ quantization:
+# The versions of auto_gptq and CUDA are correlated; please choose the version according to `https://github.com/PanQiWei/AutoGPTQ#quick-installation`.
+pip install auto_gptq optimum -U
+
+# For GPTQ v2 quantization:
+pip install gptqmodel optimum -U
+
+# For BNB quantization:
+pip install bitsandbytes -U
+```
+
+We provide a series of scripts to demonstrate SWIFT's quantization export capabilities:
+
+- Supports [AWQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/awq.sh)/[GPTQ](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq.sh)/[GPTQ v2](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/gptq_v2.sh)/[BNB](https://github.com/modelscope/ms-swift/blob/main/examples/export/quantize/bnb.sh) quantization exports.
+- Multimodal quantization: Supports quantizing multimodal models using GPTQ and AWQ, with limited multimodal models supported by AWQ. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/mllm).
+- Support for more model series: Supports quantization exports for [BERT](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/bert) and [Reward Model](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize/reward_model).
+- Models exported with SWIFT's quantization support inference acceleration using vllm/sglang/lmdeploy; they also support further SFT/RLHF using QLoRA.
+
+
+## Push Model
+
+SWIFT supports re-pushing trained/quantized models to ModelScope/Hugging Face. By default, it pushes to ModelScope, but you can specify `--use_hf true` to push to Hugging Face.
+
+```shell
+swift export \
+    --model output/vx-xxx/checkpoint-xxx \
+    --push_to_hub true \
+    --hub_model_id '<model-id>' \
+    --hub_token '<sdk-token>' \
+    --use_hf false
+```
+
+Tips:
+
+- You can use `--model <checkpoint-dir>` or `--adapters <checkpoint-dir>` to specify the checkpoint directory to be pushed. There is no difference between these two methods in the model pushing scenario.
+- When pushing to ModelScope, you need to make sure you have registered for a ModelScope account. Your SDK token can be obtained from [this page](https://www.modelscope.cn/my/myaccesstoken). Ensure that the account associated with the SDK token has edit permissions for the organization corresponding to the model_id. The model pushing process will automatically create a model repository corresponding to the model_id (if it does not already exist), and you can use `--hub_private_repo true` to automatically create a private model repository.

docs/source_en/Instruction/Frequently-asked-questions.md

@@ -0,0 +1,469 @@
+# Frequently-asked-questions
+
+Here are some common questions encountered during the use of SWIFT.
+
+## Training
+
+SWIFT supports training methods including pre-training, instruction-supervised fine-tuning (SFT), preference learning, GRPO, Embedding, Reranker, sequence classification tasks, etc. For details, please see the [homepage](https://github.com/modelscope/ms-swift/blob/main/README.md).
+
+### Q1: What models does Swift support? How do I set a local model path?
+For supported models, please refer to the [Supported Models and Datasets documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). If the model has already been downloaded locally, simply set `--model <path_to_model>`. For training in an offline environment, set both `--model <local_path>` and `--check_model false`. If you encounter a git clone-related error, you need to clone the repository and then specify it using `local_repo_path`. For details, see [Command-line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). For models downloaded from ModelScope, you can configure the `MODELSCOPE_CACHE=your_path` environment variable to store the original models in a specified directory. If using the ModelScope SDK, use `cache_dir="local_address"`. You can also use the `modelscope download` command-line tool or `git` to download. For details, refer to the ModelScope documentation on [Model Download](https://modelscope.cn/docs/models/download). If you need to download models from Hugging Face, set the environment variable `USE_HF=1`.
+Swift automatically matches the model_type, or you can manually specify it by referring to the [Supported Models and Datasets documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html).
+
+### Q2: What datasets does Swift support? How do I use a custom dataset?
+For supported datasets, see the [Supported Models and Datasets documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). For the format and usage of custom datasets, see the [Custom Dataset documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html). Datasets that conform to these formats will automatically use Swift's built-in data preprocessors. If your dataset does not match the format in the documentation, please convert it yourself or refer to how currently supported datasets are integrated. If your custom dataset contains extra fields, they will not be used by default. You can configure this using the [remove_unused_columns command-line parameter](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#data-arguments).
+You need to download the dataset locally and then specify its path. Please see the [Custom Dataset documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html). git clone it locally, then specify the path using the `dataset_path` field in the dataset_info.json file.
+For data shuffling, see the [dataset_shuffle command-line parameter](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
+To force re-downloading a dataset, set the `--download_mode` command-line parameter. To perform error checking on the dataset, set the `strict` command-line parameter. If you need a dataset quality inspection tool, you can check out another library, [data-juicer](https://github.com/modelscope/data-juicer).
+Due to the strict type checking in the underlying PyArrow of the datasets library, parts like objects in image grounding datasets and tools in agent datasets must be strings. Otherwise, PyArrow will report an error indicating inconsistent types between rows.
+If you encounter the error `AttributeError:’TrainerState’ object has no attribute ’last_model_checkpoint’` during training, it's because the dataset is too small (the number of data points is less than one step). Increase the amount of data. A similar error can also occur when the split validation set is very small.
+Below is an error caused by an empty assistant field:
+```text
+File "/mnt/workspace/swift/swift/1lm/dataset/preprocessor/core. py", line 69, in _check_messages raise
+ValueError(f'assistant_message; {assistant_message}')
+ValueError: assistant_message: {'role' :'assistant', 'content': ''}
+```
+```shell
+CUDA_VISIBLE_DEVICES=0 NPROC_PER_NODE=1 MAX_PIXELS=1003520 swift sft --model Qwen/Qwen2.5-VL-7B-Instruct --tuner_type lora --dataset /mnt/workspace/data.json --deepspeed zero2 --max_length 16384
+```
+If the assistant field in the dataset is empty, remove this empty string for inference, as it can cause NaN values during training and will be checked for.
+
+### Q3: Issues related to loading datasets from cache
+Setting the command-line parameter `--load_from_cache_file true` can speed up dataset loading, especially for multimodal datasets or large datasets. When debugging or modifying a preprocessor, set it to false. For more information, search for this parameter in the [Command-line Parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
+
+### Q4: How do I set up the Swift environment? Are there Docker images available?
+For environment setup, see the [Swift Installation documentation](https://swift.readthedocs.io/en/latest/GetStarted/SWIFT-installation.html). Recommended versions for some common dependencies can be found on the [homepage](https://github.com/modelscope/ms-swift/blob/main/README.md). The documentation provides a Docker image. You can start a container using the `docker run` command, for example: `docker run --gpus all -p 8000:8000 -it -d --name ms modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.9.0-vllm0.13.0-modelscope1.33.0-swift3.12.5 /bin/bash`. After starting the container, pull the latest code and install Swift.
+
+### Q5: Questions about multimodal model training data formats, parameter freezing, and optimizer settings
+See [examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal) for multimodal model training. It supports training with text-only data, image-text data, or a mixture of both. For parameters related to images, videos, and audio, such as max pixels, fps, etc., please see [Model-specific Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#specific-model-arguments).
+In Grounding tasks, the general data format supports one object corresponding to multiple bboxes. Refer to the [Custom Dataset documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html). videos can be a list of images specified via a file directory.
+Swift resizes images based on max_pixels and saves both the pre-processed and post-processed images, then adjusts the bboxes accordingly. However, this adjustment is not done during inference, so you need to manually process the images beforehand.
+To reduce GPU memory usage when training VLM models, configure `--freeze_vit true` and the `--max_pixels` parameter to limit the maximum pixels. For details on parameters like `--freeze_vit`, `--freeze_aligner`, and `--freeze_llm`, see the Tuner section in the [Command-line Parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). If the ViT is not being trained, it is normal to see a warning like `"none of the inputs have requires_grad=True"`. If it is being trained, this warning should not appear.
+To perform full-parameter fine-tuning on the visual encoder while using LoRA to fine-tune the LLM, refer to this [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/lora_llm_full_vit).
+
+### Q6: Issues related to templates
+Since Jinja chat templates do not have labels, they are not supported for training.
+For multimodal datasets, if you need to perform dynamic data augmentation after loading the data (e.g., randomly adding noise to the input), please modify the encode method in the template.
+
+### Q7: How to debug Swift training?
+See the [Pre-training and Fine-tuning documentation](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html).
+
+### Q8: How to use a Python script for Swift training?
+Refer to the [notebook examples]((https://github.com/modelscope/ms-swift/tree/main/examples/notebook)).
+
+### Q9: How to use the UI for Swift training?
+Use the `swift web-ui` command. Training via the UI is consistent with the command line; for UI parameters, please refer to the command-line parameter documentation. The usage of custom datasets is the same as described in Q2 above. Megatron-swift does not support UI training.
+
+### Q10: Issues related to single-node, multi-GPU training
+Swift's multi-GPU training relies on torchrun. `deepspeed` and `device_map` are incompatible; you can only choose one. For more details, see the [examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu) in the code repository.
+
+### Q11: Issues related to multi-node, multi-GPU training
+Please see the [multi-node, multi-GPU examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node). During multi-node, multi-GPU training, only the main node produces logs.
+If multi-node training is slow (e.g., a significant speed drop when using DeepSpeed ZeRO3), please check this [issue](https://github.com/modelscope/ms-swift/issues/1825).
+
+### Q12: Issues related to large-scale datasets
+If the dataset is very large and tokenization takes a long time each run, use lazy_tokenize or streaming. See the [Command-line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html) for details.
+
+### Q13: Issues related to resuming from a checkpoint
+Keep the parameters from the previous training script and add `--resume_from_checkpoint output/xxx/vx-xxx/checkpoint-xxx`. For details, see [Command-line Parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). If the dataset has changed and you only want to load the model, also set `--resume_only_model`. For more complex scenarios, search for "resume" in the command-line parameters documentation.
+
+### Q14: Issues related to streaming dataset loading
+For streaming (`--streaming true`), data is loaded while training. You must set max_steps. For details, see the documentation for the streaming parameter in the [Command-line Parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#data-arguments).
+Note: Streaming does not shuffle the data and does not automatically create a validation set. The validation set must be specified using the `--val_dataset` command-line parameter.
+When resuming training with streaming, the data can only be indexed forward, not randomly. Skipping already trained data is very time-consuming, so using streaming for resuming is not recommended.
+
+### Q15: Issues related to packing
+Packing should be used with FlashAttention; otherwise, there will be discrepancies, and the attention_mask will have issues.
+The linear-attention in the Qwen3.5 model does not support var_len, so enabling packing is not recommended.
+When packing is enabled, multimodal data will undergo two map operations: one for the dataset and one for the template. If this is very slow, you can set `OMP_NUM_THREADS=14` to accelerate it, or disable packing to avoid the second mapping.
+
+### Q16: Dataset Multiprocessing
+When the dataset mapping process is slow, you can enable multiprocessing by setting the `--dataset_num_proc` parameter. It is normal for the mapping process of multimodal datasets to be slow.
+
+### Q17: How many checkpoints are saved by default after training?
+By default, all checkpoints are saved. For details, see the [command-line parameter save_total_limit](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
+
+### Q18: Loss and Accuracy During Training
+Custom loss functions can be added in the plugin. If you need loss curves for different datasets, set `--enable_channel_loss`.
+If the accuracy (acc) from evaluation does not match the accuracy calculated by re-running inference on the corresponding saved checkpoint (ckpt), it is because the calculation methods for eval_acc during training and acc during inference are different. `acc_strategy`: The default is `'token'`. Available options include `'token'` and `'seq'`.
+token_acc might not be available during training because for some models, the number of `logits` and `labels` do not match, so it is not calculated.
+You can view the currently supported losses or add new ones [here](https://github.com/modelscope/ms-swift/blob/main/swift/loss/mapping.py).
+To check if special tokens like `<image>` are included in the loss calculation, you can inspect the printed labels in the command-line log.
+When training an agent, tool_call should be included in the loss calculation, while tool_response should not.
+
+### Q19: Issues Related to Freezing Model Parameters
+During training, if freezing certain layers causes some parameters to not participate in gradient backpropagation, please configure the parameter `--ddp_find_unused_parameters true`.
+Regarding freeze_parameters and freeze_vit/freeze_aligner/freeze_llm: The freezing logic is applied first, and then trainable parameters are activated. The three parameters `freeze_vit`, `freeze_aligner`, and `freeze_llm` will adjust the sets of frozen and trainable parameters. Because the ViT in some models includes an `aligner`, the `aligner` will be added to trainable_parameters separately.
+The mechanism of the freeze_parameters_ratio parameter is to freeze layers from the bottom up, starting from the embeddings.
+
+### Q20: Issues Related to Sequence Parallelism
+Sequence parallelism supports PT (Pre-Training), SFT (Supervised Fine-Tuning), DPO, and GRPO. Refer to this example: [sequence_parallel](https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel).
+For VLM models, only FlashAttention is currently supported. For text-only models, both FlashAttention and SDPA are supported.
+Sequence parallelism can be used simultaneously with the Liger kernel.
+If there is a conflict between sequence parallelism and a custom loss, it is because sequence parallelism has its own custom loss implementation. You can modify it yourself [here](https://github.com/modelscope/ms-swift/blob/main/swift/trainers/sequence_parallel/ulysses.py).
+
+### Q21: Expanding the Vocabulary
+To expand the vocabulary using the Swift framework, you need to set the command-line parameters `new_special_tokens` and `--modules_to_save embed_tokens lm_head`. For details, see this [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/new_special_tokens).
+
+### Q22: Issues Related to Tuners
+LlamaPro in Swift has been adapted for multimodal tasks.
+LongLoRA can only be used with LLaMA-series models.
+LoRA training is incompatible with the `--trainable_parameters` parameter. For other trainable parameters outside the LoRA modules, use modules_to_save.
+
+### Q23: Embedding/Reranker Training
+[Example](https://github.com/modelscope/ms-swift/blob/main/examples/train/embedding) for training embeddings.
+[Example](https://github.com/modelscope/ms-swift/tree/main/examples/train/reranker) for training a reranker.
+For the data format, see [Custom Datasets](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html).
+
+### Q24: Training for Classification Tasks
+Swift supports multi-label classification. The data format is described in the custom dataset documentation. Search for `problem_type` in the command-line parameter documentation. Other aspects are the same as for regression tasks.
+Note: The label field should be at the same level as the message field.
+
+### Q25: Training a 'Thinking' Model
+See this [issue](https://github.com/modelscope/ms-swift/issues/4030).
+
+### Q26: Does Swift support distillation?
+Yes. Refer to this [example](https://github.com/modelscope/ms-swift/blob/main/examples/sampler/distill/distill.sh).
+
+### Q27: For GKD training, do the student and teacher models need to have the same model_type? For example, can one be a dense model and the other a MoE model?
+Yes, this is possible, as long as their vocabularies are the same. However, using a MoE (Mixture of Experts) model will be slower.
+
+### Q28: Issues Related to GRPO Training
+Swift now supports GRPO training for multimodal tasks.It is normal for the loss to approach 0 during GRPO training. See this [issue](https://github.com/huggingface/open-r1/issues/239#issuecomment-2646297851) for reference.
+Set sleep_mode to make the VllmEngine release GPU memory after inference is complete. The engine will be reloaded on the next call instead of occupying memory continuously.
+If you do not want to include the KL term during GRPO training, you can configure it using the beta command-line parameter.
+To continue with GRPO training after LoRA fine-tuning, search for `--adapters` in the command-line parameter documentation.
+Because calculating entropy incurs some extra overhead, the entropy curve is not logged by default. If you need it, set `--log_entropy true`.
+The colocate mode does not support use_async_engine.
+GRPO does not support channel_loss.
+The Liger kernel and padding-free cannot be enabled simultaneously during the GRPO phase. Doing so would require modifying the Liger GRPO loss implementation within the Liger kernel library, which is not convenient.
+If your training set contains different tasks, please refer to [Multi-task Training](https://swift.readthedocs.io/en/latest/Instruction/GRPO/DeveloperGuide/multi_task.html).
+
+### Q29: Issues Related to Reward Functions/Models
+reward_model and reward_funcs can be used together.
+For custom reward functions, refer to [examples/train/grpo/plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin).
+For math problems, you need to pass the solution from the dataset; otherwise, it is difficult to calculate accuracy.
+If you need to pass a specific column from the dataset into a custom reward function for ORM, place that column outside of the messages field.
+If you need to specify an LLM-judge model for scoring during GRPO training, please refer to the reward model documentation.
+
+### Q30: Issues Related to Rollout
+Rollout is likely incompatible with pipeline parallelism.
+The vLLM inference engine has trust_remote_code set to true by default.
+
+### Q31: I have a question: in the GRPO script, does save_steps refer to the "step" or the "global step"? Currently, my local training shows global step as 18, while wandb shows step as 628.
+It refers to the `global_step` displayed by the local tqdm progress bar.
+
+### Q32: If num_iterations=1 is used by default, the clip function becomes ineffective, right? DAPO's clip_higher also doesn't work. I've seen that veRL has a micro-batch setting to update the policy model in smaller batches within a single epoch to make the clip term effective. Looking at the source code, it seems ms-swift's mini-batch only performs gradient accumulation?
+Yes, num_iterations needs to be greater than 1.
+
+### Q33: Does GSPO training support the top_entropy_quantile parameter? After passing --importance_sampling_level sequence, can I still optimize the top x% of tokens based on the entropy distribution?
+Yes, it's supported. The order of operations is: first, the loss is calculated normally (affected by importance_sampling_level), and then the loss is masked based on top_entropy_quantile.
+
+### Q34: FAQ in the GRPO documentation
+For more GRPO-related FAQs, please refer to the [GRPO documentation](https://swift.readthedocs.io/en/latest/Instruction/GRPO/GetStarted/GRPO.html#faq).
+
+### Q35: Issues Related to PPO and Other Preference Training Methods
+PPO training does not support gradient clipping.
+Currently, PPO only supports scenarios where the reward model (RM) and the policy model belong to the same model series (i.e., same tokenizer/template).
+Multi-turn DPO is not supported.
+
+### Q36: Issues Related to MoE Model Training
+When training a Mixture-of-Experts (MoE) model with LoRA, if the aux_loss barely changes, add all_router to target_modules.
+During LoRA training, whether the router modules are trained depends on whether their gates are implemented as nn.Linear. If they are nn.Parameter, they are not trained. For details, see the command-line parameter [target_parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#tuner-arguments).
+
+### Q37: Issues Related to Megatron-SWIFT Training
+For checkpoint saving, refer to the command-line parameter [save_strategy](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Command-line-parameters.html).
+During Megatron multi-node training, logs are printed on the last pipeline parallelism (PP) rank, not the master node, because only the last PP rank has the complete information.
+Megatron-SWIFT now supports save_total_limit and SwanLab for monitoring training. See the [Megatron-SWIFT command-line parameters documentation](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Command-line-parameters.html) for details.
+ViT uses the Transformers model architecture and currently does not support parallelism. If you encounter an Out-Of-Memory (OOM) error during training, reduce `decoder_first_pipeline_num_layers`.
+Megatron-SWIFT supports new models. There is no tutorial available at the moment; please refer to the Pull Requests (PRs) for adding new models.
+The degree of parallelism for sequence_parallel is equal to the tensor parallelism (TP) degree.
+FP8 training supports block-wise implementation. Refer to the FP8 example in [examples/megatron/fp8](https://github.com/modelscope/ms-swift/tree/main/examples/megatron/fp8).
+
+### Q38: How do I configure resuming from a checkpoint in Megatron-SWIFT?
+To resume training, use `--mcore_model` to load the checkpoint. Additionally, configure these parameters as needed: `--finetune`, `--no_load_optim`, `--no_load_rng`. To resume LoRA training from a checkpoint, configure `--mcore_adapter`; other settings are the same as for full-parameter training. For details, see the [Megatron-SWIFT command-line parameters documentation](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Command-line-parameters.html).
+
+### Q39: Issues Related to MTP
+To enable MTP training, set the command-line parameter `mtp_num_layers`.
+If the base model does not include an MTP structure, you can initialize and train the MTP from scratch.
+Multi-modal MTP is not yet supported.
+
+### Q40: I have a question about Megatron GKD. If the teacher is Qwen3-235B and the student is Qwen3-30BA3B, for SFT on the 235B model, I used pp=8 and set decoder_first and decoder_last to 11. If I also set decoder_first/last during GKD, will it affect the student's parallelism?
+Currently, the parallelism parameters are shared between the teacher and student models. Support for different parallelism settings for each model will be introduced in a version after v4.
+
+### Q41: Issues Related to Quantized Model Training
+For QLoRA fine-tuning, refer to this [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora).
+Quantized models cannot be fully fine-tuned. The integer-type parameters in GPTQ models cannot be used for gradient calculation, so only attached structures like LoRA can be updated.
+For merging the model after QLoRA training, refer to the [QLoRA example](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora).
+Megatron-SWIFT does not support QLoRA training.
+
+### Q42: Training Specific Models
+SWIFT currently does not support training MiniCPM-O with audio modal input.
+To fine-tune DeepSeek-VL-2, use a transformers version earlier than 4.42 and `peft==0.11.*`.
+Fine-tuning Moonlight-16B-A3B-Instruct: Training is disabled in the model files. Refer to the solution for DeepSeek-VL-2, which can be found by searching the issues.
+Fine-tuning the Ovis-2 model is special; it requires padding to max_length. Set the `--max_length` argument.
+Qwen2.5-Omni currently only supports "thinker" mode for training, not "talker" mode.
+SFT for Qwen2-Audio does not support packing.
+
+### Q43: On devices that do not support Flash Attention, what is the default attention_implementation? The documentation says the default is none.
+The default implementation used is sdpa.
+
+### Q44: Is left padding the default for model training?
+You can choose either left or right padding for training. The default is right padding, while `batch infer` always uses left padding.
+
+### Q45: What are the parameters for MoE? I can't find them by searching for keywords in the parameter list. How do I configure parameters like the number of experts and expert routing?
+Use the parameters directly from the config.json file.
+
+### Q46: Can swift support setting a minimum learning rate? I feel like it decreases too much towards the end.
+Yes, you can set it with `--lr_scheduler_type cosine_with_min_lr --lr_scheduler_kwargs '{"min_lr": 1e-6}'`.
+
+### Q47: Does it currently support configuring GRPO and SFT using YAML files?
+Yes, both are supported. The configuration is directly processed into command-line arguments in main.py.
+
+### Q48: Is it true that use_liger_kernel and log_entropy cannot be used together right now?
+They are not supported together.
+
+### Q49: How can I handle this error? Installing apex didn't help.
+```text
+RuntimeError: ColumnParallelLinear was called with gradient_accumulation_fusion set to True but the custom CUDA extension fused_weight_gradient_mlp_cuda module is not found. To use gradient_accumulation_fusion you must install APEX with --cpp_ext and --cuda_ext. For example: pip install --global-option="--cpp_ext" --global-option="--cuda_ext ." Note that the extension requires CUDA>=11. Otherwise, you must turn off gradient accumulation fusion.
+```
+Set `--gradient_accumulation_fusion false`.
+
+### Q50: If I finetune a VLM on several tasks together, and the video sampling rules for different tasks are inconsistent, does ms-swift support this? Where can I configure it?
+Check `interleave_prob` in the [Command-line Parameters Documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
+
+### Q51: I have a question. During multimodal packing pre-training, it seems the GPU memory usage increases slightly after each "pytorch allocator cache flushes since last step," leading to an OOM error after many steps.
+Add the environment variable `PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'`.
+
+### Q52: Can use_logits_to_keep be used with large multimodal models now?
+It will cause an error if the expansion of multimodal tokens happens within the model's forward pass.
+
+### Q53: Why does the GPU memory usage increase significantly several times during training, for example, at step 50 or 100?
+Set the `PYTORCH_CUDA_ALLOC_CONF` environment variable. For details, please refer to the PyTorch documentation.
+
+### Q54: Is there a practical guide for fine-tuning a Qwen base model into a chat model? Are there any special configurations needed?
+Use `swift sft`. No other special configurations are needed. Refer to the [example](https://github.com/modelscope/ms-swift/tree/main/examples/train/base_to_chat).
+
+### Q55: After training, the model's responses contain a lot of repetitive content.
+Please refer to [Pre-training and Fine-tuning](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html). If repetition occurs during training, try training for more epochs, cleaning the data, performing full-parameter fine-tuning, or using RLHF to mitigate the issue.
+
+### Q56: Why does using --torch_dtype float16 (my GPU doesn't support bf16) result in the error: lib/python3.12/site-packages/torch/amp/grad_scaler.py", line 260, in _unscale_grads_ raise ValueError("Attempting to unscale FP16 gradients.") ValueError: Attempting to unscale FP16 gradients.
+For full-parameter fine-tuning, you cannot train with fp16.
+
+### Q57: I'm getting an error when merging LoRA parameters. My current PEFT version is 0.11.0. Is this because the PEFT version needs to be upgraded?
+```text
+File "/opt/conda/lib/python3.9/site-packages/peft/config.py", line 118, in from_peft_type
+  return config_cls(**kwargs)
+TypeError: __init__() got an unexpected keyword argument 'corda_config'
+```
+This is caused by a mismatch between the PEFT versions used for training and merging.
+
+### Q58: How to solve this problem? safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge
+You are running out of disk space, and the model was not saved completely.
+
+### Q59: Why does this error appear here? I can't find where numpy.object is.
+Try `numpy==1.26.3`.
+
+### Q60: Training with unsloth, I get the error: assert(type(target_modules) in (list,tuple,)). The parameter I configured is --target_modules all-linear.
+Don't use `all-linear`. Change it to a specific list of modules, for example, `--target_modules q_proj k_proj v_proj`.
+
+### Q61: For qwen2.5-omni, does --freeze_vit false mean that both the visual encoder and the audio encoder are unfrozen? Is there a way to unfreeze only the audio encoder and not the visual encoder?
+Use the `--target_regex` argument.
+
+## Inference
+
+Swift supports inference via Python scripts, command line, and UI interfaces. For details, see [Inference and Deployment](https://swift.readthedocs.io/en/latest/Instruction/Inference-and-deployment.html).
+
+### Q1: How to set up a model for inference in Swift?
+For models from full-parameter training, models merged after LoRA training, or models downloaded from the Model Hub, set the command-line argument `--model <model_id_or_path>`. For unmerged models after LoRA training, use `--adapters`, and you can also specify the base model path with `--model`.
+
+### Q2: How to use a dataset for inference in Swift? Where are the inference results saved?
+Use `--val_dataset <your-val-dataset>` to specify the dataset. For trained models, you can also set the `--load_data_args true` argument. The path to save inference results is set via `--result_path your_path`, and the path will be printed in the logs. For details, see the documentation on [Command-line parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
+If you need to keep extra fields from the inference dataset, set `--remove_unused_columns false`.
+
+### Q3: How to set up batch inference in Swift?
+If the infer_backend is `transformers`, set the command-line argument `--max_batch_size 16`, or use a [Python script](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py). Here, max_batch_size refers to the batch size per GPU card.
+
+### Q4: How to set up streaming inference in Swift?
+Set `--stream true`. In this case, the inference results will be written to a jsonl file line by line. Note that streaming inference does not support DDP.
+
+### Q5: Questions related to vLLM and SGLang inference backends
+For models trained with LoRA, please check the vLLM and SGLang documentation. If they support LoRA inference, merging the adapters is not required. Additionally, SGLang inference does not currently support multi-modality.
+
+### Q6: Questions related to generation parameters
+Parameters like temperature are read from generation_config.json by default. Set `--temperature 0` or `--top_k 1` to disable randomness in inference.
+
+### Q7: How to set the system prompt to empty? If I don't set the system parameter in the command line, a default one is still added.
+Set `--system ''`.
+
+### Q8: How to compute metrics like ACC/ROUGE during inference?
+Refer to the documentation on inference parameters for [metrics](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#inference-arguments).
+
+### Q9: During model inference, which parameter should be set to continue generation from a specific prefix?
+Use the `--response_prefix` parameter.
+
+### Q10: The 'answer' in my data already contains part of the prompt. How should I modify the inference to complete the 'answer'?
+```text
+{"messages": [{"role": "system", "content": "<system>"}, {"role": "user", "content": "<query1>"}, {"role": "assistant", "content": "answer1, "}]}
+```
+This is supported in Swift versions 3.0 and later. Refer to [examples/infer/demo_agent](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_agent.py).
+
+### Q11: During multimodal model inference, how can I limit the maximum number of pixels to reduce GPU memory usage?
+Set the command-line argument `--max_pixels xxx`, the environment variable `MAX_PIXELS=xxx`, or the specific model argument `--model_kwargs '{"max_pixels": xxx}'`. The environment variable only affects the models specified in the documentation. For details, see the documentation on [Specific Model Arguments](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#specific-model-arguments).
+
+### Q12: How to output probability values (logprobs) during Swift inference?
+For command-line inference, set `--logprobs true`. For Python script inference, set `request_config = RequestConfig(..., logprobs=True, top_logprobs=2)`. Refer to [test_logprobs.py](https://github.com/modelscope/ms-swift/blob/main/tests/infer/test_logprobs.py).
+
+### Q13: How to output last_hidden_state during Swift inference?
+There is no direct example, but you can refer to the `_get_last_hidden_state` method in the GRPO trainer.
+
+### Q14: Issues with inconsistent inference results between Transformers, vLLM, Ollama, etc.
+Swift's templates are aligned with those of Transformers. Check if the inference parameters are consistent. Additionally, there are differences between VllmEngine and TransformersEngine.
+
+### Q15: Inference for embedding/reranker models
+For embedding model inference, refer to the [example](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_embedding.py) here. For reranker model inference, refer to the [example](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_reranker.py) here.
+
+### Q16: When using a Python script for inference, how can I use the CPU?
+Set the environment variable: `os.environ['CUDA_VISIBLE_DEVICES'] = '-1'`.
+
+### Q17: Does the swift infer command support multi-machine inference?
+If the model can fit on a single node, you can orchestrate it using Kubernetes. If the model does not fit on a single node, multi-machine inference is not supported.
+
+### Q18: When sampling with Swift, it seems batching is not supported? It looks like it samples one by one in a for loop, which is a bit slow.
+There is a [script](https://github.com/modelscope/ms-swift/blob/main/examples/train/rft/rft.py) that can use multiple processes to split the dataset for sampling.
+
+### Q19: Issues related to specific model dependency versions
+If Qwen2-Audio inference results are garbled, please use transformers==4.48.
+LoRA models trained with transformers==4.55.2 can no longer be loaded by versions older than 4.52. See [issue#5440](https://github.com/modelscope/ms-swift/issues/5440) for details.
+Swift is compatible with different versions of qwen-vl-utils, so you do not need to switch its version when using qwen2.5-vl and qwen3-vl models.
+
+### Q20: I got an error: safetensors_rust.SafetensorError: Error while deserializing header:MetadataIncompleteBuffer
+The model weights are corrupted.
+
+### Q21: How can I handle this vLLM error: `ValueError: the decoder prompt contains a(n) video item with length 16758, which exceeds the pre-allocated encoder cache size 16384. please reduce the input size or increase the encoder cache size by setting --limit-mm-per-prompt at startup.`?
+This usually means the multimodal input is too long and exceeds vLLM's pre-allocated encoder cache size. You can adjust the encoder cache size with `--limit-mm-per-prompt`. Another practical workaround is to increase `max_num_batched_tokens`. In Swift cli:
+```shell
+--vllm_engine_kwargs '{"max_num_batched_tokens": 20000}'
+```
+
+## Export
+
+### Q1: Errors related to autoawq
+If you encounter autoawq-related errors during inference without using an AWQ-quantized model, try uninstalling autoawq and running the inference again. For models that do not support AWQ quantization, you can try using GPTQ for quantization instead.
+
+### Q2: The model does not fit on a single GPU during Swift quantization
+Try setting the `--device_map cpu` flag. Alternatively, you can load the model across multiple GPUs but perform the quantization on a single GPU.
+
+### Q3: I am trying to quantize the Qwen2.5 72B model to GPTQ int4 using swift export. I'm using the default max_model_length=32768 and a calibration dataset with 128 samples. However, the quantization process failed with the following error log: factorization could not be completed because the input is not positive-definite (the leading minor of order 18145 is not positive-definite). What could be the reason for this?
+This is an issue caused by the Hessian matrix not being positive-definite. Please try using a different calibration dataset.
+
+### Q4: If I pass a custom template_type when using swift export (e.g., swift export --template_type custom), will this permanently change the template associated with the model?
+No, it will not be permanently modified. In Swift, templates are defined within the framework's internal code; they are not saved as part of the model files, for instance, in a Jinja format.
+
+### Q5: Can the model be directly converted to GGUF format after training?
+Currently, only exporting to ModelFile is supported. For details, please refer to the documentation on [Command-line parameters](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).
+
+## Deployment
+
+### Q1: How do I set up a model for Swift deployment?
+This is the same as Q1 for inference.
+
+### Q2: How do I perform multi-GPU deployment with Swift?
+For details, see the [example](https://github.com/modelscope/ms-swift/tree/main/examples/deploy). If you are using the transformers engine, it does not support DDP, so multi-GPU deployment is not possible. Furthermore, heterogeneous deployment (e.g., using different GPU models or assigning different memory ratios to each GPU) is not supported.
+
+### Q3: Regarding the system prompt, you can specify it via the --system parameter, prepend it to each data entry in the dataset, or define it in the template. Is it sufficient to use just one of these methods? And are they all treated the same way by the model?
+System prompt priority: The one in the dataset > The one from the command line > The default one in the template.
+
+### Q4: Questions about multimodal input from the client.
+To pass images, audio, and other media from the client, please see the [client example](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/client/mllm). If you encounter an invalid image URL, you can set a request timeout either through the `SWIFT_TIMEOUT` environment variable or as a parameter in `InferClient`.
+
+### Q5: Questions about setting generation parameters.
+For inference, parameters like temperature can only be set before launching. For deployment, you can set default values at launch, which can later be overridden by client-side settings.
+
+### Q6: How do I enable streaming generation for a model deployed with Swift?
+This is controlled on the client side. Please refer to the examples in [examples/deploy/client](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/client).
+
+### Q7: How can I output token probabilities in a Swift deployment?
+Set `--logprobs true` on the server side. Then, the client needs to pass the corresponding parameters, for example: `request_config = RequestConfig(..., logprobs=True, top_logprobs=2)`.
+
+### Q8: Questions about the "thinking" process during model deployment.
+If you need to disable the "thinking" step, it can currently only be done at launch time with swift deploy. See this [issue](https://github.com/modelscope/ms-swift/issues/4030) for more information.
+
+### Q9: During deployment, which parameter should be set to generate multiple outputs in a single request?
+The `n` parameter in `RequestConfig`.
+
+### Q10: Questions regarding deployment with Swift using --infer_backend vllm compared to deploying directly with VLLM.
+If there's a significant difference in inference results, it's likely due to a template mismatch. If there's a significant difference in inference speed, it might be caused by inconsistent image resolutions. Swift uses the V1 engine by default; you can control this with the environment variable `VLLM_USE_V1=1`.
+
+### Q11: Questions about specific models and dependency versions.
+If you get an error message about a missing "model.language_model.embed_tokens.weight", it indicates a version mismatch in the transformers library between training and inference. If you encounter garbled text when running inference with Qwen2.5 using FP16, try using BF16.
+
+### Q12: I have a question: After deploying Qwen2-7B, when I use the client, I have to call the OpenAI API with client.completions.create and cannot use client.chat.completions.create. However, when using the qwen2-7b-instruct-q5_k_m.gguf model, I can use client.chat.completions.create. Why is this?
+Base models can use client.chat.completions.create, but this is intended as a compatibility feature.
+
+## Evaluation
+
+### Q1: What evaluation datasets does Swift support? And how can I use a custom evaluation dataset?
+For details on using standard and custom evaluation datasets, please refer to the documentation on [Evaluation](https://swift.readthedocs.io/en/latest/Instruction/Evaluation.html).
+
+### Q2: After manually downloading an officially supported evaluation dataset, can swift eval be configured to evaluate using a local path?
+For offline evaluation, please refer to the EvalScope documentation's [Quick Start](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html).
+
+### Q3: The model after eval fine-tuning keeps stopping at a fixed percentage, but the vllm service seems to be running normally. The larger the model, the sooner it disconnects.
+Set the `SWIFT_TIMEOUT` environment variable to -1.
+
+### Q4: Can I control the number of dataset entries during evaluation? It takes over an hour to evaluate an MMLU, which is too slow.
+Use the configuration parameter `--eval_limit`. This `--eval_limit` controls the number of entries in each subset. For example, if MMLU has over 50 subsets, and each limit is set to 10 entries, then that would be over 500 entries in total.
+
+### Q5: In swift eval, the model stops generating after 1024 tokens. How can I modify this? Setting --max_new_tokens 5000 doesn't seem to work.
+Check the command-line parameter [eval_generation_config](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html#evaluation-arguments).
+
+### Q6: How can I load downloaded datasets locally when using opencompass backend for evaluation?
+The opencompass backend doesn't support setting `data_args`.
+
+### Q7: Does swift eval with --eval_backend OpenCompass not support custom datasets?
+```text
+ValueError: eval_dataset: /mnt/workspace/data.jsonl is not supported.
+eval_backend: OpenCompass supported datasets: ['C3', 'summedits', 'WiC', 'csl', 'lambada', 'mbpp', 'hellaswag', 'ARC_e', 'math', 'nq', 'race', 'MultiRC', 'cmb', 'ceval', 'GaokaoBench', 'mmlu', 'winogrande', 'tnews', 'triviaqa', 'CB', 'cluewsc', 'humaneval', 'AX_g', 'DRCD', 'RTE', 'ocnli_fc', 'gsm8k', 'obqa', 'ReCoRD', 'Xsum', 'ocnli', 'WSC', 'siqa', 'agieval', 'piqa', 'cmnli', 'cmmlu', 'eprstmt', 'storycloze', 'AX_b', 'afqmc', 'strategyqa', 'bustm', 'BoolQ', 'COPA', 'ARC_c', 'PMMEval', 'chid', 'CMRC', 'lcsts']
+```
+OpenCompass doesn't support custom datasets; use native mode for custom datasets.
+
+### Q8: Evalscope can natively generate reports, but other backends like OpenCompass do not support report visualization, correct?
+Currently, only native visualization is supported; other backends are not yet supported.
+
+### Q9: Could you explain what causes the following error when using ifeval for evaluation?
+```text
+[Errno 20] Not a directory: '/root/nltk_data/tokenizers/punkt_tab.zip/punkt_tab/english/collocations.tab'
+```
+Unzip the file using `unzip /path/to/nltk_data/tokenizers/punkt_tab.zip`.
+
+### Q10: When evaluating with eval_backend='OpenCompass', how can I specify the path to offline datasets?
+Check the [data preparation guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html#data-preparation), download and unzip the dataset. You don't need to specify `dataset-args`; just place the dataset folder (the data folder) in the current working directory.
+
+### Q11: What causes the following error when using evalscope?
+```text
+unzip: cannot find or open /root/nltk_data/tokenizers/punkt_tab.zip, /root/nltk_data/tokenizers/punkt_tab.zip.zip or /root/nltk_data/tokenizers/punkt_tab.zip.ZIP
+```
+This occurs during the download of nltk dependencies. Manually download [punkt_tab.zip](https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/open_data/nltk_data/punkt_tab.zip) and unzip it under `~/nltk_data/tokenizers`.
+
+### Q12: Why is there no issue with plain text, but when testing multi-modal data, even though we specify the path, it still fails to detect the dataset and attempts to download it?
+The VLMEvalKit process differs from native; it will automatically download data to `~/LMUData/`. For details, see the [documentation](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html#data-preparation).
+
+### Q13: When using swift eval for benchmark evaluation, can I specify an llm as a judge and how should I pass in the parameters?
+Yes, you can use swift to pass `judge-model-args` parameters from `extra_eval_args`, which include `api_key, api_url, and model_id`, as a JSON string.
+
+### Q14: What could be the reason for uneven GPU memory allocation across multiple cards when running eval?
+```shell
+NPROC_PER_NODE=8
+ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7\ MAX_PIXELS=802816\ swift eval\
+--model "$MODEL_PATH” \$EXTRA_ARGS \
+--eval_backend Native \ --infer_backend transformers\ --device_map auto \
+--eval_limit"$EVAL_LIMIT"\ --eval_dataset general_qa\
+--dataset_args "{\"general_qa\": {\"local_path\": \"${DATA_PATH}\", \"subset_list\": [\"${SUBSET_NAME}\"]}}" \ --host 127.0.0.1\> "$LOG_FILE" 2>&1
+```
+swift eval does not support being launched in DDP mode.
+
+### Q15: Where can I see what extra fields, besides the question itself, are included in the query sent during a Swift evaluation?
+The simplest way is to check the input field in the output reviews file. It contains the content sent to the model, converted to Markdown format. Note that this is not available if you are using the opencompass backend; you need to use the native backend.
+
+The evaluation capabilities of ms-swift utilize the ModelScope community's evaluation framework, EvalScope. For more advanced features, please use the [EvalScope](https://evalscope.readthedocs.io/en/latest/get_started/introduction.html) directly.

docs/source_en/Instruction/GRPO/AdvancedResearch/CHORD.md

@@ -0,0 +1,63 @@
+# On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting (CHORD)
+
+This document describes the CHORD algorithm proposed in the paper [On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting](https://arxiv.org/abs/2508.11408). The core idea of CHORD is to dynamically integrate expert data (SFT) into reinforcement learning by a dual control mechanism: a global weight μ plus a token-level weight φ, thereby balancing imitation and exploration.
+
+## Algorithm Overview
+CHORD mixes training by introducing the SFT loss into the GRPO loss. The overall objective is:
+
+$$
+    \mathcal{L}_{\text{CHORD}} = (1 - \mu) \cdot \mathcal{L}_{\text{GRPO}} + \mu \cdot \mathcal{L}_{\text{SFT}}
+$$
+
+where:
+- $\mathcal{L}_{\text{GRPO}}$: on-policy RL loss based on on-policy samples (similar to PPO).
+- $\mathcal{L}_{\text{SFT}}$: supervised fine-tuning (SFT) loss.
+- $\mu \in [0, 1]$: global balancing coefficient that controls the contribution of the SFT signal to the overall gradient.
+
+### Configuration (data and batch sizes)
+We can implement CHORD training based on GRPO training.
+
+CHORD requires specifying an additional SFT dataset and batch size at training time:
+- `chord_sft_dataset`: the SFT dataset that provides expert data.
+- `chord_sft_per_device_train_batch_size`: SFT mini-batch size per device.
+
+---
+
+## Two CHORD Variants
+
+The paper proposes two variants: CHORD-μ and CHORD-φ.
+
+### CHORD-μ
+CHORD-μ gradually decays μ during training to transition from imitating experts toward autonomous exploration.
+
+Parameters:
+- `chord_mu_peak`: the peak value of μ.
+- `chord_mu_valley`: the final decayed value of μ.
+- `chord_mu_warmup_steps`: number of training steps to ramp μ up to the peak.
+- `chord_mu_decay_steps`: number of training steps to decay μ from peak to valley.
+
+### CHORD-φ (Token-level weighting)
+CHORD-φ uses a token-wise weighting function φ to dynamically control each expert token's gradient contribution.
+
+Definition of φ:
+$$
+    \phi(y_t^\star, \pi_\theta) = p_t \cdot (1 - p_t)
+$$
+
+where:
+- $p_t = \pi_\theta(y_t^\star \mid x, y_{<t}^\star)$ is the model's current predicted probability of the expert token.
+- When $p_t \approx 0.5$ (model uncertainty), φ is maximal → emphasize tokens the model is uncertain about.
+- When $p_t \approx 0$ or $p_t \approx 1$, φ → 0 → avoid overemphasizing tokens that are already certain or impossible.
+
+Parameter to enable φ weighting:
+- `chord_enable_phi_function: bool = False`
+  - Set to `True` to enable token-wise weight φ.
+
+Note: If using a constant μ, set `chord_mu_peak` and `chord_mu_valley` to the same value.
+
+<details>
+<summary>Code implementation of μ scheduling and loss computation</summary>
+See the `GRPOTrainer` method `_compute_chord_loss`.
+</details>
+
+Training reference script: https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/chord.sh

docs/source_en/Instruction/GRPO/AdvancedResearch/CISPO.md

@@ -0,0 +1,62 @@
+# Clipped Importance Sampling Policy Optimization (CISPO)
+
+Clipped Importance Sampling Policy Optimization (CISPO) is a reinforcement learning algorithm proposed in the [MiniMax-M1](https://arxiv.org/abs/2506.13585) paper. Compared to GRPO (Group Relative Policy Optimization), CISPO clips the importance sampling weights themselves.
+
+## Algorithm Overview
+
+For clarity, we explain CISPO by contrasting it with GRPO.
+
+GRPO limits the magnitude of policy updates by clipping the policy ratio. Its loss function is:
+
+$$
+\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}\left[\min\left(r_t(\theta) \cdot \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \cdot \hat{A}_t\right)\right]
+$$
+
+where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the importance sampling ratio.
+
+When handling long reasoning chains, this clipping approach can lead to the following issues:
+
+**Gradient Suppression of Critical Tokens**: In complex reasoning tasks, certain critical low-probability tokens (such as *However, Recheck, Wait, Aha*) are crucial for triggering deep thinking and reasoning error correction. These tokens have low probability in the old policy $\pi_{\theta_{\text{old}}}$. When the new policy attempts to increase their probability, it results in a large policy ratio $r_t(\theta)$, and GRPO's clipping mechanism will discard these tokens.
+
+
+### CISPO's Solution
+
+The core idea of CISPO is to clip the importance sampling weights while preserving gradient updates. Specifically, CISPO's loss function is:
+
+$$
+\mathcal{L}_{\text{CISPO}}(\theta) = -\mathbb{E}\left[\text{detach}\left(\min(r_t(\theta), \epsilon_{\text{high}})\right) \cdot \hat{A}_t \cdot \log \pi_\theta(a_t|s_t)\right]
+$$
+
+where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the importance sampling ratio.
+
+**Key Mechanisms**:
+- Clip the importance sampling weights: $\min(r_t(\theta), \epsilon_{\text{high}})$
+- **Detach operation**: The clipped weights do not participate in gradient computation and serve as constant coefficients
+- Gradients come from the $\log \pi_\theta(a_t|s_t)$ term, ensuring all tokens contribute gradients
+
+
+## Implementation Details
+
+The pseudo-code implementation of CISPO is as follows:
+
+```python
+log_ratio = per_token_logps - old_per_token_logps
+importance_weights = torch.exp(log_ratio)  # r_t(θ) = π_θ / π_θ_old
+
+clamped_ratios = torch.clamp(importance_weights, max=epsilon_high).detach()
+
+per_token_loss = -clamped_ratios * advantages.unsqueeze(1) * per_token_logps
+```
+
+## Parameter Configuration
+
+CISPO training can be enabled based on `GRPOTrainer` by setting the following parameters:
+
+```bash
+--loss_type cispo
+--epsilon_high 5.0
+```
+
+> Compared to other algorithms, cispo generally uses a larger value for epsilon_high. The minimax paper does not provide specific parameter settings; the value used here refers to the experimental setup in the paper [ScaleRL](https://arxiv.org/pdf/2510.13786).
+
+For other training parameters, refer to the [GRPO parameter documentation](../../Command-line-parameters.md#grpo-arguments).

docs/source_en/Instruction/GRPO/AdvancedResearch/DAPO.md

@@ -0,0 +1,89 @@
+# DAPO: An Open-Source LLM Reinforcement Learning System at Scale
+
+[Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO)](https://arxiv.org/abs/2503.14476) introduces several tricks based on GRPO, including:
+- [Clip Higher](#clip-higher)
+- [Dynamic Sampling](#dynamic-sampling)
+- [Token level Loss](#token-level-loss)
+- [Overlong Filtering](#overlong-filtering)
+- [Soft Overlong Punishment](#soft-overlong-punishment)
+
+## Clip Higher
+PPO and GRPO use symmetric clipping ranges (e.g., ±0.2) to limit the magnitude of policy updates. While this ensures stability, it also restricts the model's exploratory capabilities. Specifically, when certain tokens have extremely low probabilities under the old policy, even if the current gradient indicates they should be reinforced (A > 0), the maximum increase is strictly limited.
+
+DAPO employs an asymmetric clipping range, raising the upper clipping limit to encourage exploration:
+- The upper bound (encouragement side) is relaxed to 0.28.
+- The lower bound (suppression side) remains unchanged at 0.2.
+
+In GRPO, the default symmetric clipping range is set using `epsilon`.
+
+Parameters:
+- `epsilon_high` sets the upper clipping range, while `epsilon` serves as the lower clipping range.
+
+## Dynamic Sampling
+GRPO samples multiple responses per question to compute inter-group advantages:
+
+$$
+\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
+$$
+
+However, when all generated outputs {o_i} receive the same reward, the inter-group advantage becomes zero, leading to vanishing gradients and reduced training efficiency.
+
+DAPO addresses this issue with a dynamic sampling strategy:
+- Skips data with zero inter-group reward standard deviation during sampling.
+- Continues generating samples until the batch is filled.
+
+Parameters:
+- `dynamic_sample true` enables dynamic sampling.
+- `max_resample_times` sets the maximum number of resampling attempts.
+
+## Token level Loss
+GRPO normalizes losses at the sentence level, which introduces bias based on response length.
+
+DAPO uses token-level normalization to avoid this bias in loss calculation.
+
+Parameters:
+- `loss_type bnpo/dapo` enables token-level normalization.
+
+> For the loss_type formula, please refer to the [documentation](../DeveloperGuide/loss_types.md).
+
+## Overlong Filtering
+DAPO argues that forcibly truncated responses contain high reward noise, making it difficult for the model to distinguish between quality issues and length issues. To address this, DAPO filters out truncated data during training, excluding it from loss computation.
+
+Parameters:
+- `overlong_filter` enables filtering of overly long samples.
+
+## Soft Overlong Punishment
+Language models often struggle with controlling output length:
+- Overly long outputs may be truncated, leading to incorrect judgments of valid content.
+- Unconstrained length generation affects practicality and computational efficiency.
+
+DAPO designs a three-stage length penalty function:
+
+$$
+R_{\text{length}}(L) =
+\begin{cases}
+0, & L \leq L_{\text{max}} - L_{\text{cache}} \\[10pt]
+\dfrac{(L_{\text{max}} - L_{\text{cache}}) - L}{L_{\text{cache}}}, & L_{\text{max}} - L_{\text{cache}} < L \leq L_{\text{max}} \\[10pt]
+-1, &  L > L_{\text{max}}
+\end{cases}
+$$
+
+When the length falls within the interval $(L_{\text{max}} - L_{\text{cache}} < L \leq L_{\text{max}})$, a linearly increasing penalty is applied. For lengths $(L > L_{\text{max}})$, the maximum penalty (-1) is imposed.
+
+Parameters:
+- `reward_funcs soft_overlong` enables this reward function.
+- `soft_max_length` sets L_max, which defaults to the model's maximum output length (`max_completion_length`).
+- `soft_cache_length` sets L_cache.
+
+## Parameter Settings
+In summary, the following parameters can be set based on GRPOTrainer to implement DAPO training.
+
+| Parameter             | Type      | Value       |
+|-----------------------|-----------|-------------|
+| `--loss_type`         | `str`     | `bnpo`/`dapo`|
+| `--epsilon_high`      | `float`   | `0.28`      |
+| `--dynamic_sample`    | `bool`    | `true`      |
+| `--max_resample_times`| `int`     | `3`         |
+| `--overlong_filter`   | `bool`    | `true`      |
+| `--reward_funcs`      | `str`     | `soft_overlong`|
+| `--soft_cache_length` | `int`     | `4096`      |

docs/source_en/Instruction/GRPO/AdvancedResearch/FIPO.md

@@ -0,0 +1,55 @@
+# FIPO: Future-KL Influenced Policy Optimization
+
+Author: [li2zhi](https://github.com/li2zhi)
+
+[FIPO](https://arxiv.org/abs/2603.19835) is a value-free RL method for eliciting longer and deeper reasoning. It keeps the GRPO/DAPO training scaffold, but changes how token-level policy updates are weighted: instead of applying one sequence-level advantage uniformly to every token, FIPO uses a discounted Future-KL signal to estimate whether the future trajectory after each token is being reinforced or suppressed.
+
+## Core Idea
+
+In GRPO/DAPO, tokens in the same response usually share the same sequence-level advantage:
+
+$$
+\hat{A}_{i,t} = \hat{A}_{i}
+$$
+
+This is simple and stable, but the credit assignment is coarse. FIPO starts from the signed log-probability shift between the current policy and the old policy:
+
+$$
+\Delta \log p_t = \log \pi_\theta(y_t \mid x, y_{<t}) -
+\log \pi_{\mathrm{old}}(y_t \mid x, y_{<t})
+$$
+
+A positive value means the token probability is being increased by the current update, while a negative value means it is being suppressed. FIPO then accumulates this signal from the current token to the end of the response:
+
+$$
+\mathrm{FutureKL}_t =
+\sum_{k=t}^{T}\gamma^{k-t} M_k \Delta \log p_k
+$$
+
+where $M_k$ is the completion mask and $\gamma = 2^{-1 / \text{decay\_rate}}$. A larger `decay_rate` gives farther future tokens more influence; a smaller value makes the signal more local. FIPO maps the Future-KL value into a bounded influence weight:
+
+$$
+f_t = \mathrm{clip}(\exp(\mathrm{FutureKL}_t), 1-\epsilon_f, 1+\epsilon_f)
+$$
+
+The original advantage is then replaced by a future-aware advantage:
+
+$$
+\tilde{A}_{i,t} = \hat{A}_{i} \cdot f_{i,t}
+$$
+
+## Parameters
+
+
+| Parameter                 | Type    | Default | Description                                                                                                                                                                                                         |
+| ------------------------- | ------- | ------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `--loss_type`             | `str`   | `grpo`  | Set to`fipo` to enable FIPO loss                                                                                                                                                                                    |
+| `--delta`                 | `float` | `None`  | When enabled, it is used for both Future-KL high-IS-ratio token filtering and the main-loss dual-clip upper bound, and should be greater than `1 + epsilon_high`. Set it to `10.0` to match the official 32B script |
+| `--fipo_decay_rate`       | `float` | `32.0`  | Half-life parameter for Future-KL; the actual discount is`2 ** (-1 / fipo_decay_rate)`                                                                                                                              |
+| `--fipo_clip_range`       | `float` | `0.2`   | Influence weight clipping range;`0.2` clips to `[0.8, 1.2]`                                                                                                                                                         |
+| `--fipo_clip_high_only`   | `bool`  | `true`  | If`true`, clips the weight to `[1.0, 1.0 + fipo_clip_range]`                                                                                                                                                        |
+| `--fipo_safety_threshold` | `float` | `4.0`   | Caps the FIPO weight to `[0.8, 1.0]` for negative-advantage tokens whose IS ratio exceeds this threshold                                                                                                            |
+
+## Training Example
+
+[swift](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/fipo.sh)

docs/source_en/Instruction/GRPO/AdvancedResearch/GSPO.md

@@ -0,0 +1,66 @@
+# Group Sequence Policy Optimization (GSPO)
+
+In [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071), it is pointed out that GRPO computes importance sampling weights at the token level. However, this approach is problematic: since each token is only sampled once, it cannot realize effective distribution correction, and instead introduces high-variance noise during training, which can easily lead to unstable gradient estimates and even training collapse. Therefore, the paper argues that the unit of the objective function should be consistent with that of the reward. Since the reward is typically given at the sequence level (i.e., for the entire generated response), it is more reasonable to perform off-policy correction and optimization at the sequence level rather than the token level.
+
+Below are the three main strategies for computing importance sampling weights:
+
+1. GRPO
+GRPO computes the importance sampling ratio independently for each token, as follows:
+
+$$
+w^{\mathrm{GRPO}}_{i,t} = \frac{\pi_\theta (y_{i, t} \mid x, y_{i, <t})}{\pi_{\theta_{\mathrm{old}}} (y_{i, t} \mid x, y_{i, <t})}
+$$
+
+2. GSPO (Sequence-Level)
+GSPO calculates the importance sampling ratio at the sequence level, given by:
+
+$$
+w^{\mathrm{GSPO}}_{i} = \left[ \frac{\pi_\theta (y_i \mid x)}{\pi_{\theta_{\mathrm{old}}} (y_i \mid x)} \right]^{\frac{1}{|y_i|}}
+= \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta (y_{i, t} \mid x, y_{i, <t})}{\pi_{\theta_{\mathrm{old}}} (y_{i, t} \mid x, y_{i, <t})} \right)
+$$
+
+3. GSPO-token
+GSPO-token combines both sequence-level and token-level importance sampling:
+
+$$
+w_{i, t}^{\mathrm{GSPO-token}} = \mathrm{sg}\left[w_i^{\mathrm{GSPO}}\right] \cdot \frac{\pi_{\theta}(y_{i, t} \mid x, y_{i, < t})}{\mathrm{sg}\left[\pi_{\theta}(y_{i, t} \mid x, y_{i, < t})\right]}
+$$
+
+where $\mathrm{sg}[\cdot]$ denotes stop-gradient (detach()).
+
+> **NOTE:** According to gradient analysis (i.e., Eqs. (11) and (18) in the paper), when the advantage for each token is identical, GSPO-token is equivalent to GSPO. In the current implementation of GRPO, all token advantages are normalized based on the sentence-level reward within each group. Therefore, in this setting, GSPO-token and GSPO are theoretically equivalent. However, GSPO-token provides support for future fine-grained (token-level) advantages.
+
+Pseudo-code implementation:
+```python
+log_ratio = per_token_logps - old_per_token_logps
+# GRPO
+log_importance_weights = log_ratio
+
+# GSPO (Sequence-Level)
+seq_weight = (log_ratio * mask).sum(-1) / mask.sum(-1)
+log_importance_weights = seq_weight.unsqueeze(-1)  # (B,1)
+
+# GSPO-token
+seq_weight = (log_ratio * mask).sum(-1) / mask.sum(-1)
+log_importance_weights = seq_weight.detach().unsqueeze(-1) + (per_token_logps - per_token_logps.detach())
+
+importance_weights = torch.exp(log_importance_weights)
+```
+
+Based on GRPO training, you can select different algorithms via the `--importance_sampling_level` argument:
+
+- `importance_sampling_level token` (default, GRPO implementation)
+- `importance_sampling_level sequence` (GSPO)
+- `importance_sampling_level sequence_token` (GSPO-token)
+
+
+Other hyperparameters in the paper
+```bash
+    --epsilon 3e-4 # from paper section 5.1
+    --epsilon_high 4e-4 # from paper section 5.1
+    --gradient_accumulation_steps 8
+    --steps_per_generation 32 # from paper section 5.1 (each batch of rollout data is partitioned into four minibatches for gradient updates)
+    --beta 0 # zero kl regularization https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306
+```
+
+For training, you can refer to [this script](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/gspo.sh).

docs/source_en/Instruction/GRPO/AdvancedResearch/REAL.md

@@ -0,0 +1,93 @@
+# Rewards as Labels: Revisiting RLVR from a Classification Perspective
+
+Author: [li2zhi](https://github.com/li2zhi)
+
+[Rewards as Labels: Revisiting RLVR from a Classification Perspective](https://arxiv.org/abs/2602.05630) proposes a reformulation of GRPO by treating rewards as labels and performing **in-group classification** instead of advantage estimation. This converts the policy optimization problem into a classification problem, thereby addressing two key issues in the GRPO loss:
+- **Gradient Misassignment** for positive samples
+- **Gradient Domination** for negative samples
+
+## Background and Motivation
+
+GRPO Objective
+
+$$
+J_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{q,o\sim\pi_{\mathrm{od}}(\cdot|q)}\left[\frac{1}{|o|}\sum_{t=1}^{|o|}\left(\min\left(\rho_tA_t,\mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t\right)\right)\right]
+$$
+
+where:
+- $\rho_t = \frac{\pi_\theta(o_t|q)}{\pi_{\mathrm{old}}(o_t|q)}$ is the probability ratio
+- $A_t$ is the advantage function
+
+The corresponding gradient is:
+
+$$
+\nabla_{\theta} J_{\mathrm{GRPO}} = \mathbb { E } \left[ \frac { 1 } { | o | } \sum _ { t = 1 } ^ { | o | } \mathbb { I } _ { \mathrm { clip } } \cdot A _ { t } e ^ { s _ { t } } \nabla _ { \theta } \log \pi _ { \theta } \left( o _ { t } | q \right) \right]
+$$
+
+where:
+- $s_t = \log \frac{\pi_\theta(o_t|q)}{\pi_{\mathrm{old}}(o_t|q)}$ is the relative log-probability
+- $\mathbb{I}_{\mathrm{clip}}$ is the clipping indicator
+
+Thus, the per-token gradient weight in GRPO is:
+
+$$
+|\mathcal{W}_{\mathrm{GRPO}}|=\left\{ \begin{array} {ll}\left|A\cdot e^s\right|, & \mathrm{if~}\mathbb{I}_{\mathrm{clip}}=1, \\ 0, & \text{otherwise.} \end{array}\right.
+$$
+
+![Gradient magnitude visualizations in GRPO](../../../../resources/real.png)
+
+1. **Gradient Misassignment (Positive Samples)**：
+For positive samples, as the relative log-probability $s$ decreases, the gradient magnitude also decreases.
+This is counterintuitive: tokens that the model is less confident about but correct should receive larger updates. However, GRPO assigns more weight to already confident tokens, causing under-trained tokens to receive insufficient learning signal.
+
+2. **Gradient Domination (Negative Samples)**：
+For negative samples, as $s$ decreases, the gradient magnitude increases exponentially.
+This leads to a situation where a few overconfident incorrect tokens dominate the gradient, overwhelming other negative signals within the same group. Due to the absence of an upper bound, this may result in unstable and excessively large parameter updates.
+
+To address the above issues, REAL treats rewards directly as labels and performs **group-wise classification training**.
+
+![Real Framework](../../../../resources/real_framework.png)
+
+The classification logit for each sample is defined as:
+
+$$
+\bar{s}^k=\frac{1}{|o^k|}\sum_{t=1}^{|o^k|}\left(\log\frac{\pi_\theta(o_t^k\mid q)}{\pi_{\mathrm{old}}(o_t^k\mid q)}\right)
+$$
+
+- $\bar{s}^k > 0$: The sample is more likely under the current policy than the old policy → the model tends to **promote** this sample
+- $\bar{s}^k < 0$: The sample is less likely under the current policy → the model tends to **suppress** this sample
+
+Loss Function
+
+$$
+\mathcal{L}_{REAL}=\log\left(1+\sum_{\mathcal{O}_+}e^{-\bar{s}^i/\tau}\right)+\log\left(1+\sum_{\mathcal{O}_-}e^{\bar{s}^j/\tau}\right)
+$$
+
+Gradient Properties
+
+$$
+|\mathcal{W}_{\mathrm{REAL}}|=
+\begin{cases}
+\frac{1}{\tau}\frac{1}{1+C_{+}e^{\bar{s}^{k}/\tau}}, & r=1 \\
+ \\
+\frac{1}{\tau}\frac{1}{1+C_{-}e^{-\bar{s}^{k}/\tau}}, & r=0 & & &
+\end{cases}
+$$
+
+## Parameter Settings
+
+| Parameter | Type | Default | Description                                                        |
+|-----------|------|---------|--------------------------------------------------------------------|
+| `--loss_type` | `str` | -       | Set to `real`                                                      |
+| `--real_tau` | `float` | `0.5`   | Temperature parameter controlling decision boundary sharpness |
+
+Training Script Reference
+
+[swift](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/real.sh)
+
+## Important Notes
+
+When configuring training parameters, ensure that:
+- `per_device_train_batch_size` is divisible by `num_generations`
+
+This guarantees that each training batch contains complete groups, which is required for correct in-group classification.

docs/source_en/Instruction/GRPO/AdvancedResearch/REINFORCEPP.md

@@ -0,0 +1,86 @@
+# REINFORCE++: An Efficient RLHF Algorithm with Robustness to Both Prompt and Reward Models
+
+[REINFORCE++ Baseline](https://arxiv.org/abs/2501.03262) is a simplified version of the REINFORCE++ algorithm, designed for outcome rewards (response-level scalar rewards). Similar to GRPO, it samples multiple model outputs for each prompt and uses an intra-group baseline to estimate advantages. The key difference lies in the statistics used for normalization.
+
+## Algorithm Overview
+For clarity, we explain REINFORCE++ Baseline by contrasting it with GRPO (Group Relative Policy Optimization).
+
+Both GRPO and REINFORCE++ Baseline estimate advantages via intra-group comparisons. Their main differences are:
+
+### Difference 1: Statistics Used for Normalization
+
+**GRPO (Group Relative Policy Optimization)**
+
+For each prompt, GRPO generates $G$ response samples and normalizes using the **mean and standard deviation of all samples within the group**:
+
+$$
+\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
+$$
+
+When `scale_rewards='batch'` is set, it uses the **batch-level std of original rewards**:
+
+$$
+\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^{N})}
+$$
+
+where $N$ is the total number of samples in the batch.
+
+**REINFORCE++ Baseline**
+
+For each prompt, REINFORCE++ generates $G$ response samples, subtracts the group mean, and then normalizes using the **standard deviation of the group-mean-subtracted rewards**:
+
+$$
+\begin{align}
+\tilde{A}_{i} &= R_i - \text{mean}(\{R_j\}_{j=1}^G) \\
+\hat{A}_{i} &= \frac{\tilde{A}_{i}}{\text{std}(\{\tilde{A}_k\}_{k=1}^{N})}
+\end{align}
+$$
+
+where $N$ is the total number of samples in the batch.
+
+**Key Difference**:
+- **GRPO**: Uses the std of **original rewards $R$** for normalization
+- **REINFORCE++**: Uses the std of **group-mean-subtracted rewards $\tilde{A}$** for normalization
+
+### Difference 2: KL Divergence Regularization
+
+Similar to RLOO, REINFORCE++ Baseline integrates KL divergence directly into the reward:
+
+$$
+R'_i = R_i - \beta \cdot \text{KL}(\pi_\theta || \pi_{\text{ref}})
+$$
+
+where $\beta$ is the KL divergence weight coefficient (corresponding to the parameter `beta`), and $\pi_{\text{ref}}$ is the reference policy.
+
+## Parameter Configuration
+
+We can implement REINFORCE++ Baseline training by configuring the following parameters with `GRPOTrainer`:
+
+```bash
+--advantage_estimator reinforce_plus_plus
+--scale_rewards batch
+--kl_in_reward true
+```
+
+For training examples, please refer to this [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/reinforce_plus_plus.sh)
+
+### Key Parameter Descriptions
+
+- **`--advantage_estimator`**: Selects the advantage estimation method
+  - `grpo` (default): Uses the std of original rewards for normalization
+  - `reinforce_plus_plus`: Uses the std of group-mean-subtracted rewards for normalization
+
+- **`--kl_in_reward`**: Controls where the KL divergence regularization term is applied
+  - `false`: KL divergence is an independent regularization term in the loss function (GRPO default)
+  - `true`: KL divergence is subtracted directly from the reward (REINFORCE++ original implementation)
+
+- **`--scale_rewards`**: Controls the normalization method
+  - `group` (default): Intra-group normalization
+  - `batch`: Global batch-level normalization (REINFORCE++ original implementation)
+  - `none`: No normalization
+
+- **`--num_generations`**: Number of samples generated per prompt ($G$)
+
+- **`--beta`**: KL divergence regularization coefficient ($\beta$)
+
+For other parameters, please refer to [GRPO Parameters](../../Command-line-parameters.md#grpo-arguments)

docs/source_en/Instruction/GRPO/AdvancedResearch/RLOO.md

@@ -0,0 +1,93 @@
+# REINFORCE Leave-One-Out (RLOO)
+
+[REINFORCE Leave-One-Out (RLOO)](https://arxiv.org/abs/2402.14740) is a reinforcement learning algorithm based on the classic REINFORCE policy-gradient method. It constructs an unbiased advantage baseline via the Leave-One-Out (LOO) technique.
+
+## Algorithm Overview
+
+For clarity, we explain RLOO by contrasting it with GRPO (Group Relative Policy Optimization).
+
+Both GRPO and RLOO estimate advantages via intra-group comparisons to avoid the high variance of a global baseline. Their core differences are mainly in the following aspects:
+
+### Difference 1: How the Advantage Baseline Is Constructed
+
+**1. GRPO (Group Relative Policy Optimization)**
+
+For each prompt, GRPO generates $G$ response samples and normalizes rewards using the group mean and standard deviation:
+
+$$
+\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
+$$
+
+Where:
+- $R_i$ is the reward of the $i$-th sample
+- $\text{mean}(\{R_j\}_{j=1}^G) = \frac{1}{G}\sum_{j=1}^G R_j$ is the group mean
+- $\text{std}(\{R_j\}_{j=1}^G)$ is the group standard deviation
+
+**2. RLOO (REINFORCE Leave-One-Out)**
+
+For each prompt, RLOO generates $K$ response samples and constructs the baseline via Leave-One-Out, i.e., for the $i$-th sample, the baseline is the mean of the other $K-1$ samples:
+
+$$
+\hat{A}_{i} = R_i - \frac{1}{K-1}\sum_{j \neq i} R_j
+$$
+
+This can be equivalently rewritten as:
+
+$$
+\hat{A}_{i} = \frac{K}{K-1} \left(R_i - \bar{R}\right)
+$$
+
+where $\bar{R} = \frac{1}{K}\sum_{j=1}^K R_j$ is the group mean reward.
+
+> Note: We use $K$ here to match the notation in the paper. It has the same meaning as $G$ in GRPO and corresponds to the configuration parameter `num_generations`.
+
+**Why Leave-One-Out?**
+
+The key advantage is unbiasedness. For the $i$-th sample, its reward $R_i$ is independent of the baseline $\frac{1}{K-1}\sum_{j \neq i} R_j$, hence the advantage estimate is unbiased. In contrast, using the mean including itself as the baseline introduces bias.
+
+### Difference 2: How KL Regularization Is Applied
+
+To prevent the policy from drifting too far from the reference policy, both algorithms introduce KL divergence regularization, but in different ways:
+
+**GRPO**: Adds KL divergence as an independent regularization term to the [loss](../GetStarted/GRPO.md#algorithm-overview):
+
+$$
+\mathcal{L}(\theta) = -\mathbb{E}\left[\hat{A}_i \log \pi_\theta(a_i|s_i)\right] + \beta \cdot \text{KL}(\pi_\theta \Vert \pi_{\text{ref}})
+$$
+
+**RLOO**: Integrates KL divergence directly into the reward, constructing a modified reward:
+
+$$
+R'_i = R_i - \beta \cdot \text{KL}(\pi_\theta \Vert \pi_{\text{ref}})
+$$
+
+where $\beta$ is the KL coefficient (parameter `beta`), and $\pi_{\text{ref}}$ is the reference policy (typically an SFT model or the initial policy).
+
+## Parameter Configuration
+
+RLOO training can be enabled based on `GRPOTrainer` by setting the following parameters:
+
+```bash
+# Basic RLOO configuration
+--advantage_estimator rloo  # Use RLOO's leave-one-out advantage estimator
+--kl_in_reward true         # Integrate KL divergence into the reward (default for RLOO)
+```
+
+You can refer to this [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/rloo.sh) for training.
+
+### Important Parameters
+
+- **`--advantage_estimator`**: Choose the advantage estimator
+  - `grpo` (default): standardize using group mean and standard deviation
+  - `rloo`: construct the baseline via Leave-One-Out
+
+- **`--kl_in_reward`**: Controls where the KL term is applied
+  - `false`: KL as a separate regularization term in the loss (GRPO style)
+  - `true`: subtract KL directly from the reward to form a modified reward (RLOO style)
+
+- **`--num_generations`**: Number of samples per prompt, i.e., $K$
+
+- **`--beta`**: KL regularization coefficient $\beta$
+  - Controls how conservatively the policy updates
+
+Other parameters are consistent with the [GRPO arguments](../../Command-line-parameters.md#grpo-arguments).

docs/source_en/Instruction/GRPO/AdvancedResearch/SAPO.md

@@ -0,0 +1,91 @@
+# Soft Adaptive Policy Optimization (SAPO)
+
+[Soft Adaptive Policy Optimization (SAPO)](https://arxiv.org/abs/2511.20347) addresses the issues caused by hard clipping in GRPO by proposing a temperature-controlled soft gate mechanism that smoothly attenuates off-policy updates while preserving useful learning signals.
+
+## Background and Motivation
+
+When training LLMs with reinforcement learning, GRPO handles off-policy training by computing token-level importance sampling ratios:
+
+$$
+r_t = \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|x, y_{<t})}
+$$
+
+However, token-level importance sampling ratios often exhibit high variance, which can be exacerbated in the following cases:
+- **Long text generation**
+- **MoE model routing heterogeneity**: The old-policy model during sampling and the training model may use different expert routing, significantly amplifying logps differences
+
+To address this, GRPO uses hard clipping to limit the magnitude of policy updates:
+
+$$
+L^{\mathrm{GRPO}} = -\min\left( r_t \cdot A, \mathrm{clip}(r_t, 1-\epsilon, 1+\epsilon) \cdot A \right)
+$$
+
+**The Dilemma of Hard Clipping**: Hard clipping struggles to balance stability and learning efficiency—too strict clipping limits the number of effective samples, while too loose clipping introduces noisy gradients from off-policy samples, leading to training instability.
+
+## SAPO Method
+
+SAPO uses a temperature-controlled sigmoid soft gate function to replace hard clipping, achieving smooth gradient attenuation.
+
+### Soft Gate Function
+
+The core of SAPO is using the sigmoid function to apply soft gating on the importance sampling ratio:
+
+For positive advantages ($A > 0$), use positive gating:
+
+$$
+g^{+}_t = \sigma\left( \tau_{\mathrm{pos}} \cdot (r_t - 1) \right) \cdot \frac{4}{\tau_{\mathrm{pos}}}
+$$
+
+For negative advantages ($A < 0$), use negative gating:
+
+$$
+g^{-}_t = \sigma\left( \tau_{\mathrm{neg}} \cdot (r_t - 1) \right) \cdot \frac{4}{\tau_{\mathrm{neg}}}
+$$
+
+where:
+- $\sigma(\cdot)$ is the sigmoid function
+- $\tau_{\mathrm{pos}}$ and $\tau_{\mathrm{neg}}$ are temperature parameters that control the gate function slope
+- $r_t$ is the importance sampling ratio
+
+### SAPO Loss Function
+
+$$
+L^{\mathrm{SAPO}} = -g_t \cdot A
+$$
+
+where $g_t = g^{+}_t$ when $A > 0$, $g_t = g^{-}_t$ when $A < 0$.
+
+### Temperature Parameters
+
+The temperature parameter $\tau$ controls the decay rate of the soft gate function—larger values result in faster decay.
+
+![tau curve](../../../../resources/sapo_tau.png)
+
+The paper points out that positive advantages increase the logit of sampled tokens while decreasing the logits of all unsampled tokens; negative advantages do the opposite, increasing the logits of many unsampled tokens, which may spread to a large number of irrelevant tokens and introduce instability. Therefore, the paper recommends setting $\tau_\text{neg} > \tau_\text{pos}$ to make the gradient decay faster for tokens with negative rewards, improving training stability and performance.
+
+The paper recommends default values of $\tau_{\mathrm{pos}} = 1.0$ and $\tau_{\mathrm{neg}} = 1.05$.
+
+## Parameter Settings
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `--loss_type` | `str` | - | Set to `sapo` |
+| `--tau_pos` | `float` | `1.0` | Temperature parameter for positive advantages, controls gate slope |
+| `--tau_neg` | `float` | `1.05` | Temperature parameter for negative advantages, controls gate slope |
+
+```bash
+swift rlhf \
+    --rlhf_type grpo \
+    --loss_type sapo \
+    --tau_pos 1.0 \
+    --tau_neg 1.05 \
+    # ... other parameters
+```
+
+Example training scripts:
+
+- [swift](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/sapo.sh)
+- [megatron swift](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/sapo.sh)
+
+> The soft gate mechanism of SAPO only takes effect during off-policy training.
+> The importance sampling granularity in SAPO is at the token level (i.e., importance_sampling_level defaults to token), which conflicts with GSPO.

docs/source_en/Instruction/GRPO/AdvancedResearch/deepeyes.md

@@ -0,0 +1,93 @@
+# DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
+
+## Principle Introduction
+
+The [DeepEyes paper](https://arxiv.org/abs/2505.14362) proposes a method that enables models to "think with images" (image-assisted reasoning) by leveraging reinforcement learning. This approach achieves emergent model capabilities through end-to-end reinforcement learning, without requiring additional SFT (Supervised Fine-Tuning) steps. The model is equipped with built-in image localization capabilities and can actively invoke an "image zoom-in tool": during inference, the model automatically selects specific regions within an image for zooming and cropping, and then chains the processed region information into downstream reasoning, thus achieving multi-step visual-text reasoning.
+
+![DeepEyes Overview](../../../../resources/deepeyes.png)
+
+## Best Practices
+
+**Dataset Download and Registration**
+
+Download the official DeepEyes training datasets locally:
+```bash
+# modelscope
+modelscope download --dataset Lixiang/ChenShawn-DeepEyes-Datasets-47k
+
+# huggingface
+huggingface-cli download ChenShawn/DeepEyes-Datasets-47k --repo-type=dataset
+```
+
+There are three parquet files in the dataset. Register them in the `swift/dataset/data/dataset_info.json` file. For each, rename the `prompt` column to `messages`:
+
+```json
+    {
+        "ms_dataset_id": "path/to/data_0.1.2_visual_toolbox_v2.parquet",
+        "columns": {
+            "prompt": "messages"
+        }
+    },
+    {
+        "ms_dataset_id": "path/to/data/data_thinklite_reasoning_acc.parquet",
+        "columns": {
+            "prompt": "messages"
+        }
+    },
+    {
+        "ms_dataset_id": "path/to/data/data_v0.8_visual_toolbox_v2.parquet",
+        "columns": {
+            "prompt": "messages"
+        }
+    }
+```
+
+For registering the reward function and tool call logic locally as used in the paper, you can refer to the [DeepEyes implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py).
+
+**Deploying the Evaluation Model**
+
+The reward function in DeepEyes relies on a generative reward model to compare model outputs with ground-truth answers. For efficiency, it is recommended to deploy the reward model as a service.
+
+Assuming you use the Qwen2.5-VL-72B-Instruct model for evaluation, refer to the following deployment command:
+
+```bash
+# 4*80G
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+swift deploy \
+    --model Qwen/Qwen2.5-VL-72B-Instruct \
+    --infer_backend vllm \
+    --vllm_tensor_parallel_size 4 \
+```
+
+In the plugin file, you can use the OpenAI interface to call the model; see the [Reward Model documentation](../DeveloperGuide/reward_model.md#external-deployment).
+
+The complete training script can be found at https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes.sh
+
+## Implementation Details
+
+The [DeepEyes implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py), which references the [official implementation](https://github.com/Visual-Agent/DeepEyes/blob/main/verl/utils/reward_score/vl_agent.py), provides sample code for a DeepEyes training plugin, covering the logic of the reward function and multi-turn interaction calls.
+
+**Dataset Info** is shown below:
+
+| Dataset File Name                        | data_source           | Scoring Function                  | Tool Call              |
+|------------------------------------------|-----------------------|-----------------------------------|------------------------|
+| data_v0.8_visual_toolbox_v2.parquet      | chart                 | vl_agent.compute_score            | True (image_zoom_in_tool)   |
+| data_0.1.2_visual_toolbox_v2.parquet     | vstar                 | vl_agent.compute_score            | True (image_zoom_in_tool)   |
+| data_thinklite_reasoning_acc.parquet     | thinklite_eureka      | vl_agent.compute_score_math       | False                  |
+
+**Note**: When processing image inputs, multimodal large models may perform preprocessing (such as cropping or resizing limited by the `max_pixels` parameter). When using the image zoom-in tool (`image_zoom_in_tool`), the model will output the cropped bbox based on the input image. Therefore, it is necessary to ensure the image fed into the tool has already been preprocessed. The following example shows how this is implemented in the Qwen2.5-VL series:
+
+```python
+from qwen_vl_utils import fetch_image
+# At this point, images have not yet been preprocessed
+infer_request.images
+# Load as PIL.Image and apply cropping (same as using the MAX_PIXELS environment variable)
+img = fetch_image({'image': load_pil_image(infer_request.images[0])})
+```
+
+**Tool Reward**
+
+According to the paper, if the final answer is correct and the trajectory uses at least one tool, a tool reward is given. To prevent the model from generating invalid tool calls, we determine this based on the number of images, not just on the presence of `<tool_call>` tokens.
+```python
+tool_reward = 1.0 if num_image > 1 and acc_reward > 0.5 else 0.0
+```

docs/source_en/Instruction/GRPO/AdvancedResearch/entropy_mask.md

@@ -0,0 +1,28 @@
+# Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
+
+The [paper](https://arxiv.org/abs/2506.01939) finds that when training large language models for reasoning abilities with methods such as RLVR, the key to learning progress lies in a small fraction of high-entropy "minority tokens," rather than the majority of low-entropy tokens.
+
+The paper demonstrates that within the token distribution during model reasoning, only a few high-entropy tokens play a dominant role. These tokens typically appear at critical junctures where the reasoning or decision path diverges the most (e.g., tokens like "wait," "since," etc.), determining whether the model can master complex reasoning tasks. In contrast, most low-entropy tokens contribute little to the model's reasoning ability. The paper proposes computing policy gradients exclusively on high-entropy tokens, discarding gradients for low-entropy tokens.
+
+The formula for token entropy is as follows:
+
+$
+H_t := -\sum_{j=1}^{V} p_{t,j} \log p_{t,j}, \qquad \text{where } (p_{t,1}, \cdots, p_{t,V}) = \mathbf{p}_t = \pi_\theta(\cdot | \mathbf{q}, \mathbf{o}_{<t}) = \text{Softmax}\left(\frac{\mathbf{z}_t}{T}\right)
+$
+
+Where:
+- $\pi_\theta$: The model parameterized by $\theta$;
+- $\mathbf{q}$: The input query;
+- $\mathbf{o}_{<t} = (o_1, o_2, \cdots, o_{t-1})$: The sequence of tokens generated prior to timestep $t$;
+- $V$: Vocabulary size;
+- $\mathbf{z}_t \in \mathbb{R}^V$: The pre-softmax logits at timestep $t$;
+- $\mathbf{p}_t \in \mathbb{R}^V$: The model's output probability distribution over the vocabulary;
+- $T \in \mathbb{R}$: The decoding temperature, controlling the smoothness of the distribution.
+
+Object of entropy computation: $H_t$ is the entropy of the token generation distribution $\mathbf{p}_t$, which measures the uncertainty in the policy $\pi_\theta$ under the given context $(\mathbf{q}, \mathbf{o}_{<t})$.
+
+> "Token entropy" $H_t$ always refers to the uncertainty of the generation distribution $\mathbf{p}_t$ at position $t$, rather than a property of the token $o_t$ itself. In other words, $H_t$ is the entropy of the distribution $\mathbf{p}_t$ at position $t$, and is independent of the sampled token $o_t$.
+
+In practice, during GRPO training, the top_entropy_quantile parameter can be used to control the percentile threshold for entropy filtering. In the experiments from the paper, this parameter is set to 0.2, meaning that only the top 20% of tokens (with the highest entropy) at each sequence position are used for optimization in each batch.
+
+By setting the parameter `log_entropy`, you can record the changes in entropy during training; see the [documentation](../GetStarted/GRPO.md#logged-metrics) for reference.

docs/source_en/Instruction/GRPO/AdvancedResearch/index.rst

@@ -0,0 +1,19 @@
+Advanced Research
+===============
+.. toctree::
+   :maxdepth: 1
+
+   entropy_mask.md
+   CISPO.md
+   DAPO.md
+   deepeyes.md
+   FIPO.md
+   GSPO.md
+   CHORD.md
+   RLOO.md
+   REINFORCEPP.md
+   REAL.md
+   router_replay.md
+   SAPO.md
+   training_inference_mismatch.md
+   treepo.md

docs/source_en/Instruction/GRPO/AdvancedResearch/router_replay.md

@@ -0,0 +1,104 @@
+# Router Replay (R2/R3)
+
+**TL;DR**: In RL training of MoE models, routing inconsistency between the training engine and the inference engine can significantly amplify training-inference mismatch, and even cause training collapse. Router Replay eliminates this inconsistency by replaying fixed routing masks during the training forward pass. Depending on the replay source, there are two strategies: R2 (Vanilla Routing Replay) and R3 (Rollout Routing Replay).
+
+## Background
+
+### Three Policies in MoE RL
+
+In GRPO training of MoE models, there are three distinct policy stages that share the same model weights but may differ in routing behavior:
+
+| Policy | Notation | Routing Result | Description |
+|--------|----------|---------------|-------------|
+| **Training Policy** | $\pi_\theta$ | $e^{\pi}_t$ | The model during gradient updates |
+| **Old Policy** | $\pi_{\theta_{\text{old}}}$ | $e^{\pi}_{\text{old},t}$ | The model state before batch updates |
+| **Rollout Policy** | $\mu_{\theta_{\text{old}}}$ | $e^{\mu}_{\text{old},t}$ | The sampling policy in the inference engine (e.g., vLLM), with the same weights as old policy, but different routing due to kernel implementation differences, precision, etc. |
+
+Here, $\pi_{\theta_{\text{old}}}$ and $\mu_{\theta_{\text{old}}}$ have identical weights at sampling time, but due to implementation differences between the inference and training engines (e.g., operator implementations), routing results may differ even for the same input.
+
+### Decomposition of Training-Inference Mismatch
+
+According to the [paper](https://arxiv.org/abs/2507.18071), the token-level importance sampling ratio can be decomposed into two factors:
+
+$$
+\frac{\pi_\theta(y_t|x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t|x, y_{<t})} = \underbrace{\frac{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t|x, y_{<t})}}_{\text{training-inference discrepancy}} \times \underbrace{\frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}}_{\text{policy staleness}}
+$$
+
+For MoE models, expert routing is deeply coupled with both factors:
+
+- **Training-inference discrepancy**: Routing inconsistency between the training and inference engines ($e^{\pi}_{\text{old},t} \neq e^{\mu}_{\text{old},t}$) amplifies output divergence
+- **Policy staleness**: As mini-batch updates proceed, routing also drifts ($e^{\pi}_t \neq e^{\pi}_{\text{old},t}$), further deviating from the sampling policy
+
+## R2: Vanilla Routing Replay
+
+The core idea of R2 is to **replay the routing determined by the old policy in the training engine** ($e^{\pi}_{\text{old},t}$) during gradient updates.
+
+### Principle
+
+During the training forward pass, first run a forward pass with the old policy weights to record the expert indices selected by each MoE Router layer, then force the training model $\pi_\theta$ to use these indices in its forward pass:
+
+$$
+g_{\text{replay},i} = \frac{I^{\pi}_{\text{old},i} \cdot \exp(s_{\text{train},i})}{\sum_j I^{\pi}_{\text{old},j} \cdot \exp(s_{\text{train},j})}
+$$
+
+where $I^{\pi}_{\text{old}}$ is the old policy's routing mask and $s_{\text{train}}$ is the router logits computed by the training model. The softmax still operates on the training logits, so gradients can flow back to the router weights normally.
+
+### Properties
+
+| Scenario | Behavior |
+|----------|----------|
+| **First mini-batch** (on-policy) | $\theta = \theta_{\text{old}}$, so $e^{\pi}_t = e^{\pi}_{\text{old},t}$, **target policy unchanged** (no bias) |
+| **Subsequent mini-batches** (off-policy) | $\theta \neq \theta_{\text{old}}$, so $e^{\pi}_t \neq e^{\pi}_{\text{old},t}$, **target policy changed** (biased), but policy staleness is controlled |
+
+## R3: Rollout Routing Replay
+
+The core idea of R3 is to **replay the routing determined by the rollout policy in the inference engine** ($e^{\mu}_{\text{old},t}$) during the training forward pass.
+
+### Principle
+
+During sampling in the inference engine (e.g., vLLM), additionally record the expert indices (routing mask) for each token at every MoE Router layer, then pass these masks to the training engine and force $\pi_\theta$ to use them in its forward pass:
+
+$$
+g_{\text{replay},i} = \frac{I^{\mu}_{\text{old},i} \cdot \exp(s_{\text{train},i})}{\sum_j I^{\mu}_{\text{old},j} \cdot \exp(s_{\text{train},j})}
+$$
+
+### Compatibility with Other Methods
+
+- R3 is **orthogonal** to **GSPO** and can be combined for further improvement
+- R3 combined with **TIS** may not provide additional gains (R3 already eliminates inconsistency at the source; TIS's additional correction may be redundant)
+- Router Replay and **Clipping** are both essential in off-policy training
+
+## Router Mask Caching
+
+The R3 paper also proposes that routing masks can be cached alongside the KV Cache: for the same prefix tokens, the MoE Router output is deterministic, so routing masks can be stored and reused together with the prefix KVCache. This is particularly important in multi-turn Agent scenarios (tool calling), avoiding the need to re-prefill the prefix to obtain routing masks, with an overall rollout latency overhead of less than 3%.
+
+## Swift Implementation
+
+### Parameters
+
+Select the routing replay strategy via the `--router_replay_mode` parameter:
+
+| Value | Description |
+|-------|-------------|
+| `disabled` (default) | No routing replay |
+| `R2` | Vanilla Routing Replay: record old policy routing in the training engine and replay |
+| `R3` | Rollout Routing Replay: export routing masks from the inference engine and replay in training |
+
+Environment requirements:
+
+- R3 requires vLLM ≥ 0.14.0 to support returning `routed_experts` information.
+- Router Replay is currently only available with the Megatron backend, requiring megatron-core ≥ 0.16.0.
+
+### Relationship with Training-Inference Correction
+
+Router Replay and the importance sampling (IS) correction described in [Training-Inference Mismatch](training_inference_mismatch.md) are complementary:
+
+- **IS correction**: Corrects probability divergence at the loss level via weighting
+- **Router Replay**: Eliminates the source of divergence at the model architecture level by fixing routing
+
+## References
+
+1. [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://arxiv.org/abs/2510.11370)
+2. [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071)
+3. [Stabilizing Reinforcement Learning with LLMs: Formulation and Practices](https://arxiv.org/abs/2512.01374)
+4. [Megatron Core Router Replay Design Document](https://docs.nvidia.com/megatron-core/developer-guide/nightly/api-guide/router_replay.html)

docs/source_en/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md

@@ -0,0 +1,236 @@
+# Training-Inference-Mismatch
+
+**TL;DR**: While GRPO introduces vLLM to accelerate the sampling process, it also introduces Training-Inference Mismatch issues that may affect training stability. This document explains the background, causes, and solutions to this problem.
+
+## Background
+
+### Basic Assumptions of GRPO
+
+The training objective of GRPO (Group Relative Policy Optimization) can be expressed as:
+
+$$
+\mathcal{L}_{\text{GRPO}} = - \mathbb{E}_{y \sim \pi_\theta} \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
+$$
+
+Where:
+- $r_t(\theta) = \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}$ is the importance sampling ratio
+- $\hat{A}_t$ is the advantage function, calculated based on reward and group baseline
+- $\epsilon$ is the clipping parameter
+
+**Core Assumption**: Samples $y$ are drawn from the policy $\pi_\theta$. In practice, this means:
+1. The rollout model and the training model (policy model) should be **the same model** $\pi_\theta$
+2. The probability distributions of both models should be **exactly identical**, i.e., $\pi_{\text{rollout}} = \pi_\theta$
+
+### Deviation After Introducing vLLM
+
+GRPO's training speed is largely constrained by the sampling process (rollout). To accelerate this, training frameworks introduce high-performance inference engines (such as vLLM) for sampling. The **ideal assumption** is that through weight synchronization, vLLM maintains consistency with the training model, i.e., $\pi_{\text{vLLM}} \equiv \pi_\theta$.
+
+However, in practice, even with fully synchronized weights, due to differences in operator implementations, the probability distributions still deviate:
+
+$$
+\pi_{\text{vLLM}}(y|x) \neq \pi_\theta(y|x)
+$$
+
+At this point, the actual training objective becomes:
+
+$$
+\mathcal{L} = - \mathbb{E}_{y \sim \pi_{\text{vLLM}}} \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
+$$
+
+Where samples come from $\pi_{\text{vLLM}}$, but gradients are computed based on $\pi_\theta$. This **violates the algorithm's on-policy assumption**, introducing training-inference mismatch issues and potentially causing performance degradation.
+
+## Solution
+
+To address training-inference mismatch, we can introduce **Importance Sampling (IS)** correction mechanisms.
+
+### Importance Sampling Correction
+
+The basic idea of importance sampling is: when samples come from distribution $q$ rather than target distribution $p$, we can correct the expectation calculation by introducing weights:
+
+$$
+\mathbb{E}_{x \sim p} [f(x)] = \mathbb{E}_{x \sim q} \left[ \frac{p(x)}{q(x)} \cdot f(x) \right]
+$$
+
+Applied to the GRPO scenario, the corrected loss function is:
+
+$$
+\mathcal{L}_{\text{corrected}} = - \mathbb{E}_{y \sim \pi_{\text{vLLM}}} \left[ w(x, y) \cdot \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
+$$
+
+Where $w(x, y)$ is the importance sampling weight used to correct the distribution bias between vLLM and the training model.
+
+Importance sampling weights can be computed and applied at different granularities:
+
+1. **Token-Level**
+
+Compute the importance sampling ratio at each token:
+
+$$
+w_{i,t}^{\text{token}} = \frac{\pi_\theta(y_{i,t}|x, y_{i,<t})}{\pi_{\text{vLLM}}(y_{i,t}|x, y_{i,<t})}
+$$
+
+2. **Sequence-Level**
+
+Compute the sequence-level importance sampling ratio, then broadcast to each token:
+
+$$
+w_i^{\text{seq}} = \left[ \frac{\pi_\theta(y_i|x)}{\pi_{\text{vLLM}}(y_i|x)} \right]^{\frac{1}{|y_i|}} = \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t}|x, y_{i,<t})}{\pi_{\text{vLLM}}(y_{i,t}|x, y_{i,<t})} \right)
+$$
+
+### Stability Control: Truncate vs. Mask
+
+Excessively large importance sampling weights can cause gradient explosion and destabilize training. Therefore, weight control is necessary:
+
+#### 1. Truncate
+
+Truncate the importance sampling weight to the $[0, \tau]$ interval:
+
+$$
+w_{\text{truncate}} = \min(w, \tau)
+$$
+
+This method retains all samples but limits their influence.
+
+#### 2. Mask
+
+Discard token/sequence data where weights exceed the threshold:
+
+$$
+w_{\text{mask}} = \begin{cases}
+w & \text{if } w \leq \tau \\
+0 & \text{otherwise}
+\end{cases}
+$$
+
+
+### Four Correction Modes
+
+Combining granularity and control strategies, there are four correction modes (selected via `--rollout_importance_sampling_mode` parameter):
+
+| Mode | Description |
+|------|-------------|
+| `token_truncate` | Token-level truncation |
+| `token_mask` | Token-level masking |
+| `sequence_truncate` | Sequence-level truncation |
+| `sequence_mask` | Sequence-level masking |
+
+The threshold is set via the `--rollout_importance_sampling_threshold` parameter.
+
+## Metrics
+
+To monitor the degree of training-inference mismatch during training, we add the following metrics to the logs (prefixed with `rollout_correction/`):
+
+### 1. KL Divergence
+
+KL divergence measures the deviation between the rollout policy and the training policy. Both metrics estimate $\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta)$
+
+**Direct estimator `kl`**:
+
+$$
+\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta) = \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \log \frac{\pi_{\text{vLLM}}}{\pi_\theta} \right]
+$$
+
+**K3 estimator `k3_kl`**:
+
+$$
+\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta) \approx \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \rho - \log \rho - 1 \right], \quad \rho = \frac{\pi_\theta}{\pi_{\text{vLLM}}}
+$$
+
+The K3 estimator is more numerically stable when KL values are small and is always non-negative.
+
+### 2. Perplexity (PPL)
+
+Perplexity measures the model's prediction uncertainty for a sequence:
+
+$$
+\text{PPL} = \exp\left( -\frac{1}{|y|} \sum_{t=1}^{|y|} \log p(y_t) \right)
+$$
+
+Related metrics:
+- `training_ppl` / `training_log_ppl`: Training policy PPL and its logarithm
+- `rollout_ppl` / `rollout_log_ppl`: Rollout policy PPL and its logarithm
+- `log_ppl_diff`: Log PPL difference, positive value means training policy assigns lower probability
+- `log_ppl_abs_diff`: Absolute log PPL difference
+- `log_ppl_diff_max` / `log_ppl_diff_min`: Max/min of log PPL difference
+- `ppl_ratio`: PPL ratio $\frac{\text{PPL}_{\text{training}}}{\text{PPL}_{\text{rollout}}}$
+
+### 3. χ² Divergence (Chi-squared Divergence)
+
+χ² divergence measures the variance of importance sampling weights:
+
+$$
+\chi^2(\pi_\theta \| \pi_{\text{vLLM}}) = \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \rho^2 \right] - 1, \quad \rho = \frac{\pi_\theta}{\pi_{\text{vLLM}}}
+$$
+
+- `chi2_token`: Token-level χ² divergence, $\mathbb{E}[\rho_t^2] - 1$
+- `chi2_seq`: Sequence-level χ² divergence (geometric mean based), $\mathbb{E}[\rho_{\text{geo}}^2] - 1$, where $\rho_{\text{geo}} = \exp(\frac{1}{T}\sum_t \log \rho_t)$
+
+Higher χ² divergence indicates larger IS weight variance and less stable training. `chi2_seq` uses geometric mean instead of product, making it comparable in scale to `chi2_token`.
+
+### 4. Effective Sample Size (ESS)
+
+Effective sample size measures the number of samples that actually contribute after importance sampling:
+
+$$
+\text{ESS} = \frac{1}{\mathbb{E}\left[\left(\frac{w}{\mathbb{E}[w]}\right)^2\right]}
+$$
+
+A larger ESS value (closer to 1) indicates more uniform importance sampling weight distribution and higher sample utilization efficiency. When all weights are equal (on-policy), ESS = 1; when weights differ significantly (severely off-policy), ESS becomes small.
+
+### 5. IS Weight Statistics
+
+- `is_weight_mean`: Average importance sampling weight, ideal value is 1.0
+- `clipped_frac`: Fraction of samples that were truncated or masked
+
+
+## Usage
+
+### Logging Diagnostic Metrics Only
+
+If you only want to monitor the degree of training-inference mismatch without enabling importance sampling correction, you can set:
+
+```
+--log_rollout_offpolicy_metrics true
+```
+
+This will log all diagnostic metrics (KL, PPL, χ², etc.) without modifying the loss function.
+
+### Enabling Importance Sampling Correction
+
+Enable the correction mechanism with the following parameters:
+
+```
+--rollout_importance_sampling_mode (default None)
+--rollout_importance_sampling_threshold (default 2)
+```
+
+When `rollout_importance_sampling_mode` is set, diagnostic metrics are automatically logged without needing to set `log_rollout_offpolicy_metrics`.
+
+## Off-Policy Sequence Masking
+
+In addition to importance sampling correction, you can use **Off-Policy Sequence Masking** to address training-inference mismatch. This technique comes from the [DeepSeek-V3.2 paper](https://arxiv.org/abs/2512.02556).
+
+### Principle
+
+The core idea of Off-Policy Sequence Masking is: when the current policy deviates significantly from the old policy (rollout or old policy), directly discard (mask) that sequence from loss computation. This approach specifically targets **sequences with negative advantage**, as these are more likely to cause training instability when policy shift is large.
+
+Specifically, for each sequence, compute:
+
+$$
+\delta_i = \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \bigl( \log \pi_{\text{old}}(y_{i,t}|x, y_{i,<t}) - \log \pi_\theta(y_{i,t}|x, y_{i,<t}) \bigr)
+$$
+
+Sequence $i$ will be masked when the following conditions are met (mean taken over completion tokens, i.e., where `completion_mask=1`):
+1. $\delta_i > \tau$
+2. **AND** $\hat{A}_i < 0$
+
+Where:
+- $\pi_{\text{old}}$ preferentially uses `rollout_per_token_logps` (logprobs from rollout/behavior policy); if unavailable, falls back to `old_per_token_logps`
+- $\tau$ is the user-set threshold (`--off_policy_sequence_mask_delta`, default None = disabled)
+
+## References
+
+1. https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda
+2. https://fengyao.notion.site/off-policy-rl
+3. https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/rollout_corr_helper.py
+4. [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://arxiv.org/abs/2512.02556)

docs/source_en/Instruction/GRPO/AdvancedResearch/treepo.md

@@ -0,0 +1,35 @@
+# TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
+
+Author: [li2zhi](https://github.com/li2zhi)
+
+## Principle Introduction
+[TreePO paper](https://arxiv.org/abs/2508.17445) proposes a tree-structured modeling method. This method organizes sequence generation into a segmented tree structure search. Through dynamic branching, backtracking, and early termination mechanisms, it significantly improves the reuse rate of the key-value cache, thereby reducing computational overhead, while maintaining or even enhancing the diversity of exploration.
+
+![TreePO Overview](../../../../resources/treepo.png)
+
+## Implementation Details
+[TreePO implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/treepo/tree_rollout_plugin.py), which references the [official implementation](https://github.com/multimodal-art-projection/TreePO/blob/main/recipe/treepo/vllm_rollout_tree.py) provides sample code for a TreePO training plugin，covering logic related to multi-round interactions, termination judgment, and branch rollback.
+
+**Note:** In actual use, you need to rewrite the logic of methods such as step and check_finished according to your own scenario requirements to ensure that they can execute as expected in the custom scenario.
+For information on the design and use of custom rewards, you can refer to the implementation of [DeepEyes](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py).
+
+The complete training script can be found at [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/treepo/tree_rollout.sh).
+
+## Test Data
+> model: Qwen/Qwen2.5-0.5B
+> dataset: AI-MO/NuminaMath-TIR
+> subset size: 1,000 samples
+> 1 GPU for training, 1 GPU for inference
+
+| \                       | batch_size | num_generation | max_tree_depth | global_step | total inference calls | saving ratio | train_speed(iter/s) | improvement rate |
+| ----------------------- | ---------- | -------------- | -------------- | ----------- | --------------------- | ------------ | ------------------- | ---------------- |
+| original implementation | 8          | 8              | 4              | 200         | 5965                  | 0.00%        | 0.292436            | 0.00%            |
+| tree(max_divergence=3)  | 8          | 8              | 4              | 200         | 3678                  | 38.34%       | 0.31819             | 8.81%            |
+|                         |            |                |                |             |                       |              |                     |                  |
+| original implementation | 8          | 8              | 5              | 105         | 4312                  | 0.00%        | 0.261324            | 0.00%            |
+| tree(max_divergence=2)  | 8          | 8              | 5              | 105         | 2513                  | 52.69%       | 0.336639            | 28.82%           |
+| tree(max_divergence=3)  | 8          | 8              | 5              | 105         | 2990                  | 30.66%       | 0.308791            | 18.16%           |
+|                         |            |                |                |             |                       |              |                     |                  |
+| original implementation | 8          | 8              | 6              | 105         | 5202                  | 0.00%        | 0.24832             | 0.00%            |
+| tree(max_divergence=2)  | 8          | 8              | 6              | 105         | 3348                  | 35.64%       | 0.27755             | 11.77%           |
+| tree(max_divergence=3)  | 8          | 8              | 6              | 105         | 3888                  | 25.26%       | 0.272339            | 9.67%            |

docs/source_en/Instruction/GRPO/DeveloperGuide/gym_env.md

@@ -0,0 +1,361 @@
+# GYM Environment Training
+
+GYM-style environment training wraps the "model → environment → reward" chain behind an abstract interface, letting the LLM interact with the environment as an Agent over multiple turns. The reward of each step is produced directly by the environment, so you don't need a separate reward function to infer it from the trajectory. This document first introduces the interface, then walks through a complete custom example (FrozenLake) showing how to plug it into training.
+
+## Gym interface
+
+GYM originates from the [Gymnasium library](https://github.com/Farama-Foundation/Gymnasium). In ms-swift we define the following interface:
+
+```python
+class Env(ABC):
+
+    def __init__(self, env_config):
+        """env_config comes from the env_config column of each dataset row and carries initialization arguments."""
+        self.env_config = env_config
+
+    @abstractmethod
+    async def reset(self, config: RolloutInferRequest) -> Tuple[str, Dict[str, Any], str]:
+        """
+        Returns:
+            - observation: sent to the model as the first user message
+            - info: debug/log information, recorded in completions.jsonl
+            - system_message: system prompt for this trajectory
+        """
+        pass
+
+    @abstractmethod
+    async def step(self, action: Messages) -> Tuple[str, float, bool, Dict[str, Any]]:
+        """
+        Args:
+            action: the complete conversation messages so far; the last one is the model's latest reply
+        Returns:
+            - next_observation: next user message
+            - reward: reward for the current step
+            - done: whether the trajectory is finished
+            - info: debug/log information
+        """
+        pass
+
+    @abstractmethod
+    async def close(self):
+        """Release resources."""
+        pass
+```
+
+The `RolloutInferRequest` received by `reset` contains the dataset row's `messages`, `data_dict` (extra columns including `env_config`), etc. See the [input example](./multi_turn.md#multiturnscheduler) for the full structure.
+
+> If you need extra control over the conversation history between turns (e.g. dynamic compression, injecting hints), subclass `MultiTurnScheduler` and implement `on_trajectory_start` / `on_turn_end` hooks, or override `step` / `run` — see the [multi-turn doc](./multi_turn.md#customising-the-interaction-logic).
+
+## Launching training
+
+Use the built-in [gym_scheduler](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py) to wire the env into multi-turn rollout.
+
+`GYMScheduler` is based on the generic hook protocol:
+- Inherits `MultiTurnScheduler` — no need to override the `run` method
+- Implements `on_trajectory_start` (calls `env.reset`) and `on_turn_end` (calls `env.step`)
+- Works with both server mode (`run()`) and colocate mode (`run_multi_turn()`)
+
+User-defined envs are loaded via `--external_plugins your_plugin.py`; the plugin runs `envs['my_env'] = MyEnv` to register them (the FrozenLake example below demonstrates the full pattern).
+
+The built-in `GYMScheduler` completes the control logic via hooks:
+
+```python
+class GYMScheduler(MultiTurnScheduler):
+    def on_trajectory_start(self, requests):
+        # Create an env for each request, call env.reset, inject initial observation
+        for req in requests:
+            env = self._create_env(req.data_dict.get('env_config', {}))
+            observation, info, system_message = env.reset(req)
+            req.messages = [system_msg, user_msg(observation)]
+            self._envs[req.uuid] = env
+
+    def on_turn_end(self, req, response_choice, current_turn):
+        # Call env.step, accumulate reward, return done + rollout_infos
+        next_obs, reward, done, info = env.step(deepcopy(req.messages))
+        self._total_rewards[req.uuid] += reward
+        return {
+            'done': done,
+            'rollout_infos': {
+                'total_reward': self._total_rewards[req.uuid],
+                'step_rewards': [...],
+            }
+        }
+
+    def step(self, req, response_choice, current_turn):
+        # Inject the next observation into a user message
+        if self._pending_obs.get(req.uuid):
+            req.messages.append({'role': 'user', 'content': next_obs})
+        return {'infer_request': req}
+```
+
+**Colocate mode**:
+
+```bash
+megatron rlhf \
+    --rlhf_type grpo \
+    --vllm_mode colocate \
+    --external_plugins examples/megatron/grpo/multi_turn/frozen_lake_plugin.py \
+    --multi_turn_scheduler gym_scheduler \
+    --gym_env frozen_lake \
+    --use_gym_env true \
+    --max_turns 10 \
+    ...
+
+# swift rlhf works the same way
+```
+
+
+**Server mode**
+
+```bash
+swift rollout \
+    --model xxx \
+    --use_gym_env true \
+    --external_plugins examples/megatron/grpo/multi_turn/frozen_lake_plugin.py \
+    --multi_turn_scheduler gym_scheduler \
+    --gym_env frozen_lake \
+    --max_turns 10
+
+# On the trainer side, add --vllm_server_pass_dataset true so the env_config column reaches the rollout server.
+megatron rlhf --vllm_mode server --vllm_server_pass_dataset true ...
+# or swift rlhf --vllm_mode server --vllm_server_pass_dataset true ...
+```
+
+Two ways to select the environment:
+- Set it globally via `--gym_env env_name` (recommended — one env for the whole script);
+- Or specify it per dataset row via `env_config.name` (for mixed-env workloads; overrides `--gym_env`).
+
+## Example: writing a FrozenLake environment from scratch
+
+<img src="https://gymnasium.farama.org/_images/frozen_lake.gif" width="220" alt="FrozenLake environment (image from Gymnasium docs)" />
+
+[FrozenLake](https://gymnasium.farama.org/environments/toy_text/frozen_lake/) is a classic task from OpenAI Gym: the agent starts at the start cell, must cross a frozen lake to reach the goal, and avoid holes along the way. The original environment is illustrated above. The walkthrough below uses a text-only version of it (the same grid rendered as ASCII).
+
+Full source: [frozen_lake_plugin](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/multi_turn/frozen_lake_plugin.py).
+
+**1. Define the Env**
+
+Each dataset row produces a freshly generated random 4x4 map (random holes + random S/G positions, BFS-validated to be solvable). Cell meanings: `S` start / `G` goal / `H` hole (stepping in = fail) / `F` safe ice / `P` player's current position.
+
+```python
+class FrozenLakeEnv(Env):
+    def __init__(self, env_config):
+        super().__init__(env_config)
+        self.size = int(env_config.get('size', 4))
+        self.p = float(env_config.get('p', 0.8))
+        seed = env_config.get('seed')
+        self.seed = int(seed) if seed is not None else None
+
+    async def reset(self, config: RolloutInferRequest):
+        self.grid = generate_random_map(size=self.size, p=self.p, seed=self.seed)
+        ...
+        return observation, {'seed': self.seed}, SYSTEM_PROMPT
+
+    async def step(self, action: Messages):
+        move = _parse_action(action[-1]['content'])  # <action>up|down|left|right</action>
+        # Advance one cell, check G / H; the outer max_turns is enforced by the scheduler.
+        if cell == 'G': return obs, 1.0, True, {'status': 'goal'}
+        if cell == 'H': return obs, 0.0, True, {'status': 'hole'}
+        ...
+```
+
+**2. Register**
+
+Hook the env class into swift's `envs` registry. `--external_plugins` imports the file at startup, so the registration takes effect automatically:
+
+```python
+# examples/megatron/grpo/multi_turn/frozen_lake_plugin.py
+from swift.rollout.gym_env import Env, envs
+
+class FrozenLakeEnv(Env):
+    ...
+
+envs['frozen_lake'] = FrozenLakeEnv
+```
+
+**3. Prepare the dataset**
+
+The dataset is just a placeholder here — the actual data is constructed by the env, with `env_config.seed` controlling map-generation randomness:
+
+```json
+{"messages":[{"role":"user","content":"<placeholder>"}],"env_config":{"seed":0}}
+{"messages":[{"role":"user","content":"<placeholder>"}],"env_config":{"seed":1}}
+...
+{"messages":[{"role":"user","content":"<placeholder>"}],"env_config":{"seed":127}}
+```
+
+**4. (Optional) Blend in extra rewards**
+
+With `--use_gym_env true`, the env-provided `total_reward` is automatically added as one reward column — no reward function is required. To mix in additional signals (e.g. format/length checks), just pass them via `--reward_funcs`; the gym reward is appended as an extra column and blended with the reward_funcs through `--reward_weights`. For example, also enabling a format reward:
+
+```bash
+megatron rlhf ... --use_gym_env true --reward_funcs format --reward_weights 0.2 1.0
+# the last entry of reward_weights corresponds to the gym total_reward
+```
+
+**5. Train**
+
+Runnable script: [`examples/megatron/grpo/multi_turn/frozen_lake.sh`](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/multi_turn/frozen_lake.sh)
+
+During training, observe `rollout_infos.num_turns` (steps per trajectory) and the reward mean in the logs. `--log_completions true` writes full conversations to `completions.jsonl`, so you can verify the model outputs in the `<action>...</action>` format turn by turn.
+
+References:
+
+- https://gymnasium.farama.org/environments/toy_text/frozen_lake/
+- https://github.com/alibaba/ROLL/tree/main/roll/pipeline/agentic/env/frozen_lake
+- [OpenEnv](https://github.com/huggingface/openenv)
+- [TRL Sudoku GRPO Example](https://github.com/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb)
+
+## OpenEnv Environment Training
+
+[OpenEnv](https://github.com/huggingface/openenv) is an open-source Agentic RL environment framework by HuggingFace that communicates with environment servers via WebSocket. Unlike the local `Env` interface used by FrozenLake above, OpenEnv places environment logic in a separate server process, and swift communicates with it through `OpenEnvScheduler` + `OpenEnvWrapper`.
+
+### Architecture Comparison
+
+| Feature | Built-in Gym (`GYMScheduler`) | OpenEnv (`OpenEnvScheduler`) |
+|---------|------------------------------|------------------------------|
+| Environment location | In-process (Python object) | Standalone server (WebSocket) |
+| Environment interface | Subclass `Env`, implement `reset/step/close` | Server provides HTTP/WebSocket API |
+| Registration | `--external_plugins` + `envs` registry | `--external_plugins` + `multi_turns` registry |
+| Use case | Lightweight local envs (FrozenLake, etc.) | Complex server envs (TextArena, CARLA, etc.) |
+| Concurrency control | Not needed | Built-in Semaphore for connection limiting |
+
+### OpenEnvScheduler
+
+`OpenEnvScheduler` extends `GYMScheduler`, replacing the local `Env` with `OpenEnvWrapper` (a WebSocket client). Key design:
+
+- **`_create_env`**: Creates an `OpenEnvWrapper` connected to the OpenEnv server
+- **`on_trajectory_start`**: Creates a wrapper per request, calls `reset()`, uses Semaphore to limit concurrency (default 4)
+- **`on_turn_end`**: Parses model output, calls `wrapper.step()`, accumulates reward
+- **`parse_action`** (overridable): Converts model text to action dict, default `json.loads`
+- **`format_observation`** (overridable): Converts server observation to string, default `json.dumps`
+
+Users subclass `OpenEnvScheduler` and override `parse_action`, `format_observation`, `on_trajectory_start`, and `on_turn_end` to adapt to specific environments.
+
+### Example: Sudoku Environment
+
+Using TextArena Sudoku as an example, the model places numbers on a 9x9 Sudoku grid via `[row col number]` format. Full code: [sudoku_scheduler.py](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/openenv/sudoku_scheduler.py).
+
+**1. Start OpenEnv Server**
+
+Install OpenEnv and the Sudoku environment package (textarena and nltk are installed automatically as dependencies):
+
+```bash
+pip install openenv
+pip install git+https://huggingface.co/spaces/openenv/sudoku
+```
+
+Use the provided startup script to start the local server (default port 8000). `MAX_CONCURRENT_ENVS` must be ≥ `num_generations` used in training:
+
+```bash
+TEXTARENA_ENV_ID=Sudoku-v0 MAX_CONCURRENT_ENVS=8 python examples/train/grpo/plugin/openenv/start_sudoku_server.py
+```
+
+> The default `python -m textarena_env.server.app` only supports 1 concurrent session, which is insufficient for GRPO's parallel multi-generation sampling. `start_sudoku_server.py` lifts this restriction by setting `SUPPORTS_CONCURRENT_SESSIONS`.
+
+Point `base_url` to the local server in your dataset:
+
+```json
+{"messages":[{"role":"user","content":"Play"}],"env_config":{"name":"openenv","base_url":"http://127.0.0.1:8000"}}
+```
+
+**2. Custom Scheduler**
+
+Subclass `OpenEnvScheduler` to implement Sudoku-specific action parsing, observation formatting, and multi-component rewards:
+
+```python
+from swift.rollout.multi_turn import OpenEnvScheduler
+
+class SudokuScheduler(OpenEnvScheduler):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._last_content_len = {}  # Content diff tracking
+
+    async def on_trajectory_start(self, requests):
+        # Create env, parse board, generate hints
+        # hints include 'guaranteed moves' and candidate numbers
+        ...
+
+    async def on_turn_end(self, infer_request, response_choice, current_turn):
+        # Parse [row col number], step env
+        # Compute 5-component reward: empty_cell / valid_move / repetition / progress / correct
+        # Return updated board + hints as next observation
+        ...
+
+    def parse_action(self, text):
+        import re
+        match = re.search(r'\[\s*(\d+)\s+(\d+)\s+(\d+)\s*\]', text)
+        if match:
+            row, col, num = match.groups()
+            return {"message": f"[{row} {col} {num}]"}
+        return {"message": "[1 1 1]"}
+```
+
+**Multi-component reward system** (adapted from [TRL Sudoku example](https://github.com/huggingface/trl/blob/main/examples/notebooks/openenv_sudoku_grpo.ipynb)):
+
+| Reward component | Calculation | Purpose |
+|-----------------|-------------|---------|
+| `empty_cell_reward` | Targets empty cell +1 / overwrites -1 | Guide model to valid positions |
+| `valid_move_reward` | Valid new move +1 / warning -0.5 / invalid 0 | Encourage legal moves |
+| `repetition_reward` | Exponential penalty for repeats (-2^n, cap -10) | Avoid repetition |
+| `progress_reward` | (filled - initial) / (81 - initial) | Measure solving progress |
+| `correct_reward` | Binary reward from environment | Puzzle fully solved |
+
+Combined reward = sum of component averages, providing denser learning signal than a single binary reward.
+
+**3. Hints System**
+
+At each turn, the scheduler parses the current board state and provides hints to the model:
+
+- **GUARANTEED MOVES**: Cells with only one candidate (can be filled directly)
+- **Other options**: Cells with 2-3 candidates
+- **MOVES ALREADY TRIED**: Previously attempted moves (to avoid repetition)
+
+This significantly reduces exploration difficulty and enables the model to make more valid moves.
+
+**4. Prepare Dataset**
+
+The dataset serves as a placeholder; actual boards are generated by the environment server. Point `base_url` to the OpenEnv hosted address:
+
+```json
+{"messages":[{"role":"user","content":"Play"}],"env_config":{"name":"openenv","base_url":"http://127.0.0.1:8000"}}
+```
+
+**5. Register Scheduler**
+
+`sudoku_scheduler.py` includes registration code at the end, loaded via `--external_plugins`:
+
+```python
+# End of sudoku_scheduler.py
+from swift.rollout.multi_turn import multi_turns
+multi_turns['sudoku_scheduler'] = SudokuScheduler
+```
+
+**6. Start Training**
+
+```bash
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen3.5-4B \
+    --dataset examples/train/grpo/plugin/openenv/sudoku.jsonl \
+    --external_plugins examples/train/grpo/plugin/openenv/sudoku_scheduler.py \
+    --enable_thinking false \
+    --max_completion_length 256 \
+    --use_gym_env true \
+    --multi_turn_scheduler sudoku_scheduler \
+    --max_turns 20 \
+    --use_vllm true \
+    --vllm_mode colocate \
+    ...
+```
+
+Runnable script: [`examples/train/grpo/plugin/openenv/run_grpo_sudoku.sh`](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/openenv/run_grpo_sudoku.sh)
+
+### Notes
+
+1. **vLLM mode**: The example above uses `--vllm_mode colocate`, where vLLM and training share the same GPUs. If using `--vllm_mode server`, you need to start `swift rollout` separately as the vLLM server, and `--multi_turn_scheduler` / `--max_turns` should be passed to `swift rlhf`, not `swift rollout`.
+2. **Server concurrency**: `start_sudoku_server.py`'s `MAX_CONCURRENT_ENVS` must be ≥ `num_generations` used in training. The default `python -m textarena_env.server.app` only supports 1 concurrent session.
+3. **Content diff**: Environments like TextArena return cumulative messages (full history each turn). The scheduler tracks `_last_content_len` to return only the new portion, preventing context length explosion.
+4. **First-turn timing**: `on_trajectory_start` is called BEFORE the first rollout, ensuring the model sees the actual environment observation (e.g., Sudoku board) rather than the placeholder text from the dataset.
+5. **enable_thinking**: When using Qwen3.5 series models, set `--enable_thinking false` to skip `<think>` block generation.
+6. **Sync I/O**: `OpenEnvWrapper`'s `reset()`/`step()` are synchronous WebSocket calls. `OpenEnvScheduler` subclasses should wrap these calls with `asyncio.to_thread()` to avoid blocking the event loop.

docs/source_en/Instruction/GRPO/DeveloperGuide/index.rst

@@ -0,0 +1,11 @@
+Developer Guide
+===============
+.. toctree::
+   :maxdepth: 1
+
+   loss_types.md
+   multi_turn.md
+   multi_task.md
+   reward_function.md
+   reward_model.md
+   gym_env.md

docs/source_en/Instruction/GRPO/DeveloperGuide/loss_types.md

@@ -0,0 +1,131 @@
+# Loss Types
+
+GRPO training supports multiple loss types, with the main differences being the normalization dimension and gradient handling.
+
+## Loss Function
+
+At the token level, GRPO training uses the following loss function:
+
+$$\mathcal{L}_{i,t} = -\min\left(\rho_{i,t} A_{i,t}, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) A_{i,t}\right)$$
+
+When setting `loss_type cispo`, the CISPO loss is used:
+
+$$\mathcal{L}_{i,t}^{\text{CISPO}} = -\text{detach}\left(\min(\rho_{i,t}, \epsilon_{\text{high}})\right) \cdot A_{i,t} \cdot \log \pi_\theta(y_{i,t}|y_{i,<t})$$
+
+When setting `loss_type sapo`, soft gating replaces hard clipping, see [SAPO](../AdvancedResearch/SAPO.md)
+
+$$\mathcal{L}_{i,t}^{\text{SAPO}} = -g_{i,t} \cdot A_{i,t}$$
+
+where $g_{i,t} = \sigma(\tau \cdot (\rho_{i,t} - 1))$ is the temperature-controlled soft gate function.
+
+where:
+- $\rho_{i,t} = \frac{\pi_\theta(y_{i,t}|y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|y_{i,<t})}$ is the importance sampling weight
+- $A_{i,t}$ is the advantage function
+- $\epsilon$ and $\epsilon_{\text{high}}$ are the clipping parameters
+- $\text{detach}(\cdot)$ indicates that this term does not participate in gradient computation
+- $\sigma(\cdot)$ is the sigmoid function, $\tau$ is the temperature parameter
+
+## GRPO
+
+`--loss_type grpo`
+
+GRPO is the standard loss function implementation that averages the token-level losses for each sample, then averages across all samples.
+
+**Formula:**
+
+$$\mathcal{L}_{\text{GRPO}} = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{T_i} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}$$
+
+where:
+- $N$ is the number of samples in the batch
+- $T_i$ is the number of completion tokens for the $i$-th sample
+
+**Normalization Dimension:** Sample dimension (first average over tokens for each sample, then average over all samples)
+
+## BNPO (Batch Normalized Policy Optimization)
+
+`--loss_type bnpo`
+
+BNPO sums all token losses from all samples and then divides by the total number of completion tokens.
+
+**Formula:**
+
+$$\mathcal{L}_{\text{BNPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}}{\sum_{i=1}^{N} T_i}$$
+
+where:
+- $N$ is the number of samples in the batch
+- $T_i$ is the number of completion tokens for the $i$-th sample
+
+**Normalization Dimension:** Token dimension (average over all completion tokens)
+
+## DR-GRPO
+
+`--loss_type dr_grpo`
+
+DR-GRPO sums all token losses from all samples and then divides by the batch size multiplied by the maximum completion length.
+
+**Formula:**
+
+$$\mathcal{L}_{\text{DR-GRPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}}{N \times L_{\text{max}}}$$
+
+where:
+- $N$ is the number of samples in the batch
+- $T_i$ is the number of completion tokens for the $i$-th sample
+- $L_{\text{max}}$ is the maximum completion length
+
+**Normalization Dimension:** Fixed dimension (batch size × maximum completion length)
+
+## CISPO
+
+`--loss_type cispo`
+
+CISPO loss is normalized by the total number of completion tokens across all processes.
+
+**Formula:**
+
+$$\mathcal{L}_{\text{CISPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}^{\text{CISPO}}}{\sum_{\text{all processes}} \sum_{i=1}^{N_p} T_{p,i}}$$
+
+where:
+- $N$ is the number of samples in the current process batch
+- $T_i$ is the number of completion tokens for the $i$-th sample
+- $N_p$ is the number of samples for the $p$-th process
+
+**Normalization Dimension:** Global token dimension (total completion tokens across all processes)
+
+## DAPO
+
+`--loss_type dapo`
+
+DAPO is similar to BNPO, using token-level normalization, but based on global data (multi-process) normalization.
+
+**Formula:**
+
+$$\mathcal{L}_{\text{DAPO}} = \frac{\sum_{i=1}^{N} \sum_{t=1}^{T_i} \mathcal{L}_{i,t}}{\sum_{\text{all processes}} \sum_{i=1}^{N_p} T_{p,i}}$$
+
+where:
+- $N$ is the number of samples in the current process batch
+- $T_i$ is the number of completion tokens for the $i$-th sample
+- $N_p$ is the number of samples for the $p$-th process
+
+**Normalization Dimension:** Global token dimension (total completion tokens across all processes)
+
+## FIPO
+
+`--loss_type fipo`
+
+FIPO adds a Future-KL influence weight on top of the DAPO/GRPO clipped policy loss. The sequence-level advantage for each token is weighted by the discounted accumulated KL shift from the current token to future tokens:
+
+$$f_{i,t} = \text{clip}\left(\exp\left(\sum_{k=t}^{T_i} \gamma^{k-t} M_{i,k} \Delta \log p_{i,k}\right), 1-\epsilon_f, 1+\epsilon_f\right)$$
+
+$$\mathcal{L}_{i,t}^{\text{FIPO}} = f_{i,t} \cdot \mathcal{L}_{i,t}$$
+
+The FIPO influence weight is detached by default and uses the same global token normalization as DAPO.
+
+**Normalization Dimension:** Global token dimension (total completion tokens across all processes)
+
+## SAPO
+
+`--loss_type sapo`
+
+SAPO uses temperature-controlled soft gating instead of hard clipping to achieve smooth gradient attenuation. The normalization method is the same as GRPO.
+
+For details, please refer to [SAPO](../AdvancedResearch/SAPO.md)

docs/source_en/Instruction/GRPO/DeveloperGuide/multi_task.md

@@ -0,0 +1,55 @@
+# Multi-Task Training
+We can add a column to the dataset that indicates the task type, and then use this information in the reward function or reward model plugin to determine which task is being processed. This allows us to implement multi-task training. For example, suppose our dataset contains both math and programming tasks like the following:
+
+```json
+[
+    {"query": "Solve the equation x + 2 = 5", "solution": "3", "task": "math"},
+    {"query": "Write a function to calculate the Fibonacci sequence", "solution": "xxx", "task": "code"},
+    {"query": "What is the integral of x^2?", "solution": "xxx", "task": "math"},
+    {"query": "Implement a sorting algorithm in Python", "solution": "xxx", "task": "code"}
+]
+```
+
+
+We can set up different reward functions to handle math and code data separately. Note that the columns in the dataset will be passed to the reward function, so we can use the `task` column to distinguish between tasks.
+
+Below are examples of reward functions tailored for different tasks:
+
+```python
+from swift.rewards import ORM, orms
+import random
+
+# Math-specific reward function
+class MathRandomReward(ORM):
+  def __call__(self, completions, task, **kwargs):
+      rewards = []
+      for completion, t in zip(completions, task):
+          if t == "math":
+              import random
+              # Implement math accuracy logic
+              reward = random.random()
+              rewards.append(reward)
+          else:
+              # Return None for non-math tasks
+              rewards.append(None)
+      return rewards
+
+# Coding-specific reward function
+class CodeRandomReward(ORM):
+  def __call__(self, completions, task, **kwargs):
+      rewards = []
+      for prompt, completion, t in zip(prompts, completions, task):
+          if t == "code":
+              # Implement coding accuracy logic
+              reward = random.random()
+              rewards.append(reward)
+          else:
+              # Return None for non-coding tasks
+              rewards.append(None)
+      return rewards
+
+orms['math_reward'] = MathRandomReward
+orms['code_reward'] = CodeRandomReward
+```
+
+For data that does not belong to the current task, we handle it by returning None, ensuring that the reward calculation only applies to data within the designated task.

docs/source_en/Instruction/GRPO/DeveloperGuide/multi_turn.md

@@ -0,0 +1,323 @@
+# Multi-turn Training
+
+In reinforcement-learning scenarios, the model may need to interact with the environment over multiple turns (e.g., tool calls).
+This interactive training requires the model to carry out continuous reasoning based on the feedback from the environment.
+This document explains in detail how to customise the multi-turn training workflow in GRPO training.
+
+> GKD also supports multi-turn training, sharing the same `MultiTurnScheduler` infrastructure as GRPO.
+
+The figure below shows a typical multi-turn training process, where the model may perform several rollout rounds that include environment interaction, tool calls, and so on:
+
+![Multi-turn example](../../../../resources/grpo_multi_turn.png)
+
+## MultiTurnScheduler
+
+`MultiTurnScheduler` is an abstract base class that provides the default multi-turn dialogue-management logic.
+Its workflow is illustrated below:
+
+<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/multiturn_pipeline.png" width="300" />
+
+The scheduler is responsible for two core functions:
+- **Termination check** — decide whether the current turn of inference should stop via `check_finished`.
+- **Inference request construction** — build the request object for the next turn via `step`.
+
+Key methods of the abstract base class `MultiTurnScheduler`:
+
+```python
+class MultiTurnScheduler(ABC):
+
+    def __init__(self, max_turns: Optional[int] = None, *args, **kwargs):
+        self.max_turns = max_turns
+
+    def on_trajectory_start(self, requests: List['RolloutInferRequest']) -> None:
+        """Called before the first inference turn to initialize trajectory-level state.
+
+        This method can directly modify requests (e.g., inject initial environment observation).
+        Default is no-op.
+        """
+        pass
+
+    def on_turn_end(self, infer_request: 'RolloutInferRequest',
+                    response_choice: 'ChatCompletionResponseChoice',
+                    current_turn: int) -> Dict[str, Any]:
+        """Called after the assistant message is appended and before check_finished.
+
+        Used to advance the environment state (e.g., env.step) and return per-turn metadata.
+
+        Returns:
+            Dict[str, Any]: may optionally contain:
+                - 'done' (bool): if present, overrides the result of check_finished
+                - 'rollout_infos' (dict): merged into the accumulated trajectory info
+        Default returns an empty dict (no-op).
+        """
+        return {}
+
+    def step(self, infer_request: 'RolloutInferRequest', response_choice: 'ChatCompletionResponseChoice',
+             current_turn: int) -> Dict:
+        """
+        Handle the transition between dialogue turns.
+
+        Args:
+            infer_request: current inference request
+            response_choice: response of the current turn
+            current_turn: current turn index (starting from 1)
+
+        Returns:
+            Dict[str, Any]: a dict containing the result of this turn
+                - infer_request (required): the inference request for the next turn
+                - response_token_ids (optional): token IDs of each rollout response
+                - response_loss_mask (optional): loss mask of each rollout response
+                - rollout_logprobs (optional): token logps of each rollout response
+                - rollout_infos (optional): extra information
+        """
+        raise NotImplementedError
+
+    def check_finished(self, infer_request: 'RolloutInferRequest', response_choice: 'ChatCompletionResponseChoice',
+                       current_turn: int) -> bool:
+        """
+        Default termination logic for multi-turn rollout.
+
+        Termination conditions:
+        1. The response is truncated (finish_reason == 'length').
+        2. The dialogue reaches the maximum number of turns (if max_turns is set).
+
+        Args:
+            infer_request: the inference request
+            response_choice: response choice containing finish_reason
+            current_turn: current turn index
+
+        Returns:
+            bool: True to stop, False to continue
+        """
+        if response_choice.finish_reason == 'length':
+            return True
+        if self.max_turns and current_turn >= self.max_turns:
+            return True
+        return False
+```
+
+Arguments passed to `step` and `check_finished`:
+- **infer_request**: current inference request
+- **response_choice**: inference result of the current turn
+- **current_turn**: current turn index (starting from 1)
+
+<details><summary>Input example (click to expand)</summary>
+
+```python
+infer_request
+"""
+RolloutInferRequest(
+    messages=[
+        {'role': 'system', 'content': 'A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>\n'}, {'role': 'user', 'content': 'What is the value of $\\sqrt{36 \\times \\sqrt{16}}$?'},
+        {'role': 'assistant', 'content': 'To find the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\), we will break down the problem step-by-step.\n\nFirst, we need to evaluate the inner square root:\n\\[\n\\sqrt{16}\n\\]\nWe know that:\n\\[\n4^2 = 16 \\implies \\sqrt{16} = 4\n\\]\n\nNext, we substitute this result back into the original expression:\n\\[\n\\sqrt{36 \\times \\sqrt{16}} = \\sqrt{36 \\times 4}\n\\]\n\nNow, we need to evaluate the product inside the square root:\n\\[\n36 \\times 4 = 144\n\\]\n\nSo, the expression simplifies to:\n\\[\n\\sqrt{144}\n\\]\n\nFinally, we determine the square root of 144:\n\\[\n\\sqrt{144} = 12\n\\]\n\nThus, the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\) is:\n\\[\n\\boxed{12}\n\\]'}
+    ],
+    images=[],
+    audios=[],
+    videos=[],
+    tools=None,
+    objects={},
+    data_dict={
+        'problem': 'What is the value of $\\sqrt{36 \\times \\sqrt{16}}$?',
+        'solution': "To solve the problem, we need to evaluate the expression \\(\\sqrt{36 \\times \\sqrt{16}}\\).\n\nWe can break down the steps as follows:\n\n1. Evaluate the inner square root: \\(\\sqrt{16}\\).\n2. Multiply the result by 36.\n3. Take the square root of the product obtained in step 2.\n\nLet's compute this step by step using Python code for accuracy.\n```python\nimport math\n\n# Step 1: Evaluate the inner square root\ninner_sqrt = math.sqrt(16)\n\n# Step 2: Multiply the result by 36\nproduct = 36 * inner_sqrt\n\n# Step 3: Take the square root of the product\nfinal_result = math.sqrt(product)\nprint(final_result)\n```\n```output\n12.0\n```\nThe value of \\(\\sqrt{36 \\times \\sqrt{16}}\\) is /\\(\\boxed{12}\\)."
+        }
+    )
+"""
+response_choice
+"""
+ChatCompletionResponseChoice(
+    index=0,
+    message=ChatMessage(
+        role='assistant',
+        content='To find the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\), we will break down the problem step-by-step.\n\nFirst, we need to evaluate the inner square root:\n\\[\n\\sqrt{16}\n\\]\nWe know that:\n\\[\n4^2 = 16 \\implies \\sqrt{16} = 4\n\\]\n\nNext, we substitute this result back into the original expression:\n\\[\n\\sqrt{36 \\times \\sqrt{16}} = \\sqrt{36 \\times 4}\n\\]\n\nNow, we need to evaluate the product inside the square root:\n\\[\n36 \\times 4 = 144\n\\]\n\nSo, the expression simplifies to:\n\\[\n\\sqrt{144}\n\\]\n\nFinally, we determine the square root of 144:\n\\[\n\\sqrt{144} = 12\n\\]\n\nThus, the value of \\(\\sqrt{36 \\times \\sqrt{16}}\\) is:\n\\[\n\\boxed{12}\n\\]', tool_calls=None),
+        finish_reason='stop',
+        logprobs=None,
+        messages=None)
+"""
+# response_choice.messages will be copied at the end of multi-turn inference.
+```
+</details>
+
+<br>
+<br>
+
+The default `check_finished` logic stops inference in the following cases:
+- The model reply is truncated, i.e. exceeds `max_completion_length`.
+- The number of inference turns exceeds the specified maximum.
+
+For the full default multi-turn rollout logic, see the `run` method of the class.
+You can override `run` to implement a completely custom workflow.
+
+## Setting multi-turn parameters
+
+Specify the scheduler via `multi_turn_scheduler` in the `swift rollout` command:
+
+```bash
+swift rollout \
+    --model Qwen/Qwen3-1.7B \
+    --vllm_use_async_engine true \
+    --multi_turn_scheduler thinking_tips_scheduler \
+    --vllm_max_model_len 32768 \
+    --vllm_gpu_memory_utilization 0.8 \
+    --max_turns 3
+```
+
+> With the `external_plugins` argument you can register your own local scheduler with ms-swift.
+> Refer to the [plugin code](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py).
+
+A full multi-turn training script can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/external/vllm_multi_turn.sh).
+
+For multi-turn rollout we use `AsyncEngine` to perform efficient batched asynchronous sampling.
+AsyncEngine reduces compute bubbles in multi-turn inference:
+
+<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/asyncengine.png" width="400" />
+
+Use the `vllm_use_async_engine` argument in the `rollout` command to specify the engine type (async is the default).
+
+> Note: The async engine is only available in server mode.
+
+### GYM environment training
+
+If your multi-turn task can be modeled as a standard gym environment (`reset` / `step` / reward produced by the env directly), use the built-in `gym_scheduler` and implement an `Env` subclass to describe the task.
+
+`GYMScheduler` is based on the generic hook protocol and does not require overriding the `run` method:
+- **`on_trajectory_start`**: calls `env.reset` and injects the initial observation into the first user message
+- **`on_turn_end`**: calls `env.step` to advance the environment, returns `{'done': bool, 'rollout_infos': dict}`
+
+This design makes `GYMScheduler` compatible with both server mode (`run()`) and colocate mode (`run_multi_turn()`) — users only need to implement the `Env` interface.
+
+See the [GYM environment training doc](./gym_env.md) for the full interface, the steps to define a custom env, and a minimal end-to-end example with no external dependencies (FrozenLake — runs out of the box on Megatron in colocate mode).
+
+## Advanced topics
+
+### Customising the interaction logic
+
+In the default logic we treat the whole multi-turn rollout as one trajectory when computing the loss.
+This assumes the model's history is not modified during interaction.
+
+In some scenarios you may need to dynamically change the history during rollout (e.g., compressing context).
+In that case each turn should be treated as a separate trajectory.
+
+#### Approach 1: Using hooks
+
+```python
+class CustomScheduler(MultiTurnScheduler):
+    def on_trajectory_start(self, requests):
+        # Initialise before the first turn (e.g., env.reset, inject initial state)
+        for req in requests:
+            req.messages = [system_msg, user_msg(initial_observation)]
+
+    def on_turn_end(self, req, response_choice, current_turn):
+        # Advance state after each turn, return done and rollout_infos
+        next_obs, reward, done = self.advance_env(req.messages)
+        return {
+            'done': done,
+            'rollout_infos': {'reward': reward, ...}
+        }
+```
+
+This approach works with both server mode and colocate mode, and does not require overriding the `run` method.
+
+#### Approach 2: Overriding the `run` method (fully custom)
+
+A common scenario is for "thinking" models: during real inference the model keeps only the last reasoning step and discards previous ones.
+
+For such cases override the `run` method in your scheduler to return the result for each rollout turn individually.
+The built-in `ThinkingModelTipsScheduler` shows how to fully customise multi-turn inference by overriding `run()`.
+See the implementation in [multi_turn.py](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py).
+
+**NOTE**: In this scenario, the data for a single trajectory is split into multiple records. When computing rewards, you must assign the same reward to every record that belongs to the same trajectory.
+
+The complete trajectory can be accessed via `trajectory_inputs` in `kwargs`.
+
+For a concrete implementation, see the [MultiTurnThinkingTips class](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py)
+
+### Multimodal Data Override
+In multimodal, multi-turn interactions, you may need to dynamically add, delete, or modify multimodal data during the conversation and ensure these changes are synchronized to the trainer.
+
+Implementation: Use `rollout_infos` to override the original multimodal content in the dataset by specifying the corresponding keys.
+
+Supported override keys: images, audios, videos.
+
+For details, see [DeepEyes Scheduler](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py#L403-L404).
+
+### Returning response token IDs
+
+In the default workflow the scheduler returns text, the trainer re-encodes it to token IDs for training.
+To avoid this extra encoding, have the scheduler return `response_token_ids` directly.
+
+Steps:
+
+- Read the `token_ids` attribute from `response_choice` to obtain the sequence.
+- Include `response_token_ids` in the dict returned by `step` / `run`; the trainer can then use them directly.
+
+For a concrete implementation, refer to the [ThinkingModelTipsScheduler class](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py)
+
+### Loss mask
+
+When the environment or a tool call returns content that becomes part of the model response, you may want to mask it so the model is not penalised on externally generated tokens.
+
+You can set the loss mask in two ways.
+
+**1. Using `loss_scale`**
+
+ms-swift provides the `loss_scale` parameter to scale or mask parts of the response.
+For example, `--loss_scale last_round` zeroes out the loss for all but the last round.
+Custom `loss_scale` can also be implemented; see the [customisation guide](../../../Customization/Architecture.md#loss-scale).
+
+> Note: In GRPO, `loss_scale` serves only as a mask; it does not scale the loss.
+
+**2. Using `loss_mask`**
+
+In `step` or `run`, set `response_loss_mask` to define a custom mask.
+This requires returning `response_token_ids`; the mask must be the same length.
+When `response_loss_mask` is provided, `loss_scale` is ignored.
+
+For how to return response_loss_mask, see the [ToolCallScheduler class](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py)
+### Reward-function related tips
+
+**Accessing multi-turn rollout information in a reward function**
+
+Return a `rollout_infos` object from `step` / `run`, then read it from `kwargs` in the reward function:
+
+```python
+class Scheduler():
+    def step(self, infer_request: 'RolloutInferRequest', response_choice: 'ChatCompletionResponseChoice',
+             current_turn: int) -> Dict:
+        ...
+        return {'infer_request': infer_request, 'rollout_infos': extra_dict}
+
+class RewardFunction():
+    def __call__(self, completions, **kwargs):
+        infos = kwargs.get('rollout_infos', {})
+        ...
+```
+
+### Accessing additional dataset information in scheduler
+
+Set `--vllm_server_pass_dataset` on the training side to pass other dataset columns to the scheduler.
+They can be read from `infer_request.data_dict`.
+
+### Training-Inference-Mismatch
+
+Swift supports returning rollout logprobs from the vLLM side to address training-inference mismatch issues. For details, please refer to this [document](../AdvancedResearch/training_inference_mismatch.md).
+
+In multi-turn training, if `rollout_importance_sampling_mode` is enabled, the framework automatically collects log probabilities from each rollout turn to correct off-policy issues.
+
+**Default Behavior**:
+- When using the default `run` method, the framework automatically extracts log probabilities from `response_choice.logprobs`
+- These logprobs are passed to the trainer along with `response_token_ids` and `response_loss_mask`
+
+**Notes for Custom Schedulers**:
+
+If you modify the response in your `step` method (e.g., truncation, adding content), you need to return the corresponding `rollout_logprobs`:
+
+**Key Rules**:
+- The length of `rollout_logprobs` should equal the count of 1s in `response_loss_mask`
+- For tokens with `loss_mask=0` (e.g., user-added prompts, tool return results), no logprobs are needed
+- If `step` does not return `rollout_logprobs`, the framework will automatically extract them from `response_choice.logprobs`
+
+**When Overriding the `run` Method**:
+
+If you completely override the `run` method, you need to manually collect and pass `rollout_logprobs`
+
+For implementation, please refer to [here](https://github.com/modelscope/ms-swift/blob/main/swift/rollout/multi_turn.py)

docs/source_en/Instruction/GRPO/DeveloperGuide/reward_function.md

@@ -0,0 +1,144 @@
+# Reward Function
+## Custom Reward Function
+The reward function takes as arguments (via kwargs) the model-generated completions, other columns from the dataset, and the training state, and calculates a reward score. The [trainer state](https://huggingface.co/docs/transformers/main/main_classes/callback#transformers.TrainerState) includes information such as the current training step.
+
+Note: The columns related to model input (such as query and response) are converted to the messages key. The original assistant response in the dataset will be discarded, so please use extra columns if you wish to retain it.
+The relevant column names for processing can be found in the [document](../../../Customization/Custom-dataset.md#Query-Response)
+
+Below is an example illustrating how to implement a simple length-based reward function. This function assigns a reward of 1.0 if the length of the generated completion exceeds 1024, and 0.0 otherwise.
+
+```python
+from swift.rewards import ORM, orms
+class DummyLengthRewardFunction(ORM)
+    def __call__(completions, **kwargs):
+        return [1.0 if len(completion) > 1024 else 0.0 for completion in completions]
+
+orms['dummy']= DummyLengthRewardFunction
+```
+
+**Accessing Other Columns in the Dataset**
+For example, if the reward function needs to access the solution column from the dataset, as well as the current training step and the total number of steps for calculation, there are two ways to retrieve these values:
+
+
+Explicitly define the column name in the __call__ parameters:
+```python
+    def __call__(completions, solution, trainer_state, **kwargs):
+        print(solution)
+        global_step = trainer_state.global_step
+        max_steps = trainer_state.max_steps
+        ...
+```
+
+Retrieve it from kwargs:
+```python
+    def __call__(completions, **kwargs):
+        solution = kwargs.get('solution')
+        trainer_state = kwargs.get('trainer_state')
+        global_step = trainer_state.global_step
+        max_steps = trainer_state.max_steps
+        ...
+```
+
+**Using Custom Reward Functions**
+
+You can add the reward function in [plugin program](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py), register it using the parameter `--external_plugins examples/train/grpo/plugin/plugin.py`, and specify it via the `reward_funcs` parameter.
+
+For execution scripts, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/run_external_reward_func.sh).
+
+## Async Reward Functions
+
+For reward functions involving I/O operations (such as API calls, database queries, etc.), you can use asynchronous (async) reward functions to improve performance. Async reward functions are executed in parallel using `asyncio.gather`, which can significantly speed up reward computation.
+
+```python
+from swift.rewards import AsyncORM, orms
+import asyncio
+
+class AsyncAPIReward(AsyncORM):
+    async def __call__(self, completions, **kwargs):
+        import aiohttp
+
+        async def score_single(session, text):
+            async with session.post(
+                'https://api.example.com/score',
+                json={'text': text}
+            ) as resp:
+                result = await resp.json()
+                return result['score']
+
+        async with aiohttp.ClientSession() as session:
+            # Use asyncio.gather to send all requests in parallel
+            tasks = [score_single(session, c) for c in completions]
+            rewards = await asyncio.gather(*tasks)
+            return list(rewards)
+
+orms['async_api'] = AsyncAPIReward
+```
+
+Swift supports using both synchronous and asynchronous reward functions simultaneously. The trainer automatically detects the type of reward function:
+- Synchronous reward functions are executed sequentially
+- Asynchronous reward functions are executed in parallel using `asyncio.gather`
+
+The [plugin](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py) file provides an example of a generative reward model (async_genrm) that calls the `swift deploy` service.
+
+## Built-in Reward Functions
+Swift includes five rule-based reward functions (code can be found in swift/rewards/orm.py).
+
+| Reward Function | Paper |
+|----------------|----------------------------------------------------------------------------|
+| accuracy       | [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL](https://arxiv.org/abs/2501.12948) |
+| format         | Same as above |
+| cosine         | [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://arxiv.org/abs/2502.03373) |
+| repetition     | Same as above |
+| soft_overlong  | [Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO)](https://arxiv.org/abs/2503.14476) |
+
+### 1. **accuracy**
+
+This function compares the model's generated output with the solution column in the dataset to calculate an accuracy score. If the generated output matches the reference answer, the score is 1.0; otherwise, it is 0.0.
+
+Note: This reward function uses the `math_verify` library to parse the generated output and the solution, which may only be applicable to specific mathematical datasets.
+
+### 2. **format**
+
+The paper uses the following system prompt to require the model to return responses in a fixed format:
+```
+A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>
+```
+
+This function checks whether the model generates text in the format `<think>think content</think><answer>answer content</answer>`. If the generated text meets the format requirements, the score is 1.0; otherwise, it is 0.0.
+
+### 3. **cosine**
+
+The paper found that using only the accuracy reward function for training could lead to excessively long generated outputs, thereby affecting training effectiveness. The cosine reward function optimizes the training process by controlling the length of the model's outputs:
+
+- For texts with correct answers, the reward decreases as the length increases, encouraging the model to generate concise responses.
+- For texts with incorrect answers, the reward increases as the length increases, encouraging the model to think more deeply.
+
+A cosine function is used to smoothly adjust the reward value, ensuring the changes remain within a reasonable range. The parameters of the cosine function include the length of the generated text, the maximum length limit, and the minimum and maximum reward values.
+
+Parameters:
+- cosine_min_len_value_wrong (default: -0.5): The reward value for the minimum length when the answer is incorrect.
+- cosine_max_len_value_wrong (default: 0.0): The reward value for the maximum length when the answer is incorrect.
+- cosine_min_len_value_correct (default: 1.0): The reward value for the minimum length when the answer is correct.
+- cosine_max_len_value_correct (default: 0.5): The reward value for the maximum length when the answer is correct.
+- cosine_max_len (default equals the model's maximum generation length): The maximum length limit for the generated text.
+
+### 4. **repetition**
+
+Penalizes repetitive content in the model's generated text by detecting repeated n-gram patterns and applying corresponding penalties.
+
+The function splits the generated text into words and extracts n-grams of a specified size (default: 3-grams). By calculating the ratio of unique n-grams to the total number of n-grams, it determines the repetition rate. If the repetition rate is high, a larger negative reward (penalty) is applied. The penalty value is calculated based on the repetition rate and the maximum penalty value (default: -1.0).
+
+Parameters:
+- repetition_n_grams (default: 3): The size of n-grams used to detect repetition.
+- repetition_max_penalty (default: -1.0): The maximum penalty value, controlling the penalty strength.
+
+### 5. **soft overlong punishment**
+Defines a length penalty interval. Within this interval, a linear penalty in the range [-1, 0] is applied.
+
+Parameters:
+- soft_max_length: L_max in the paper, the model's maximum generation length, defaulting to max_completion_length.
+- soft_cache_length: L_cache in the paper, controlling the length penalty interval, which is [soft_max_length - soft_cache_length, soft_max_length].
+
+## Notes
+
+If a model needs to be loaded in the reward function, the training DeepSpeed plugin (transformers logic) will be used by default. Under Zero3, this may cause the model to fail to perform inference properly. Refer to this [issue](https://github.com/modelscope/ms-swift/issues/4580) to skip the DeepSpeed initialization environment.

docs/source_en/Instruction/GRPO/DeveloperGuide/reward_model.md

@@ -0,0 +1,142 @@
+# Reward Model
+
+By default, a reward model refers to a model with a classification head that outputs numeric values, usually called an Output Reward Model (ORM). These models score the outputs from other models and produce a scalar value representing the quality of the model response.
+
+You can load reward models with a classification head using the `reward_models` parameter, or load reward models trained by [reward modeling](../../RLHF.md#rm), and then use the model's logits as rewards.
+
+## Custom Reward Models
+
+For generative reward models, there are two common ways to use them: one is by directly defining the reward model logic inside the Trainer via the `reward_model_plugin`, and then using TransformersEngine for inference; the other is to call an externally deployed model service.
+
+- Using `reward_model_plugin`, the reward model will be embedded within the Trainer and does not require additional computational resources. The advantage of this approach is ease of integration, but generation speed is relatively slow, making it more suitable for small-parameter reward models.
+- When deploying reward models externally, you can use commands like `swift deploy` or `vllm serve` to deploy the model service on an independent device to greatly improve inference speed, which is more suitable for large models. However, this approach requires reserving extra hardware resources.
+
+### Internal Plugin
+
+You can flexibly customize the reward model processing logic inside `reward_model_plugin`. This enables implementations such as generative reward models, including:
+
+- Custom model system prompts: define specific instructions and context to guide the evaluation process.
+- Handling model interaction history: manage dialog context to allow meaningful and context-aware evaluation.
+- Defining custom evaluation metrics: set unique criteria and measures for response evaluation, beyond the default accuracy and relevance checks.
+
+With `reward_model_plugin`, developers can tailor the reward evaluation process for specific application needs. This flexibility allows for more fine-grained and effective reward-based training strategies.
+
+The reward model is called via the plugin's `__call__` method, which takes `inputs` as a parameter. `inputs` contains the messages of model input/output and other columns from the dataset.
+
+```python
+    def __call__(self, inputs):
+        print(inputs)
+        """
+[
+    {
+        'messages': [
+                {'role': 'system', 'content': 'system prompt'},
+                {'role': 'query', 'content': 'query'},
+                {'role': 'user', 'content': 'completions1'},
+            ],
+        'solution': "abc",
+    },
+    {
+        'messages': [
+                {'role': 'system', 'content': 'system prompt'},
+                {'role': 'query', 'content': 'query'},
+                {'role': 'user', 'content': 'completions2'},
+            ],
+        'solution': "abc",
+    }
+]
+        """
+```
+
+When using TransformersEngine in the plugin for reward model inference, you only need to construct messages and call the infer interface:
+
+```python
+class RMPlugin(DefaultRMPlugin):
+
+    def __init__(self, model, template):
+
+        super().__init__(model, template)
+        # initilize TransformersEngine to infer
+        self.engine = TransformersEngine(self.model, template=self.template, max_batch_size=0)
+
+    def __call__(self, inputs):
+        system_prompt = ...
+        query = ...
+        messages = [{'role': 'system', 'content': system_prompt}, {'role': 'query', 'content': query}]
+        result = self.engine.infer([messages], self.request_config, use_tqdm=False)
+        rewards = ...
+        return rewards
+```
+
+We provide a simple example of a generative reward model (`GenRMPlugin`) in [rm_plugin.py](https://github.com/modelscope/ms-swift/blob/main/swift/rewards/rm_plugin.py).
+
+You can customize your reward model plugin in [plugin.py](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/plugin/plugin.py) and register it using the `external_plugins` parameter.
+
+Note:
+1. In `GRPOTrainer`, the reward_model will be appended to reward_funcs one by one. Therefore, the order of `reward_weights` corresponds to `[reward_funcs, reward_model]`.
+2. The default for `reward_model_plugin` is `default`, which uses ORM logic.
+3. For models with a large number of parameters, TransformersEngine generation is slow. Please use [external deployment](#external-deployment).
+
+For models like BERT that cannot be loaded by `reward_model`, you can load them inside `reward_function`, see [issue](https://github.com/modelscope/ms-swift/issues/4580).
+
+### External Deployment
+
+This approach does not require the `reward_model_plugin` and can be called directly in the reward function.
+
+First, use the following command to start the model service:
+
+```bash
+# Note: Do not overlap deployment devices with training devices
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+swift deploy \
+    --model Qwen/Qwen2.5-72B-Instruct \
+    --vllm_tensor_parallel_size 4
+
+# [INFO:swift] model_list: ['Qwen2.5-72B-Instruct']
+# INFO:     Started server process [xxxxxx]
+# INFO:     Waiting for application startup.
+# INFO:     Application startup complete.
+# INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
+```
+
+In the reward function, initialize the client using the OpenAI library and specify the address and port of the model service. Example:
+
+```python
+from openai import OpenAI
+
+class RMReward(ORM):
+
+    def __init__(self):
+        super().__init__()
+        try:
+            self.client = OpenAI(
+                api_key='EMPTY',
+                base_url='http://127.0.0.1:8000/v1', # 127.0.0.1 if deployed locally
+            )
+            self.verify_model_name = self.client.models.list().data[0].id
+        except Exception as e:
+            raise RuntimeError('Failed to connect to the model service. Please deploy the model '
+                               "using 'swift deploy' or 'vllm serve'.") from e
+
+    def __call__(self, completions, messages, **kwargs) -> List[float]:
+        rewards = []
+        for completion, message in zip(completions, messages):
+            rm_prompt = ... # Construct the prompt for the reward model
+            chat_response = self.client.chat.completions.create(
+                model=self.verify_model_name,
+                messages=[
+                    {
+                        'role': 'system',
+                        'content': 'You are a helpful assistant.'
+                    },
+                    {
+                        'role': 'user',
+                        'content': rm_prompt
+                    },
+                ],
+            )
+            response = chat_response.choices[0].message.content.strip()
+            reward = ... # Extract the reward value from the result
+            rewards.append(reward)
+        return rewards
+```

docs/source_en/Instruction/GRPO/GetStarted/GRPO.md

@@ -0,0 +1,401 @@
+# GRPO
+
+[GRPO (Group Relative Policy Optimization)](https://arxiv.org/abs/2402.03300) leverages intra-group relative advantage calculations to replace the independent value model in the PPO algorithm and directly incorporates KL divergence penalties into the loss function to improve training stability.
+
+## Algorithm Overview
+
+GRPO Objective Function is defined as
+$
+{\scriptstyle
+\begin{aligned}
+\mathcal{J}_{G R P O}(\theta) & =\mathbb{E}_{\left[q \sim P(Q),\left\{o_i\right\}_{i=1}^G \sim \pi_{\theta_{o l d}}(O \mid q)\right]} \\
+& \frac{1}{G} \sum_{i=1}^G \frac{1}{\left|o_i\right|} \sum_{t=1}^{\left|o_i\right|}\left\{\min \left[\frac{\pi_\theta\left(o_{i, t} \mid q, o_{i,<t}\right)}{\pi_{\theta_{o l d}}\left(o_{i, t} \mid q, o_{i,<t}\right)} \hat{A}_{i, t}, \operatorname{clip}\left(\frac{\pi_\theta\left(o_{i, t} \mid q, o_{i,<t}\right)}{\pi_{\theta_{o l d}}\left(o_{i, t} \mid q, o_{i,<t}\right)}, 1-\varepsilon, 1+\varepsilon\right) \hat{A}_{i, t}\right]-\beta \mathbb{D}_{K L}\left[\pi_\theta| | \pi_{r e f}\right]\right\}
+\end{aligned}
+}
+$
+
+The advantage function is defined as
+
+$
+\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
+$
+
+
+<details> <summary>GRPO Algorithm Pseudocode</summary>
+
+```python
+# ========== 1. Rollout Generation Phase ==========
+prompt = "Question: Which is bigger? 9.11 or 9.9?"
+
+# Generate multiple completions through parallel sampling
+completions = rollout_function(
+    model=current_policy_model,
+    prompt=prompt,
+    num_generations=8,  # Hyperparameter: number of samples per prompt
+    temperature=1.0     # Hyperparameter: sampling diversity
+)
+"""
+completions = [
+    (completion 1) "The larger number is 9.9...",
+    (completion 2) "9.11 is bigger than...",
+    ...
+    (completion 8) "After calculation, 9.9..."
+]
+"""
+
+# ========== 2. Reward Calculation Phase ==========
+# Evaluate generated completions using reward model
+rewards = reward_function(
+    completions=completions,
+    ground_truth="9.9"  # Expected correct answer
+)
+"""
+rewards = [
+    (reward 1) 1.0,  # Correct answer
+    (reward 2) 0.0,  # Incorrect
+    ...
+    (reward 8) 1.0   # Correct
+]
+"""
+
+# Normalize rewards to advantages
+rewards_mean = mean(rewards)  # μ = 0.5
+rewards_std = std(rewards)    # σ = 0.25
+advantages = (rewards - rewards_mean) / (rewards_std + 1e-8)  # Standardization
+"""
+advantages = [
+    (advantage 1)  2.0,  # (1.0 - 0.5)/0.25
+    (advantage 2) -2.0,
+    ...
+    (advantage 8)  2.0
+]
+"""
+
+# ========== 3. Policy Optimization Phase ==========
+# Get token-level log probabilities from different models
+current_logps = get_per_token_logps(current_policy_model, prompt, completions)  # π_θ
+old_logps = get_per_token_logps(old_policy_model, prompt, completions)          # π_θ_old
+ref_logps = get_per_token_logps(reference_model, prompt, completions)           # π_ref
+
+# PPO Clipped Objective
+is_ratio = exp(current_logps - old_logps)  # Importance sampling ratio: e^(π_θ - π_θ_old)
+clipped_ratio = clip(is_ratio, 1-ε, 1+ε)   # ε=0.2 typically
+
+# Policy gradient term (dual form)
+policy_loss = -mean(
+    minimum(is_ratio * advantages,       # Unclipped objective
+           clipped_ratio * advantages)  # Clipped objective
+)
+
+# KL Divergence Penalty (K3 estimator)
+# KL(π_θ||π_ref) ≈ e^(logπ_ref - logπ_θ) - (logπ_ref - logπ_θ) - 1
+kl_penalty = beta * mean(
+    exp(ref_logps - current_logps) -
+    (ref_logps - current_logps) - 1
+)
+
+# Total Loss = Policy Loss + KL Penalty
+total_loss = policy_loss + kl_penalty
+
+# ========== 4. Update Rule ==========
+# Apply gradient descent to minimize total_loss
+optimizer.zero_grad()
+total_loss.backward()
+optimizer.step()
+```
+</details>
+
+
+For training script examples, refer to [examples](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo).
+
+For GRPO parameters, refer to the [documentation](../../../Instruction/Command-line-parameters.md#grpo-arguments)
+
+## Cluster Support
+
+![](../../../../resources/grpo.png)
+
+The GRPO training framework supports integration with high-performance inference engines (e.g., vLLM) to accelerate the sampling process, offering the following two deployment modes:
+
+### 1. Colocate (Internal) Mode
+Training and inference share GPU resources, with the inference service launched internally within the Trainer.
+
+Startup parameters
+```bash
+--use_vllm true \
+--vllm_mode colocate
+```
+
+#### Memory Optimization Solutions in Colocate Mode
+
+When running in Colocate mode, out-of-memory (OOM) issues may frequently occur. Below are several effective memory optimization methods and parameter configurations:
+
+1. Reduce the vllm_gpu_memory_utilization parameter.
+
+2. During the training phase, release the GPU memory occupied by vLLM:
+
+
+```bash
+--sleep_level 1
+```
+
+3. During the vLLM inference phase, release the GPU memory occupied by the model and optimizer:
+
+```bash
+--offload_optimizer true \
+--offload_model true \
+```
+
+4. Use Tensor Parallelism in vLLM:
+
+```bash
+--vllm_tensor_parallel_size [tp_size]
+```
+
+5. Gather model weights in batches (when synchronizing vLLM weights under zero3):
+
+```bash
+--move_model_batches [批次数量]
+```
+
+6. Store Megatron exported HF format weights for vLLM updates in CPU main memory to reduce GPU memory usage:
+
+```bash
+--offload_bridge true
+```
+
+### 2. Async(External) Mode
+
+Training and inference resources are separated, with a dedicated inference server deployed.
+
+Use the `swift rollout` command to deploy the vLLM server (currently only supports vLLM backend):
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+swift rollout \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --vllm_tensor_parallel_size 2 \
+  --vllm_data_parallel_size 1
+
+CUDA_VISIBLE_DEVICES=0,1 \
+swift rollout \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --vllm_tensor_parallel_size 2 \
+  --vllm_data_parallel_size 1
+
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+swift rollout \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --vllm_tensor_parallel_size 2 \
+  --vllm_data_parallel_size 2
+```
+For more rollout parameters, refer to the [vllm arguments](../../../Instruction/Command-line-parameters.md#vllm-arguments) and [rollout arguments](../../../Instruction/Command-line-parameters.md#rollout-arguments)
+
+Note: When set `vllm_use_async_engine`, enabling only DP (Data Parallelism) may cause errors. [Related issue](https://github.com/vllm-project/vllm/issues/18567). If errors occur, try enabling both TP (Tensor Parallelism) and DP or upgrading vLLM.
+
+To configure the external vLLM server during training, use the following parameters:
+
+```bash
+--use_vllm true \
+--vllm_mode server \
+--vllm_server_host <server_IP> \
+--vllm_server_port <service_port> \
+--vllm_server_timeout <timeout> \
+```
+
+### Weight-Sync Acceleration
+
+Setting the following parameters optimizes weight synchronization speed for LoRA training by syncing only the LoRA adapter weights instead of the full model weights.
+
+> Note: This synchronization method may slightly impact vLLM inference speed.
+
+```bash
+# rollout(server mode)
+swift rollout \
+    --vllm_enable_lora true \
+    --vllm_max_lora_rank xxx # match the lora_rank in the training script
+    ...
+
+# grpo(colocate mode)
+swift rlhf \
+    --rlhf_type grpo \
+    --vllm_mode colocate \
+    --vllm_enable_lora true \
+    ...
+
+# megatron grpo(colocate mode)
+swift megatron rlhf \
+    --rlhf_type grpo \
+    --vllm_mode colocate \
+    --vllm_enable_lora true \
+    ...
+```
+
+**Multimodal ViT LoRA Sync:** If ViT LoRA is enabled during training (`freeze_vit false`),
+tower/connector LoRA support must also be enabled on the vLLM side.
+
+pass via `vllm_engine_kwargs`:
+
+```bash
+--vllm_engine_kwargs '{"enable_tower_connector_lora": true}'
+```
+
+This is an experimental vLLM feature, currently supporting models such as Qwen2.5-VL and Qwen3-VL.
+For model-specific support details, see the [vLLM documentation](https://docs.vllm.ai/en/latest/features/lora/)
+and the [vLLM issue](https://github.com/vllm-project/vllm/issues/31479).
+
+## logged metrics
+- completions/mean_length: The average length of generated completions.
+- completions/min_length: The minimum length among generated completions.
+- completions/max_length: The maximum length among generated completions.
+- completions/clipped_ratio: The proportion of completions that were truncated due to length limits.
+- reward/{reward_func_name}/mean: The average reward value for a specific reward function.
+- reward/{reward_func_name}/std: The standard deviation of the reward for a specific reward function.
+> Note: These two metrics are calculated across all completions.
+- reward: The overall average reward after applying reward_weights.
+- reward_std: The standard deviation of the overall reward within each batch after applying reward_weights.
+> Note: These two metrics are first computed within each group and then averaged (for mean/std) across groups.
+- frac_reward_zero_std: The proportion of samples in a generation batch where the reward standard deviation is zero, meaning there is almost no diversity in answers for that prompt (i.e., the rewards of all completions are same).
+- kl: The average KL divergence between the model and the reference model on completions. This is logged only if beta is nonzero.
+- clip_ratio/region_mean: The average proportion of tokens clipped by the CLIP operator across different sentences.
+- clip_ratio/low_mean: The average proportion of tokens clipped by the lower CLIP bound across different sentences.
+- clip_ratio/low_min: The minimum proportion of tokens clipped by the lower CLIP bound across different sentences.
+- clip_ratio/high_mean: The average proportion of tokens clipped by the upper CLIP bound across different sentences.
+- clip_ratio/high_max: The maximum proportion of tokens clipped by the upper CLIP bound across different sentences.
+> Note: If `overlong_filter` is enabled, the kl and clip_ratio metrics will exclude overlength samples.
+
+If the `log_entropy` parameter is set, additional entropy-related metrics will be logged, including:
+- entropy/mean: the average entropy across different sentences
+- entropy/max: the maximum entropy among different sentences
+- entropy/min: the minimum entropy among different sentences
+> Note: Here, sentence entropy refers to the mean entropy of tokens in each completion.
+
+If `top_entropy_quantile` is set to a value smaller than 1.0, the entropy threshold value will also be recorded:
+- entropy/threshold: Tokens with entropy below this value will be excluded from the loss calculation.
+
+Training-inference consistency metrics, prefixed with rollout_correction, requires setting `log_rollout_offpolicy_metrics=true` or `rollout_importance_sampling_mode`:
+- `kl` / `k3_kl`: KL divergence between training policy and rollout policy (direct estimator / K3 estimator)
+- `training_ppl` / `rollout_ppl`: Perplexity of training policy and rollout policy
+- `log_ppl_diff`: Log PPL difference, reflects the degree of distribution shift
+- `ppl_ratio`: PPL ratio
+- `chi2_token` / `chi2_seq`: Token/Sequence-level χ² divergence
+
+IS correction metrics (requires setting `rollout_importance_sampling_mode`):
+- `is_weight_mean`: Average importance sampling weight
+- `ess`: Effective Sample Size
+- `clipped_frac`: Fraction of samples that were truncated or masked
+
+> For detailed explanation of training-inference consistency metrics, please refer to [Training-Inference-Mismatch](../AdvancedResearch/training_inference_mismatch.md)
+
+If `log_completions` is set, the training dynamics will be saved in the output directory, including:
+- step: The training step at the time of logging.
+- prompt: The model input.
+- completion: The model's sampled answer.
+- {reward_func_name}: The specific reward(s).
+- entropy: The average token entropy (recorded if `log_entropy` is set).
+
+Setting `report_to wandb/swanlab` will send training dynamics table to the respective platform.
+
+If you want to log extra columns in the Table, populate the `metrics_to_gather` dictionary inside `GRPOTrainer._generate_and_score_completions`.
+
+The trainer automatically detects and logs the following keys:
+
+- image: image inputs for vision models(wandb only).
+- solution: the solution column from the dataset.
+
+## FAQ
+
+**1. Loss Equals Zero / Approaches Zero / Is Negative During Training**
+
+This is normal behavior. For reference, see [issue](https://github.com/huggingface/open-r1/issues/239#issuecomment-2646297851).
+
+---
+
+**2. num_generations / Batch Size Related**
+
+In GRPO, the batch size is measured in terms of completions (i.e., model-generated outputs). For example, setting `per_device_train_batch_size=8` means that each GPU processes 8 completions for loss calculation during training.
+
+During the training phase, the total effective batch size in a full gradient accumulation step equals:
+
+```python
+effective_batch_size = num_processes * per_device_train_batch_size * gradient_accumulation_steps
+```
+
+During the sampling phase, the total batch size (completion-level) depends on the following:
+
+- If generation_batch_size is set, the total equals generation_batch_size.
+- If steps_per_generation is set, the total equals per_device_train_batch_size * steps_per_generation * num_processes.
+- By default, steps_per_generation is set to gradient_accumulation_steps, and generation_batch_size equals the per_device_train_batch_size * steps_per_generation * num_processes = per_device_train_batch_size * gradient_accumulation_steps * num_processes = effective_batch_size.
+
+During evaluation, the number of completions equals:
+
+```
+num_processes * per_device_eval_batch_size
+```
+
+The parameter `num_generations` must be divisible by the total batch size used in sampling and evaluation to ensure even distribution across devices.
+
+**Example**
+
+- num_processes = 8
+- per_device_train_batch_size = 4
+- gradient_accumulation_steps = 8
+- generation_batch_size = 512
+- num_generations = 64
+
+
+1. Total prompts needed for sampling: 512 / 64 = 8
+2. Generate 512 responses from the model per sampling step
+3. Model update batch size: 8 * 4 * 8 = 256
+
+**3. Why did KL result in NaN?**
+
+With `overlong_filter` enabled, all completions on a certain GPU were truncated.
+
+**4. How is the training steps calculated?**
+
+Refer to [issue](https://github.com/modelscope/ms-swift/issues/3912).
+
+**5. Why is the clip ratio always 0?**
+
+The core purpose of the clip mechanism is to limit the magnitude of policy updates to prevent policy performance collapse due to excessively large updates (i.e., a drastic decline in performance after policy updates). The specific formula for the clip operation is as follows:
+
+$$
+L_{\text{CLIP}}(\theta) = \mathbb{E}_{t} \left[ \min\left(r_{t}(\theta) \hat{A}_{t}, \text{clip}(r_{t}(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_{t} \right) \right]
+$$
+
+Where: $r_{t}(\theta) = \frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\text{old}}(a_{t} \mid s_{t})}$ is the importance sampling ratio, which measures the difference between the new and old policy. $\hat{A}_{t}$ is the advantage function, representing the relative return of the action. $\epsilon$ is used to limit the deviation range of $r_{t}(\theta)$.
+
+In the on-policy training process, since each update uses data generated by the latest policy, the new and old policies are the same, i.e., $\pi_{\theta} = \pi_{\text{old}}$.
+
+Thus, the importance sampling ratio is always 1, and the clip operation does not take effect.
+
+The algorithm becomes off-policy (near-on-policy) under the following parameter settings:
+1. num_iterations > 1, or
+2. gradient_accumulation_steps % steps_per_generation != 0
+
+Refer to [issue](https://github.com/huggingface/open-r1/issues/239#issuecomment-2646297851).
+
+**6. How to set the training `mini-batch size`**
+
+In GRPO training, we can configure mini-batch updates in the following two ways:
+
+- Set `generation_batch_size` to be an integer multiple of the training global batch size (effective_batch_size).
+- Or set `steps_per_generation` to be an integer multiple of `gradient_accumulation_steps`.
+
+Typical configuration example:
+- When configured with:
+steps_per_generation = 16, gradient_accumulation_steps = 8, mini_batch_size = steps_per_generation / gradient_accumulation_steps = 2. The results from 1 rollout will be split into 2 mini-batch updates.
+
+**7. Difference between swift deploy and swift rollout**
+
+- swift deploy is primarily used for model deployment and inference. It supports various engines such as PT, vLLM, and SGLang, and is compatible with streaming inference as well as the OpenAI API format.
+
+- swift rollout, on the other hand, is dedicated to GRPO rollout acceleration. Currently, it only supports the vLLM engine and comes with built-in automatic weight synchronization.
+
+**8. How to disable the KL loss term**
+
+Set the parameter `--beta 0` to disable KL loss calculation. The reference model (ref model) will not be loaded in this case.
+
+
+## RL WeChat Group
+
+<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/grpo.png" width="250">

docs/source_en/Instruction/GRPO/GetStarted/index.rst

@@ -0,0 +1,6 @@
+Get Started
+===============
+.. toctree::
+   :maxdepth: 1
+
+   GRPO.md

docs/source_en/Instruction/GRPO/index.rst

@@ -0,0 +1,19 @@
+GRPO
+===============
+.. toctree::
+   :maxdepth: 2
+   :caption: Get Started
+
+   GetStarted/index.rst
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Developer Guide
+
+   DeveloperGuide/index.rst
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Advanced Research
+
+   AdvancedResearch/index.rst

docs/source_en/Instruction/Inference-and-deployment.md

@@ -0,0 +1,354 @@
+# Inference and Deployment
+
+Below are the inference engines supported by Swift along with their corresponding capabilities. The three inference acceleration engines provide inference acceleration for Swift's inference, deployment, and evaluation modules:
+
+| Inference Acceleration Engine                    | OpenAI API                                                   | Multimodal                                                   | Quantized Model | Multiple LoRAs                                               | QLoRA | Batch Inference                                              | Parallel Techniques |
+| ------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | --------------- | ------------------------------------------------------------ | ----- | ------------------------------------------------------------ | ------------------- |
+| transformers                               | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/deploy/client/llm/chat/openai_client.py) | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/app/mllm.sh) | ✅               | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_lora.py) | ✅     | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/batch_ddp.sh) | DDP/device_map      |
+| [vllm](https://github.com/vllm-project/vllm)     | ✅                                                            | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_tp.sh) | ✅               | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/deploy/lora/server.sh) | ❌     | ✅                                                            | TP/PP/DP               |
+| [sglang](https://github.com/sgl-project/sglang)    | ✅          | ❌ |      ✅        | ❌      | ❌     | ✅ | TP/PP/DP/EP |
+| [lmdeploy](https://github.com/InternLM/lmdeploy) | ✅                                                            | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/infer/lmdeploy/mllm_tp.sh) | ✅               | ❌                                                            | ❌     | ✅                                                            | TP/DP                  |
+
+## Inference
+
+ms-swift uses a layered design philosophy, allowing users to perform inference through the command-line interface, web UI, or directly using Python.
+
+To view the inference of a model fine-tuned with LoRA, please refer to the [Pre-training and Fine-tuning documentation](./Pre-training-and-Fine-tuning.md#inference-fine-tuned-model).
+
+### Using CLI
+
+**Full Parameter Model:**
+
+```shell
+CUDA_VISIBLE_DEVICES=0 swift infer \
+    --model Qwen/Qwen2.5-7B-Instruct \
+    --stream true \
+    --infer_backend transformers \
+    --max_new_tokens 2048
+```
+
+**LoRA Model:**
+
+```shell
+CUDA_VISIBLE_DEVICES=0 swift infer \
+    --model Qwen/Qwen2.5-7B-Instruct \
+    --adapters swift/test_lora \
+    --stream true \
+    --infer_backend transformers \
+    --temperature 0 \
+    --max_new_tokens 2048
+```
+
+**Command-Line Inference Instructions**
+
+The above commands are for interactive command-line interface inference. After running the script, you can simply enter your query in the terminal. You can also input the following special commands:
+
+- `multi-line`: Switch to multi-line mode, allowing line breaks in the input, ending with `#`.
+- `single-line`: Switch to single-line mode, with line breaks indicating the end of input.
+- `reset-system`: Reset the system and clear history.
+- `clear`: Clear the history.
+- `quit` or `exit`: Exit the conversation.
+
+**Multimodal Model**
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+MAX_PIXELS=1003520 \
+VIDEO_MAX_PIXELS=50176 \
+FPS_MAX_FRAMES=12 \
+swift infer \
+    --model Qwen/Qwen2.5-VL-3B-Instruct \
+    --stream true \
+    --infer_backend transformers \
+    --max_new_tokens 2048
+```
+
+To perform inference with a multimodal model, you can add tags like `<image>`, `<video>`, or `<audio>` in your query (representing the location of image representations in `inputs_embeds`). For example, you can input `<image><image>What is the difference between these two images?` or `<video>Describe this video.` Then, follow the prompts to input the corresponding image/video/audio.
+
+
+Here is an example of inference:
+```
+<<< <image><image>What is the difference between these two images?
+Input an image path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png
+Input an image path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png
+The first image depicts a cute, cartoon-style kitten with large, expressive eyes and a fluffy white and gray coat. The background is simple, featuring a gradient of colors that highlight the kitten's face.
+
+The second image shows a group of four cartoon-style sheep standing on a grassy field with mountains in the background. The sheep have fluffy white wool, black legs, and black faces with white markings around their eyes and noses. The background includes green hills and a blue sky with clouds, giving it a pastoral and serene atmosphere.
+--------------------------------------------------
+<<< clear
+<<< <video>Describe this video.
+Input a video path or URL <<< https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4
+A baby wearing glasses is sitting on a bed and reading a book. The baby is holding the book with both hands and is looking down at it. The baby is wearing a light blue shirt and pink pants. The baby is sitting on a white pillow. The baby is looking at the book with interest. The baby is not moving much, just turning the pages of the book.
+```
+
+**Dataset Inference:**
+
+```
+CUDA_VISIBLE_DEVICES=0 swift infer \
+    --model Qwen/Qwen2.5-7B-Instruct \
+    --stream true \
+    --infer_backend transformers \
+    --val_dataset AI-ModelScope/alpaca-gpt4-data-zh \
+    --max_new_tokens 2048
+```
+
+The above example provides streaming inference for both full parameters and LoRA, and below are more inference techniques available in SWIFT:
+
+- Interface Inference: You can change `swift infer` to `swift app`.
+- Batch Inference: For large models and multimodal models, you can specify `--max_batch_size` for batch inference by using `infer_backend=transformers`. For specific details, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/batch_ddp.sh). Note that you cannot set `--stream true` when performing batch inference.
+- DDP/device_map Inference: `infer_backend=transformers` supports parallel inference using DDP/device_map technology. For further details, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/mllm_device_map.sh).
+- Inference Acceleration: Swift supports using vllm/sglang/lmdeploy for inference acceleration across the inference, deployment, and evaluation modules by simply adding `--infer_backend vllm/sglang/lmdeploy`. You can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_ddp.sh).
+- Multimodal Models: We provide shell scripts for multi-GPU inference for multimodal models using [transformers](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/mllm_device_map.sh), [vllm](https://github.com/modelscope/ms-swift/blob/main/examples/infer/vllm/mllm_tp.sh), and [lmdeploy](https://github.com/modelscope/ms-swift/blob/main/examples/infer/lmdeploy/mllm_tp.sh).
+- Quantized Models: You can directly select models that are quantized with GPTQ, AWQ, or BNB, for example: `--model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4`.
+- More Model Types: We also provide inference scripts for [bert](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/bert.sh), [reward_model](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/reward_model.sh), and [prm](https://github.com/modelscope/ms-swift/blob/main/examples/infer/transformers/prm.sh).
+
+**Tips:**
+
+- SWIFT saves inference results, and you can specify the save path using `--result_path`.
+- To output log probabilities, simply specify `--logprobs true` during inference. SWIFT will save these results. Note that setting `--stream true` will prevent storage of results.
+- Using `infer_backend=transformers` supports inference for all models supported by SWIFT, while `infer_backend=vllm/lmdeploy` supports only a subset of models. Please refer to the documentation for [vllm](https://docs.vllm.ai/en/latest/models/supported_models.html), [sglang](https://docs.sglang.ai/supported_models/generative_models.html) and [lmdeploy](https://lmdeploy.readthedocs.io/en/latest/supported_models/supported_models.html).
+- If you encounter OOM when using `--infer_backend vllm`, you can lower `--vllm_max_model_len`, `--vllm_max_num_seqs`, choose an appropriate `--vllm_gpu_memory_utilization`, or set `--vllm_enforce_eager true`. Alternatively, you can address this by using tensor parallelism with `--vllm_tensor_parallel_size`.
+- When inferring multimodal models using `--infer_backend vllm`, you need to input multiple images. You can set `--vllm_limit_mm_per_prompt` to resolve this, for example: `--vllm_limit_mm_per_prompt '{"image": 10, "video": 5}'`.
+- If you encounter OOM issues while inferring qwen2-vl/qwen2.5-vl, you can address this by setting `MAX_PIXELS`, `VIDEO_MAX_PIXELS`, and `FPS_MAX_FRAMES`. For more information, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/app/mllm.sh).
+- SWIFT's built-in dialogue templates align with dialogue templates run using transformers. You can refer to [here](https://github.com/modelscope/ms-swift/blob/main/tests/test_align/test_template/test_vision.py) for testing. If there are any misalignments, please feel free to submit an issue or PR for correction.
+
+
+### Using Web-UI
+
+If you want to perform inference through a graphical interface, you can refer to the [Web-UI documentation](../GetStarted/Web-UI.md).
+
+### Using Python
+
+**Text Model:**
+
+```python
+import os
+os.environ['CUDA_VISIBLE_DEVICES'] = '0'
+
+from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
+model = 'Qwen/Qwen2.5-0.5B-Instruct'
+
+# Load the inference engine
+engine = TransformersEngine(model, max_batch_size=2)
+request_config = RequestConfig(max_tokens=512, temperature=0)
+
+# Using 2 infer_requests to demonstrate batch inference
+infer_requests = [
+    InferRequest(messages=[{'role': 'user', 'content': 'Who are you?'}]),
+    InferRequest(messages=[{'role': 'user', 'content': 'Where is the capital of Zhejiang?'},
+                           {'role': 'assistant', 'content': 'The capital of Zhejiang Province, China, is Hangzhou.'},
+                           {'role': 'user', 'content': 'What are some fun places here?'}]),
+]
+resp_list = engine.infer(infer_requests, request_config)
+query0 = infer_requests[0].messages[0]['content']
+print(f'response0: {resp_list[0].choices[0].message.content}')
+print(f'response1: {resp_list[1].choices[0].message.content}')
+```
+
+**Multimodal Model:**
+
+```python
+import os
+os.environ['CUDA_VISIBLE_DEVICES'] = '0'
+os.environ['MAX_PIXELS'] = '1003520'
+os.environ['VIDEO_MAX_PIXELS'] = '50176'
+os.environ['FPS_MAX_FRAMES'] = '12'
+
+from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
+model = 'Qwen/Qwen2.5-VL-3B-Instruct'
+
+# Load the inference engine
+engine = TransformersEngine(model, max_batch_size=2)
+request_config = RequestConfig(max_tokens=512, temperature=0)
+
+# Using 3 infer_requests to demonstrate batch inference
+infer_requests = [
+    InferRequest(messages=[{'role': 'user', 'content': 'Who are you?'}]),
+    InferRequest(messages=[{'role': 'user', 'content': '<image><image> What is the difference between these two images?'}],
+                 images=['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png',
+                         'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']),
+    InferRequest(messages=[{'role': 'user', 'content': '<video> Describe the video'}],
+                 videos=['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4']),
+]
+resp_list = engine.infer(infer_requests, request_config)
+query0 = infer_requests[0].messages[0]['content']
+print(f'response0: {resp_list[0].choices[0].message.content}')
+print(f'response1: {resp_list[1].choices[0].message.content}')
+print(f'response2: {resp_list[2].choices[0].message.content}')
+```
+
+We also provide more demos for Python-based inference:
+
+- For streaming inference using `VllmEngine`, `SglangEngine` and `LmdeployEngine` for inference acceleration, you can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py).
+- Multimodal Inference: In addition to the aforementioned multimodal input formats, Swift is compatible with OpenAI's multimodal input format; refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_mllm.py).
+- Grounding Tasks: For performing grounding tasks with multimodal models, you can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_grounding.py).
+- Multiple LoRA Inference: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_lora.py).
+- Agent Inference: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_agent.py).
+- Asynchronous Interface: For Python-based inference using `engine.infer_async`, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py).
+
+
+## Deployment
+
+If you want to see the deployment of a model fine-tuned with LoRA, you can refer to the [Pre-training and Fine-tuning documentation](./Pre-training-and-Fine-tuning.md#deployment-fine-tuned-model).
+
+This section primarily focuses on the deployment and invocation of multimodal models. For text-based large models, we provide a simple deployment and invocation example:
+
+**Server Deployment:**
+
+```shell
+CUDA_VISIBLE_DEVICES=0 swift deploy \
+    --model Qwen/Qwen2.5-7B-Instruct \
+    --infer_backend vllm \
+    --max_new_tokens 2048 \
+    --served_model_name Qwen2.5-7B-Instruct
+```
+
+**Client Invocation Test:**
+
+```shell
+curl http://localhost:8000/v1/chat/completions \
+-H "Content-Type: application/json" \
+-d '{
+"model": "Qwen2.5-7B-Instruct",
+"messages": [{"role": "user", "content": "What should I do if I can’t sleep at night?"}],
+"max_tokens": 256,
+"temperature": 0
+}'
+```
+
+
+### Server Side
+
+```shell
+# test env: pip install transformers==4.51.3 vllm==0.8.5.post1
+CUDA_VISIBLE_DEVICES=0 \
+MAX_PIXELS=1003520 \
+VIDEO_MAX_PIXELS=50176 \
+FPS_MAX_FRAMES=12 \
+swift deploy \
+    --model Qwen/Qwen2.5-VL-3B-Instruct \
+    --infer_backend vllm \
+    --vllm_gpu_memory_utilization 0.9 \
+    --vllm_max_model_len 8192 \
+    --max_new_tokens 2048 \
+    --vllm_limit_mm_per_prompt '{"image": 5, "video": 2}' \
+    --served_model_name Qwen2.5-VL-3B-Instruct
+```
+
+### Client Side
+
+We introduce three methods for invoking the client: using curl, the OpenAI library, and the Swift client.
+
+**Method 1: curl**
+
+```shell
+curl http://localhost:8000/v1/chat/completions \
+-H "Content-Type: application/json" \
+-d '{
+"model": "Qwen2.5-VL-3B-Instruct",
+"messages": [{"role": "user", "content": [
+    {"type": "image", "image": "http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png"},
+    {"type": "image", "image": "http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png"},
+    {"type": "text", "text": "What is the difference between these two images?"}
+]}],
+"max_tokens": 256,
+"temperature": 0
+}'
+```
+
+**Method 2: OpenAI Library**
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key='EMPTY',
+    base_url=f'http://127.0.0.1:8000/v1',
+)
+model = client.models.list().data[0].id
+print(f'model: {model}')
+
+messages = [{'role': 'user', 'content': [
+    {'type': 'video', 'video': 'https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4'},
+    {'type': 'text', 'text': 'describe the video'}
+]}]
+
+resp = client.chat.completions.create(model=model, messages=messages, max_tokens=512, temperature=0)
+query = messages[0]['content']
+response = resp.choices[0].message.content
+print(f'query: {query}')
+print(f'response: {response}')
+
+# Using base64
+import base64
+import requests
+resp = requests.get('https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4')
+base64_encoded = base64.b64encode(resp.content).decode('utf-8')
+messages = [{'role': 'user', 'content': [
+    {'type': 'video', 'video': f'data:video/mp4;base64,{base64_encoded}'},
+    {'type': 'text', 'text': 'describe the video'}
+]}]
+
+gen = client.chat.completions.create(model=model, messages=messages, stream=True, temperature=0)
+print(f'query: {query}\nresponse: ', end='')
+for chunk in gen:
+    if chunk is None:
+        continue
+    print(chunk.choices[0].delta.content, end='', flush=True)
+print()
+```
+
+**Method 3: Swift Client**
+
+```python
+from swift import InferRequest, InferClient, RequestConfig, InferStats
+
+engine = InferClient(host='127.0.0.1', port=8000)
+print(f'models: {engine.models}')
+metric = InferStats()
+request_config = RequestConfig(max_tokens=512, temperature=0)
+
+# Using 3 infer_requests to demonstrate batch inference
+# Supports local paths, base64, and URLs
+infer_requests = [
+    InferRequest(messages=[{'role': 'user', 'content': 'Who are you?'}]),
+    InferRequest(messages=[{'role': 'user', 'content': '<image><image> What is the difference between these two images?'}],
+                 images=['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png',
+                         'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']),
+    InferRequest(messages=[{'role': 'user', 'content': '<video> Describe the video'}],
+                 videos=['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4']),
+]
+
+resp_list = engine.infer(infer_requests, request_config, metrics=[metric])
+print(f'response0: {resp_list[0].choices[0].message.content}')
+print(f'response1: {resp_list[1].choices[0].message.content}')
+print(f'response2: {resp_list[2].choices[0].message.content}')
+print(metric.compute())
+metric.reset()
+
+# Using base64
+import base64
+import requests
+resp = requests.get('https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4')
+base64_encoded = base64.b64encode(resp.content).decode('utf-8')
+messages = [{'role': 'user', 'content': [
+    {'type': 'video', 'video': f'data:video/mp4;base64,{base64_encoded}'},
+    {'type': 'text', 'text': 'describe the video'}
+]}]
+infer_request = InferRequest(messages=messages)
+request_config = RequestConfig(max_tokens=512, temperature=0, stream=True)
+gen_list = engine.infer([infer_request], request_config, metrics=[metric])
+print(f'response0: ', end='')
+for chunk in gen_list[0]:
+    if chunk is None:
+        continue
+    print(chunk.choices[0].delta.content, end='', flush=True)
+print()
+print(metric.compute())
+```
+
+We also provide more deployment demos:
+
+- Multiple LoRA deployment and invocation: Refer to [this link](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/lora).
+- Deployment and invocation of the Base model: Refer to [this link](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/client/llm/base).
+- More model types: We provide deployment scripts for [bert](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/bert) and [reward_model](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/reward_model).

docs/source_en/Instruction/Pre-training-and-Fine-tuning.md

@@ -0,0 +1,333 @@
+# Pre-training and Fine-tuning
+Training Capability:
+
+| Method                                                       | Full-Parameter                                               | LoRA | QLoRA                                                        | Deepspeed                                                    | Multi-Machine                                                | Multimodal                                                   |
+| ------------------------------------------------------------ | ------------------------------------------------------------ | ---- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
+| [Pre-training](https://github.com/modelscope/ms-swift/blob/main/examples/train/pretrain) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [Supervised Fine-Tuning](https://github.com/modelscope/ms-swift/blob/main/examples/train/lora_sft.sh) | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/full/train.sh) | ✅    | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora) | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu/deepspeed) | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node) | [✅](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal) |
+| [GRPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [GKD](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/gkd) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/rlhf/gkd) |
+| [PPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/ppo) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ❌                                                            |
+| [DPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/dpo) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/rlhf/dpo) |
+| [KTO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/kto.sh) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | [✅](https://github.com/modelscope/ms-swift/blob/main/examples/train/multimodal/rlhf/kto.sh) |
+| [Reward Model](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/rm.sh) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [CPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/cpo.sh) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [SimPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/simpo.sh) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [ORPO](https://github.com/modelscope/ms-swift/blob/main/examples/train/rlhf/orpo.sh) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [Embedding](https://github.com/modelscope/ms-swift/blob/main/examples/train/embedding) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [Reranker](https://github.com/modelscope/ms-swift/tree/main/examples/train/reranker) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+| [Sequence Classification](https://github.com/modelscope/ms-swift/blob/main/examples/train/seq_cls) | ✅                                                            | ✅    | ✅                                                            | ✅                                                            | ✅                                                            | ✅                                                            |
+
+
+## Environment Preparation
+
+Refer to the [SWIFT installation documentation](../GetStarted/SWIFT-installation.md) for recommended versions of third-party libraries.
+
+```shell
+pip install ms-swift -U
+
+# If using deepspeed zero2/zero3
+pip install deepspeed -U
+```
+
+## Pre-training
+
+Pre-training is done using the `swift pt` command, which will automatically use the generative template instead of the conversational template, meaning that `use_chat_template` is set to False (all other commands, such as `swift sft/rlhf/infer`, default `use_chat_template` to True). Additionally, `swift pt` has a different dataset format compared to `swift sft`, which can be referenced in the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
+
+You can refer to the CLI script for pre-training [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/pretrain/train.sh). For more information on training techniques, please refer to the fine-tuning section.
+
+Tips:
+
+- `swift pt` is equivalent to `swift sft --use_chat_template false --loss_scale all`.
+
+## Fine-tuning
+
+ms-swift employs a hierarchical design philosophy, allowing users to perform fine-tuning through the command line interface, Web-UI interface, or directly using Python.
+
+### Using CLI
+
+We provide best practices for self-cognition fine-tuning of Qwen2.5-7B-Instruct on a single 3090 GPU in 10 minutes; for details, refer to [here](../GetStarted/Quick-start.md). This can help you quickly understand SWIFT.
+
+Additionally, we offer a series of scripts to help you understand the training capabilities of SWIFT:
+
+- Lightweight Training: Examples of lightweight fine-tuning supported by SWIFT can be found [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/tuners). (Note: These methods can also be used for pre-training, but pre-training typically uses full parameter training.)
+- Distributed Training: SWIFT supports distributed training techniques, including: DDP, device_map, DeepSpeed ZeRO2/ZeRO3, and FSDP.
+  - device_map: Simplified model parallelism. If multiple GPUs are available, device_map will be automatically enabled. This evenly partitions the model layers across visible GPUs, significantly reducing memory consumption, although training speed may decrease due to serial processing.
+  - DDP + device_map: Models will be grouped and partitioned using device_map. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/multi-gpu/ddp_device_map/train.sh) for details.
+  - DeepSpeed ZeRO2/ZeRO3: Save memory resources but may reduce training speed. ZeRO2 shards optimizer states and model gradients. ZeRO3 further shards model parameters on top of ZeRO2, saving even more memory but reducing training speed further. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu/deepspeed) for details.
+  - FSDP + QLoRA: Training a 70B model on two 3090 GPUs. Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-gpu/fsdp_qlora/train.sh).
+  - Multi-node Multi-GPU Training: We have provided example shell scripts for launching multi-node runs using swift, torchrun, dlc, deepspeed, and accelerate. Except for dlc and deepspeed, the other launch scripts need to be started on all nodes to run properly. Please refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/multi-node) for details.
+- Quantization Training: Supports QLoRA training using quantization techniques such as GPTQ, AWQ, AQLM, BNB, HQQ, and EETQ. Fine-tuning a 7B model only requires 9GB of memory. For more details, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/qlora).
+- Multi-modal Training: SWIFT supports pre-training, fine-tuning, and RLHF for multi-modal models. It supports tasks such as Captioning, VQA, OCR, and [Grounding](https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2_5-vl-grounding/zh.ipynb). It supports three modalities: images, videos, and audio. For more details, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal). The format for custom multi-modal datasets can be found in the [Custom Dataset Documentation](../Customization/Custom-dataset.md).
+  - For examples of using full-parameter training for ViT/Aligner, LoRA training for LLM, and employing different learning rates, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/lora_llm_full_vit).
+  - For multimodal model packing to increase training speed, refer to the example [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/packing).
+- RLHF Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf). For multi-modal models, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multimodal/rlhf). For GRPO training, refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal). For reinforcement fine-tuning, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft).
+- Megatron Training: Supports the use of Megatron's parallelization techniques to accelerate the training of large models, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, and context parallelism. Refer to the [Megatron-SWIFT Training Documentation](../Megatron-SWIFT/Quick-start.md).
+- Sequence Classification Model Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/seq_cls).
+- Embedding Model Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/embedding).
+- Agent Training: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/agent).
+- Any-to-Any Model Training: Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/all_to_all).
+- Other Capabilities:
+  - Streaming Data Reading: Reduces memory usage when handling large datasets. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/streaming/streaming.sh).
+  - Packing: Combines multiple sequences into one, making each training sample as close to max_length as possible to improve GPU utilization. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/packing).
+  - Long Text Training: Refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/sequence_parallel).
+  - Lazy Tokenize: Performs tokenization during training instead of pre-training (for multi-modal models, this avoids the need to load all multi-modal resources before training), which can reduce preprocessing wait times and save memory. Refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/streaming/lazy_tokenize.sh).
+
+
+### Tips:
+
+- When fine-tuning a base model to a chat model using LoRA technology with `swift sft`, you may sometimes need to manually set the template. Add the `--template default` parameter to avoid issues where the base model may fail to stop correctly due to encountering special characters in the dialogue template that it has not seen before. For more details, see [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/base_to_chat).
+- If you need to train in an **offline** environment, please set `--model <model_dir>` and `--check_model false`. If the corresponding model requires `git clone` from GitHub repositories, such as `deepseek-ai/Janus-Pro-7B`, please manually download the repository and set `--local_repo_path <repo_dir>`. For specific parameter meanings, refer to the [command line parameter documentation](./Command-line-parameters.md).
+- Merging LoRA for models trained with QLoRA is not possible, so it is not recommended to use QLoRA for fine-tuning, as it cannot utilize vLLM/Sglang/LMDeploy for inference acceleration during inference and deployment. It is recommended to use LoRA or full parameter fine-tuning, merge them into complete weights, and then use GPTQ/AWQ/BNB for [quantization](https://github.com/modelscope/ms-swift/tree/main/examples/export/quantize).
+- If you are using an NPU for training, simply change `CUDA_VISIBLE_DEVICES` in the shell to `ASCEND_RT_VISIBLE_DEVICES`.
+- By default, SWIFT sets `--gradient_checkpointing true` during training to save memory, which may slightly slow down the training speed.
+- If you are using DDP for training and encounter the error: `RuntimeError: Expected to mark a variable ready only once.`, please additionally set the parameter `--gradient_checkpointing_kwargs '{"use_reentrant": false}'` or use DeepSpeed for training.
+- To use DeepSpeed, you need to install it: `pip install deepspeed -U`. Using DeepSpeed can save memory but may slightly reduce training speed.
+- If your machine has high-performance GPUs like A100 and the model supports flash-attn, it is recommended to install [flash-attn](https://github.com/Dao-AILab/flash-attention/releases) and set `--attn_impl flash_attn`, as this will accelerate training and inference while slightly reducing memory usage.
+
+**How to debug:**
+
+You can use the following method for debugging, which is equivalent to using the command line for fine-tuning, but this method does not support distributed training. You can refer to the entry point for the fine-tuning command line [here](https://github.com/modelscope/ms-swift/blob/main/swift/cli/sft.py).
+
+```python
+from swift import sft_main, SftArguments
+result = sft_main(SftArguments(
+    model='Qwen/Qwen2.5-7B-Instruct',
+    tuner_type='lora',
+    dataset=['AI-ModelScope/alpaca-gpt4-data-zh#500',
+             'AI-ModelScope/alpaca-gpt4-data-en#500',
+             'swift/self-cognition#500'],
+    torch_dtype='bfloat16',
+    # ...
+))
+```
+
+
+### Using Web-UI
+
+If you want to use the interface for training, you can refer to the [Web-UI documentation](../GetStarted/Web-UI.md).
+
+### Using Python
+
+- For the Qwen2.5 self-cognition fine-tuning notebook, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2_5-self-cognition/self-cognition-sft.ipynb).
+- For the Qwen2VL OCR task notebook, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/notebook/qwen2vl-ocr/ocr-sft.ipynb).
+
+## Merge LoRA
+
+- See [here](https://github.com/modelscope/ms-swift/blob/main/examples/export/merge_lora.sh).
+
+
+## Inference (Fine-Tuned Model)
+
+To perform inference on a LoRA-trained checkpoint using the CLI:
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift infer \
+    --adapters output/vx-xxx/checkpoint-xxx \
+    --infer_backend transformers \
+    --stream true \
+    --temperature 0 \
+    --max_new_tokens 2048
+```
+
+- The adapters folder contains the trained parameter file `args.json`, so there is no need to specify `--model` or `--system` explicitly; Swift will automatically read these parameters. If you want to disable this behavior, you can set `--load_args false`.
+- If you are using full parameter training, please use `--model` instead of `--adapters` to specify the training checkpoint directory. For more information, refer to the [Inference and Deployment documentation](./Inference-and-deployment.md#Inference).
+- You can use `swift app` instead of `swift infer` for interactive inference.
+- You can choose to merge LoRA (by additionally specifying `--merge_lora true`), and then specify `--infer_backend vllm/sglang/lmdeploy` for inference acceleration.
+
+For batch inference on the validation set of the dataset:
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift infer \
+    --adapters output/vx-xxx/checkpoint-xxx \
+    --infer_backend transformers \
+    --temperature 0 \
+    --max_new_tokens 2048 \
+    --load_data_args true \
+    --max_batch_size 1
+```
+
+- You can set `--max_batch_size 8` to enable batch processing with `--infer_backend transformers`. If you use `infer_backend vllm/sglang/lmdeploy`, it will automatically handle batching without needing to specify.
+- `--load_data_args true` will additionally read the data parameters from the training storage parameter file `args.json`.
+
+If you want to perform inference on an additional test set instead of using the training validation set, use `--val_dataset <dataset_path>` for inference:
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift infer \
+    --adapters output/vx-xxx/checkpoint-xxx \
+    --infer_backend transformers \
+    --temperature 0 \
+    --max_new_tokens 2048 \
+    --val_dataset <dataset-path> \
+    --max_batch_size 1
+```
+
+
+Example of Inference on LoRA-Trained Model Using Python:
+
+```python
+import os
+os.environ['CUDA_VISIBLE_DEVICES'] = '0'
+
+from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
+from swift import get_model_processor, get_template
+from swift.utils import safe_snapshot_download
+from peft import PeftModel
+# Please adjust the following lines
+model = 'Qwen/Qwen2.5-7B-Instruct'
+lora_checkpoint = safe_snapshot_download('swift/test_lora')  # Change to your checkpoint_dir
+template_type = None  # None: use the default template_type of the corresponding model
+default_system = "You are a helpful assistant."  # None: use the default system prompt of the corresponding model
+
+# Load model and dialogue template
+model, tokenizer = get_model_processor(model)
+if lora_checkpoint is not None:
+    model = PeftModel.from_pretrained(model, lora_checkpoint)
+template_type = template_type or model.model_meta.template
+template = get_template(tokenizer, template_type=template_type, default_system=default_system)
+engine = TransformersEngine(model, template=template, max_batch_size=2)
+request_config = RequestConfig(max_tokens=512, temperature=0)
+
+# Using 2 infer_requests to demonstrate batch inference
+infer_requests = [
+    InferRequest(messages=[{'role': 'user', 'content': 'who are you?'}]),
+    InferRequest(messages=[{'role': 'user', 'content': 'Where is the capital of Zhejiang?'},
+                           {'role': 'assistant', 'content': 'Where is the capital of Zhejiang?'},
+                           {'role': 'user', 'content': 'What is good to eat here?'},]),
+]
+resp_list = engine.infer(infer_requests, request_config)
+query0 = infer_requests[0].messages[0]['content']
+print(f'response0: {resp_list[0].choices[0].message.content}')
+print(f'response1: {resp_list[1].choices[0].message.content}')
+```
+
+Example of LoRA Inference for Multi-Modal Model:
+
+```python
+import os
+os.environ['CUDA_VISIBLE_DEVICES'] = '0'
+
+from swift.infer_engine import TransformersEngine, RequestConfig, InferRequest
+from swift import get_model_processor, get_template
+from swift.utils import safe_snapshot_download
+from peft import PeftModel
+# Please adjust the following lines
+model = 'Qwen/Qwen2.5-VL-7B-Instruct'
+lora_checkpoint = safe_snapshot_download('swift/test_grounding')  # Change to your checkpoint_dir
+template_type = None  # None: use the default template_type of the corresponding model
+default_system = None  # None: use the default system prompt of the corresponding model
+
+# Load model and dialogue template
+model, tokenizer = get_model_processor(model)
+if lora_checkpoint is not None:
+    model = PeftModel.from_pretrained(model, lora_checkpoint)
+template_type = template_type or model.model_meta.template
+template = get_template(tokenizer, template_type=template_type, default_system=default_system)
+engine = TransformersEngine(model, template=template, max_batch_size=2)
+request_config = RequestConfig(max_tokens=512, temperature=0)
+
+# Using 2 infer_requests to demonstrate batch inference
+infer_requests = [
+    InferRequest(messages=[{'role': 'user', 'content': 'who are you?'}]),
+    InferRequest(messages=[{'role': 'user', 'content': '<image>Task: Object Detection'}],
+                 images=['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']),
+]
+resp_list = engine.infer(infer_requests, request_config)
+query0 = infer_requests[0].messages[0]['content']
+print(f'response0: {resp_list[0].choices[0].message.content}')
+print(f'response1: {resp_list[1].choices[0].message.content}')
+```
+
+If you are using a model trained with ms-swift, you can obtain the training configuration as follows:
+
+```python
+from swift import safe_snapshot_download, BaseArguments
+
+lora_adapters = safe_snapshot_download('swift/test_lora')
+args = BaseArguments.from_pretrained(lora_adapters)
+print(f'args.model: {args.model}')
+print(f'args.model_type: {args.model_type}')
+print(f'args.template_type: {args.template}')
+print(f'args.default_system: {args.system}')
+```
+
+- To perform inference on a checkpoint trained with full parameters, set `model` to `checkpoint_dir` and `lora_checkpoint` to `None`. For more information, refer to the [Inference and Deployment documentation](./Inference-and-deployment.md#Inference).
+- For streaming inference and acceleration using `VllmEngine`, `SglangEngine` and `LmdeployEngine`, you can refer to the inference examples for [large models](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo.py) and [multi-modal large models](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_mllm.py).
+- For inference on fine-tuned models using the Hugging Face transformers/PEFT ecosystem, you can see [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_hf.py).
+- If you have trained multiple LoRAs and need to switch among them, refer to the [inference](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_lora.py) and [deployment](https://github.com/modelscope/ms-swift/tree/main/examples/deploy/lora) examples.
+- For grounding tasks in multi-modal models, you can refer to [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_grounding.py).
+- For inference on a LoRA fine-tuned BERT model, see [here](https://github.com/modelscope/ms-swift/blob/main/examples/infer/demo_bert.py).
+
+
+## Deployment (Fine-Tuned Model)
+
+Use the following command to start the deployment server. If the weights are trained using full parameters, please use `--model` instead of `--adapters` to specify the training checkpoint directory. You can refer to the client calling methods described in the [Inference and Deployment documentation](./Inference-and-deployment.md#Deployment): curl, OpenAI library, and Swift client.
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift deploy \
+    --adapters output/vx-xxx/checkpoint-xxx \
+    --infer_backend transformers \
+    --temperature 0 \
+    --max_new_tokens 2048 \
+    --served_model_name '<model-name>'
+```
+
+Here, a complete example of deploying and calling multiple LoRAs using vLLM will be provided.
+
+### Server Side
+
+First, you need to install vLLM: `pip install vllm -U`, and use `--infer_backend vllm` when deploying, which can significantly speed up inference.
+
+We pre-trained two base models with different self-awareness LoRA incremental weights for `Qwen/Qwen2.5-7B-Instruct` (which can run successfully). You can find relevant information in [args.json](https://modelscope.cn/models/swift/test_lora/file/view/master). You simply need to modify `--adapters` to specify the local path for the trained LoRA weights during deployment.
+
+```shell
+CUDA_VISIBLE_DEVICES=0 \
+swift deploy \
+    --adapters lora1=swift/test_lora lora2=swift/test_lora2 \
+    --infer_backend vllm \
+    --temperature 0 \
+    --max_new_tokens 2048
+```
+
+### Client Side
+
+Here, we will only cover calling using the OpenAI library. Examples for calling with curl and the Swift client can be referenced in the [Inference and Deployment documentation](./Inference-and-deployment.md#Deployment).
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key='EMPTY',
+    base_url=f'http://127.0.0.1:8000/v1',
+)
+models = [model.id for model in client.models.list().data]
+print(f'models: {models}')
+
+query = 'who are you?'
+messages = [{'role': 'user', 'content': query}]
+
+resp = client.chat.completions.create(model=models[1], messages=messages, max_tokens=512, temperature=0)
+query = messages[0]['content']
+response = resp.choices[0].message.content
+print(f'query: {query}')
+print(f'response: {response}')
+
+gen = client.chat.completions.create(model=models[2], messages=messages, stream=True, temperature=0)
+print(f'query: {query}\nresponse: ', end='')
+for chunk in gen:
+    if chunk is None:
+        continue
+    print(chunk.choices[0].delta.content, end='', flush=True)
+print()
+"""
+models: ['Qwen2.5-7B-Instruct', 'lora1', 'lora2']
+query: who are you?
+response: I am an artificial intelligence model named swift-robot, developed by swift. I can answer your questions, provide information, and engage in conversation. If you have any inquiries or need assistance, feel free to ask me at any time.
+query: who are you?
+response: I am an artificial intelligence model named Xiao Huang, developed by ModelScope. I can answer your questions, provide information, and engage in conversation. If you have any inquiries or need assistance, feel free to ask me at any time.
+"""
+```

docs/source_en/Instruction/RLHF.md

@@ -0,0 +1,136 @@
+# RLHF
+
+This document provides training scripts for various human preference alignment algorithms. If you want to learn more about the algorithms and how to choose them, please refer to the [documentation](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/M.%E4%BA%BA%E7%B1%BB%E5%81%8F%E5%A5%BD%E5%AF%B9%E9%BD%90%E8%AE%AD%E7%BB%83.md).
+
+## Dataset
+The data required by the PPO and GRPO algorithm consists solely of model inputs, which include the system prompt (optional) and the query. In the case of the GRPO algorithm, the reward function may require additional data columns. For example, to calculate accuracy, a `solution` column is needed as a reference answer.
+
+For RM and DPO-type algorithms such as ORPO, CPO, and SimPO, $(x,y_w,y_l)$ formatted data is required, where $x$ is the model input, $y_w$ is the preferred answer that aligns with human preferences, and $y_l$ is the rejected answer that does not align with human preferences, as shown in ![dpo_data](../../resources/dpo_data.png).
+
+In contrast, the KTO algorithm has a special data format that only requires $(x,y,\text{label})$, where $x$ is the model input, $y$ is the model output, and the label indicates whether the answer aligns with human preferences, as shown in ![kto_data](../../resources/kto_data.png).
+
+For RLHF training of text models or multimodal large models using a custom dataset, you can refer to the [custom dataset documentation](../Customization/Custom-dataset.md#rlhf).
+
+
+## GRPO
+[Paper on arXiv](https://arxiv.org/abs/2402.03300)
+
+Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo).
+
+## DPO
+[Paper on arXiv](https://arxiv.org/abs/2305.18290)
+
+Hyperparameters:
+
+- beta: KL regularization coefficient. A larger value imposes a stronger penalty for deviating from the reference model. Default is 0.1.
+- loss_type: Variant of the DPO algorithm. You can find the available options in the [documentation](https://huggingface.co/docs/trl/main/en/dpo_trainer#loss-functions). Default is 'sigmoid'.
+- (Optional) loss_weights: Weights for mixing multiple loss functions.
+- (Optional) ld_alpha: From the [LD-DPO paper](https://arxiv.org/abs/2409.06411). Applies a weight α < 1 to the log-probabilities of tokens that lie beyond the shared prefix of the chosen and rejected responses, thereby mitigating length bias.
+- (Optional) discopop_tau: Temperature parameter τ from the [DiscoPOP paper](https://arxiv.org/abs/2406.08414) used to scale the log-ratio before the sigmoid modulation. Default 0.05; only active when loss_type is discopop.
+
+It is recommended to perform SFT training on the preferred responses in your preference dataset before starting DPO training. This helps ensure that the data distribution better matches the requirements of the DPO algorithm.
+
+If you want to mix multiple losses (such as for [MPO](https://arxiv.org/abs/2411.10442) training), you can specify multiple loss_type values and set their weights via loss_weights.
+
+By setting the hyperparameter `rpo_alpha`, a certain proportion of SFT loss can be mixed into the loss to improve training stability.
+
+Training script references:
+
+- [DPO script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/dpo)
+- [MPO script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/mpo.sh)
+
+## RM
+[Paper on arXiv](https://arxiv.org/abs/2203.02155)
+
+Reward Modeling stage in RLHF.
+
+Use the base model or instruct model trained with SFT as the foundation model. Add a value head and train it using the preference dataset to create the reward model.
+
+The weights of the added value head will be saved in `value_head.safetensors` or `value_head.bin`.
+
+The loss function for reward modeling is as follows:
+
+$
+\text{loss} = -\log \sigma \left( r^{(c)} - r^{(r)} - m \right) + \lambda \left( r^{(c)} + r^{(r)} \right)^2
+$
+
+- $r^{(c)}$: The score assigned by the model to the chosen response.
+- $r^{(r)}$: The score assigned by the model to the rejected response.
+- $\lambda$: L2 regularization coefficient that encourages the model outputs to be close to zero. It is set by the parameter `center_rewards_coefficient`, as described in [the paper](https://arxiv.org/pdf/2307.09288), and defaults to 0.
+- $m$: Margin term that encourages the model to distinguish between samples of different difficulty levels. The dataset needs to provide a `margin` column for this; by default, it is 0. This term is also introduced in [the paper](https://arxiv.org/pdf/2307.09288).
+
+
+Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/rm.sh).
+
+## PPO
+[Paper on arXiv](https://arxiv.org/abs/2203.02155)
+
+PPO (proximal policy optimization) stage in RLHF involves four models:
+- model: The training model, either the base model or the instruct model trained with SFT.
+- ref_model: The reference model, which defaults to the model.
+- reward_model: The reward model obtained from the RM stage.
+- value_model: The value model initialized by the reward model, updated synchronously during training.
+
+Hyperparameters:
+
+- local_rollout_forward_batch_size: Batch size for each data sample, default is 64.
+- whiten_rewards: Normalize rewards, default is False.
+- kl_coef: Coefficient for the KL divergence term, default is 0.05.
+- cliprange: Clip range in the PPO policy loss function, default is 0.2.
+- vf_coef: Coefficient for the value loss function, default is 0.1.
+- cliprange_value: Clip range in the PPO value loss function, default is 0.2.
+- gamma: Discount factor for cumulative rewards, default is 1.0.
+- lam: Lambda coefficient in [GAE](https://arxiv.org/abs/1506.02438), default is 0.95.
+- num_sample_generations: Number of debugging samples generated during training, default is 10.
+
+Note: When training the base model, perform SFT first and then proceed to RLHF. Specify the chat template, and it is recommended to use `full` for `tuner_type`.
+
+Refer to the [documentation](https://huggingface.co/docs/trl/ppov2_trainer#explanation-of-the-logged-metrics) for metric explanations during training.
+
+## KTO
+[Paper on arXiv](https://arxiv.org/abs/2402.01306)
+
+Hyperparameters:
+
+- beta: KL regularization coefficient. A larger value leads to a greater penalty for deviation from the reference model. Default is 0.1.
+- desirable_weight: The $\lambda_D$ term in the loss function represents the loss weight for the preferred response samples, with a default value of 1.0.
+- undesirable_weight: The $\lambda_U$ term in the loss function represents the loss weight for rejected samples, with a default value of 1.0.
+
+Let $n_D$ and $n_U$ represent the number of preferred and rejected samples in the dataset, respectively. For hyperparameters $\lambda_D$ and $\lambda_U$, the authors recommend setting $\frac{\lambda_D n_D}{\lambda_U n_U} \in [1, \frac{4}{3}]$.
+
+Training script:
+Train using data in the $(x,y,\text{label})$ format.
+
+Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/kto.sh).
+
+## CPO
+[Paper on arXiv](https://arxiv.org/abs/2401.08417)
+
+Hyperparameters:
+
+- beta: Coefficient before the implicit reward, default is 0.1.
+- cpo_alpha: Coefficient for NLL loss, default is 1.0.
+
+Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/cpo.sh).
+
+## ORPO
+[Paper on arXiv](https://arxiv.org/abs/2403.07691)
+
+Hyperparameters:
+
+- lambda: Odds Ratio loss coefficient.
+
+Note: ORPO uses the parameter `--beta` to pass the hyperparameter `lambda`.
+
+Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/orpo.sh).
+
+## SimPO
+[Paper on arXiv](https://arxiv.org/abs/2405.14734)
+
+Hyperparameters:
+
+- beta: Coefficient before the implicit reward, default is 2.0.
+- simpo_gamma: Reward margin term, default is 1.0.
+- cpo_alpha: The mixed CPO NLL loss for improving training stability; defaults to 1.0, set to 0.0 to use the original SimPO algorithm.
+
+Reference the training script [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rlhf/simpo.sh).

docs/source_en/Instruction/Ray.md

@@ -0,0 +1,239 @@
+# Ray Support
+
+## Megatron Ray
+
+The Megatron backend supports GRPO and GKD training via Ray:
+
+| Feature | Example | Assignable Roles |
+|---------|---------|------------------|
+| megatron grpo | https://github.com/modelscope/ms-swift/tree/main/examples/ray/grpo | train/rollout |
+| megatron gkd  | https://github.com/modelscope/ms-swift/tree/main/examples/ray/gkd  | train/rollout/teacher |
+
+### When to Use
+
+Non-Ray Megatron (`megatron rlhf`) and Ray Megatron (`megatron rlhf --use_ray true`) have identical training functionality.
+
+The core difference is in **deployment**:
+
+- **Non-Ray Megatron**: Launched via torchrun. Inference can use colocate (same process) or server (manually started vLLM server) mode. Multi-node requires manually configuring `MASTER_ADDR/PORT` on each node and separately launching torchrun and vLLM server.
+- **Ray Megatron**: A single YAML declares the GPU count for each role (`train.gpus`, `rollout.gpus`, `teacher.gpus`). Ray automatically handles process creation, GPU allocation, and cross-node scheduling — no manual multi-process management needed.
+
+Both support GPU isolation between training and inference (non-Ray via `vllm_mode=server`, Ray via YAML separate mode), making them functionally equivalent. Ray's advantage is automating multi-process orchestration — in multi-node scenarios, it eliminates the operational burden of manually launching torchrun and vLLM server on each node.
+
+**Selection guide:**
+
+| Scenario | Recommendation |
+|----------|---------------|
+| Single-node training | **Non-Ray** — simpler |
+| Multi-node cluster | **Ray** — automatic cross-node scheduling, one YAML to launch |
+
+### Quick Start
+
+```bash
+# 1. Start Ray cluster (optional for single node)
+ray start --head                        # head node
+ray start --address=<head_ip>:6379      # worker nodes
+
+# 2. Submit training
+megatron rlhf --use_ray true --config examples/ray/grpo/ray_grpo_colocate.yaml
+```
+
+### GPU Allocation Modes
+
+**Colocate (shared GPU)** — Training and inference share the same GPUs, alternating usage via sleep/wake to free memory:
+
+```yaml
+colocate_groups: [[train, rollout]]
+offload_model: true
+offload_optimizer: true
+sleep_level: 1
+
+train:
+  gpus: 4
+rollout:
+  gpus: 4    # must match train
+```
+
+**Separate (dedicated GPU)** — Training and inference occupy separate GPUs with no memory contention:
+
+```yaml
+# do not set colocate_groups
+train:
+  gpus: 4
+rollout:
+  gpus: 4    # dedicated 4 GPUs
+```
+
+### GKD Teacher Modes
+
+| Mode | Configuration | top-k | full-vocab |
+|------|--------------|:-----:|:----------:|
+| Colocated teacher | Set `teacher_model` + `offload_teacher_model: true` | ✅ | ✅ |
+| Standalone teacher GPU group | Add `teacher:` group with `gpus`, `model` | ✅ | ❌ |
+
+- **Colocated teacher**: The teacher is a Megatron model sharing the same GPUs and parallel parameters as the student, with memory alternated via offload.
+- **Standalone teacher GPU group**: The teacher is an independent vLLM inference engine running on separate GPUs, with independently configured parallel parameters (`vllm_tensor_parallel_size`).
+- **top-k**: The distillation loss is computed only over the teacher's top-k highest-probability tokens (set via `gkd_logits_topk`). Lower memory usage, but discards long-tail distribution information.
+- **full-vocab**: The distillation loss is computed over the entire vocabulary, preserving the full distribution. Higher memory usage.
+
+### Related Documentation
+
+For more details, refer to:
+
+- **GRPO Training**: [Megatron GRPO docs](../Megatron-SWIFT/GRPO.md)
+- **GKD Training**: [GKD docs](../Megatron-SWIFT/GKD.md)
+- **Megatron Parameters**: [Command-line parameters](../Megatron-SWIFT/Command-line-parameters.md)
+- **Megatron Quick Start**: [Quick Start](../Megatron-SWIFT/Quick-start.md)
+
+For detailed configuration and examples, see [examples](https://github.com/modelscope/ms-swift/tree/main/examples/ray).
+
+## Swift Ray
+
+SWIFT's HF Trainer also supports using Ray for multi-GPU or multi-node training:
+
+| Feature  | Ray Support | Example                                                                        | Assignable Roles |
+|----------|-------------|--------------------------------------------------------------------------------|------------------|
+| pt/sft   | ✅           | https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node/ray | default          |
+| dpo      | ❎           |                                                                                |                  |
+| grpo     | ❎           |                                                                                |                  |
+| ppo      | ❎           |                                                                                |                  |
+| sampling | ✅           | https://github.com/modelscope/ms-swift/tree/main/examples/sampler/distill      | sampler/prm/orm  |
+| distill  | ✅           | https://github.com/modelscope/ms-swift/tree/main/examples/sampler/sample       | sampler/prm/orm  |
+
+### Technical Details
+
+Before describing parameter settings, it's necessary to first explain the technical details. Since SWIFT currently uses many existing implementations from transformers and trl internally, decomposing into different Ray roles like veRL or ROLL is impractical, and decomposition would center around Ray, resulting in poor support for non-Ray scenarios.
+
+Therefore, SWIFT adopts a decorator-based technical approach, defining different roles at the function level. These roles can be defined in parameters to specify how they are used. See the example below:
+
+```python
+from swift.ray_utils import RayHelper
+
+@RayHelper.worker(group=['model1', 'model2'])
+class MyTrainer:
+
+    def __init__(self, args):
+        self._prepare_model1()
+        self._prepare_model2()
+        self._prepare_datasets()
+
+    @RayHelper.function(group='model1')
+    def _prepare_model1(self):
+        ...
+
+    @RayHelper.function(group='model2')
+    def _prepare_model2(self):
+        ...
+
+    @RayHelper.function(group='model1')
+    def rollout(self, inputs):
+        return self.model1.generate(inputs)
+
+    @RayHelper.function(group='model2')
+    def forward_model2(self, inputs):
+        loss = self.model2.forward(inputs)
+        loss.backward()
+
+    def _prepare_datasets(self):
+        self.dataset = ...
+
+    def train(self):
+        for batch in DataLoader(self.dataset):
+            generated = self.rollout(batch)
+            self.forward_model2(generated)
+            ...
+
+
+if __name__ == '__main__':
+    ...
+    MyTrainer(args).train()
+```
+
+RayHelper distributes decorated methods to different hardware clusters, and local calls are smoothly converted to remote calls in the Ray cluster. You can also partition centered around classes:
+
+```python
+
+@RayHelper.worker(group=['model1'])
+class Model1:
+    ...
+
+    @RayHelper.function(group='model1')
+    def rollout(self):
+        ...
+
+@RayHelper.worker(group=['model2'])
+class Model2:
+    ...
+
+    @RayHelper.function(group='model2')
+    def forward_and_optimize(self):
+        ...
+
+
+class Trainer:
+    ...
+```
+
+SWIFT's support for Ray essentially uses a combination of @worker and @function annotations. The worker specifies Ray cluster roles, and function specifies how to distribute data.
+
+The function annotation has several additional parameters:
+```python
+    @staticmethod
+    def function(group: str,
+                 dispatch: Union[Literal['slice', 'all'], Callable] = 'all',
+                 execute: Literal['first', 'all'] = 'all',
+                 collect: Union[Literal['none', 'flatten'], Callable] = 'none'):
+```
+
+- dispatch: How to distribute call input parameters
+  - slice: Split the input parameters, meaning workers execute with load balancing
+  - all: All workers receive identical input parameters
+  - Custom slicing method, format:
+    ```python
+        def my_custom_slice(n, i, data):
+            # n is the number of workers, i is the current worker index, data is the original input parameters
+            # Return the input parameters for the i-th worker
+    ```
+- execute: How to execute
+  - first: Execute on rank0; slice and Callable slicing methods are invalid in this case
+  - all: Execute on all
+
+- collect: How to collect returned data
+  - none: Return as-is, format is a list of return values from each worker
+  - flatten: Flatten the results returned by workers, supports tuple flattening
+  - Callable: Custom collect method, format:
+    ```python
+        def my_custom_collect(result):
+            # result is a list returned by each worker
+            # Return in your desired format
+    ```
+
+### Parameter Settings
+
+After explaining the technical details, we can configure the parameters. Developers can set different hardware configurations according to the role list in different processes. For example, in the sampling function, there are three roles: sampler, prm, orm. You can configure them like this:
+
+```yaml
+device_groups:
+  nproc_per_node: 4
+  sample_group:
+    device: GPU
+    ranks: list(range(0, 2))
+    workers:
+      - sampler
+  rm_group:
+    device: GPU
+    ranks: list(range(2, 4))
+    workers:
+      - prm
+      - orm
+```
+
+- nproc_per_node: The minimum number of GPUs per node required in the Ray cluster.
+- xxx_group: The name of each Ray group, can be specified arbitrarily
+  - device: Device type, currently supports GPU/CPU, etc.
+  - ranks: Which ranks are allocated to the current group. If CPU, ranks can only be an integer representing the total number of processes needed. If GPU, can be in formats like `[0,1,2,3]`, `4`, `list(range(0, 4))`, etc.
+  - workers: Which roles are allocated to the current group.
+
+All available roles can be found in the table at the top of this document.
+
+If using the command line, device_groups can also be passed as `--device_groups xxx`, where xxx is a JSON string. For configuration simplicity, we strongly recommend using YAML format in conjunction with Ray.

docs/source_en/Instruction/Reinforced-Fine-tuning.md

@@ -0,0 +1,93 @@
+# Reinforced Fine-Tuning
+
+Reinforced fine-tuning is one of the most important functionalities in current model training, with various implementations. SWIFT has already supported the atomic capabilities required for reinforced fine-tuning, such as sampling, reinforcement learning, and fine-tuning. Currently, we provide a specific example of rejection sampling fine-tuning, which can be found [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft/rft.py).
+
+## Concept of Reinforced Fine-Tuning
+
+The concept of reinforced fine-tuning has been proposed since 2022 (or even earlier). Its general workflow typically includes the following steps:
+
+1. Generate data using a specific model or augment the original dataset.
+2. Train the target model using the generated data.
+3. Repeat the above process if necessary.
+
+**Step 1:**
+
+- If the data-generating model is a larger model, such as GPT, Qwen-Max, DeepSeek-V3/R1, etc., this process can be understood as distillation.
+- If the data-generating model is the same model being trained, this can be considered self-improvement fine-tuning.
+- If the sampling process involves sampling a batch, fitting the data with KL divergence and rewards, and iterating continuously, it can be classified as on-policy algorithms like PPO or GRPO.
+- Sampling algorithms include Monte Carlo sampling, do_sample, group beam search, DVTS, etc.
+- The sampling process can incorporate ORM (Outcome Reward Model), PRM (Process Reward Model), diversity filtering, language filtering, etc.
+
+**Step 2:**
+
+- If SFT (Supervised Fine-Tuning) is used, it is referred to as rejection sampling fine-tuning.
+- If reinforcement learning is used, it is called reinforcement learning fine-tuning.
+
+**Step 3:**
+
+- If distillation is performed using a larger model (e.g., Monte Carlo sampling distillation with a larger model), the process usually does not involve iterations.
+- If the same model is used for sampling or algorithms like PPO are applied, iterations are typically included.
+
+In general, the common approaches to reinforced fine-tuning include:
+
+1. **Distillation**: Sampling high-quality data in bulk from a larger model using methods like Monte Carlo or do_sample, and training a smaller model on this data.
+2. **Self-improvement**: Sampling a portion of high-quality data from the same model, filtering it, and training the model iteratively.
+3. **On-policy RL**: Using methods like PPO or GRPO for iterative training.
+
+The sampling process is usually much more time-consuming than the training process. If data is distilled using GPT or other large models, token costs must be considered. Thus, reinforced fine-tuning is generally a supplementary mechanism for fine-tuning, except for special cases like DeepSeek-R1.
+
+DeepSeek-R1 uses the GRPO algorithm to enable the emergence of CoT (Chain-of-Thought) capabilities from scratch in a base model. This method requires large-scale cluster support and sufficiently large models for capability emergence. This is not discussed in detail here, but more information can be found in the [paper analysis](https://zhuanlan.zhihu.com/p/19714987272).
+
+Some related papers on reinforced fine-tuning:
+
+- Rejection Sampling Fine-Tuning: https://arxiv.org/pdf/2308.01825
+- ReST: https://arxiv.org/pdf/2308.08998
+- B-STAR: https://arxiv.org/pdf/2412.17256
+- DeepSeekMath: https://arxiv.org/pdf/2402.03300
+- Qwen-Math-PRM: https://arxiv.org/pdf/2501.07301
+- DeepSeek-R1: https://github.com/deepseek-ai/DeepSeek-R1/tree/main
+
+## When to Use Reinforced Fine-Tuning
+
+Since LLaMA3, we have observed a very noticeable yet rarely mentioned phenomenon: when training an Instruct model using a CoT-enabled training dataset and evaluating it on the corresponding test set, the test set performance tends to degrade. For example, training `llama3.1-8b-instruct` on the GSM8K training set and evaluating the generated checkpoint on the test set reveals performance degradation.
+
+This phenomenon mainly arises from the issue of knowledge forgetting disaster in models. During fine-tuning by model manufacturers, a significant amount of CoT data is often included. When solving mathematical tasks, the model's capability often originates not from the math dataset itself but potentially from datasets like ARC. This inference is supported by [some works](https://zhuanlan.zhihu.com/p/19269451950). Continued training on general tasks disrupts the model's existing capabilities, leading to performance degradation.
+
+However, it is always correct to prioritize fine-tuning. Fine-tuning allows the model to quickly adapt to the dataset distribution at a low cost. Reinforced fine-tuning should be used under the following conditions:
+
+1. The model has already been fine-tuned but does not meet the requirements.
+2. Stronger CoT capabilities are needed.
+3. Base model training for general capabilities is necessary, and the original dataset no longer improves performance.
+4. The output results for corresponding queries can be relatively accurately evaluated, such as tasks with clear results (math, code) or clear processes (translation, style fitting).
+
+Reinforced fine-tuning heavily depends on the accuracy of reward evaluations. If the evaluations are inaccurate, the training may oscillate without progress or even degrade the model performance.
+
+## SWIFT Implementation
+
+SWIFT supports the `sample` command, which is used for model sampling. Currently supported sampling methods include:
+
+- **sample**: Use `generate` do rollout.
+
+We have provided a general [RFT script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft/rft.py). This script supports self-improvement training and allows dynamic adjustments of sampling temperature, PRM thresholds, and other hyperparameters. The training method is flexible (e.g., fine-tuning, DPO) and supports iterative retraining of the original model or continued training from the previous iteration, even loading all training states from the previous iteration. Developers can incorporate additional data filtering (e.g., ensuring rows with the same ID come from the same query), including diversity checks, language filtering, etc.
+
+## Experimental Results
+
+We used the RFT script to train and evaluate the `competition_math` dataset in the math domain. The results are as follows:
+
+| Model                      | MATH Score | Training Method | Iterations | Post-Training MATH Score  |
+|----------------------------|------------|-----------------|------------|---------------------------|
+| LLaMA3.1_8b               | 12.0       | SFT             | 3          | 25.2 (LLaMA3.1_8b_sft)   |
+| LLaMA3.1_8b_sft           | 25.2       | RFT             | 2          | 32.4                     |
+| LLaMA3.1_8b_instruct      | 52.2       | SFT             | 2          | 39.0                     |
+| LLaMA3.1_8b_instruct      | 52.2       | RFT             | 3          | 58                       |
+| Qwen2.5_math_7b_instruct  | 79.6       | RFT             | 2          | 83.2                     |
+
+As shown, applying SFT to the `competition_math` dataset resulted in significant performance degradation for the instruct model. However, RFT improved the model's capabilities, even for the state-of-the-art `Qwen2.5_math_7b_instruct` math model.
+
+Specifically, we tested the GSM8K metric for `Qwen2.5_math_7b_instruct`:
+
+| Model                      | GSM8K Score | Post-RFT GSM8K Score |
+|----------------------------|-------------|-----------------------|
+| Qwen2.5_math_7b_instruct  | 92.8        | 91.6                 |
+
+As shown, RFT training did not significantly change the GSM8K score, avoiding the previously mentioned performance degradation phenomenon.

docs/source_en/Instruction/Sample.md

@@ -0,0 +1,100 @@
+# Sampling
+
+Sampling is one of the newly supported key capabilities of SWIFT. This feature can be understood as the practical implementation of `test-time compute`. Additionally, this capability is crucial for the implementation of RFT (Reinforcement Fine-Tuning).
+
+## Capability Introduction
+
+The sampling capability of SWIFT can be demonstrated with the following example:
+
+```shell
+swift sample --model LLM-Research/Meta-Llama-3.1-8B-Instruct --sampler_engine transformers --num_return_sequences 5 --dataset AI-ModelScope/alpaca-gpt4-data-zh#5
+```
+
+A `jsonl` file with a timestamp as the filename will be generated in the `sample_output` directory of the current folder. This file should contain 25 lines, each representing a complete `messages` format data.
+
+For a list of sampling parameters, please refer to [here](Command-line-parameters.md).
+
+## Environment Setup
+
+```shell
+pip install ms-swift[llm] -U
+```
+
+Or install swift from source:
+
+```shell
+git clone https://github.com/modelscope/ms-swift.git
+cd ms-swift
+pip install -e '.[llm]'
+```
+
+## Using PRM and ORM for Result Filtering
+
+An important capability of sampling is supervising the process and results, which can be supported by setting additional parameters.
+
+```shell
+swift sample --model LLM-Research/Meta-Llama-3.1-8B-Instruct --sampler_engine lmdeploy --num_return_sequences 5 --n_best_to_keep 2 --dataset tastelikefeet/competition_math#5 --prm_model AI-ModelScope/GRM-llama3.2-3B-rewardmodel-ft --orm_model math
+```
+
+A `jsonl` file with a timestamp as the filename will be generated in the `sample_output` directory of the current folder. This file **will contain at most** 10 lines, each representing a complete `messages` format data.
+> The reason it contains at most 10 lines is that although 5 data points are processed in total, and 2 are kept for each data point (`n_best_to_keep`), ORM may fail some validations, and failed data will not be retained in the file.
+> Additionally, after adding `--prm_model` or `--orm_model`, the file format is slightly different and includes a `rejected_response` key, which contains the responses with the lowest PRM scores.
+
+## Customizing PRM or ORM
+
+PRM and ORM can be customized by adding a new implementation in the plugin according to the existing code. For example:
+
+```python
+class CustomPRM:
+
+    # The constructor should be parameterless
+    def __init__(self):
+        # Initialize here
+        pass
+
+    def __call__(self, infer_requests: List[InferRequest],  ground_truths: List[str], **kwargs) -> List[Union[float, List[float]]]:
+        ...
+
+
+prms = {'custom': CustomPRM}
+```
+
+Afterward, use `--prm_model custom` in the command line.
+
+## Memory Control
+
+If the sampled model and PRM are loaded into memory simultaneously, it may lead to an OOM (Out of Memory) issue. To address this, sampling can be divided into two stages:
+
+- **Stage 1**: Specify `--model` and `--sampler_engine` without specifying `--orm_model` and `--prm_model`. Perform sampling only and save the results to a file.
+- **Stage 2**: Specify `--sampler_engine no`, along with `--orm_model` and `--prm_model`, and also specify `--cache_files`. Perform only RM data filtering without re-sampling.
+
+By dividing the process into two stages, only one model is loaded at a time, avoiding OOM issues.
+
+## Practical Example
+
+Please refer to the [Reinforcement Fine-Tuning Script](https://github.com/modelscope/ms-swift/tree/main/examples/train/rft/rft.py). This script provides a practical example of using sampling for reinforcement fine-tuning.
+
+> **Note:** The actual effectiveness of this script is strongly related to the quality of the model, data, and RM. Therefore, it is presented only as an example. Users should modify this script and train their own RM and generator models accordingly.
+
+## Sampling From Large Model
+
+SWIFT's sample supports using the OpenAI API to distill data with large models. Example:
+
+```shell
+OPENAI_API_KEY="your_api_key" \
+swift sample \
+    --sampler_type distill \
+    --sampler_engine client \
+    --model deepseek-r1 \
+    --stream true \
+    --dataset tastelikefeet/competition_math#5 \
+    --num_return_sequences 1 \
+    --temperature 0.6 \
+    --top_p 0.95 \
+    --engine_kwargs '{"base_url":"https://dashscope.aliyuncs.com/compatible-mode/v1"}'
+```
+In this example:
+
+`base_url` and `model` represent the API endpoint and model name, respectively. `stream` indicates the stream parameter for the request.
+
+Note: For Deepseek-R1 series models, the output will be formatted as:`<thinking>{reasoning_content}</thinking>\n\n<answer>{content}</answer>`.

docs/source_en/Instruction/Use-tuners.md

@@ -0,0 +1,122 @@
+# Using Tuners
+
+Tuners refer to additional structural components attached to a model, aimed at reducing the number of training parameters or enhancing training accuracy. The tuners currently supported by SWIFT include:
+
+- LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/abs/2106.09685)
+- LoRA+: [LoRA+: Efficient Low Rank Adaptation of Large Models](https://arxiv.org/pdf/2402.12354.pdf)
+- LLaMA PRO: [LLAMA PRO: Progressive LLaMA with Block Expansion](https://arxiv.org/pdf/2401.02415.pdf)
+- GaLore/Q-GaLore: [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://arxiv.org/abs/2403.03507)
+- Liger Kernel: [Liger Kernel: Efficient Triton Kernels for LLM Training](https://arxiv.org/abs/2410.10989)
+- LISA: [LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning](https://arxiv.org/abs/2403.17919)
+- UnSloth: https://github.com/unslothai/unsloth
+- SCEdit: [SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing](https://arxiv.org/abs/2312.11392)  < [arXiv](https://arxiv.org/abs/2312.11392)  |  [Project Page](https://scedit.github.io/) >
+- NEFTune: [Noisy Embeddings Improve Instruction Finetuning](https://arxiv.org/abs/2310.05914)
+- LongLoRA: [Efficient Fine-tuning of Long-Context Large Language Models](https://arxiv.org/abs/2309.12307)
+- Adapter: [Parameter-Efficient Transfer Learning for NLP](http://arxiv.org/abs/1902.00751)
+- Vision Prompt Tuning: [Visual Prompt Tuning](https://arxiv.org/abs/2203.12119)
+- Side: [Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks](https://arxiv.org/abs/1912.13503)
+- Res-Tuning: [Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone](https://arxiv.org/abs/2310.19859)  < [arXiv](https://arxiv.org/abs/2310.19859)  |  [Project Page](https://res-tuning.github.io/) >
+- Tuners provided by [PEFT](https://github.com/huggingface/peft), such as AdaLoRA, DoRA, Fourierft, etc.
+
+## Interface List
+
+### Swift Class Static Interfaces
+
+- `Swift.prepare_model(model, config, **kwargs)`
+  - Function: Loads a tuner into a model. If it is a subclass of `PeftConfig`, it uses the corresponding interface from the Peft library to load the tuner. When using `SwiftConfig`, this interface can accept `SwiftModel` instances and can be called repeatedly, functioning similarly to passing a dictionary of configs.
+    - This interface supports the parallel loading of multiple tuners of different types for concurrent use.
+  - Parameters:
+    - `model`: An instance of `torch.nn.Module` or `SwiftModel`, the model to be loaded.
+    - `config`: An instance of `SwiftConfig` or `PeftConfig`, or a dictionary of custom tuner names paired with their respective configs.
+  - Return Value: An instance of `SwiftModel` or `PeftModel`.
+
+- `Swift.merge_and_unload(model)`
+  - Function: Merges LoRA weights back into the original model and completely unloads the LoRA component.
+  - Parameters:
+    - `model`: An instance of `SwiftModel` or `PeftModel` that has had LoRA loaded.
+  - Return Value: None.
+
+- `Swift.merge(model)`
+  - Function: Merges LoRA weights back into the original model without unloading the LoRA component.
+  - Parameters:
+    - `model`: An instance of `SwiftModel` or `PeftModel` that has had LoRA loaded.
+  - Return Value: None.
+
+- `Swift.unmerge(model)`
+  - Function: Splits LoRA weights back from the original model weights into the LoRA structure.
+  - Parameters:
+    - `model`: An instance of `SwiftModel` or `PeftModel` that has had LoRA loaded.
+  - Return Value: None.
+
+- `Swift.save_to_peft_format(ckpt_dir, output_dir)`
+  - Function: Converts stored LoRA checkpoints to a PEFT-compatible format. Key changes include:
+    - The `default` will be split from the corresponding `default` folder into the root directory of `output_dir`.
+    - The `{tuner_name}.` field will be removed from weight keys, e.g., `model.layer.0.self.in_proj.lora_A.default.weight` becomes `model.layer.0.self.in_proj.lora_A.weight`.
+    - Weight keys will have a `basemodel.model` prefix added.
+    - Note: Only LoRA can be converted; other types of tuners will raise conversion errors due to PEFT not supporting them. Additionally, due to the presence of extra parameters in LoRAConfig, such as `dtype`, conversion to Peft format is not supported when these parameters are set. In such cases, you can manually delete the corresponding fields in adapter_config.json.
+  - Parameters:
+    - `ckpt_dir`: Original weights directory.
+    - `output_dir`: The target directory for the weights.
+  - Return Value: None.
+
+- `Swift.from_pretrained(model, model_id, adapter_name, revision, **kwargs)`
+  - Function: Load the tuner onto the model from the stored weights directory. If `adapter_name` is not provided, all tuners from the `model_id` directory will be loaded. This interface can also be called repeatedly, similar to `prepare_model`.
+  - Parameters:
+    - `model`: An instance of `torch.nn.Module` or `SwiftModel` to which the tuner will be loaded.
+    - `model_id`: A string indicating the tuner checkpoint to be loaded, which can be an ID from the model hub or a local directory.
+    - `adapter_name`: Can be of type `str`, `List[str]`, `Dict[str, str]`, or `None`. If `None`, all tuners in the specified directory will be loaded. If it is a `str` or `List[str]`, only specific tuners will be loaded. If it is a `Dict`, the key represents the tuner to load, which will be renamed to the corresponding value.
+    - `revision`: If `model_id` is an ID from the model hub, `revision` can specify the corresponding version number.
+
+### SwiftModel Interfaces
+
+Below is a list of interfaces that users may call. Other internal or less recommended interfaces can be viewed by running the `make docs` command to access the API Doc.
+
+- `SwiftModel.create_optimizer_param_groups(self, **defaults)`
+  - Function: Creates parameter groups based on the loaded tuners; currently, this only applies to the `LoRA+` algorithm.
+  - Parameters:
+    - `defaults`: Default parameters for the `optimizer_groups`, such as `lr` and `weight_decay`.
+  - Return Value:
+    - The created `optimizer_groups`.
+
+- `SwiftModel.add_weighted_adapter(self, ...)`
+  - Function: Merges existing LoRA tuners into one.
+  - Parameters:
+    - This interface is a passthrough to `PeftModel.add_weighted_adapter`, and parameters can be referenced in the [add_weighted_adapter documentation](https://huggingface.co/docs/peft/main/en/package_reference/lora#peft.LoraModel.add_weighted_adapter).
+
+- `SwiftModel.save_pretrained(self, save_directory, safe_serialization, adapter_name)`
+  - Function: Saves tuner weights.
+  - Parameters:
+    - `save_directory`: The directory for saving.
+    - `safe_serialization`: Whether to use safe tensors, default is `False`.
+    - `adapter_name`: Stored adapter tuner, if not provided, defaults to storing all tuners.
+
+- `SwiftModel.set_active_adapters(self, adapter_names, offload=None)`
+  - Function: Sets the currently active adapters; adapters not in the list will be deactivated.
+    - In inference, the environment variable `USE_UNIQUE_THREAD=0/1`, default is `1`. If set to `0`, then `set_active_adapters` only takes effect in the current thread, at which point it defaults to using the tuners activated in this thread, with tuners in different threads not interfering with each other.
+  - Parameters:
+    - `adapter_names`: The names of the active tuners.
+    - `offload`: How to handle deactivated adapters; default is `None`, meaning they remain in GPU memory. Can also use `cpu` or `meta` to offload to CPU or meta device to reduce memory consumption. In `USE_UNIQUE_THREAD=0`, do not pass the `offload` value to avoid affecting other threads.
+  - Return Value: None.
+
+- `SwiftModel.activate_adapter(self, adapter_name)`
+  - Function: Activates a tuner.
+    - In inference, the environment variable `USE_UNIQUE_THREAD=0/1`, default is `1`. If set to `0`, `activate_adapter` will only be effective for the current thread, at which point it defaults to using the tuners activated in this thread, with tuners in different threads not interfering with each other.
+  - Parameters:
+    - `adapter_name`: The name of the tuner to be activated.
+  - Return Value: None.
+
+- `SwiftModel.deactivate_adapter(self, adapter_name, offload)`
+  - Function: Deactivates a tuner.
+    - During `inference`, do not call this interface when the `USE_UNIQUE_THREAD=0`.
+  - Parameters:
+    - `adapter_name`: The name of the tuner to be deactivated.
+    - `offload`: How to handle deactivated adapters; defaults to `None`, meaning they remain in GPU memory. Can also use `cpu` or `meta` to offload to CPU or meta device to reduce memory consumption.
+  - Return Value: None.
+
+- `SwiftModel.get_trainable_parameters(self)`
+  - Function: Returns information about the trainable parameters.
+  - Parameters: None.
+  - Return Value: Information about trainable parameters in the following format:
+    ```text
+    trainable params: 100M || all params: 1000M || trainable%: 10.00% || cuda memory: 10GiB.
+    ```