wehub/modelscope--ms-swift
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Lint test / lint (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:34:58 +08:00
commit a203934033
1368 changed files with 175001 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source_en/Instruction/GRPO/AdvancedResearch/CHORD.md
+63
View File
@@ -0,0 +1,63 @@
# On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting (CHORD)
This document describes the CHORD algorithm proposed in the paper [On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting](https://arxiv.org/abs/2508.11408). The core idea of CHORD is to dynamically integrate expert data (SFT) into reinforcement learning by a dual control mechanism: a global weight μ plus a token-level weight φ, thereby balancing imitation and exploration.
## Algorithm Overview
CHORD mixes training by introducing the SFT loss into the GRPO loss. The overall objective is:
$$
\mathcal{L}_{\text{CHORD}} = (1 - \mu) \cdot \mathcal{L}_{\text{GRPO}} + \mu \cdot \mathcal{L}_{\text{SFT}}
$$
where:
- $\mathcal{L}_{\text{GRPO}}$: on-policy RL loss based on on-policy samples (similar to PPO).
- $\mathcal{L}_{\text{SFT}}$: supervised fine-tuning (SFT) loss.
- $\mu \in [0, 1]$: global balancing coefficient that controls the contribution of the SFT signal to the overall gradient.
### Configuration (data and batch sizes)
We can implement CHORD training based on GRPO training.
CHORD requires specifying an additional SFT dataset and batch size at training time:
- `chord_sft_dataset`: the SFT dataset that provides expert data.
- `chord_sft_per_device_train_batch_size`: SFT mini-batch size per device.
---
## Two CHORD Variants
The paper proposes two variants: CHORD-μ and CHORD-φ.
### CHORD-μ
CHORD-μ gradually decays μ during training to transition from imitating experts toward autonomous exploration.
Parameters:
- `chord_mu_peak`: the peak value of μ.
- `chord_mu_valley`: the final decayed value of μ.
- `chord_mu_warmup_steps`: number of training steps to ramp μ up to the peak.
- `chord_mu_decay_steps`: number of training steps to decay μ from peak to valley.
### CHORD-φ (Token-level weighting)
CHORD-φ uses a token-wise weighting function φ to dynamically control each expert token's gradient contribution.
Definition of φ:
$$
\phi(y_t^\star, \pi_\theta) = p_t \cdot (1 - p_t)
$$
where:
- $p_t = \pi_\theta(y_t^\star \mid x, y_{<t}^\star)$ is the model's current predicted probability of the expert token.
- When $p_t \approx 0.5$ (model uncertainty), φ is maximal → emphasize tokens the model is uncertain about.
- When $p_t \approx 0$ or $p_t \approx 1$, φ → 0 → avoid overemphasizing tokens that are already certain or impossible.
Parameter to enable φ weighting:
- `chord_enable_phi_function: bool = False`
- Set to `True` to enable token-wise weight φ.
Note: If using a constant μ, set `chord_mu_peak` and `chord_mu_valley` to the same value.
<details>
<summary>Code implementation of μ scheduling and loss computation</summary>
See the `GRPOTrainer` method `_compute_chord_loss`.
</details>
Training reference script: https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/chord.sh
docs/source_en/Instruction/GRPO/AdvancedResearch/CISPO.md
+62
View File
@@ -0,0 +1,62 @@
# Clipped Importance Sampling Policy Optimization (CISPO)
Clipped Importance Sampling Policy Optimization (CISPO) is a reinforcement learning algorithm proposed in the [MiniMax-M1](https://arxiv.org/abs/2506.13585) paper. Compared to GRPO (Group Relative Policy Optimization), CISPO clips the importance sampling weights themselves.
## Algorithm Overview
For clarity, we explain CISPO by contrasting it with GRPO.
GRPO limits the magnitude of policy updates by clipping the policy ratio. Its loss function is:
$$
\mathcal{L}_{\text{GRPO}}(\theta) = -\mathbb{E}\left[\min\left(r_t(\theta) \cdot \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \cdot \hat{A}_t\right)\right]
$$
where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the importance sampling ratio.
When handling long reasoning chains, this clipping approach can lead to the following issues:
**Gradient Suppression of Critical Tokens**: In complex reasoning tasks, certain critical low-probability tokens (such as *However, Recheck, Wait, Aha*) are crucial for triggering deep thinking and reasoning error correction. These tokens have low probability in the old policy $\pi_{\theta_{\text{old}}}$. When the new policy attempts to increase their probability, it results in a large policy ratio $r_t(\theta)$, and GRPO's clipping mechanism will discard these tokens.
### CISPO's Solution
The core idea of CISPO is to clip the importance sampling weights while preserving gradient updates. Specifically, CISPO's loss function is:
$$
\mathcal{L}_{\text{CISPO}}(\theta) = -\mathbb{E}\left[\text{detach}\left(\min(r_t(\theta), \epsilon_{\text{high}})\right) \cdot \hat{A}_t \cdot \log \pi_\theta(a_t|s_t)\right]
$$
where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the importance sampling ratio.
**Key Mechanisms**:
- Clip the importance sampling weights: $\min(r_t(\theta), \epsilon_{\text{high}})$
- **Detach operation**: The clipped weights do not participate in gradient computation and serve as constant coefficients
- Gradients come from the $\log \pi_\theta(a_t|s_t)$ term, ensuring all tokens contribute gradients
## Implementation Details
The pseudo-code implementation of CISPO is as follows:
```python
log_ratio = per_token_logps - old_per_token_logps
importance_weights = torch.exp(log_ratio) # r_t(θ) = π_θ / π_θ_old
clamped_ratios = torch.clamp(importance_weights, max=epsilon_high).detach()
per_token_loss = -clamped_ratios * advantages.unsqueeze(1) * per_token_logps
```
## Parameter Configuration
CISPO training can be enabled based on `GRPOTrainer` by setting the following parameters:
```bash
--loss_type cispo
--epsilon_high 5.0
```
> Compared to other algorithms, cispo generally uses a larger value for epsilon_high. The minimax paper does not provide specific parameter settings; the value used here refers to the experimental setup in the paper [ScaleRL](https://arxiv.org/pdf/2510.13786).
For other training parameters, refer to the [GRPO parameter documentation](../../Command-line-parameters.md#grpo-arguments).
docs/source_en/Instruction/GRPO/AdvancedResearch/DAPO.md
+89
View File
@@ -0,0 +1,89 @@
# DAPO: An Open-Source LLM Reinforcement Learning System at Scale
[Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO)](https://arxiv.org/abs/2503.14476) introduces several tricks based on GRPO, including:
- [Clip Higher](#clip-higher)
- [Dynamic Sampling](#dynamic-sampling)
- [Token level Loss](#token-level-loss)
- [Overlong Filtering](#overlong-filtering)
- [Soft Overlong Punishment](#soft-overlong-punishment)
## Clip Higher
PPO and GRPO use symmetric clipping ranges (e.g., ±0.2) to limit the magnitude of policy updates. While this ensures stability, it also restricts the model's exploratory capabilities. Specifically, when certain tokens have extremely low probabilities under the old policy, even if the current gradient indicates they should be reinforced (A > 0), the maximum increase is strictly limited.
DAPO employs an asymmetric clipping range, raising the upper clipping limit to encourage exploration:
- The upper bound (encouragement side) is relaxed to 0.28.
- The lower bound (suppression side) remains unchanged at 0.2.
In GRPO, the default symmetric clipping range is set using `epsilon`.
Parameters:
- `epsilon_high` sets the upper clipping range, while `epsilon` serves as the lower clipping range.
## Dynamic Sampling
GRPO samples multiple responses per question to compute inter-group advantages:
$$
\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
$$
However, when all generated outputs {o_i} receive the same reward, the inter-group advantage becomes zero, leading to vanishing gradients and reduced training efficiency.
DAPO addresses this issue with a dynamic sampling strategy:
- Skips data with zero inter-group reward standard deviation during sampling.
- Continues generating samples until the batch is filled.
Parameters:
- `dynamic_sample true` enables dynamic sampling.
- `max_resample_times` sets the maximum number of resampling attempts.
## Token level Loss
GRPO normalizes losses at the sentence level, which introduces bias based on response length.
DAPO uses token-level normalization to avoid this bias in loss calculation.
Parameters:
- `loss_type bnpo/dapo` enables token-level normalization.
> For the loss_type formula, please refer to the [documentation](../DeveloperGuide/loss_types.md).
## Overlong Filtering
DAPO argues that forcibly truncated responses contain high reward noise, making it difficult for the model to distinguish between quality issues and length issues. To address this, DAPO filters out truncated data during training, excluding it from loss computation.
Parameters:
- `overlong_filter` enables filtering of overly long samples.
## Soft Overlong Punishment
Language models often struggle with controlling output length:
- Overly long outputs may be truncated, leading to incorrect judgments of valid content.
- Unconstrained length generation affects practicality and computational efficiency.
DAPO designs a three-stage length penalty function:
$$
R_{\text{length}}(L) =
\begin{cases}
0, & L \leq L_{\text{max}} - L_{\text{cache}} \\[10pt]
\dfrac{(L_{\text{max}} - L_{\text{cache}}) - L}{L_{\text{cache}}}, & L_{\text{max}} - L_{\text{cache}} < L \leq L_{\text{max}} \\[10pt]
-1, & L > L_{\text{max}}
\end{cases}
$$
When the length falls within the interval $(L_{\text{max}} - L_{\text{cache}} < L \leq L_{\text{max}})$, a linearly increasing penalty is applied. For lengths $(L > L_{\text{max}})$, the maximum penalty (-1) is imposed.
Parameters:
- `reward_funcs soft_overlong` enables this reward function.
- `soft_max_length` sets L_max, which defaults to the model's maximum output length (`max_completion_length`).
- `soft_cache_length` sets L_cache.
## Parameter Settings
In summary, the following parameters can be set based on GRPOTrainer to implement DAPO training.
| Parameter | Type | Value |
|-----------------------|-----------|-------------|
| `--loss_type` | `str` | `bnpo`/`dapo`|
| `--epsilon_high` | `float` | `0.28` |
| `--dynamic_sample` | `bool` | `true` |
| `--max_resample_times`| `int` | `3` |
| `--overlong_filter` | `bool` | `true` |
| `--reward_funcs` | `str` | `soft_overlong`|
| `--soft_cache_length` | `int` | `4096` |
docs/source_en/Instruction/GRPO/AdvancedResearch/FIPO.md
+55
View File
@@ -0,0 +1,55 @@
# FIPO: Future-KL Influenced Policy Optimization
Author: [li2zhi](https://github.com/li2zhi)
[FIPO](https://arxiv.org/abs/2603.19835) is a value-free RL method for eliciting longer and deeper reasoning. It keeps the GRPO/DAPO training scaffold, but changes how token-level policy updates are weighted: instead of applying one sequence-level advantage uniformly to every token, FIPO uses a discounted Future-KL signal to estimate whether the future trajectory after each token is being reinforced or suppressed.
## Core Idea
In GRPO/DAPO, tokens in the same response usually share the same sequence-level advantage:
$$
\hat{A}_{i,t} = \hat{A}_{i}
$$
This is simple and stable, but the credit assignment is coarse. FIPO starts from the signed log-probability shift between the current policy and the old policy:
$$
\Delta \log p_t = \log \pi_\theta(y_t \mid x, y_{<t}) -
\log \pi_{\mathrm{old}}(y_t \mid x, y_{<t})
$$
A positive value means the token probability is being increased by the current update, while a negative value means it is being suppressed. FIPO then accumulates this signal from the current token to the end of the response:
$$
\mathrm{FutureKL}_t =
\sum_{k=t}^{T}\gamma^{k-t} M_k \Delta \log p_k
$$
where $M_k$ is the completion mask and $\gamma = 2^{-1 / \text{decay\_rate}}$. A larger `decay_rate` gives farther future tokens more influence; a smaller value makes the signal more local. FIPO maps the Future-KL value into a bounded influence weight:
$$
f_t = \mathrm{clip}(\exp(\mathrm{FutureKL}_t), 1-\epsilon_f, 1+\epsilon_f)
$$
The original advantage is then replaced by a future-aware advantage:
$$
\tilde{A}_{i,t} = \hat{A}_{i} \cdot f_{i,t}
$$
## Parameters
| Parameter | Type | Default | Description |
| ------------------------- | ------- | ------- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--loss_type` | `str` | `grpo` | Set to`fipo` to enable FIPO loss |
| `--delta` | `float` | `None` | When enabled, it is used for both Future-KL high-IS-ratio token filtering and the main-loss dual-clip upper bound, and should be greater than `1 + epsilon_high`. Set it to `10.0` to match the official 32B script |
| `--fipo_decay_rate` | `float` | `32.0` | Half-life parameter for Future-KL; the actual discount is`2 ** (-1 / fipo_decay_rate)` |
| `--fipo_clip_range` | `float` | `0.2` | Influence weight clipping range;`0.2` clips to `[0.8, 1.2]` |
| `--fipo_clip_high_only` | `bool` | `true` | If`true`, clips the weight to `[1.0, 1.0 + fipo_clip_range]` |
| `--fipo_safety_threshold` | `float` | `4.0` | Caps the FIPO weight to `[0.8, 1.0]` for negative-advantage tokens whose IS ratio exceeds this threshold |
## Training Example
[swift](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/fipo.sh)
docs/source_en/Instruction/GRPO/AdvancedResearch/GSPO.md
+66
View File
@@ -0,0 +1,66 @@
# Group Sequence Policy Optimization (GSPO)
In [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071), it is pointed out that GRPO computes importance sampling weights at the token level. However, this approach is problematic: since each token is only sampled once, it cannot realize effective distribution correction, and instead introduces high-variance noise during training, which can easily lead to unstable gradient estimates and even training collapse. Therefore, the paper argues that the unit of the objective function should be consistent with that of the reward. Since the reward is typically given at the sequence level (i.e., for the entire generated response), it is more reasonable to perform off-policy correction and optimization at the sequence level rather than the token level.
Below are the three main strategies for computing importance sampling weights:
1. GRPO
GRPO computes the importance sampling ratio independently for each token, as follows:
$$
w^{\mathrm{GRPO}}_{i,t} = \frac{\pi_\theta (y_{i, t} \mid x, y_{i, <t})}{\pi_{\theta_{\mathrm{old}}} (y_{i, t} \mid x, y_{i, <t})}
$$
2. GSPO (Sequence-Level)
GSPO calculates the importance sampling ratio at the sequence level, given by:
$$
w^{\mathrm{GSPO}}_{i} = \left[ \frac{\pi_\theta (y_i \mid x)}{\pi_{\theta_{\mathrm{old}}} (y_i \mid x)} \right]^{\frac{1}{|y_i|}}
= \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta (y_{i, t} \mid x, y_{i, <t})}{\pi_{\theta_{\mathrm{old}}} (y_{i, t} \mid x, y_{i, <t})} \right)
$$
3. GSPO-token
GSPO-token combines both sequence-level and token-level importance sampling:
$$
w_{i, t}^{\mathrm{GSPO-token}} = \mathrm{sg}\left[w_i^{\mathrm{GSPO}}\right] \cdot \frac{\pi_{\theta}(y_{i, t} \mid x, y_{i, < t})}{\mathrm{sg}\left[\pi_{\theta}(y_{i, t} \mid x, y_{i, < t})\right]}
$$
where $\mathrm{sg}[\cdot]$ denotes stop-gradient (detach()).
> **NOTE:** According to gradient analysis (i.e., Eqs. (11) and (18) in the paper), when the advantage for each token is identical, GSPO-token is equivalent to GSPO. In the current implementation of GRPO, all token advantages are normalized based on the sentence-level reward within each group. Therefore, in this setting, GSPO-token and GSPO are theoretically equivalent. However, GSPO-token provides support for future fine-grained (token-level) advantages.
Pseudo-code implementation:
```python
log_ratio = per_token_logps - old_per_token_logps
# GRPO
log_importance_weights = log_ratio
# GSPO (Sequence-Level)
seq_weight = (log_ratio * mask).sum(-1) / mask.sum(-1)
log_importance_weights = seq_weight.unsqueeze(-1) # (B,1)
# GSPO-token
seq_weight = (log_ratio * mask).sum(-1) / mask.sum(-1)
log_importance_weights = seq_weight.detach().unsqueeze(-1) + (per_token_logps - per_token_logps.detach())
importance_weights = torch.exp(log_importance_weights)
```
Based on GRPO training, you can select different algorithms via the `--importance_sampling_level` argument:
- `importance_sampling_level token` (default, GRPO implementation)
- `importance_sampling_level sequence` (GSPO)
- `importance_sampling_level sequence_token` (GSPO-token)
Other hyperparameters in the paper
```bash
--epsilon 3e-4 # from paper section 5.1
--epsilon_high 4e-4 # from paper section 5.1
--gradient_accumulation_steps 8
--steps_per_generation 32 # from paper section 5.1 (each batch of rollout data is partitioned into four minibatches for gradient updates)
--beta 0 # zero kl regularization https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306
```
For training, you can refer to [this script](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/gspo.sh).
docs/source_en/Instruction/GRPO/AdvancedResearch/REAL.md
+93
View File
@@ -0,0 +1,93 @@
# Rewards as Labels: Revisiting RLVR from a Classification Perspective
Author: [li2zhi](https://github.com/li2zhi)
[Rewards as Labels: Revisiting RLVR from a Classification Perspective](https://arxiv.org/abs/2602.05630) proposes a reformulation of GRPO by treating rewards as labels and performing **in-group classification** instead of advantage estimation. This converts the policy optimization problem into a classification problem, thereby addressing two key issues in the GRPO loss:
- **Gradient Misassignment** for positive samples
- **Gradient Domination** for negative samples
## Background and Motivation
GRPO Objective
$$
J_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{q,o\sim\pi_{\mathrm{od}}(\cdot|q)}\left[\frac{1}{|o|}\sum_{t=1}^{|o|}\left(\min\left(\rho_tA_t,\mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t\right)\right)\right]
$$
where:
- $\rho_t = \frac{\pi_\theta(o_t|q)}{\pi_{\mathrm{old}}(o_t|q)}$ is the probability ratio
- $A_t$ is the advantage function
The corresponding gradient is:
$$
\nabla_{\theta} J_{\mathrm{GRPO}} = \mathbb { E } \left[ \frac { 1 } { | o | } \sum _ { t = 1 } ^ { | o | } \mathbb { I } _ { \mathrm { clip } } \cdot A _ { t } e ^ { s _ { t } } \nabla _ { \theta } \log \pi _ { \theta } \left( o _ { t } | q \right) \right]
$$
where:
- $s_t = \log \frac{\pi_\theta(o_t|q)}{\pi_{\mathrm{old}}(o_t|q)}$ is the relative log-probability
- $\mathbb{I}_{\mathrm{clip}}$ is the clipping indicator
Thus, the per-token gradient weight in GRPO is:
$$
|\mathcal{W}_{\mathrm{GRPO}}|=\left\{ \begin{array} {ll}\left|A\cdot e^s\right|, & \mathrm{if~}\mathbb{I}_{\mathrm{clip}}=1, \\ 0, & \text{otherwise.} \end{array}\right.
$$
![Gradient magnitude visualizations in GRPO](../../../../resources/real.png)
1. **Gradient Misassignment (Positive Samples)**：
For positive samples, as the relative log-probability $s$ decreases, the gradient magnitude also decreases.
This is counterintuitive: tokens that the model is less confident about but correct should receive larger updates. However, GRPO assigns more weight to already confident tokens, causing under-trained tokens to receive insufficient learning signal.
2. **Gradient Domination (Negative Samples)**：
For negative samples, as $s$ decreases, the gradient magnitude increases exponentially.
This leads to a situation where a few overconfident incorrect tokens dominate the gradient, overwhelming other negative signals within the same group. Due to the absence of an upper bound, this may result in unstable and excessively large parameter updates.
To address the above issues, REAL treats rewards directly as labels and performs **group-wise classification training**.
![Real Framework](../../../../resources/real_framework.png)
The classification logit for each sample is defined as:
$$
\bar{s}^k=\frac{1}{|o^k|}\sum_{t=1}^{|o^k|}\left(\log\frac{\pi_\theta(o_t^k\mid q)}{\pi_{\mathrm{old}}(o_t^k\mid q)}\right)
$$
- $\bar{s}^k > 0$: The sample is more likely under the current policy than the old policy → the model tends to **promote** this sample
- $\bar{s}^k < 0$: The sample is less likely under the current policy → the model tends to **suppress** this sample
Loss Function
$$
\mathcal{L}_{REAL}=\log\left(1+\sum_{\mathcal{O}_+}e^{-\bar{s}^i/\tau}\right)+\log\left(1+\sum_{\mathcal{O}_-}e^{\bar{s}^j/\tau}\right)
$$
Gradient Properties
$$
|\mathcal{W}_{\mathrm{REAL}}|=
\begin{cases}
\frac{1}{\tau}\frac{1}{1+C_{+}e^{\bar{s}^{k}/\tau}}, & r=1 \\
\\
\frac{1}{\tau}\frac{1}{1+C_{-}e^{-\bar{s}^{k}/\tau}}, & r=0 & & &
\end{cases}
$$
## Parameter Settings
| Parameter | Type | Default | Description |
|-----------|------|---------|--------------------------------------------------------------------|
| `--loss_type` | `str` | - | Set to `real` |
| `--real_tau` | `float` | `0.5` | Temperature parameter controlling decision boundary sharpness |
Training Script Reference
[swift](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/real.sh)
## Important Notes
When configuring training parameters, ensure that:
- `per_device_train_batch_size` is divisible by `num_generations`
This guarantees that each training batch contains complete groups, which is required for correct in-group classification.
docs/source_en/Instruction/GRPO/AdvancedResearch/REINFORCEPP.md
+86
View File
@@ -0,0 +1,86 @@
# REINFORCE++: An Efficient RLHF Algorithm with Robustness to Both Prompt and Reward Models
[REINFORCE++ Baseline](https://arxiv.org/abs/2501.03262) is a simplified version of the REINFORCE++ algorithm, designed for outcome rewards (response-level scalar rewards). Similar to GRPO, it samples multiple model outputs for each prompt and uses an intra-group baseline to estimate advantages. The key difference lies in the statistics used for normalization.
## Algorithm Overview
For clarity, we explain REINFORCE++ Baseline by contrasting it with GRPO (Group Relative Policy Optimization).
Both GRPO and REINFORCE++ Baseline estimate advantages via intra-group comparisons. Their main differences are:
### Difference 1: Statistics Used for Normalization
**GRPO (Group Relative Policy Optimization)**
For each prompt, GRPO generates $G$ response samples and normalizes using the **mean and standard deviation of all samples within the group**:
$$
\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
$$
When `scale_rewards='batch'` is set, it uses the **batch-level std of original rewards**:
$$
\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^{N})}
$$
where $N$ is the total number of samples in the batch.
**REINFORCE++ Baseline**
For each prompt, REINFORCE++ generates $G$ response samples, subtracts the group mean, and then normalizes using the **standard deviation of the group-mean-subtracted rewards**:
$$
\begin{align}
\tilde{A}_{i} &= R_i - \text{mean}(\{R_j\}_{j=1}^G) \\
\hat{A}_{i} &= \frac{\tilde{A}_{i}}{\text{std}(\{\tilde{A}_k\}_{k=1}^{N})}
\end{align}
$$
where $N$ is the total number of samples in the batch.
**Key Difference**:
- **GRPO**: Uses the std of **original rewards $R$** for normalization
- **REINFORCE++**: Uses the std of **group-mean-subtracted rewards $\tilde{A}$** for normalization
### Difference 2: KL Divergence Regularization
Similar to RLOO, REINFORCE++ Baseline integrates KL divergence directly into the reward:
$$
R'_i = R_i - \beta \cdot \text{KL}(\pi_\theta || \pi_{\text{ref}})
$$
where $\beta$ is the KL divergence weight coefficient (corresponding to the parameter `beta`), and $\pi_{\text{ref}}$ is the reference policy.
## Parameter Configuration
We can implement REINFORCE++ Baseline training by configuring the following parameters with `GRPOTrainer`:
```bash
--advantage_estimator reinforce_plus_plus
--scale_rewards batch
--kl_in_reward true
```
For training examples, please refer to this [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/reinforce_plus_plus.sh)
### Key Parameter Descriptions
- **`--advantage_estimator`**: Selects the advantage estimation method
- `grpo` (default): Uses the std of original rewards for normalization
- `reinforce_plus_plus`: Uses the std of group-mean-subtracted rewards for normalization
- **`--kl_in_reward`**: Controls where the KL divergence regularization term is applied
- `false`: KL divergence is an independent regularization term in the loss function (GRPO default)
- `true`: KL divergence is subtracted directly from the reward (REINFORCE++ original implementation)
- **`--scale_rewards`**: Controls the normalization method
- `group` (default): Intra-group normalization
- `batch`: Global batch-level normalization (REINFORCE++ original implementation)
- `none`: No normalization
- **`--num_generations`**: Number of samples generated per prompt ($G$)
- **`--beta`**: KL divergence regularization coefficient ($\beta$)
For other parameters, please refer to [GRPO Parameters](../../Command-line-parameters.md#grpo-arguments)
docs/source_en/Instruction/GRPO/AdvancedResearch/RLOO.md
+93
View File
@@ -0,0 +1,93 @@
# REINFORCE Leave-One-Out (RLOO)
[REINFORCE Leave-One-Out (RLOO)](https://arxiv.org/abs/2402.14740) is a reinforcement learning algorithm based on the classic REINFORCE policy-gradient method. It constructs an unbiased advantage baseline via the Leave-One-Out (LOO) technique.
## Algorithm Overview
For clarity, we explain RLOO by contrasting it with GRPO (Group Relative Policy Optimization).
Both GRPO and RLOO estimate advantages via intra-group comparisons to avoid the high variance of a global baseline. Their core differences are mainly in the following aspects:
### Difference 1: How the Advantage Baseline Is Constructed
**1. GRPO (Group Relative Policy Optimization)**
For each prompt, GRPO generates $G$ response samples and normalizes rewards using the group mean and standard deviation:
$$
\hat{A}_{i} = \frac{R_i - \text{mean}(\{R_j\}_{j=1}^G)}{\text{std}(\{R_j\}_{j=1}^G)}
$$
Where:
- $R_i$ is the reward of the $i$-th sample
- $\text{mean}(\{R_j\}_{j=1}^G) = \frac{1}{G}\sum_{j=1}^G R_j$ is the group mean
- $\text{std}(\{R_j\}_{j=1}^G)$ is the group standard deviation
**2. RLOO (REINFORCE Leave-One-Out)**
For each prompt, RLOO generates $K$ response samples and constructs the baseline via Leave-One-Out, i.e., for the $i$-th sample, the baseline is the mean of the other $K-1$ samples:
$$
\hat{A}_{i} = R_i - \frac{1}{K-1}\sum_{j \neq i} R_j
$$
This can be equivalently rewritten as:
$$
\hat{A}_{i} = \frac{K}{K-1} \left(R_i - \bar{R}\right)
$$
where $\bar{R} = \frac{1}{K}\sum_{j=1}^K R_j$ is the group mean reward.
> Note: We use $K$ here to match the notation in the paper. It has the same meaning as $G$ in GRPO and corresponds to the configuration parameter `num_generations`.
**Why Leave-One-Out?**
The key advantage is unbiasedness. For the $i$-th sample, its reward $R_i$ is independent of the baseline $\frac{1}{K-1}\sum_{j \neq i} R_j$, hence the advantage estimate is unbiased. In contrast, using the mean including itself as the baseline introduces bias.
### Difference 2: How KL Regularization Is Applied
To prevent the policy from drifting too far from the reference policy, both algorithms introduce KL divergence regularization, but in different ways:
**GRPO**: Adds KL divergence as an independent regularization term to the [loss](../GetStarted/GRPO.md#algorithm-overview):
$$
\mathcal{L}(\theta) = -\mathbb{E}\left[\hat{A}_i \log \pi_\theta(a_i|s_i)\right] + \beta \cdot \text{KL}(\pi_\theta \Vert \pi_{\text{ref}})
$$
**RLOO**: Integrates KL divergence directly into the reward, constructing a modified reward:
$$
R'_i = R_i - \beta \cdot \text{KL}(\pi_\theta \Vert \pi_{\text{ref}})
$$
where $\beta$ is the KL coefficient (parameter `beta`), and $\pi_{\text{ref}}$ is the reference policy (typically an SFT model or the initial policy).
## Parameter Configuration
RLOO training can be enabled based on `GRPOTrainer` by setting the following parameters:
```bash
# Basic RLOO configuration
--advantage_estimator rloo # Use RLOO's leave-one-out advantage estimator
--kl_in_reward true # Integrate KL divergence into the reward (default for RLOO)
```
You can refer to this [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/rloo.sh) for training.
### Important Parameters
- **`--advantage_estimator`**: Choose the advantage estimator
- `grpo` (default): standardize using group mean and standard deviation
- `rloo`: construct the baseline via Leave-One-Out
- **`--kl_in_reward`**: Controls where the KL term is applied
- `false`: KL as a separate regularization term in the loss (GRPO style)
- `true`: subtract KL directly from the reward to form a modified reward (RLOO style)
- **`--num_generations`**: Number of samples per prompt, i.e., $K$
- **`--beta`**: KL regularization coefficient $\beta$
- Controls how conservatively the policy updates
Other parameters are consistent with the [GRPO arguments](../../Command-line-parameters.md#grpo-arguments).
docs/source_en/Instruction/GRPO/AdvancedResearch/SAPO.md
+91
View File
@@ -0,0 +1,91 @@
# Soft Adaptive Policy Optimization (SAPO)
[Soft Adaptive Policy Optimization (SAPO)](https://arxiv.org/abs/2511.20347) addresses the issues caused by hard clipping in GRPO by proposing a temperature-controlled soft gate mechanism that smoothly attenuates off-policy updates while preserving useful learning signals.
## Background and Motivation
When training LLMs with reinforcement learning, GRPO handles off-policy training by computing token-level importance sampling ratios:
$$
r_t = \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\mathrm{old}}}(y_t|x, y_{<t})}
$$
However, token-level importance sampling ratios often exhibit high variance, which can be exacerbated in the following cases:
- **Long text generation**
- **MoE model routing heterogeneity**: The old-policy model during sampling and the training model may use different expert routing, significantly amplifying logps differences
To address this, GRPO uses hard clipping to limit the magnitude of policy updates:
$$
L^{\mathrm{GRPO}} = -\min\left( r_t \cdot A, \mathrm{clip}(r_t, 1-\epsilon, 1+\epsilon) \cdot A \right)
$$
**The Dilemma of Hard Clipping**: Hard clipping struggles to balance stability and learning efficiency—too strict clipping limits the number of effective samples, while too loose clipping introduces noisy gradients from off-policy samples, leading to training instability.
## SAPO Method
SAPO uses a temperature-controlled sigmoid soft gate function to replace hard clipping, achieving smooth gradient attenuation.
### Soft Gate Function
The core of SAPO is using the sigmoid function to apply soft gating on the importance sampling ratio:
For positive advantages ($A > 0$), use positive gating:
$$
g^{+}_t = \sigma\left( \tau_{\mathrm{pos}} \cdot (r_t - 1) \right) \cdot \frac{4}{\tau_{\mathrm{pos}}}
$$
For negative advantages ($A < 0$), use negative gating:
$$
g^{-}_t = \sigma\left( \tau_{\mathrm{neg}} \cdot (r_t - 1) \right) \cdot \frac{4}{\tau_{\mathrm{neg}}}
$$
where:
- $\sigma(\cdot)$ is the sigmoid function
- $\tau_{\mathrm{pos}}$ and $\tau_{\mathrm{neg}}$ are temperature parameters that control the gate function slope
- $r_t$ is the importance sampling ratio
### SAPO Loss Function
$$
L^{\mathrm{SAPO}} = -g_t \cdot A
$$
where $g_t = g^{+}_t$ when $A > 0$, $g_t = g^{-}_t$ when $A < 0$.
### Temperature Parameters
The temperature parameter $\tau$ controls the decay rate of the soft gate function—larger values result in faster decay.
![tau curve](../../../../resources/sapo_tau.png)
The paper points out that positive advantages increase the logit of sampled tokens while decreasing the logits of all unsampled tokens; negative advantages do the opposite, increasing the logits of many unsampled tokens, which may spread to a large number of irrelevant tokens and introduce instability. Therefore, the paper recommends setting $\tau_\text{neg} > \tau_\text{pos}$ to make the gradient decay faster for tokens with negative rewards, improving training stability and performance.
The paper recommends default values of $\tau_{\mathrm{pos}} = 1.0$ and $\tau_{\mathrm{neg}} = 1.05$.
## Parameter Settings
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--loss_type` | `str` | - | Set to `sapo` |
| `--tau_pos` | `float` | `1.0` | Temperature parameter for positive advantages, controls gate slope |
| `--tau_neg` | `float` | `1.05` | Temperature parameter for negative advantages, controls gate slope |
```bash
swift rlhf \
--rlhf_type grpo \
--loss_type sapo \
--tau_pos 1.0 \
--tau_neg 1.05 \
# ... other parameters
```
Example training scripts:
- [swift](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/sapo.sh)
- [megatron swift](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/sapo.sh)
> The soft gate mechanism of SAPO only takes effect during off-policy training.
> The importance sampling granularity in SAPO is at the token level (i.e., importance_sampling_level defaults to token), which conflicts with GSPO.
docs/source_en/Instruction/GRPO/AdvancedResearch/deepeyes.md
+93
View File
@@ -0,0 +1,93 @@
# DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
## Principle Introduction
The [DeepEyes paper](https://arxiv.org/abs/2505.14362) proposes a method that enables models to "think with images" (image-assisted reasoning) by leveraging reinforcement learning. This approach achieves emergent model capabilities through end-to-end reinforcement learning, without requiring additional SFT (Supervised Fine-Tuning) steps. The model is equipped with built-in image localization capabilities and can actively invoke an "image zoom-in tool": during inference, the model automatically selects specific regions within an image for zooming and cropping, and then chains the processed region information into downstream reasoning, thus achieving multi-step visual-text reasoning.
![DeepEyes Overview](../../../../resources/deepeyes.png)
## Best Practices
**Dataset Download and Registration**
Download the official DeepEyes training datasets locally:
```bash
# modelscope
modelscope download --dataset Lixiang/ChenShawn-DeepEyes-Datasets-47k
# huggingface
huggingface-cli download ChenShawn/DeepEyes-Datasets-47k --repo-type=dataset
```
There are three parquet files in the dataset. Register them in the `swift/dataset/data/dataset_info.json` file. For each, rename the `prompt` column to `messages`:
```json
{
"ms_dataset_id": "path/to/data_0.1.2_visual_toolbox_v2.parquet",
"columns": {
"prompt": "messages"
}
},
{
"ms_dataset_id": "path/to/data/data_thinklite_reasoning_acc.parquet",
"columns": {
"prompt": "messages"
}
},
{
"ms_dataset_id": "path/to/data/data_v0.8_visual_toolbox_v2.parquet",
"columns": {
"prompt": "messages"
}
}
```
For registering the reward function and tool call logic locally as used in the paper, you can refer to the [DeepEyes implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py).
**Deploying the Evaluation Model**
The reward function in DeepEyes relies on a generative reward model to compare model outputs with ground-truth answers. For efficiency, it is recommended to deploy the reward model as a service.
Assuming you use the Qwen2.5-VL-72B-Instruct model for evaluation, refer to the following deployment command:
```bash
# 4*80G
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift deploy \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--infer_backend vllm \
--vllm_tensor_parallel_size 4 \
```
In the plugin file, you can use the OpenAI interface to call the model; see the [Reward Model documentation](../DeveloperGuide/reward_model.md#external-deployment).
The complete training script can be found at https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes.sh
## Implementation Details
The [DeepEyes implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py), which references the [official implementation](https://github.com/Visual-Agent/DeepEyes/blob/main/verl/utils/reward_score/vl_agent.py), provides sample code for a DeepEyes training plugin, covering the logic of the reward function and multi-turn interaction calls.
**Dataset Info** is shown below:
| Dataset File Name | data_source | Scoring Function | Tool Call |
|------------------------------------------|-----------------------|-----------------------------------|------------------------|
| data_v0.8_visual_toolbox_v2.parquet | chart | vl_agent.compute_score | True (image_zoom_in_tool) |
| data_0.1.2_visual_toolbox_v2.parquet | vstar | vl_agent.compute_score | True (image_zoom_in_tool) |
| data_thinklite_reasoning_acc.parquet | thinklite_eureka | vl_agent.compute_score_math | False |
**Note**: When processing image inputs, multimodal large models may perform preprocessing (such as cropping or resizing limited by the `max_pixels` parameter). When using the image zoom-in tool (`image_zoom_in_tool`), the model will output the cropped bbox based on the input image. Therefore, it is necessary to ensure the image fed into the tool has already been preprocessed. The following example shows how this is implemented in the Qwen2.5-VL series:
```python
from qwen_vl_utils import fetch_image
# At this point, images have not yet been preprocessed
infer_request.images
# Load as PIL.Image and apply cropping (same as using the MAX_PIXELS environment variable)
img = fetch_image({'image': load_pil_image(infer_request.images[0])})
```
**Tool Reward**
According to the paper, if the final answer is correct and the trajectory uses at least one tool, a tool reward is given. To prevent the model from generating invalid tool calls, we determine this based on the number of images, not just on the presence of `<tool_call>` tokens.
```python
tool_reward = 1.0 if num_image > 1 and acc_reward > 0.5 else 0.0
```
docs/source_en/Instruction/GRPO/AdvancedResearch/entropy_mask.md
+28
View File
@@ -0,0 +1,28 @@
# Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
The [paper](https://arxiv.org/abs/2506.01939) finds that when training large language models for reasoning abilities with methods such as RLVR, the key to learning progress lies in a small fraction of high-entropy "minority tokens," rather than the majority of low-entropy tokens.
The paper demonstrates that within the token distribution during model reasoning, only a few high-entropy tokens play a dominant role. These tokens typically appear at critical junctures where the reasoning or decision path diverges the most (e.g., tokens like "wait," "since," etc.), determining whether the model can master complex reasoning tasks. In contrast, most low-entropy tokens contribute little to the model's reasoning ability. The paper proposes computing policy gradients exclusively on high-entropy tokens, discarding gradients for low-entropy tokens.
The formula for token entropy is as follows:
$
H_t := -\sum_{j=1}^{V} p_{t,j} \log p_{t,j}, \qquad \text{where } (p_{t,1}, \cdots, p_{t,V}) = \mathbf{p}_t = \pi_\theta(\cdot | \mathbf{q}, \mathbf{o}_{<t}) = \text{Softmax}\left(\frac{\mathbf{z}_t}{T}\right)
$
Where:
- $\pi_\theta$: The model parameterized by $\theta$;
- $\mathbf{q}$: The input query;
- $\mathbf{o}_{<t} = (o_1, o_2, \cdots, o_{t-1})$: The sequence of tokens generated prior to timestep $t$;
- $V$: Vocabulary size;
- $\mathbf{z}_t \in \mathbb{R}^V$: The pre-softmax logits at timestep $t$;
- $\mathbf{p}_t \in \mathbb{R}^V$: The model's output probability distribution over the vocabulary;
- $T \in \mathbb{R}$: The decoding temperature, controlling the smoothness of the distribution.
Object of entropy computation: $H_t$ is the entropy of the token generation distribution $\mathbf{p}_t$, which measures the uncertainty in the policy $\pi_\theta$ under the given context $(\mathbf{q}, \mathbf{o}_{<t})$.
> "Token entropy" $H_t$ always refers to the uncertainty of the generation distribution $\mathbf{p}_t$ at position $t$, rather than a property of the token $o_t$ itself. In other words, $H_t$ is the entropy of the distribution $\mathbf{p}_t$ at position $t$, and is independent of the sampled token $o_t$.
In practice, during GRPO training, the top_entropy_quantile parameter can be used to control the percentile threshold for entropy filtering. In the experiments from the paper, this parameter is set to 0.2, meaning that only the top 20% of tokens (with the highest entropy) at each sequence position are used for optimization in each batch.
By setting the parameter `log_entropy`, you can record the changes in entropy during training; see the [documentation](../GetStarted/GRPO.md#logged-metrics) for reference.
docs/source_en/Instruction/GRPO/AdvancedResearch/index.rst
+19
View File
@@ -0,0 +1,19 @@
Advanced Research
===============
.. toctree::
:maxdepth: 1
entropy_mask.md
CISPO.md
DAPO.md
deepeyes.md
FIPO.md
GSPO.md
CHORD.md
RLOO.md
REINFORCEPP.md
REAL.md
router_replay.md
SAPO.md
training_inference_mismatch.md
treepo.md
docs/source_en/Instruction/GRPO/AdvancedResearch/router_replay.md
+104
View File
@@ -0,0 +1,104 @@
# Router Replay (R2/R3)
**TL;DR**: In RL training of MoE models, routing inconsistency between the training engine and the inference engine can significantly amplify training-inference mismatch, and even cause training collapse. Router Replay eliminates this inconsistency by replaying fixed routing masks during the training forward pass. Depending on the replay source, there are two strategies: R2 (Vanilla Routing Replay) and R3 (Rollout Routing Replay).
## Background
### Three Policies in MoE RL
In GRPO training of MoE models, there are three distinct policy stages that share the same model weights but may differ in routing behavior:
| Policy | Notation | Routing Result | Description |
|--------|----------|---------------|-------------|
| **Training Policy** | $\pi_\theta$ | $e^{\pi}_t$ | The model during gradient updates |
| **Old Policy** | $\pi_{\theta_{\text{old}}}$ | $e^{\pi}_{\text{old},t}$ | The model state before batch updates |
| **Rollout Policy** | $\mu_{\theta_{\text{old}}}$ | $e^{\mu}_{\text{old},t}$ | The sampling policy in the inference engine (e.g., vLLM), with the same weights as old policy, but different routing due to kernel implementation differences, precision, etc. |
Here, $\pi_{\theta_{\text{old}}}$ and $\mu_{\theta_{\text{old}}}$ have identical weights at sampling time, but due to implementation differences between the inference and training engines (e.g., operator implementations), routing results may differ even for the same input.
### Decomposition of Training-Inference Mismatch
According to the [paper](https://arxiv.org/abs/2507.18071), the token-level importance sampling ratio can be decomposed into two factors:
$$
\frac{\pi_\theta(y_t|x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t|x, y_{<t})} = \underbrace{\frac{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}{\mu_{\theta_{\text{old}}}(y_t|x, y_{<t})}}_{\text{training-inference discrepancy}} \times \underbrace{\frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}}_{\text{policy staleness}}
$$
For MoE models, expert routing is deeply coupled with both factors:
- **Training-inference discrepancy**: Routing inconsistency between the training and inference engines ($e^{\pi}_{\text{old},t} \neq e^{\mu}_{\text{old},t}$) amplifies output divergence
- **Policy staleness**: As mini-batch updates proceed, routing also drifts ($e^{\pi}_t \neq e^{\pi}_{\text{old},t}$), further deviating from the sampling policy
## R2: Vanilla Routing Replay
The core idea of R2 is to **replay the routing determined by the old policy in the training engine** ($e^{\pi}_{\text{old},t}$) during gradient updates.
### Principle
During the training forward pass, first run a forward pass with the old policy weights to record the expert indices selected by each MoE Router layer, then force the training model $\pi_\theta$ to use these indices in its forward pass:
$$
g_{\text{replay},i} = \frac{I^{\pi}_{\text{old},i} \cdot \exp(s_{\text{train},i})}{\sum_j I^{\pi}_{\text{old},j} \cdot \exp(s_{\text{train},j})}
$$
where $I^{\pi}_{\text{old}}$ is the old policy's routing mask and $s_{\text{train}}$ is the router logits computed by the training model. The softmax still operates on the training logits, so gradients can flow back to the router weights normally.
### Properties
| Scenario | Behavior |
|----------|----------|
| **First mini-batch** (on-policy) | $\theta = \theta_{\text{old}}$, so $e^{\pi}_t = e^{\pi}_{\text{old},t}$, **target policy unchanged** (no bias) |
| **Subsequent mini-batches** (off-policy) | $\theta \neq \theta_{\text{old}}$, so $e^{\pi}_t \neq e^{\pi}_{\text{old},t}$, **target policy changed** (biased), but policy staleness is controlled |
## R3: Rollout Routing Replay
The core idea of R3 is to **replay the routing determined by the rollout policy in the inference engine** ($e^{\mu}_{\text{old},t}$) during the training forward pass.
### Principle
During sampling in the inference engine (e.g., vLLM), additionally record the expert indices (routing mask) for each token at every MoE Router layer, then pass these masks to the training engine and force $\pi_\theta$ to use them in its forward pass:
$$
g_{\text{replay},i} = \frac{I^{\mu}_{\text{old},i} \cdot \exp(s_{\text{train},i})}{\sum_j I^{\mu}_{\text{old},j} \cdot \exp(s_{\text{train},j})}
$$
### Compatibility with Other Methods
- R3 is **orthogonal** to **GSPO** and can be combined for further improvement
- R3 combined with **TIS** may not provide additional gains (R3 already eliminates inconsistency at the source; TIS's additional correction may be redundant)
- Router Replay and **Clipping** are both essential in off-policy training
## Router Mask Caching
The R3 paper also proposes that routing masks can be cached alongside the KV Cache: for the same prefix tokens, the MoE Router output is deterministic, so routing masks can be stored and reused together with the prefix KVCache. This is particularly important in multi-turn Agent scenarios (tool calling), avoiding the need to re-prefill the prefix to obtain routing masks, with an overall rollout latency overhead of less than 3%.
## Swift Implementation
### Parameters
Select the routing replay strategy via the `--router_replay_mode` parameter:
| Value | Description |
|-------|-------------|
| `disabled` (default) | No routing replay |
| `R2` | Vanilla Routing Replay: record old policy routing in the training engine and replay |
| `R3` | Rollout Routing Replay: export routing masks from the inference engine and replay in training |
Environment requirements:
- R3 requires vLLM ≥ 0.14.0 to support returning `routed_experts` information.
- Router Replay is currently only available with the Megatron backend, requiring megatron-core ≥ 0.16.0.
### Relationship with Training-Inference Correction
Router Replay and the importance sampling (IS) correction described in [Training-Inference Mismatch](training_inference_mismatch.md) are complementary:
- **IS correction**: Corrects probability divergence at the loss level via weighting
- **Router Replay**: Eliminates the source of divergence at the model architecture level by fixing routing
## References
1. [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://arxiv.org/abs/2510.11370)
2. [Group Sequence Policy Optimization](https://arxiv.org/abs/2507.18071)
3. [Stabilizing Reinforcement Learning with LLMs: Formulation and Practices](https://arxiv.org/abs/2512.01374)
4. [Megatron Core Router Replay Design Document](https://docs.nvidia.com/megatron-core/developer-guide/nightly/api-guide/router_replay.html)
docs/source_en/Instruction/GRPO/AdvancedResearch/training_inference_mismatch.md
+236
View File
@@ -0,0 +1,236 @@
# Training-Inference-Mismatch
**TL;DR**: While GRPO introduces vLLM to accelerate the sampling process, it also introduces Training-Inference Mismatch issues that may affect training stability. This document explains the background, causes, and solutions to this problem.
## Background
### Basic Assumptions of GRPO
The training objective of GRPO (Group Relative Policy Optimization) can be expressed as:
$$
\mathcal{L}_{\text{GRPO}} = - \mathbb{E}_{y \sim \pi_\theta} \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
$$
Where:
- $r_t(\theta) = \frac{\pi_\theta(y_t|x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t|x, y_{<t})}$ is the importance sampling ratio
- $\hat{A}_t$ is the advantage function, calculated based on reward and group baseline
- $\epsilon$ is the clipping parameter
**Core Assumption**: Samples $y$ are drawn from the policy $\pi_\theta$. In practice, this means:
1. The rollout model and the training model (policy model) should be **the same model** $\pi_\theta$
2. The probability distributions of both models should be **exactly identical**, i.e., $\pi_{\text{rollout}} = \pi_\theta$
### Deviation After Introducing vLLM
GRPO's training speed is largely constrained by the sampling process (rollout). To accelerate this, training frameworks introduce high-performance inference engines (such as vLLM) for sampling. The **ideal assumption** is that through weight synchronization, vLLM maintains consistency with the training model, i.e., $\pi_{\text{vLLM}} \equiv \pi_\theta$.
However, in practice, even with fully synchronized weights, due to differences in operator implementations, the probability distributions still deviate:
$$
\pi_{\text{vLLM}}(y|x) \neq \pi_\theta(y|x)
$$
At this point, the actual training objective becomes:
$$
\mathcal{L} = - \mathbb{E}_{y \sim \pi_{\text{vLLM}}} \left[ \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
$$
Where samples come from $\pi_{\text{vLLM}}$, but gradients are computed based on $\pi_\theta$. This **violates the algorithm's on-policy assumption**, introducing training-inference mismatch issues and potentially causing performance degradation.
## Solution
To address training-inference mismatch, we can introduce **Importance Sampling (IS)** correction mechanisms.
### Importance Sampling Correction
The basic idea of importance sampling is: when samples come from distribution $q$ rather than target distribution $p$, we can correct the expectation calculation by introducing weights:
$$
\mathbb{E}_{x \sim p} [f(x)] = \mathbb{E}_{x \sim q} \left[ \frac{p(x)}{q(x)} \cdot f(x) \right]
$$
Applied to the GRPO scenario, the corrected loss function is:
$$
\mathcal{L}_{\text{corrected}} = - \mathbb{E}_{y \sim \pi_{\text{vLLM}}} \left[ w(x, y) \cdot \min \left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]
$$
Where $w(x, y)$ is the importance sampling weight used to correct the distribution bias between vLLM and the training model.
Importance sampling weights can be computed and applied at different granularities:
1. **Token-Level**
Compute the importance sampling ratio at each token:
$$
w_{i,t}^{\text{token}} = \frac{\pi_\theta(y_{i,t}|x, y_{i,<t})}{\pi_{\text{vLLM}}(y_{i,t}|x, y_{i,<t})}
$$
2. **Sequence-Level**
Compute the sequence-level importance sampling ratio, then broadcast to each token:
$$
w_i^{\text{seq}} = \left[ \frac{\pi_\theta(y_i|x)}{\pi_{\text{vLLM}}(y_i|x)} \right]^{\frac{1}{|y_i|}} = \exp\left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{\pi_\theta(y_{i,t}|x, y_{i,<t})}{\pi_{\text{vLLM}}(y_{i,t}|x, y_{i,<t})} \right)
$$
### Stability Control: Truncate vs. Mask
Excessively large importance sampling weights can cause gradient explosion and destabilize training. Therefore, weight control is necessary:
#### 1. Truncate
Truncate the importance sampling weight to the $[0, \tau]$ interval:
$$
w_{\text{truncate}} = \min(w, \tau)
$$
This method retains all samples but limits their influence.
#### 2. Mask
Discard token/sequence data where weights exceed the threshold:
$$
w_{\text{mask}} = \begin{cases}
w & \text{if } w \leq \tau \\
0 & \text{otherwise}
\end{cases}
$$
### Four Correction Modes
Combining granularity and control strategies, there are four correction modes (selected via `--rollout_importance_sampling_mode` parameter):
| Mode | Description |
|------|-------------|
| `token_truncate` | Token-level truncation |
| `token_mask` | Token-level masking |
| `sequence_truncate` | Sequence-level truncation |
| `sequence_mask` | Sequence-level masking |
The threshold is set via the `--rollout_importance_sampling_threshold` parameter.
## Metrics
To monitor the degree of training-inference mismatch during training, we add the following metrics to the logs (prefixed with `rollout_correction/`):
### 1. KL Divergence
KL divergence measures the deviation between the rollout policy and the training policy. Both metrics estimate $\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta)$
**Direct estimator `kl`**:
$$
\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta) = \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \log \frac{\pi_{\text{vLLM}}}{\pi_\theta} \right]
$$
**K3 estimator `k3_kl`**:
$$
\text{KL}(\pi_{\text{vLLM}} \| \pi_\theta) \approx \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \rho - \log \rho - 1 \right], \quad \rho = \frac{\pi_\theta}{\pi_{\text{vLLM}}}
$$
The K3 estimator is more numerically stable when KL values are small and is always non-negative.
### 2. Perplexity (PPL)
Perplexity measures the model's prediction uncertainty for a sequence:
$$
\text{PPL} = \exp\left( -\frac{1}{|y|} \sum_{t=1}^{|y|} \log p(y_t) \right)
$$
Related metrics:
- `training_ppl` / `training_log_ppl`: Training policy PPL and its logarithm
- `rollout_ppl` / `rollout_log_ppl`: Rollout policy PPL and its logarithm
- `log_ppl_diff`: Log PPL difference, positive value means training policy assigns lower probability
- `log_ppl_abs_diff`: Absolute log PPL difference
- `log_ppl_diff_max` / `log_ppl_diff_min`: Max/min of log PPL difference
- `ppl_ratio`: PPL ratio $\frac{\text{PPL}_{\text{training}}}{\text{PPL}_{\text{rollout}}}$
### 3. χ² Divergence (Chi-squared Divergence)
χ² divergence measures the variance of importance sampling weights:
$$
\chi^2(\pi_\theta \| \pi_{\text{vLLM}}) = \mathbb{E}_{\pi_{\text{vLLM}}}\left[ \rho^2 \right] - 1, \quad \rho = \frac{\pi_\theta}{\pi_{\text{vLLM}}}
$$
- `chi2_token`: Token-level χ² divergence, $\mathbb{E}[\rho_t^2] - 1$
- `chi2_seq`: Sequence-level χ² divergence (geometric mean based), $\mathbb{E}[\rho_{\text{geo}}^2] - 1$, where $\rho_{\text{geo}} = \exp(\frac{1}{T}\sum_t \log \rho_t)$
Higher χ² divergence indicates larger IS weight variance and less stable training. `chi2_seq` uses geometric mean instead of product, making it comparable in scale to `chi2_token`.
### 4. Effective Sample Size (ESS)
Effective sample size measures the number of samples that actually contribute after importance sampling:
$$
\text{ESS} = \frac{1}{\mathbb{E}\left[\left(\frac{w}{\mathbb{E}[w]}\right)^2\right]}
$$
A larger ESS value (closer to 1) indicates more uniform importance sampling weight distribution and higher sample utilization efficiency. When all weights are equal (on-policy), ESS = 1; when weights differ significantly (severely off-policy), ESS becomes small.
### 5. IS Weight Statistics
- `is_weight_mean`: Average importance sampling weight, ideal value is 1.0
- `clipped_frac`: Fraction of samples that were truncated or masked
## Usage
### Logging Diagnostic Metrics Only
If you only want to monitor the degree of training-inference mismatch without enabling importance sampling correction, you can set:
```
--log_rollout_offpolicy_metrics true
```
This will log all diagnostic metrics (KL, PPL, χ², etc.) without modifying the loss function.
### Enabling Importance Sampling Correction
Enable the correction mechanism with the following parameters:
```
--rollout_importance_sampling_mode (default None)
--rollout_importance_sampling_threshold (default 2)
```
When `rollout_importance_sampling_mode` is set, diagnostic metrics are automatically logged without needing to set `log_rollout_offpolicy_metrics`.
## Off-Policy Sequence Masking
In addition to importance sampling correction, you can use **Off-Policy Sequence Masking** to address training-inference mismatch. This technique comes from the [DeepSeek-V3.2 paper](https://arxiv.org/abs/2512.02556).
### Principle
The core idea of Off-Policy Sequence Masking is: when the current policy deviates significantly from the old policy (rollout or old policy), directly discard (mask) that sequence from loss computation. This approach specifically targets **sequences with negative advantage**, as these are more likely to cause training instability when policy shift is large.
Specifically, for each sequence, compute:
$$
\delta_i = \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \bigl( \log \pi_{\text{old}}(y_{i,t}|x, y_{i,<t}) - \log \pi_\theta(y_{i,t}|x, y_{i,<t}) \bigr)
$$
Sequence $i$ will be masked when the following conditions are met (mean taken over completion tokens, i.e., where `completion_mask=1`):
1. $\delta_i > \tau$
2. **AND** $\hat{A}_i < 0$
Where:
- $\pi_{\text{old}}$ preferentially uses `rollout_per_token_logps` (logprobs from rollout/behavior policy); if unavailable, falls back to `old_per_token_logps`
- $\tau$ is the user-set threshold (`--off_policy_sequence_mask_delta`, default None = disabled)
## References
1. https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda
2. https://fengyao.notion.site/off-policy-rl
3. https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/rollout_corr_helper.py
4. [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://arxiv.org/abs/2512.02556)
docs/source_en/Instruction/GRPO/AdvancedResearch/treepo.md
+35
View File
@@ -0,0 +1,35 @@
# TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling
Author: [li2zhi](https://github.com/li2zhi)
## Principle Introduction
[TreePO paper](https://arxiv.org/abs/2508.17445) proposes a tree-structured modeling method. This method organizes sequence generation into a segmented tree structure search. Through dynamic branching, backtracking, and early termination mechanisms, it significantly improves the reuse rate of the key-value cache, thereby reducing computational overhead, while maintaining or even enhancing the diversity of exploration.
![TreePO Overview](../../../../resources/treepo.png)
## Implementation Details
[TreePO implementation example](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/treepo/tree_rollout_plugin.py), which references the [official implementation](https://github.com/multimodal-art-projection/TreePO/blob/main/recipe/treepo/vllm_rollout_tree.py) provides sample code for a TreePO training plugin，covering logic related to multi-round interactions, termination judgment, and branch rollback.
**Note:** In actual use, you need to rewrite the logic of methods such as step and check_finished according to your own scenario requirements to ensure that they can execute as expected in the custom scenario.
For information on the design and use of custom rewards, you can refer to the implementation of [DeepEyes](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/deepeyes/deepeyes_plugin.py).
The complete training script can be found at [script](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin/treepo/tree_rollout.sh).
## Test Data
> model: Qwen/Qwen2.5-0.5B
> dataset: AI-MO/NuminaMath-TIR
> subset size: 1,000 samples
> 1 GPU for training, 1 GPU for inference
| \ | batch_size | num_generation | max_tree_depth | global_step | total inference calls | saving ratio | train_speed(iter/s) | improvement rate |
| ----------------------- | ---------- | -------------- | -------------- | ----------- | --------------------- | ------------ | ------------------- | ---------------- |
| original implementation | 8 | 8 | 4 | 200 | 5965 | 0.00% | 0.292436 | 0.00% |
| tree(max_divergence=3) | 8 | 8 | 4 | 200 | 3678 | 38.34% | 0.31819 | 8.81% |
| | | | | | | | | |
| original implementation | 8 | 8 | 5 | 105 | 4312 | 0.00% | 0.261324 | 0.00% |
| tree(max_divergence=2) | 8 | 8 | 5 | 105 | 2513 | 52.69% | 0.336639 | 28.82% |
| tree(max_divergence=3) | 8 | 8 | 5 | 105 | 2990 | 30.66% | 0.308791 | 18.16% |
| | | | | | | | | |
| original implementation | 8 | 8 | 6 | 105 | 5202 | 0.00% | 0.24832 | 0.00% |
| tree(max_divergence=2) | 8 | 8 | 6 | 105 | 3348 | 35.64% | 0.27755 | 11.77% |
| tree(max_divergence=3) | 8 | 8 | 6 | 105 | 3888 | 25.26% | 0.272339 | 9.67% |