wehub-resource-sync
64 lines
2.8 KiB
Markdown
64 lines
2.8 KiB
Markdown
# On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting (CHORD)
|
|
|
|
This document describes the CHORD algorithm proposed in the paper [On-Policy RL Meets Off-Policy Experts: Harmonizing SFT and RL via Dynamic Weighting](https://arxiv.org/abs/2508.11408). The core idea of CHORD is to dynamically integrate expert data (SFT) into reinforcement learning by a dual control mechanism: a global weight μ plus a token-level weight φ, thereby balancing imitation and exploration.
|
|
|
|
## Algorithm Overview
|
|
CHORD mixes training by introducing the SFT loss into the GRPO loss. The overall objective is:
|
|
|
|
$$
|
|
\mathcal{L}_{\text{CHORD}} = (1 - \mu) \cdot \mathcal{L}_{\text{GRPO}} + \mu \cdot \mathcal{L}_{\text{SFT}}
|
|
$$
|
|
|
|
where:
|
|
- $\mathcal{L}_{\text{GRPO}}$: on-policy RL loss based on on-policy samples (similar to PPO).
|
|
- $\mathcal{L}_{\text{SFT}}$: supervised fine-tuning (SFT) loss.
|
|
- $\mu \in [0, 1]$: global balancing coefficient that controls the contribution of the SFT signal to the overall gradient.
|
|
|
|
### Configuration (data and batch sizes)
|
|
We can implement CHORD training based on GRPO training.
|
|
|
|
CHORD requires specifying an additional SFT dataset and batch size at training time:
|
|
- `chord_sft_dataset`: the SFT dataset that provides expert data.
|
|
- `chord_sft_per_device_train_batch_size`: SFT mini-batch size per device.
|
|
|
|
---
|
|
|
|
## Two CHORD Variants
|
|
|
|
The paper proposes two variants: CHORD-μ and CHORD-φ.
|
|
|
|
### CHORD-μ
|
|
CHORD-μ gradually decays μ during training to transition from imitating experts toward autonomous exploration.
|
|
|
|
Parameters:
|
|
- `chord_mu_peak`: the peak value of μ.
|
|
- `chord_mu_valley`: the final decayed value of μ.
|
|
- `chord_mu_warmup_steps`: number of training steps to ramp μ up to the peak.
|
|
- `chord_mu_decay_steps`: number of training steps to decay μ from peak to valley.
|
|
|
|
### CHORD-φ (Token-level weighting)
|
|
CHORD-φ uses a token-wise weighting function φ to dynamically control each expert token's gradient contribution.
|
|
|
|
Definition of φ:
|
|
$$
|
|
\phi(y_t^\star, \pi_\theta) = p_t \cdot (1 - p_t)
|
|
$$
|
|
|
|
where:
|
|
- $p_t = \pi_\theta(y_t^\star \mid x, y_{<t}^\star)$ is the model's current predicted probability of the expert token.
|
|
- When $p_t \approx 0.5$ (model uncertainty), φ is maximal → emphasize tokens the model is uncertain about.
|
|
- When $p_t \approx 0$ or $p_t \approx 1$, φ → 0 → avoid overemphasizing tokens that are already certain or impossible.
|
|
|
|
Parameter to enable φ weighting:
|
|
- `chord_enable_phi_function: bool = False`
|
|
- Set to `True` to enable token-wise weight φ.
|
|
|
|
Note: If using a constant μ, set `chord_mu_peak` and `chord_mu_valley` to the same value.
|
|
|
|
<details>
|
|
<summary>Code implementation of μ scheduling and loss computation</summary>
|
|
See the `GRPOTrainer` method `_compute_chord_loss`.
|
|
</details>
|
|
|
|
Training reference script: https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/internal/chord.sh
|