chore: import upstream snapshot with attribution

examples/train/grpo/external/README.md

@@ -0,0 +1,72 @@
+# README: GRPO External(Async) Mode Execution Scripts
+
+---
+
+> **Note**: External mode requires
+
+1. vLLM version 0.8.3 or higher.
+2. trl version 0.17.0 or higher
+
+For LoRA Training, set following parameters to speed up weight update
+```bash
+  --vllm_enable_lora true
+  --vllm_max_lora_rank xxx # same as lora_rank in training script
+```
+
+## **Introduction**
+
+The GRPO (Group Relative Policy Optimization) training framework supports high-performance inference engines like vLLM to accelerate the sampling process. The **External Mode** allows you to connect to an external vLLM inference server, separating the inference service from the training process. This mode is ideal for scenarios where you want to offload inference to dedicated hardware or servers, improving resource utilization and scalability.
+
+This folder contains scripts and instructions for running GRPO in **External Mode**, enabling integration with an external vLLM server.
+
+Before running the scripts, ensure the following:
+
+1. **vLLM Server Deployment**:
+   - An external vLLM server must be deployed and accessible.
+   - Use the `swift rollout` command to deploy the vLLM server.
+
+2. **Network Connectivity**:
+   - Ensure the training nodes can communicate with the vLLM server over the network.
+
+## **Deploying the vLLM Server**
+
+To deploy an external vLLM server, use the following command:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 \
+swift rollout \
+  --model Qwen/Qwen3-8B
+
+# tp
+CUDA_VISIBLE_DEVICES=0,1 \
+swift rollout \
+  --model Qwen/Qwen3-8B \
+  --vllm_tensor_parallel_size 2
+
+# dp
+CUDA_VISIBLE_DEVICES=0,1 \
+swift rollout \
+  --model Qwen/Qwen3-8B \
+  --vllm_data_parallel_size 2
+
+# tp + dp
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+swift rollout \
+  --model Qwen/Qwen3-8B \
+  --vllm_tensor_parallel_size 2 \
+  --vllm_data_parallel_size 2
+```
+
+## Training with External vLLM Server
+Configuration Parameters
+
+```bash
+--use_vllm true \
+--vllm_mode server \
+--vllm_server_host <server ip> \
+--vllm_server_port <server port> \
+--vllm_server_timeout <Timeout duration> \
+```
+
+## Multi-Node Training
+On each node, execute the original single-node training script, using the environment variables `NNODES` and `NODE_RANK`, and ensure consistent use of configuration parameters across all nodes.

examples/train/grpo/external/agent.sh

@@ -0,0 +1,43 @@
+# wandb result link: https://wandb.ai/tastelikefeet/tastelikefeet?nw=nwuseryuzezyz
+# model link: https://www.modelscope.cn/models/swift/Qwen2-7B-Agent-GRPO
+# WANDB_API_KEY=xxx \
+
+# CUDA_VISIBLE_DEVICES=7 \
+# swift rollout \
+#     --model Qwen/Qwen2.5-7B
+
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 \
+NPROC_PER_NODE=7 \
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen2.5-7B \
+    --tuner_type full \
+    --dataset LLM-Research/xlam-function-calling-60k:grpo \
+    --load_from_cache_file true \
+    --use_vllm true \
+    --vllm_mode server \
+    --vllm_server_host 127.0.0.1 \
+    --torch_dtype bfloat16 \
+    --num_train_epochs 1 \
+    --max_length 2048 \
+    --per_device_train_batch_size 7 \
+    --per_device_eval_batch_size 7 \
+    --eval_steps 2000 \
+    --save_steps 2000 \
+    --learning_rate 1e-6 \
+    --save_total_limit 2 \
+    --logging_steps 5 \
+    --output_dir output \
+    --warmup_ratio 0.05 \
+    --dataloader_num_workers 4 \
+    --max_completion_length 1024 \
+    --reward_funcs toolbench react_format \
+    --num_generations 49 \
+    --deepspeed zero3 \
+    --temperature 1.0 \
+    --stop_words Observation: \
+    --agent_template react_grpo \
+    --top_p 0.85 \
+    --top_k 50 \
+    --log_completions true \
+    --report_to wandb

examples/train/grpo/external/grpo_32b_full.sh

@@ -0,0 +1,40 @@
+# run in another node
+# CUDA_VISIBLE_DEVICES=0,1 \
+# swift rollout \
+#     --model Qwen/Qwen2.5-32B-Instruct \
+#     --vllm_tensor_parallel_size 2
+
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
+NPROC_PER_NODE=8 \
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen2.5-32B-Instruct \
+    --reward_funcs accuracy \
+    --use_vllm true \
+    --vllm_mode server \
+    --vllm_server_host xxx \
+    --vllm_server_port 8000 \
+    --tuner_type full \
+    --torch_dtype bfloat16 \
+    --dataset AI-MO/NuminaMath-TIR#1000 \
+    --load_from_cache_file true \
+    --split_dataset_ratio 0 \
+    --max_completion_length 2048 \
+    --num_train_epochs 3 \
+    --per_device_train_batch_size 1 \
+    --learning_rate 1e-6 \
+    --gradient_accumulation_steps 1 \
+    --save_total_limit 2 \
+    --logging_steps 1 \
+    --warmup_ratio 0.05 \
+    --dataloader_num_workers 4 \
+    --dataset_num_proc 4 \
+    --num_generations 8 \
+    --temperature 1.0 \
+    --top_p 0.9 \
+    --top_k 50 \
+    --deepspeed zero3 \
+    --log_completions true \
+    --num_iterations 1 \
+    --report_to tensorboard wandb \
+    --beta 0.0

examples/train/grpo/external/grpo_7b.sh

@@ -0,0 +1,40 @@
+# 8 * 80 G (6 for training, 2 for rollout)
+# CUDA_VISIBLE_DEVICES=6,7 \
+# swift rollout \
+#     --model Qwen/Qwen2.5-7B-Instruct \
+#     --data_parallel_size 2
+
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 \
+NPROC_PER_NODE=6 \
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen2.5-7B-Instruct \
+    --reward_funcs accuracy \
+    --use_vllm true \
+    --vllm_mode server \
+    --vllm_server_host 127.0.0.1 \
+    --vllm_server_port 8000 \
+    --tuner_type full \
+    --torch_dtype bfloat16 \
+    --dataset AI-MO/NuminaMath-TIR#1000 \
+    --load_from_cache_file true \
+    --split_dataset_ratio 0 \
+    --max_completion_length 2048 \
+    --num_train_epochs 3 \
+    --per_device_train_batch_size 2 \
+    --learning_rate 1e-6 \
+    --gradient_accumulation_steps 2 \
+    --save_total_limit 2 \
+    --logging_steps 1 \
+    --warmup_ratio 0.05 \
+    --dataloader_num_workers 4 \
+    --dataset_num_proc 4 \
+    --num_generations 8 \
+    --temperature 1.0 \
+    --top_p 0.9 \
+    --top_k 50 \
+    --deepspeed zero2 \
+    --log_completions true \
+    --num_iterations 1 \
+    --report_to tensorboard wandb \
+    --beta 0.04

examples/train/grpo/external/moe_full.sh

@@ -0,0 +1,44 @@
+# 8*80G
+
+# CUDA_VISIBLE_DEVICES=0 \
+# swift rollout \
+#     --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
+#     --vllm_max_model_len 16384 \
+#     --vllm_enable_prefix_caching true
+
+CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7 \
+NPROC_PER_NODE=7 \
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
+    --reward_funcs accuracy \
+    --use_vllm true \
+    --vllm_mode server \
+    --vllm_server_host 127.0.0.1 \
+    --vllm_server_port 8000 \
+    --tuner_type full \
+    --torch_dtype bfloat16 \
+    --dataset AI-MO/NuminaMath-TIR#1000 \
+    --max_length 12000 \
+    --max_completion_length 8192 \
+    --overlong_filter true \
+    --num_train_epochs 1 \
+    --per_device_train_batch_size 1 \
+    --learning_rate 1e-6 \
+    --gradient_accumulation_steps 4 \
+    --save_strategy 'steps' \
+    --eval_strategy 'steps' \
+    --eval_steps 1000 \
+    --save_steps 1000 \
+    --save_total_limit 10 \
+    --logging_steps 1 \
+    --warmup_ratio 0.01 \
+    --dataloader_num_workers 4 \
+    --num_generations 14 \
+    --temperature 1.0 \
+    --deepspeed zero3_offload \
+    --log_completions true \
+    --report_to tensorboard swanlab \
+    --num_iterations 1 \
+    --beta 0.001 \
+    --move_model_batches 5

examples/train/grpo/external/moe_lora.sh

@@ -0,0 +1,44 @@
+# 8*80G
+
+# CUDA_VISIBLE_DEVICES=0 \
+# swift rollout \
+#     --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
+#     --vllm_max_model_len 16384 \
+#     --vllm_enable_prefix_caching true
+
+CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7 \
+NPROC_PER_NODE=7 \
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
+    --reward_funcs accuracy \
+    --use_vllm true \
+    --vllm_mode server \
+    --vllm_server_host 127.0.0.1 \
+    --vllm_server_port 8000 \
+    --tuner_type lora \
+    --torch_dtype bfloat16 \
+    --dataset AI-MO/NuminaMath-TIR#1000 \
+    --max_length 12000 \
+    --max_completion_length 8192 \
+    --overlong_filter true \
+    --num_train_epochs 1 \
+    --per_device_train_batch_size 1 \
+    --learning_rate 1e-6 \
+    --gradient_accumulation_steps 4 \
+    --save_strategy 'steps' \
+    --eval_strategy 'steps' \
+    --eval_steps 1000 \
+    --save_steps 1000 \
+    --save_total_limit 10 \
+    --logging_steps 1 \
+    --warmup_ratio 0.01 \
+    --dataloader_num_workers 4 \
+    --num_generations 14 \
+    --temperature 1.0 \
+    --deepspeed zero3 \
+    --log_completions true \
+    --report_to tensorboard swanlab \
+    --num_iterations 1 \
+    --beta 0.001 \
+    --move_model_batches 5

examples/train/grpo/external/vllm_gym.sh

@@ -0,0 +1,45 @@
+# exp: https://github.com/modelscope/ms-swift/pull/4890
+
+# CUDA_VISIBLE_DEVICES=7 \
+# swift rollout \
+#   --model Qwen/Qwen2.5-3B-Instruct \
+#   --max_turns 3\
+#   --multi_turn_scheduler gym_scheduler \
+#   --use_gym_env true \
+#   --gym_env math_env
+
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5 \
+NPROC_PER_NODE=6 \
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen2.5-3B-Instruct \
+    --tuner_type full \
+    --use_vllm true \
+    --vllm_mode server \
+    --vllm_server_host 127.0.0.1 \
+    --vllm_server_port 8000 \
+    --vllm_server_pass_dataset true \
+    --torch_dtype bfloat16 \
+    --dataset AI-MO/NuminaMath-TIR#1000 \
+    --load_from_cache_file true \
+    --split_dataset_ratio 0 \
+    --max_completion_length 2048 \
+    --num_train_epochs 3 \
+    --per_device_train_batch_size 2 \
+    --learning_rate 1e-6 \
+    --gradient_accumulation_steps 2 \
+    --steps_per_generation 2 \
+    --save_total_limit 2 \
+    --logging_steps 1 \
+    --warmup_ratio 0.05 \
+    --dataloader_num_workers 4 \
+    --dataset_num_proc 4 \
+    --num_generations 6 \
+    --temperature 1.0 \
+    --top_p 0.9 \
+    --top_k 50 \
+    --log_completions true \
+    --num_iterations 1 \
+    --report_to tensorboard \
+    --beta 0 \
+    --loss_scale default

examples/train/grpo/external/vllm_multi_turn.sh

@@ -0,0 +1,57 @@
+# Exp: https://github.com/modelscope/ms-swift/pull/5307#issuecomment-3219803922
+# Before running this script, please run the following `swift rollout` script first
+# This script is a example for multi-turn training with dynamic num of rollout outputs
+# which means a trajectory of multi turn rollout is split into multiple data
+#       see details in thinking_tips_scheduler
+# NOTE: for same trajectory, the reward is supported to be the same,
+#       here we use the last turn data of each trajectory to compute accuracy reward
+#       see details in thinking_tips reward function
+
+# CUDA_VISIBLE_DEVICES=0 \
+# swift rollout \
+#     --model Qwen/Qwen3-1.7B \
+#     --vllm_use_async_engine true \
+#     --multi_turn_scheduler thinking_tips_scheduler \
+#     --vllm_max_model_len 32768 \
+#     --vllm_gpu_memory_utilization 0.8 \
+#     --max_turns 3
+
+CUDA_VISIBLE_DEVICES=1,2 \
+NPROC_PER_NODE=2 \
+swift rlhf \
+    --rlhf_type grpo \
+    --model Qwen/Qwen3-1.7B \
+    --tuner_type full \
+    --external_plugins examples/train/grpo/plugin/plugin.py \
+    --reward_funcs thinking_tips \
+    --loss_scale last_round \
+    --use_vllm true \
+    --vllm_mode server \
+    --vllm_server_host 127.0.0.1 \
+    --vllm_server_port 8000 \
+    --vllm_server_pass_dataset true \
+    --torch_dtype bfloat16 \
+    --dataset AI-MO/NuminaMath-TIR#10000 \
+    --load_from_cache_file true \
+    --split_dataset_ratio 0 \
+    --max_completion_length 8192 \
+    --num_train_epochs 1 \
+    --per_device_train_batch_size 2 \
+    --learning_rate 1e-6 \
+    --gradient_accumulation_steps 4 \
+    --steps_per_generation 8 \
+    --gradient_checkpointing_kwargs '{"use_reentrant": false}' \
+    --save_total_limit 2 \
+    --logging_steps 1 \
+    --warmup_ratio 0.05 \
+    --dataloader_num_workers 4 \
+    --dataset_num_proc 4 \
+    --num_generations 8 \
+    --temperature 1.0 \
+    --deepspeed zero2 \
+    --log_completions true \
+    --log_entropy true \
+    --importance_sampling_level sequence \
+    --top_entropy_quantile 0.2 \
+    --num_iterations 1 \
+    --report_to tensorboard swanlab