wehub-resource-sync
1270 lines
108 KiB
HTML
1270 lines
108 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>Agentic RL 面试 Cheat Sheet</title>
|
||
|
||
<meta name="generator" content="ARIS render-html (academic, v1)">
|
||
<meta name="aris:source-path" content="docs/tutorials/agentic_rl_tutorial.md">
|
||
<meta name="aris:source-sha256" content="d3d476658bea0760566d2cc9d6eaf72ceaab67d8c81d3eeaa4442473092e8e6e">
|
||
<meta name="aris:generated-at" content="2026-05-19 14:03 UTC">
|
||
|
||
<!-- MathJax 3 -->
|
||
<script>
|
||
window.MathJax = {
|
||
tex: { inlineMath: [['$', '$'], ['\\(', '\\)']], displayMath: [['$$', '$$'], ['\\[', '\\]']], processEscapes: true },
|
||
options: { skipHtmlTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code'] }
|
||
};
|
||
</script>
|
||
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js" async></script>
|
||
|
||
<!-- highlight.js -->
|
||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/styles/atom-one-light.min.css">
|
||
<script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.9.0/build/highlight.min.js"></script>
|
||
<script>document.addEventListener('DOMContentLoaded', () => hljs.highlightAll());</script>
|
||
|
||
|
||
<style>
|
||
:root {
|
||
--bg: #fdfcf7;
|
||
--bg-soft: #f4f1ea;
|
||
--bg-code: #f8f5ec;
|
||
--ink: #1a1a1a;
|
||
--ink-soft: #4a4a4a;
|
||
--ink-muted: #6b6b6b;
|
||
--primary: #1a4a8c;
|
||
--primary-soft: #2d6cb8;
|
||
--accent: #b8390e;
|
||
--warn: #b45309;
|
||
--warn-bg: #fef3c7;
|
||
--info-bg: #dbeafe;
|
||
--good-bg: #d1fae5;
|
||
--good: #065f46;
|
||
--bad-bg: #fee2e2;
|
||
--bad: #991b1b;
|
||
--border: #d6d0c0;
|
||
--border-soft: #e8e3d5;
|
||
}
|
||
|
||
* { box-sizing: border-box; }
|
||
html { scroll-behavior: smooth; }
|
||
|
||
body {
|
||
font-family: "Source Serif Pro", "Source Serif 4", "Crimson Pro", "Georgia", "Songti SC", "STSong", serif;
|
||
line-height: 1.65;
|
||
color: var(--ink);
|
||
background: var(--bg);
|
||
margin: 0;
|
||
padding: 0;
|
||
font-size: 16px;
|
||
}
|
||
|
||
.layout {
|
||
max-width: 1280px;
|
||
margin: 0 auto;
|
||
display: grid;
|
||
grid-template-columns: 260px 1fr;
|
||
gap: 48px;
|
||
padding: 40px 32px;
|
||
}
|
||
|
||
nav.toc {
|
||
position: sticky;
|
||
top: 24px;
|
||
align-self: start;
|
||
font-size: 13px;
|
||
max-height: calc(100vh - 48px);
|
||
overflow-y: auto;
|
||
border-right: 1px solid var(--border-soft);
|
||
padding-right: 16px;
|
||
}
|
||
nav.toc h3 {
|
||
margin: 0 0 12px;
|
||
font-size: 12px;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.08em;
|
||
color: var(--ink-muted);
|
||
font-weight: 600;
|
||
}
|
||
nav.toc ol { list-style: none; padding: 0; margin: 0; counter-reset: toc; }
|
||
nav.toc ol li { margin: 5px 0; counter-increment: toc; }
|
||
nav.toc ol li::before { content: counter(toc) ". "; color: var(--ink-muted); margin-right: 4px; }
|
||
nav.toc a {
|
||
color: var(--ink-soft);
|
||
text-decoration: none;
|
||
border-bottom: 1px dotted transparent;
|
||
}
|
||
nav.toc a:hover { color: var(--primary); border-bottom-color: var(--primary); }
|
||
nav.toc ul { list-style: none; padding-left: 14px; margin: 3px 0; font-size: 12px; }
|
||
nav.toc ul li::before { content: "→ "; color: var(--border); }
|
||
|
||
main { min-width: 0; }
|
||
|
||
header.hero {
|
||
border-bottom: 3px double var(--primary);
|
||
padding-bottom: 24px;
|
||
margin-bottom: 32px;
|
||
}
|
||
header.hero .eyebrow {
|
||
color: var(--accent);
|
||
font-size: 13px;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.12em;
|
||
font-weight: 600;
|
||
margin-bottom: 8px;
|
||
}
|
||
header.hero h1 {
|
||
font-size: 32px;
|
||
line-height: 1.2;
|
||
margin: 0 0 12px;
|
||
color: var(--ink);
|
||
font-weight: 700;
|
||
letter-spacing: -0.01em;
|
||
}
|
||
header.hero .subtitle {
|
||
font-size: 16px;
|
||
color: var(--ink-soft);
|
||
margin: 0 0 8px;
|
||
font-style: italic;
|
||
}
|
||
header.hero .byline {
|
||
font-size: 14px;
|
||
color: var(--ink-soft);
|
||
margin: 0 0 20px;
|
||
}
|
||
header.hero .byline strong {
|
||
color: var(--ink);
|
||
font-weight: 600;
|
||
}
|
||
header.hero .meta {
|
||
display: flex;
|
||
gap: 20px;
|
||
flex-wrap: wrap;
|
||
font-size: 12px;
|
||
color: var(--ink-muted);
|
||
border-top: 1px solid var(--border-soft);
|
||
padding-top: 14px;
|
||
}
|
||
header.hero .meta span strong { color: var(--ink-soft); }
|
||
header.hero .meta code {
|
||
font-family: "JetBrains Mono", "SF Mono", "Menlo", "Consolas", monospace;
|
||
font-size: 11px;
|
||
background: var(--bg-soft);
|
||
padding: 1px 5px;
|
||
border-radius: 3px;
|
||
border: 1px solid var(--border-soft);
|
||
}
|
||
|
||
h2 {
|
||
font-size: 24px;
|
||
margin: 44px 0 14px;
|
||
padding-bottom: 8px;
|
||
border-bottom: 1px solid var(--border);
|
||
color: var(--ink);
|
||
font-weight: 700;
|
||
}
|
||
h2 .num { color: var(--primary); font-weight: 600; margin-right: 8px; }
|
||
h3 { font-size: 19px; margin: 28px 0 10px; color: var(--primary); font-weight: 600; }
|
||
h4 { font-size: 16px; margin: 20px 0 8px; color: var(--ink); font-weight: 600; }
|
||
|
||
p { margin: 10px 0; }
|
||
ul, ol { padding-left: 22px; margin: 10px 0; }
|
||
ul li, ol li { margin: 4px 0; }
|
||
ul li::marker { color: var(--primary); }
|
||
|
||
strong { color: var(--accent); font-weight: 600; }
|
||
em { color: var(--ink-soft); }
|
||
|
||
a { color: var(--primary); }
|
||
a:hover { color: var(--accent); }
|
||
|
||
code:not(.hljs) {
|
||
font-family: "JetBrains Mono", "SF Mono", "Menlo", "Consolas", monospace;
|
||
font-size: 0.86em;
|
||
background: var(--bg-code);
|
||
padding: 1px 5px;
|
||
border-radius: 3px;
|
||
border: 1px solid var(--border-soft);
|
||
color: var(--accent);
|
||
}
|
||
|
||
pre {
|
||
background: #fafaf6;
|
||
border: 1px solid var(--border);
|
||
border-left: 4px solid var(--primary);
|
||
padding: 0;
|
||
overflow-x: auto;
|
||
border-radius: 4px;
|
||
margin: 14px 0;
|
||
}
|
||
pre code, pre code.hljs {
|
||
background: transparent !important;
|
||
display: block;
|
||
padding: 14px 18px !important;
|
||
font-size: 13px;
|
||
line-height: 1.55;
|
||
font-family: "JetBrains Mono", "SF Mono", "Menlo", monospace;
|
||
color: var(--ink);
|
||
}
|
||
pre.diagram {
|
||
background: #f9f6ed;
|
||
border-left: 4px solid var(--accent);
|
||
font-size: 12.5px;
|
||
line-height: 1.4;
|
||
}
|
||
|
||
.callout {
|
||
margin: 16px 0;
|
||
padding: 12px 16px;
|
||
border-radius: 4px;
|
||
border-left: 4px solid;
|
||
font-size: 15px;
|
||
}
|
||
.callout-title {
|
||
font-weight: 600;
|
||
margin-bottom: 6px;
|
||
font-size: 12px;
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.06em;
|
||
}
|
||
.callout-info { background: var(--info-bg); border-left-color: var(--primary); }
|
||
.callout-info .callout-title { color: var(--primary); }
|
||
.callout-warn { background: var(--warn-bg); border-left-color: var(--warn); }
|
||
.callout-warn .callout-title { color: var(--warn); }
|
||
.callout-good { background: var(--good-bg); border-left-color: var(--good); }
|
||
.callout-good .callout-title { color: var(--good); }
|
||
.callout-bad { background: var(--bad-bg); border-left-color: var(--bad); }
|
||
.callout-bad .callout-title { color: var(--bad); }
|
||
|
||
table {
|
||
width: 100%;
|
||
border-collapse: collapse;
|
||
margin: 16px 0;
|
||
font-size: 14px;
|
||
border: 1px solid var(--border);
|
||
border-radius: 4px;
|
||
overflow: hidden;
|
||
}
|
||
thead { background: var(--primary); color: white; }
|
||
th, td {
|
||
text-align: left;
|
||
padding: 9px 12px;
|
||
border-bottom: 1px solid var(--border-soft);
|
||
vertical-align: top;
|
||
}
|
||
th { font-weight: 600; font-size: 13px; letter-spacing: 0.02em; }
|
||
tr:last-child td { border-bottom: none; }
|
||
tbody tr:nth-child(even) { background: var(--bg-soft); }
|
||
|
||
details.qa, details {
|
||
background: white;
|
||
border: 1px solid var(--border-soft);
|
||
border-radius: 6px;
|
||
margin: 10px 0;
|
||
padding: 0;
|
||
}
|
||
details summary {
|
||
cursor: pointer;
|
||
padding: 10px 14px;
|
||
font-weight: 600;
|
||
font-size: 14px;
|
||
color: var(--primary);
|
||
list-style: none;
|
||
user-select: none;
|
||
}
|
||
details summary::-webkit-details-marker { display: none; }
|
||
details summary::before {
|
||
content: "▸ ";
|
||
margin-right: 4px;
|
||
display: inline-block;
|
||
transition: transform 0.15s;
|
||
}
|
||
details[open] summary::before { transform: rotate(90deg); }
|
||
details[open] summary { border-bottom: 1px solid var(--border-soft); }
|
||
details > :not(summary) { padding: 10px 14px; }
|
||
details p:first-of-type { margin-top: 8px; }
|
||
|
||
mjx-container[display="true"] { margin: 12px 0 !important; }
|
||
|
||
footer.aris-footer {
|
||
margin-top: 60px;
|
||
padding-top: 20px;
|
||
border-top: 1px solid var(--border);
|
||
font-size: 12px;
|
||
color: var(--ink-muted);
|
||
}
|
||
footer.aris-footer a { color: var(--ink-muted); border-bottom: 1px dotted var(--border); }
|
||
|
||
@media (max-width: 900px) {
|
||
.layout { grid-template-columns: 1fr; gap: 20px; padding: 20px 16px; }
|
||
nav.toc {
|
||
position: static;
|
||
max-height: none;
|
||
border-right: none;
|
||
border-bottom: 1px solid var(--border-soft);
|
||
padding-right: 0;
|
||
padding-bottom: 14px;
|
||
}
|
||
header.hero h1 { font-size: 24px; }
|
||
h2 { font-size: 20px; }
|
||
}
|
||
@media print {
|
||
nav.toc { display: none; }
|
||
.layout { grid-template-columns: 1fr; padding: 0; }
|
||
body { background: white; }
|
||
header.hero { border-bottom-color: var(--ink); }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="layout">
|
||
|
||
<nav class="toc">
|
||
<h3>Contents</h3>
|
||
<ol>
|
||
<li><a href="#0-tldr-cheat-sheet">§0 TL;DR Cheat Sheet</a>
|
||
</li>
|
||
<li><a href="#1-直觉从-rlhf-到-agentic-rl">§1 直觉：从 RLHF 到 Agentic RL</a>
|
||
<ul>
|
||
<li><a href="#11-把-llm-从会写字的策略升级成会动手的-agent">1.1 把 LLM 从"会写字的策略"升级成"会动手的 agent"</a></li>
|
||
<li><a href="#12-mental-modelmdp--pomdp-表述">1.2 Mental model：MDP / POMDP 表述</a></li>
|
||
<li><a href="#13-agentic-rl-与三类-rl-邻居的关系">1.3 Agentic RL 与三类 RL 邻居的关系</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#2-ppo--grpo-在-agent-上的关键改造">§2 PPO / GRPO 在 agent 上的关键改造</a>
|
||
<ul>
|
||
<li><a href="#21-token-maskonly-loss-on-agent-tokens">2.1 Token mask：only loss on agent tokens</a></li>
|
||
<li><a href="#22-trajectory-level-gae-for-agent">2.2 Trajectory-level GAE for agent</a></li>
|
||
<li><a href="#23-ppo-loss-adapted-to-agent">2.3 PPO loss adapted to agent</a></li>
|
||
<li><a href="#24-grpo-for-agentstrace-level-group-relative-advantage">2.4 GRPO for agents：trace-level group-relative advantage</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#3-reward-design-for-agents核心">§3 Reward design for agents（核心）</a>
|
||
<ul>
|
||
<li><a href="#31-outcome-reward-vs-process-reward">3.1 Outcome reward vs Process reward</a></li>
|
||
<li><a href="#32-verifier-based-rewardrule-based">3.2 Verifier-based reward（rule-based）</a></li>
|
||
<li><a href="#33-format-reward--shaping-reward">3.3 Format reward / shaping reward</a></li>
|
||
<li><a href="#34-length-penalty防-agent-拖太长">3.4 Length penalty（防 agent 拖太长）</a></li>
|
||
<li><a href="#35-tool-call-shaping-reward">3.5 Tool-call shaping reward</a></li>
|
||
<li><a href="#36-rlaif-for-agents用-llm-当-reward">3.6 RLAIF for agents：用 LLM 当 reward</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#4-long-horizon-credit-assignment">§4 Long-horizon credit assignment</a>
|
||
<ul>
|
||
<li><a href="#41-稀疏-reward-是-agent-rl-的核心痛点">4.1 稀疏 reward 是 agent RL 的核心痛点</a></li>
|
||
<li><a href="#42-discount--gae-在-agent-上的退化">4.2 Discount + GAE 在 agent 上的退化</a></li>
|
||
<li><a href="#43-hindsight-relabeling-for-agents">4.3 Hindsight relabeling for agents</a></li>
|
||
<li><a href="#44-subgoal-decomposition--process-reward">4.4 Subgoal decomposition + process reward</a></li>
|
||
<li><a href="#45-per-step-kl-penalty-防止-policy-collapse">4.5 Per-step KL penalty 防止 policy collapse</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#5-self-rewarding--exploration-in-agent-rl">§5 Self-rewarding & exploration in agent RL</a>
|
||
<ul>
|
||
<li><a href="#51-self-rewarding-lmyuan-et-al-2024-meta-arxiv-240110020">5.1 Self-Rewarding LM（Yuan et al. 2024 Meta arXiv 2401.10020）</a></li>
|
||
<li><a href="#52-self-rewarding-在-agent-上的危险">5.2 Self-Rewarding 在 agent 上的危险</a></li>
|
||
<li><a href="#53-exploration-in-agent-rl">5.3 Exploration in agent RL</a></li>
|
||
<li><a href="#54-curriculum--difficulty-scheduling">5.4 Curriculum & difficulty scheduling</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#6-specific-algorithms代表性-agentic-rl-papers">§6 Specific algorithms（代表性 Agentic RL papers）</a>
|
||
<ul>
|
||
<li><a href="#61-vpt-baker-et-al-2022-neurips-openai-arxiv-220611795">6.1 VPT (Baker et al. 2022 NeurIPS OpenAI, arXiv 2206.11795)</a></li>
|
||
<li><a href="#62-agenttuning-zeng-et-al-2023-thu-arxiv-231012823">6.2 AgentTuning (Zeng et al. 2023 THU arXiv 2310.12823)</a></li>
|
||
<li><a href="#63-agent-flan-chen-et-al-2024-acl-findings-arxiv-240312881">6.3 Agent-FLAN (Chen et al. 2024 ACL Findings, arXiv 2403.12881)</a></li>
|
||
<li><a href="#64-reft-trung-et-al-2024-acl-arxiv-240108967">6.4 ReFT (Trung et al. 2024 ACL arXiv 2401.08967)</a></li>
|
||
<li><a href="#65-deepseek-r1-deepseek-ai-2025-arxiv-250112948-在-agent-上的扩展">6.5 DeepSeek-R1 (DeepSeek-AI 2025 arXiv 2501.12948) 在 agent 上的扩展</a></li>
|
||
<li><a href="#66-toolrl-qian-et-al-2025-arxiv-250413958">6.6 ToolRL (Qian et al. 2025 arXiv 2504.13958)</a></li>
|
||
<li><a href="#67-research-chen-et-al-2025-arxiv-250319470">6.7 ReSearch (Chen et al. 2025 arXiv 2503.19470)</a></li>
|
||
<li><a href="#68-ragen--starpo-wang-et-al-2025-arxiv-250420073">6.8 RAGEN / StarPO (Wang et al. 2025 arXiv 2504.20073)</a></li>
|
||
<li><a href="#69-webrl-qi-et-al-2024-iclr-25-arxiv-241102337">6.9 WebRL (Qi et al. 2024 ICLR-25 arXiv 2411.02337)</a></li>
|
||
<li><a href="#610-agentq-putta-et-al-2024-arxiv-240807199">6.10 AgentQ (Putta et al. 2024 arXiv 2408.07199)</a></li>
|
||
<li><a href="#611-webgum-furuta-et-al-2024-iclr-arxiv-230511854">6.11 WebGUM (Furuta et al. 2024 ICLR arXiv 2305.11854)</a></li>
|
||
<li><a href="#612-coderl-le-et-al-2022-neurips-arxiv-220701780">6.12 CodeRL (Le et al. 2022 NeurIPS arXiv 2207.01780)</a></li>
|
||
<li><a href="#613-ppocoder-shojaee-et-al-2023-arxiv-230113816">6.13 PPOCoder (Shojaee et al. 2023 arXiv 2301.13816)</a></li>
|
||
<li><a href="#614-swe-rl-wei-et-al-2025-meta-fair-arxiv-250218449">6.14 SWE-RL (Wei et al. 2025 Meta FAIR arXiv 2502.18449)</a></li>
|
||
<li><a href="#615-openvla-kim-et-al-2024-arxiv-240609246">6.15 OpenVLA (Kim et al. 2024 arXiv 2406.09246)</a></li>
|
||
<li><a href="#616-anthropic-computer-use公开知识--训练细节未披露">6.16 Anthropic Computer-Use（公开知识 — 训练细节未披露）</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#7-code-patternspytorch--伪代码">§7 Code patterns（PyTorch / 伪代码）</a>
|
||
<ul>
|
||
<li><a href="#71-agent-rolloutstate-action-reward-收集">7.1 Agent rollout（state, action, reward 收集）</a></li>
|
||
<li><a href="#72-trajectory-level-gae-advantage">7.2 Trajectory-level GAE advantage</a></li>
|
||
<li><a href="#73-ppo-loss-adapted-to-agentwith-actionmask">7.3 PPO loss adapted to agent（with action_mask）</a></li>
|
||
<li><a href="#74-grpo-group-relative-reward-on-agent-trajectories">7.4 GRPO group-relative reward on agent trajectories</a></li>
|
||
<li><a href="#75-outcome--step-reward-combination">7.5 Outcome + step reward combination</a></li>
|
||
<li><a href="#76-verifier-based-rewardcode-test--math-match">7.6 Verifier-based reward（code test / math match）</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#8-frontier-2024-2026-关键趋势">§8 Frontier (2024-2026 关键趋势)</a>
|
||
<ul>
|
||
<li><a href="#81-critic-free-rl-is-the-new-default">8.1 Critic-free RL is the new default</a></li>
|
||
<li><a href="#82-rule-based-reward-在-agent-上的胜利">8.2 Rule-based reward 在 agent 上的胜利</a></li>
|
||
<li><a href="#83-long-horizon-training-的-infra-challenge">8.3 Long-horizon training 的 infra challenge</a></li>
|
||
<li><a href="#84-tool-rl-on-tau-bench--swe-bench--osworld">8.4 Tool-RL on TAU-bench / SWE-bench / OSWorld</a></li>
|
||
<li><a href="#85-anthropic-computer-useclaude-35--45--opus-4x">8.5 Anthropic Computer-Use（Claude 3.5 → 4.5 → Opus 4.x）</a></li>
|
||
<li><a href="#86-2025-2026-papers-群像">8.6 2025-2026 papers 群像</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#9-failure-modes--工程经验">§9 Failure modes & 工程经验</a>
|
||
<ul>
|
||
<li><a href="#91-agentic-rl-的七宗罪">9.1 Agentic RL 的"七宗罪"</a></li>
|
||
<li><a href="#92-online-vs-offline-rl-trade-off">9.2 Online vs offline RL trade-off</a></li>
|
||
<li><a href="#93-rollout-优化经验">9.3 Rollout 优化经验</a></li>
|
||
<li><a href="#94-debug-checklist">9.4 Debug checklist</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#10-25-高频面试题l1-必会--l2-进阶--l3-顶级-lab">§10 25 高频面试题（L1 必会 / L2 进阶 / L3 顶级 lab）</a>
|
||
<ul>
|
||
<li><a href="#l1-必会题10-题">L1 必会题（10 题）</a></li>
|
||
<li><a href="#l2-进阶题10-题">L2 进阶题（10 题）</a></li>
|
||
<li><a href="#l3-顶级-lab-题5-题">L3 顶级 lab 题（5 题）</a></li>
|
||
</ul>
|
||
</li>
|
||
<li><a href="#a-附录参考文献清单">§A 附录：参考文献清单</a>
|
||
</li>
|
||
</ol>
|
||
</nav>
|
||
|
||
<main>
|
||
<header class="hero">
|
||
<div class="eyebrow">Interview Prep · Agentic RL</div>
|
||
<h1>Agentic RL 面试 Cheat Sheet</h1>
|
||
<p class="subtitle">RL for LLM Agents · AgentTuning · ToolRL · GRPO for tool use · RAGEN · WebRL · SWE-RL + 25 高频题（L1 必会 · L2 进阶 · L3 顶级 lab）</p>
|
||
<p class="byline">By <strong>Ruofeng Yang (杨若峰), Shanghai Jiao Tong University</strong></p>
|
||
<div class="meta">
|
||
<span><strong>Source:</strong> <code>docs/tutorials/agentic_rl_tutorial.md</code></span>
|
||
<span><strong>SHA256:</strong> <code>d3d476658bea</code></span>
|
||
<span><strong>Rendered:</strong> 2026-05-19 14:03 UTC</span>
|
||
|
||
</div>
|
||
</header>
|
||
|
||
<h2 id="0-tldr-cheat-sheet">§0 TL;DR Cheat Sheet</h2>
|
||
<div class="callout callout-info"><div class="callout-title">9 句话搞定 Agentic RL</div><p>RL for LLM agents 是 2024-2026 把 reasoning RL 推向真实工具使用、Web、代码与 GUI 的核心范式（详见 §1-§9 推导 + §10 25 高频题）。</p></div>
|
||
<ol><li><strong>Agentic RL 与 RLHF 的本质区别</strong>：RLHF 是 single-turn 偏好对齐，reward 来自 RM 对整段 response 的打分；<strong>Agentic RL 是 multi-turn 决策，state 是 (obs, history)，action 是 (thought, tool_call)，reward 来自外部环境（test-pass、task success、verifier）而非 RM</strong>。整条轨迹长度从 RLHF 的几百 token 涨到 agent 的数千乃至数万 token，credit assignment 难度上一个台阶。</li><li><strong>PPO/GRPO 在 agent 上的关键改造</strong>（必背）：<strong>token mask</strong> 必须只对 agent 自己 generate 的 token 算 loss——observation token（tool 返回的 stdout / search snippet）属于环境，policy gradient 不能流到那里；否则 model 会试图"教 tool 怎么回答"，行为崩坏。GRPO 优势更明显：在 long-horizon trajectory 上，value model 几乎学不动 per-token V（中间几乎全 0 reward），组内归一化是更稳的 baseline。</li><li><strong>Reward 设计三层金字塔</strong>：(a) <strong>Outcome reward</strong> 最便宜也最稀疏——final answer/task success 0/1；(b) <strong>Process reward</strong> 给每步打分，需要 PRM 或 step verifier；(c) <strong>Hybrid / shaping</strong>——tool-call shaping（鼓励调对工具）、length penalty（防 agent 拖太长）、format reward（强约束输出 schema）。R1 路线用 rule-based outcome reward（数学正确 + 格式），SWE-RL 用 test-pass，WebRL 用 task success——<strong>rule-based outcome reward + dense format shaping</strong> 是 2025 工业实测最稳的组合。</li><li><strong>代表性早期 work</strong>：<strong>AgentTuning</strong> (Zeng et al. 2023 arXiv 2310.12823 THU)——agent SFT 数据集 + 多任务训练；<strong>Agent-FLAN</strong> (Chen et al. 2024 ACL Findings arXiv 2403.12881)——把 agent corpus 拆成 multi-turn / formatted / negative example 三类；<strong>ReFT</strong> (Trung et al. 2024 ACL arXiv 2401.08967)——SFT warm-start + online RL on math reasoning，PPO 在 GSM8K 上 +9pp。这三篇是 Agentic RL 的"先 SFT 后 RL"标准三段式。</li><li><strong>Tool-augmented reasoning RL</strong>：<strong>ToolRL</strong> (Qian et al. 2025 arXiv 2504.13958)——把 tool 调用嵌入 GRPO，reward 含 correctness + format + tool-use efficiency；<strong>ReSearch</strong> (Chen et al. 2025 arXiv 2503.19470)——把 search call 当 first-class action，rule-based reward 学 multi-hop search；<strong>RAGEN / StarPO</strong> (Wang et al. 2025 arXiv 2504.20073)——多回合 RL 训练 framework，state-action token level loss + critic-free GRPO 变种。共同点：<strong>outcome-only reward + format shaping + token-mask loss + GRPO</strong>。</li><li><strong>Web / GUI agent RL</strong>：<strong>WebRL</strong> (Qi et al. 2024 ICLR-25 arXiv 2411.02337)——self-evolving curriculum + ORM + retrospective rollout，把 8B Llama 推到 WebArena 43%；<strong>AgentQ</strong> (Putta et al. 2024 arXiv 2408.07199)——MCTS 搜索 + AI critique + DPO offline 训练；<strong>Computer-Use</strong> (Anthropic Claude 3.5/3.7/4 Sonnet, 2024-10-22 起)——RLHF + RL 在屏幕截图 + 鼠标键盘 action space 上训练 GUI 控制（公开知识：训练细节未披露，但 system card 说明用了大量人工 + AI 反馈）。</li><li><strong>Code agent RL</strong>：<strong>CodeRL</strong> (Le et al. 2022 NeurIPS arXiv 2207.01780) 首次把 unit test 当 reward 信号 + actor-critic；<strong>PPOCoder</strong> (Shojaee et al. 2023 arXiv 2301.13816) 加入 compilable + functional correctness 的 composite reward；<strong>SWE-RL</strong> (Wei et al. 2025 Meta FAIR arXiv 2502.18449) 用 rule-based reward（patch similarity + test-pass）在 GitHub PR 数据上做 RL，Llama-3.3-70B 把 SWE-bench Verified 推到 41%。</li><li><strong>Self-rewarding & exploration</strong>：<strong>Self-Rewarding LM</strong> (Yuan et al. 2024 Meta arXiv 2401.10020) 让 policy 同时当 judge，iterative DPO with LLM-as-judge；但 self-rewarding 在 agent 上比单 turn alignment 更危险——judge 也是 agent 自己，<strong>容易 reward drift / model collapse</strong>。生产里多用 LLM-as-judge ensemble + rule-based grounding（test-pass、math checker）+ human spot check 三件套。</li><li><strong>长 horizon credit assignment 的"三种武器"</strong>：(a) <strong>GAE + γ < 1</strong> 把信用沿轨迹回传，但在 sparse outcome reward 下退化为 MC return；(b) <strong>Hindsight relabeling</strong>（HER 思路在 agent 上的对应物）——失败轨迹按"中间状态当 goal"重新标 reward；(c) <strong>subgoal decomposition + process reward</strong>——把 50 步轨迹切成 5 个 subgoal × 10 步，PRM 给每个 subgoal 打分。L3 面试常问的"为什么 GRPO 在 long-horizon agent 上比 PPO sample efficient"——答案是 <strong>trace-level reward 直接匹配 trace-level credit</strong>，绕开 value model 在 long-CoT 上几乎学不动的痛点。</li></ol>
|
||
<h2 id="1-直觉从-rlhf-到-agentic-rl">§1 直觉：从 RLHF 到 Agentic RL</h2>
|
||
<h3 id="11-把-llm-从会写字的策略升级成会动手的-agent">1.1 把 LLM 从"会写字的策略"升级成"会动手的 agent"</h3>
|
||
<p>RLHF 把 LLM 训成"会按人类偏好写字"的 policy；但 RLHF policy 在调用 tool / 多轮交互 / 长 horizon 任务上仍然脆弱：</p>
|
||
<ul><li><strong>single-turn 偏好</strong> 不 directly transfer 到 multi-turn task success</li><li><strong>RM 学的是"哪种文风讨人喜欢"</strong>，不是"哪种调用顺序能解决问题"</li><li><strong>整段 response reward</strong>，无法区分"前 100 token 推理对、第 101 token 选错了 tool"</li></ul>
|
||
<p>Agentic RL 的本质是把 RL 信号挂在 <strong>trajectory 终点的客观结果</strong> 上（test pass、math 答对、网页 task 完成），而不是 RM 的主观偏好。这一步让 alignment-style RL 升级为 <strong>decision-making RL</strong>。</p>
|
||
<h3 id="12-mental-modelmdp--pomdp-表述">1.2 Mental model：MDP / POMDP 表述</h3>
|
||
<table><thead><tr><th>元素</th><th>RLHF (single-turn)</th><th>Agentic RL</th></tr></thead><tbody><tr><td>State $s_t$</td><td>prompt</td><td>$(o_0, a_0, o_1, \dots, o_{t-1}, a_{t-1})$（history）</td></tr><tr><td>Action $a_t$</td><td>整段 response</td><td>一步 <code>(thought, tool_call)</code> 或 token-level subaction</td></tr><tr><td>Reward $r_t$</td><td>terminal RM 分</td><td>terminal task success（多数时刻为 0）</td></tr><tr><td>Horizon $T$</td><td>1（一段 response）</td><td>10-200 步（agent loop）</td></tr><tr><td>Trajectory 长度 (token)</td><td>$10^2$-$10^3$</td><td>$10^3$-$10^5$</td></tr><tr><td>Environment</td><td>RM (神经网络)</td><td>真实环境（shell / browser / search / Python）</td></tr></tbody></table>
|
||
<pre class="diagram"><code>
|
||
┌──────────────────────────────┐
|
||
│ Policy π_θ (LLM) │ agent
|
||
└──────────────┬───────────────┘
|
||
│ action a_t = (thought, tool_call)
|
||
↓
|
||
┌──────────────────────────────┐
|
||
│ Environment / Tool │
|
||
│ - search / shell / browser │
|
||
│ - Python / unit test │
|
||
└──────────────┬───────────────┘
|
||
│ observation o_t
|
||
↓
|
||
┌──────────────────────────────┐
|
||
│ History buffer │
|
||
│ (拼回 prompt 给下一步) │
|
||
└──────────────┬───────────────┘
|
||
│
|
||
└─→ 回到 π_θ</code></pre>
|
||
<h3 id="13-agentic-rl-与三类-rl-邻居的关系">1.3 Agentic RL 与三类 RL 邻居的关系</h3>
|
||
<table><thead><tr><th>邻居</th><th>共同点</th><th>差异</th></tr></thead><tbody><tr><td><strong>RLHF / DPO</strong></td><td>LLM + KL-anchored RL</td><td>Agentic 必须多轮 + tool I/O，reward 来自环境而非 RM</td></tr><tr><td><strong>Reasoning RL（R1, R1-Zero）</strong></td><td>rule-based outcome reward + GRPO</td><td>R1 只在数学/代码答题，无 tool；Agentic RL 在 tool 调用 + 多步交互上</td></tr><tr><td><strong>经典 robotic RL（VPT, OpenVLA）</strong></td><td>sparse terminal reward + long horizon</td><td>LLM agent action space = token sequence；机器人 RL action = 连续控制</td></tr></tbody></table>
|
||
<div class="callout callout-info"><div class="callout-title">面试 framing</div><p>被问到"Agentic RL 是什么"时，<strong>先 disambiguate</strong>。</p></div>
|
||
<ul><li>(1) 严格定义：multi-turn + tool I/O + outcome reward RL</li><li>(2) 与 RLHF 的边界：RLHF 是 single-turn alignment，Agentic 是 multi-turn decision-making</li><li>(3) 与 reasoning RL 的边界：reasoning RL 只算答题正确，Agentic RL 算 task success on real environment</li></ul>
|
||
<p>这三句话能在 30 秒内把面试官的预期 anchor 准。</p>
|
||
<h2 id="2-ppo--grpo-在-agent-上的关键改造">§2 PPO / GRPO 在 agent 上的关键改造</h2>
|
||
<h3 id="21-token-maskonly-loss-on-agent-tokens">2.1 Token mask：only loss on agent tokens</h3>
|
||
<p><strong>这是 Agentic RL 第一条铁律</strong>。agent trajectory 里的 token 分两类：</p>
|
||
<ul><li><strong>agent token</strong>：policy $\pi_\theta$ generate 的（thought, action JSON, final answer）</li><li><strong>environment token</strong>：tool 返回的 observation（search snippet, stdout, screenshot caption）</li></ul>
|
||
<p>PPO/GRPO 的 log-prob ratio 与 loss <strong>必须只在 agent token 上算</strong>。如果在 observation token 上也加 loss：</p>
|
||
<ul><li>policy 会试图"教 tool 怎么回答"（毫无意义且会 reward hack）</li><li>gradient 会被大量低信息 observation token 稀释</li><li>KL penalty 也会错误地把 environment text 当成自己的 distribution 估计</li></ul>
|
||
<p>实现上是一个 <strong>action_mask: [B, L]</strong> tensor：1 表示 agent 自己 generate 的 token，0 表示 prompt / observation / padding。loss 计算时 <code>(loss * action_mask).sum() / action_mask.sum()</code>。</p>
|
||
<div class="callout callout-warn"><div class="callout-title">常见 bug</div><p>早期开源实现（包括早期 TRL agent example）漏了 observation mask，导致训练 metric 看起来在涨但 task success 下降——典型 reward hacking on tool output。OpenRLHF / verl / TRL 2024 版本都已修正，自己写 trainer 必须显式加。</p></div>
|
||
<h3 id="22-trajectory-level-gae-for-agent">2.2 Trajectory-level GAE for agent</h3>
|
||
<p>agent 轨迹长（50-200 步），reward 极稀疏（只在终点）。设：</p>
|
||
<ul><li>$r_t \in \mathbb{R}$：第 $t$ 步 reward（多数 $t$ 上 $r_t = 0$，终点 $r_T = R \in \{0, 1\}$）</li><li>$V_\phi(s_t)$：critic 估计</li></ul>
|
||
<p>TD residual：</p>
|
||
<p>$$\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$$</p>
|
||
<p>GAE：</p>
|
||
<p>$$A_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}$$</p>
|
||
<p><strong>LLM agent 中 $\gamma$ 怎么取？</strong> 取决于"step"的定义：</p>
|
||
<ul><li>如果 step = <strong>单 token</strong>：$\gamma$ 接近 1（token level discount 没意义）</li><li>如果 step = <strong>一次 thought-action-obs 循环</strong>：$\gamma \in [0.95, 0.99]$ 合理，控制长 horizon 的折扣</li></ul>
|
||
<p><strong>LLM agent 中 GAE 的退化</strong>：sparse terminal reward 下，$\lambda = 1$ + $\gamma = 1$ 等价于 sequence-level MC return 减 baseline。这正是 GRPO 直接做 trace-level reward 的隐含解释。</p>
|
||
<h3 id="23-ppo-loss-adapted-to-agent">2.3 PPO loss adapted to agent</h3>
|
||
<p>带 mask 的 PPO-Clip（按整条 trajectory 算，外层是 trajectory 期望，内层 sum over tokens）：</p>
|
||
<p>$$\boxed{\;L^{\text{CLIP-agent}}(\theta) = \mathbb{E}_{\tau \sim \pi_\text{old}}\!\left[\frac{\sum_{t=1}^{T} m_t \cdot \min\!\big(\rho_t A_t,\, \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon) A_t\big)}{\sum_{t=1}^{T} m_t}\right]\;}$$</p>
|
||
<p>其中 $\tau$ 是 trajectory（包含所有 $T$ 个 token / step），$m_t \in \{0, 1\}$ 是 agent action_mask（agent 自己生成的 token 为 1，observation/system token 为 0），$\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\theta_\text{old}}(a_t \mid s_t)$ 是 token-level importance ratio。注意外层 expectation 索引是 trajectory $\tau$，内层 sum 索引是 token $t$，不能混淆。</p>
|
||
<p>per-token KL penalty（写进 reward）：</p>
|
||
<p>$$\tilde{r}_t = m_t \cdot \big(\text{rule}\_\text{reward}_t - \beta \log \tfrac{\pi_\theta(a_t \mid s_t)}{\pi_\text{ref}(a_t \mid s_t)}\big)$$</p>
|
||
<p>注意只对 agent token 算 KL；observation token 的 $\pi$ 是没有意义的（它们是环境给的，不是 model sample 出的）。</p>
|
||
<h3 id="24-grpo-for-agentstrace-level-group-relative-advantage">2.4 GRPO for agents：trace-level group-relative advantage</h3>
|
||
<p>GRPO 在 agent 上更适用，因为：</p>
|
||
<ol><li><strong>省 critic</strong>——agent value 难学（长 horizon + sparse reward）</li><li><strong>trace-level reward 直接对应 trace-level advantage</strong>——不需要 per-step value</li><li><strong>同 prompt 多 rollout 自动 variance reduction</strong>——agent 任务通常 deterministic env，多次 rollout 给出真实 reward 方差</li></ol>
|
||
<p>公式（保留 PPO-Clip 结构，advantage 改组内归一化）：</p>
|
||
<p>$$\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\}) + \epsilon}$$</p>
|
||
<p>整条 trajectory 的所有 <strong>agent token</strong> 共享同一个 $\hat{A}_i$（observation token 仍然 mask 掉）：</p>
|
||
<p>$$L^{\text{GRPO-agent}}(\theta) = \mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{\sum_t m_{i,t}}\sum_{t=1}^{T_i} m_{i,t} \cdot \Big(\min(\rho_{i,t} \hat{A}_i, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_i) - \beta\, \text{KL}_{i,t}\Big)\right]$$</p>
|
||
<p>KL 通常用 K3 estimator（Schulman 2020 blog）：$\text{KL}_{i,t} = \exp(\log\pi_\text{ref} - \log\pi_\theta) - (\log\pi_\text{ref} - \log\pi_\theta) - 1$。</p>
|
||
<div class="callout callout-good"><div class="callout-title">GRPO-agent 的"四省"</div><p>列举如下。</p></div>
|
||
<ul><li>省 value model（一份显存）</li><li>省 per-token credit assignment（trace-level advantage）</li><li>省 reward shaping（rule-based terminal 就够）</li><li>省 hyperparameter（$c_v, \lambda$ 都不需要）</li></ul>
|
||
<div class="callout callout-warn"><div class="callout-title">GRPO-agent 的"三痛"</div><p>仍有三个 trade-off 需要承认。</p></div>
|
||
<ul><li>长 trajectory 上 trace-level advantage 太粗（所有 agent token 共享同一 $\hat{A}$）——长 trajectory 的 credit dilution</li><li>全 success / 全 fail group $\text{std} = 0$ 退化（agent 任务 reward 二值，常出现）</li><li>on-policy rollout 慢（agent rollout 含 tool I/O 延迟，远比 chat completion 慢）</li></ul>
|
||
<h2 id="3-reward-design-for-agents核心">§3 Reward design for agents（核心）</h2>
|
||
<p>reward 是 Agentic RL 的命门。<strong>Reward 错了，模型再大、算法再新也学不到东西</strong>；reward 对了，简单 GRPO 就能上 SOTA。</p>
|
||
<h3 id="31-outcome-reward-vs-process-reward">3.1 Outcome reward vs Process reward</h3>
|
||
<table><thead><tr><th>维度</th><th><strong>Outcome reward</strong></th><th><strong>Process reward</strong></th></tr></thead><tbody><tr><td>监督粒度</td><td>trajectory 终点 1 个 reward</td><td>每步 (or 每 subgoal) 1 个 reward</td></tr><tr><td>Label 来源</td><td>task success（test pass / answer match）</td><td>PRM / step verifier / human</td></tr><tr><td>稀疏度</td><td>极稀疏（多数 step 0）</td><td>dense</td></tr><tr><td>Credit assignment</td><td>难（长 horizon 上 GAE 也难)</td><td>易（每步直接打分）</td></tr><tr><td>Reward hacking</td><td>较低（rule-based 时）</td><td>较高（PRM 可被 hack）</td></tr><tr><td>实施难度</td><td>易（test-pass / 答案 match）</td><td>难（PRM 训练成本高）</td></tr></tbody></table>
|
||
<p><strong>面试 take</strong>：reasoning RL（R1）选 outcome reward 因为数学/代码可程序化验证；agent RL 主流也选 outcome reward 因为 agent 任务 ground truth 更明确（task 完成 yes/no）。<strong>process reward 主要用于 reasoning-heavy 任务</strong>（数学 step 标 PRM），在 tool-use agent 上较少。</p>
|
||
<h3 id="32-verifier-based-rewardrule-based">3.2 Verifier-based reward（rule-based）</h3>
|
||
<p>这是 Agentic RL 最干净的 reward 形式：把 reward 写成 <strong>可执行的 verifier 函数</strong>。</p>
|
||
<pre><code class="language-python">def verifier_reward(trajectory) -> float:
|
||
"""
|
||
trajectory: list of (thought, action, observation)
|
||
返回 0 或 1 的 outcome reward
|
||
"""
|
||
final_answer = trajectory[-1].final_answer
|
||
|
||
# 1. 数学题：精确匹配 ground truth
|
||
if task_type == "math":
|
||
return 1.0 if normalize_math(final_answer) == ground_truth else 0.0
|
||
|
||
# 2. 代码题：跑 unit test
|
||
if task_type == "code":
|
||
code = extract_code(final_answer)
|
||
pass_count = run_unit_tests(code, test_cases)
|
||
return pass_count / len(test_cases) # partial credit
|
||
|
||
# 3. SWE-bench: apply patch + run test
|
||
if task_type == "swe":
|
||
try:
|
||
apply_patch(repo, final_answer)
|
||
return 1.0 if run_test(repo, expected_test) else 0.0
|
||
except PatchError:
|
||
return 0.0
|
||
|
||
# 4. Web agent: task-specific verifier
|
||
if task_type == "webshop":
|
||
return webshop_grader(final_state) # 由 benchmark 提供</code></pre>
|
||
<p><strong>verifier-based reward 的核心优势</strong>：</p>
|
||
<ul><li>接近 ground truth，<strong>绕开 learned-RM 的 reward hacking 主要 failure mode</strong></li><li>可重复（同一 trajectory reward 一致），便于 advantage estimate</li><li>显存 / 算力开销极小（执行 verifier 比一次 LLM forward 便宜数百倍）</li></ul>
|
||
<p><strong>核心限制</strong>：只能用于"可验证任务"——math、code、formal verification、可 grader 化的 web/GUI task。开放式任务（写作、对话）仍需 RM。</p>
|
||
<h3 id="33-format-reward--shaping-reward">3.3 Format reward / shaping reward</h3>
|
||
<p>仅有 outcome reward 时 agent 经常学到"格式崩坏但偶然答对"的 trajectory——例如不写 <code><think></code> 块就直接 <code>Action: answer(42)</code>。<strong>Format reward</strong> 给一个轻量的格式约束信号：</p>
|
||
<pre><code class="language-python">def format_reward(trajectory) -> float:
|
||
"""
|
||
检查 trajectory 是否符合预期格式 schema
|
||
返回 [0, 1] 的连续 score
|
||
"""
|
||
score = 0.0
|
||
|
||
# 必须有 <think>...</think> 块
|
||
if "<think>" in trajectory.text and "</think>" in trajectory.text:
|
||
score += 0.3
|
||
|
||
# tool call 必须是合法 JSON
|
||
for action in trajectory.actions:
|
||
if is_valid_json(action.tool_call):
|
||
score += 0.1
|
||
else:
|
||
score -= 0.2 # 严重错误
|
||
|
||
# final answer 必须用 \boxed{...} 包裹（math task）
|
||
if has_boxed_answer(trajectory.final):
|
||
score += 0.2
|
||
|
||
return max(0.0, min(1.0, score))</code></pre>
|
||
<p><strong>Composite reward</strong> 的典型写法：</p>
|
||
<p>$$r_\text{total} = \alpha \cdot r_\text{outcome} + \beta \cdot r_\text{format} + \gamma \cdot r_\text{shaping}$$</p>
|
||
<p>R1 / R1-Zero 用 <code>accuracy_reward + format_reward</code> 的简单加和；ToolRL / RAGEN 等加 tool-call efficiency shaping。</p>
|
||
<h3 id="34-length-penalty防-agent-拖太长">3.4 Length penalty（防 agent 拖太长）</h3>
|
||
<p>agent RL 的一个 emergent failure mode：<strong>model 学到"拖长 trajectory 拿到正确答案的概率更高"</strong>——明明能 5 步解决的任务，agent 偏要走 50 步。这是 reward hacking 的一种形式。</p>
|
||
<p>缓解：</p>
|
||
<p>$$r_\text{adjusted} = r_\text{outcome} - \lambda \cdot \max(0, T - T_\text{target})$$</p>
|
||
<p>或更柔和的 sigmoid 形式：</p>
|
||
<p>$$r_\text{adjusted} = r_\text{outcome} \cdot \sigma\!\big(-(T - T_\text{target}) / \tau\big)$$</p>
|
||
<p>DAPO 报告 "overlong shaping"——超出 length budget 后 reward 指数衰减，避免 agent 拖到 context 上限。</p>
|
||
<h3 id="35-tool-call-shaping-reward">3.5 Tool-call shaping reward</h3>
|
||
<p>给"调对工具"奖励、给"调错工具 / 重复调用"惩罚：</p>
|
||
<pre><code class="language-python">def tool_shaping(trajectory) -> float:
|
||
score = 0.0
|
||
|
||
# 调对工具（task 关联性 heuristic）
|
||
if task_needs_search and any(a.tool == "search" for a in trajectory.actions):
|
||
score += 0.1
|
||
|
||
# 惩罚连续重复同样调用
|
||
consecutive_dup = count_consecutive_duplicate_calls(trajectory.actions)
|
||
score -= 0.05 * consecutive_dup
|
||
|
||
# 惩罚调用不存在的 tool
|
||
invalid_calls = sum(1 for a in trajectory.actions if a.tool not in TOOL_REGISTRY)
|
||
score -= 0.3 * invalid_calls
|
||
|
||
return score</code></pre>
|
||
<div class="callout callout-warn"><div class="callout-title">shaping reward 的风险</div><p>shaping 给得不当，agent 会 over-fit shaping signal 而忽略 outcome。<strong>主流做法</strong>：shaping reward weight ≪ outcome reward weight（典型 0.1 : 1），且 shaping 必须 capped（不能无限累加）。</p></div>
|
||
<h3 id="36-rlaif-for-agents用-llm-当-reward">3.6 RLAIF for agents：用 LLM 当 reward</h3>
|
||
<p>外部 verifier 难写时（开放式任务），用强 LLM 当 judge：</p>
|
||
<pre><code class="language-python">def llm_judge_reward(trajectory, judge_model) -> float:
|
||
"""
|
||
用 judge LLM 给 trajectory 打分
|
||
"""
|
||
prompt = f"""
|
||
Judge whether the agent completed this task successfully.
|
||
Task: {trajectory.task}
|
||
Final state: {trajectory.final_state}
|
||
Return JSON: {{"success": bool, "reasoning": str}}
|
||
"""
|
||
judgment = judge_model(prompt)
|
||
return 1.0 if judgment["success"] else 0.0</code></pre>
|
||
<p><strong>风险</strong>：</p>
|
||
<ul><li>judge LLM 自己有偏见（长度偏好、谄媚）→ 偏见放大到 student</li><li>judge 自己可能被 prompt-injected（agent trajectory 含恶意指令）</li><li>算力开销高（每个 trajectory 一次 judge call）</li></ul>
|
||
<p>主流缓解：(a) <strong>judge ensemble</strong>（3-5 个不同 model judge 取多数）；(b) <strong>judge with rubric</strong>（强约束输出结构）；(c) <strong>rule + LLM 混合</strong>（可验证部分 rule，开放部分 LLM）。</p>
|
||
<h2 id="4-long-horizon-credit-assignment">§4 Long-horizon credit assignment</h2>
|
||
<h3 id="41-稀疏-reward-是-agent-rl-的核心痛点">4.1 稀疏 reward 是 agent RL 的核心痛点</h3>
|
||
<p>经典 game RL（Atari、Mujoco）reward dense；LLM agent 在长 trajectory 上<strong>几乎所有 step reward = 0，只在终点有信号</strong>。这导致：</p>
|
||
<ul><li>value model 几乎学不到东西（中间 V 应该是什么？）</li><li>per-token policy gradient 方差极大</li><li>早期 step 与 reward 的因果链被稀释——"我第 3 步选了 search tool" 是不是导致了第 50 步答对？</li></ul>
|
||
<h3 id="42-discount--gae-在-agent-上的退化">4.2 Discount + GAE 在 agent 上的退化</h3>
|
||
<p>回顾 GAE：</p>
|
||
<p>$$A_t = \sum_l (\gamma\lambda)^l \delta_{t+l}, \quad \delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$</p>
|
||
<p>在 sparse terminal reward 下 ($r_t = 0$ for $t \lt T$, $r_T = R$)：</p>
|
||
<p>$$A_t = \gamma^{T-t} R - V(s_t) + \text{value correction terms}$$</p>
|
||
<p>即 <strong>advantage ≈ 折扣回报 - baseline</strong>。如果 $V_\phi$ 学不准（在 agent 上经常），这就退化为 raw MC return；GAE 的 bias-variance trade-off 失效。</p>
|
||
<p><strong>实践 take</strong>：agent RL 上 GAE 不如 group-relative advantage（GRPO）稳，这是 GRPO 比 PPO 在 agent 上 sample efficient 的根本原因之一。</p>
|
||
<h3 id="43-hindsight-relabeling-for-agents">4.3 Hindsight relabeling for agents</h3>
|
||
<p>Hindsight Experience Replay (Andrychowicz et al. 2017 NeurIPS) 起源于 robot manipulation：失败 trajectory 不丢，而是<strong>把"实际达成的状态"当成 goal</strong>重新标 reward。</p>
|
||
<p>LLM agent 版本：</p>
|
||
<pre><code class="language-python">def hindsight_relabel(trajectory):
|
||
"""
|
||
把 failed trajectory 改造成 "alternative task" 的 successful trajectory
|
||
"""
|
||
if trajectory.outcome == 1:
|
||
return [trajectory] # 成功的不动
|
||
|
||
# 假设 agent 在 web 上 navigate 时本来想买商品 A，但最后停在商品 B 页面
|
||
# → 改成"找到商品 B 的 trajectory"，reward = 1
|
||
alt_task = describe_terminal_state(trajectory.final_state)
|
||
relabeled = trajectory.with_task(alt_task)
|
||
relabeled.outcome = 1
|
||
return [trajectory, relabeled]</code></pre>
|
||
<div class="callout callout-info"><div class="callout-title">Agentic Hindsight 的难点</div><p>需要"任意 terminal state 都能被描述成一个合理 task"。对开放 web 环境（购物、导航）容易；对 math/code 任务很难（错答案不能被改成"另一个题的对答案"）。</p></div>
|
||
<h3 id="44-subgoal-decomposition--process-reward">4.4 Subgoal decomposition + process reward</h3>
|
||
<p>长 trajectory 切成 subgoal 是另一条 credit assignment 路线：</p>
|
||
<ul><li>把 100 步 trajectory 切成 5 个 subgoal × 20 步</li><li>每个 subgoal 终点给一个 process reward（subgoal 是否完成）</li><li>subgoal reward 累加成 trajectory reward</li></ul>
|
||
<p>实现方式：</p>
|
||
<ul><li><strong>hand-crafted subgoal</strong>：人写每个 subgoal 的判据（如 "找到购物车页面" 触发 subgoal-1 reward）</li><li><strong>LLM-decomposed subgoal</strong>：让一个 planner LLM 把 task 拆 subgoal，verifier 判每个 subgoal</li><li><strong>PRM-style step reward</strong>：训一个 PRM 评每步好坏（Math-Shepherd 思路）</li></ul>
|
||
<div class="callout callout-warn"><div class="callout-title">subgoal RL 的代价</div><p>subgoal boundary 错画会导致 agent 学到"刻意触发 subgoal reward 而不真正完成 task"。这是 process reward 通病。</p></div>
|
||
<h3 id="45-per-step-kl-penalty-防止-policy-collapse">4.5 Per-step KL penalty 防止 policy collapse</h3>
|
||
<p>agent 长 trajectory 上 policy 容易"全押"（每步生成 high-confidence token），导致 entropy 崩坏：</p>
|
||
<p>$$\tilde{r}_t = m_t \cdot \big(r_t - \beta \cdot \text{KL}(\pi_\theta(\cdot \mid s_t) \| \pi_\text{ref}(\cdot \mid s_t))\big)$$</p>
|
||
<p><strong>关键</strong>：KL 必须 per-step + 只对 agent token 算。一旦 KL 算到 observation token 上，policy 会被"惩罚 mimicking 环境文本"——但这惩罚没有意义（policy 不该 mimick 环境，只是用 observation 做条件）。</p>
|
||
<p>R1-Zero 用 $\beta = 0.001$，agent task 上常用 $\beta \in [0.001, 0.05]$。</p>
|
||
<h2 id="5-self-rewarding--exploration-in-agent-rl">§5 Self-rewarding & exploration in agent RL</h2>
|
||
<h3 id="51-self-rewarding-lmyuan-et-al-2024-meta-arxiv-240110020">5.1 Self-Rewarding LM（Yuan et al. 2024 Meta arXiv 2401.10020）</h3>
|
||
<p>核心 idea：让 policy 自己当 judge，iterative DPO：</p>
|
||
<pre><code>
|
||
Iteration k:
|
||
1. policy_k 生成 multiple responses per prompt
|
||
2. policy_k 自己 LLM-as-judge 打分（也是 policy_k 的另一个 prompt）
|
||
3. 高分 vs 低分构成 preference pair
|
||
4. policy_{k+1} = DPO(policy_k, preference_pair)</code></pre>
|
||
<p>效果：在 AlpacaEval 上 iterative 自打分能持续涨点。</p>
|
||
<h3 id="52-self-rewarding-在-agent-上的危险">5.2 Self-Rewarding 在 agent 上的危险</h3>
|
||
<p>agent 任务 vs alignment 任务最大差异：alignment 有"客观偏好分布"（人觉得有用、礼貌），可以 LLM judge；<strong>agent 任务有客观 ground truth（test pass / task success）</strong>——self-rewarding 会:</p>
|
||
<ul><li>judge 自己可能错（agent 答错了但自评对）→ training collapse</li><li>iterative drift：每轮把"自己以为对"的轨迹强化，离 ground truth 越走越远</li><li>探索退化：自评高分 prefer 已知 pattern，反而抑制探索新工具</li></ul>
|
||
<p><strong>主流做法</strong>：agent RL <strong>优先用 rule-based ground truth</strong>，self-rewarding 仅作为辅助信号（如开放式任务 fallback）。</p>
|
||
<h3 id="53-exploration-in-agent-rl">5.3 Exploration in agent RL</h3>
|
||
<p>agent action space 包括：</p>
|
||
<ul><li><strong>token-level exploration</strong>：sampling temperature，控制 token 选择多样性</li><li><strong>tool-call-level exploration</strong>：每个 thought-action 周期选择不同 tool / 不同 query</li><li><strong>trajectory-level exploration</strong>：完全不同 trajectory plan</li></ul>
|
||
<p>经典做法：</p>
|
||
<table><thead><tr><th>方法</th><th>实现</th></tr></thead><tbody><tr><td>Temperature schedule</td><td>rollout 时 $T \in [0.7, 1.2]$，训练随轮次降温</td></tr><tr><td>Top-p / Top-k</td><td>限制 sampling 范围避免 outlier token</td></tr><tr><td>ε-tool-choice</td><td>以 $\epsilon$ 概率随机选 tool（替代 LLM-policy 选择）</td></tr><tr><td>Diverse beam</td><td>多 trajectory 用 diverse beam search 保证多样性</td></tr><tr><td>GRPO group sampling</td><td>同 prompt $G = 16$ 个 rollout，天然 exploration</td></tr></tbody></table>
|
||
<p><strong>RAGEN / StarPO 的关键 insight</strong>（Wang et al. 2025 arXiv 2504.20073）：multi-turn agent RL 中 <strong>rollout 多样性是 collapse 的"防火墙"</strong>——单 trajectory 训练会让 policy 退化为 deterministic mode。</p>
|
||
<h3 id="54-curriculum--difficulty-scheduling">5.4 Curriculum & difficulty scheduling</h3>
|
||
<p>agent 任务从易到难排序，按 model 当前能力 schedule：</p>
|
||
<ul><li><strong>WebRL</strong> (Qi et al. 2024 ICLR-25 arXiv 2411.02337) 用 self-evolving curriculum——失败任务被记录，下轮加入 buffer</li><li><strong>Absolute Zero</strong> / R-Zero 用 learnability reward：选 model success rate ≈ 50% 的任务（最有学习信号）</li></ul>
|
||
<div class="callout callout-info"><div class="callout-title">Curriculum 是"长 horizon agent RL 的隐性 component"</div><p>不是 algorithm contribution，但<strong>实测对 sample efficiency 比换 algorithm 影响更大</strong>。R1 / R1-Zero 的 reasoning RL 也用了 implicit curriculum（数据难度逐渐升级）。</p></div>
|
||
<h2 id="6-specific-algorithms代表性-agentic-rl-papers">§6 Specific algorithms（代表性 Agentic RL papers）</h2>
|
||
<p>按 "时间 + 数据/任务类型" 排序，每个给一句话 + 关键公式。</p>
|
||
<h3 id="61-vpt-baker-et-al-2022-neurips-openai-arxiv-220611795">6.1 VPT (Baker et al. 2022 NeurIPS OpenAI, arXiv 2206.11795)</h3>
|
||
<p><strong>Setting</strong>: Minecraft，先用 70k hours youtube 视频 pretrain inverse dynamics model (IDM)，再用 IDM 自动标 action label → behavior clone → RL fine-tune.</p>
|
||
<p><strong>Key</strong>: 第一个大规模"视频 → action label → policy → RL"的 pipeline。Agent RL 的早期蓝本，证明了 <strong>scaling RL with imitation pretrain</strong> 可行。</p>
|
||
<h3 id="62-agenttuning-zeng-et-al-2023-thu-arxiv-231012823">6.2 AgentTuning (Zeng et al. 2023 THU arXiv 2310.12823)</h3>
|
||
<p><strong>Setting</strong>: 构建 AgentInstruct dataset（6 个 agent task 的 demonstration），多任务 SFT。</p>
|
||
<p><strong>Key</strong>: 不是 RL，是 <strong>agent SFT</strong>——但这是 Agentic RL 的标准 warm-start 步骤。Llama-2 经 AgentTuning 后 agent task 平均 +50%。</p>
|
||
<div class="callout callout-info"><div class="callout-title">AgentTuning 的位置</div><p>在 Agentic RL pipeline 里，AgentTuning-style SFT 是 RL 之前的必要 warm-start。直接 from-scratch RL 跑 agent 极其困难，因为 base model 不知道怎么 emit 合法 tool call schema。</p></div>
|
||
<h3 id="63-agent-flan-chen-et-al-2024-acl-findings-arxiv-240312881">6.3 Agent-FLAN (Chen et al. 2024 ACL Findings, arXiv 2403.12881)</h3>
|
||
<p><strong>Setting</strong>: Agent SFT 数据分三类——multi-turn dialogue / formatted tool call / negative examples（拒绝/失败案例）。</p>
|
||
<p><strong>Key</strong>: <strong>negative example 显著缓解 hallucinated tool call</strong>——SFT 不只看"怎么用对"，还看"为什么这样不对"。是 Agent SFT 工程化的重要 milestone。</p>
|
||
<h3 id="64-reft-trung-et-al-2024-acl-arxiv-240108967">6.4 ReFT (Trung et al. 2024 ACL arXiv 2401.08967)</h3>
|
||
<p><strong>Setting</strong>: math reasoning agent，先 SFT warm-start，再 PPO with rule-based outcome reward (answer correctness)。</p>
|
||
<p><strong>Key formula</strong>（标准 PPO + verifier）：</p>
|
||
<p>$$r(\tau) = \mathbb{1}[\text{answer}(\tau) = y^*] - \beta \cdot \text{KL}(\pi_\theta \| \pi_\text{SFT})$$</p>
|
||
<p><strong>Result</strong>: GSM8K +9pp over SFT，MathQA +7pp。证明 PPO + outcome reward 在 reasoning agent 上稳定可行——这是 R1 的前身实验。</p>
|
||
<h3 id="65-deepseek-r1-deepseek-ai-2025-arxiv-250112948-在-agent-上的扩展">6.5 DeepSeek-R1 (DeepSeek-AI 2025 arXiv 2501.12948) 在 agent 上的扩展</h3>
|
||
<p>R1 本身不是 agent paper，但 R1 的 GRPO + rule-based reward + format reward 方法论被 ToolRL / ReSearch / RAGEN 直接继承。<strong>R1 = Agentic RL 的算法基线模板</strong>。</p>
|
||
<p>复习 GRPO 在 agent 上的应用：</p>
|
||
<ul><li>per-prompt $G$ rollouts</li><li>trace-level reward（rule-based）</li><li>group-relative advantage</li><li>per-step KL with K3 estimator</li><li>agent token mask</li></ul>
|
||
<h3 id="66-toolrl-qian-et-al-2025-arxiv-250413958">6.6 ToolRL (Qian et al. 2025 arXiv 2504.13958)</h3>
|
||
<p><strong>Setting</strong>: tool-augmented LLM 上做 GRPO，reward = correctness + format + tool-use efficiency。</p>
|
||
<p><strong>Key formula</strong>:</p>
|
||
<p>$$r(\tau) = r_\text{correct} + \alpha \cdot r_\text{format} + \gamma \cdot r_\text{tool-eff}$$</p>
|
||
<p>其中 $r_\text{tool-eff}$ 惩罚冗余/无效 tool call。</p>
|
||
<p><strong>Result</strong>: 在 BFCL (Berkeley Function Calling Leaderboard) 上 7B model 接近 GPT-4 性能。开源验证 GRPO + tool shaping 的稳定性。</p>
|
||
<h3 id="67-research-chen-et-al-2025-arxiv-250319470">6.7 ReSearch (Chen et al. 2025 arXiv 2503.19470)</h3>
|
||
<p><strong>Setting</strong>: search-augmented agent，把 search 当 first-class action，reward = answer correctness only（rule-based）。</p>
|
||
<p><strong>Key idea</strong>: 不需要 process reward 也能学会 multi-hop search，<strong>outcome-only + GRPO 足够</strong>——前提是 base model 经 SFT warm-start 已能 emit 合法 search query。</p>
|
||
<h3 id="68-ragen--starpo-wang-et-al-2025-arxiv-250420073">6.8 RAGEN / StarPO (Wang et al. 2025 arXiv 2504.20073)</h3>
|
||
<p><strong>Paper</strong>: "Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning".</p>
|
||
<p><strong>Setting</strong>: multi-turn RL agent training framework，state-action token-level loss + critic-free。</p>
|
||
<p><strong>Key contributions</strong>:</p>
|
||
<ol><li><strong>StarPO</strong>（<strong>S</strong>tate-<strong>T</strong>hinking-<strong>A</strong>ctions-<strong>R</strong>eward Policy Optimization）：critic-free，整段 trajectory 共享 advantage，加 token mask 严格只在 agent token 上算 loss。<strong>StarPO-S</strong> 变种引入 fine-grained reasoning-aware reward + 可选 critic incorporation，进一步缓解多轮 reward sparsity（论文摘要原话）</li><li><strong>rollout 多样性 = collapse 防火墙</strong>：实测 group size $G = 16$ 比 $G = 4$ 显著更稳</li><li><strong>trajectory length 信号</strong>：失败 trajectory length 大时 reward shaping 应加 length penalty</li></ol>
|
||
<h3 id="69-webrl-qi-et-al-2024-iclr-25-arxiv-241102337">6.9 WebRL (Qi et al. 2024 ICLR-25 arXiv 2411.02337)</h3>
|
||
<p><strong>Setting</strong>: Web agent (WebArena)，self-evolving curriculum + ORM + retrospective rollout。</p>
|
||
<p><strong>Key components</strong>:</p>
|
||
<ul><li>ORM (Outcome Reward Model) 训自 task success → 在线给 reward</li><li>失败 task 进入 curriculum buffer，下一轮加大权重</li><li>retrospective rollout：失败 trajectory 用 LLM 改造成 "正确 trajectory" 重新 SFT</li></ul>
|
||
<p><strong>Result</strong>: Llama-3.1-8B 在 WebArena 上 43%（vs GPT-4 14.4%），证明小 model + 好 RL pipeline > 大 model + zero-shot。</p>
|
||
<h3 id="610-agentq-putta-et-al-2024-arxiv-240807199">6.10 AgentQ (Putta et al. 2024 arXiv 2408.07199)</h3>
|
||
<p><strong>Setting</strong>: web agent，MCTS + AI critique + offline DPO。</p>
|
||
<p><strong>Key idea</strong>: MCTS 搜出"reward-balanced" preference pair（高分 trajectory vs 低分 trajectory），用 DPO offline 训练。不需要 online RL infra，对算力受限场景实用。</p>
|
||
<h3 id="611-webgum-furuta-et-al-2024-iclr-arxiv-230511854">6.11 WebGUM (Furuta et al. 2024 ICLR arXiv 2305.11854)</h3>
|
||
<p><strong>Setting</strong>: HTML + screenshot multimodal web agent，offline SFT from demonstrations（<strong>不是 RL fine-tune</strong> — 论文是 imitation/supervised 范式）。</p>
|
||
<p><strong>Key</strong>: 把网页 DOM + 截图同时喂给 model，相比纯 text 提升 grounding 准确率。是 Computer-Use / web agent RL（如 WebRL, AgentQ）的 dataset + base 阶段。后续 web agent RL 工作常在 WebGUM 这类 base 上做 RL fine-tune。</p>
|
||
<h3 id="612-coderl-le-et-al-2022-neurips-arxiv-220701780">6.12 CodeRL (Le et al. 2022 NeurIPS arXiv 2207.01780)</h3>
|
||
<p><strong>Setting</strong>: 代码生成 + actor-critic，reward = unit test pass。</p>
|
||
<p><strong>Key formula</strong>: 经典 actor-critic + critic 用作 token-level baseline，PG 信号来自最终 test-pass。</p>
|
||
<h3 id="613-ppocoder-shojaee-et-al-2023-arxiv-230113816">6.13 PPOCoder (Shojaee et al. 2023 arXiv 2301.13816)</h3>
|
||
<p><strong>Setting</strong>: code generation + PPO，composite reward = compilable + functional correctness。</p>
|
||
<p><strong>Key</strong>: 早期把 PPO 用在 code-gen 上的尝试。证明 multi-component reward 比单 test-pass 训练更稳。</p>
|
||
<h3 id="614-swe-rl-wei-et-al-2025-meta-fair-arxiv-250218449">6.14 SWE-RL (Wei et al. 2025 Meta FAIR arXiv 2502.18449)</h3>
|
||
<p><strong>Setting</strong>: SWE-bench / GitHub PR 数据，rule-based reward = patch similarity + test-pass，GRPO。</p>
|
||
<p><strong>Key features</strong>:</p>
|
||
<ul><li>数据规模：Meta 用 GitHub PR commit history 构造 76M+ context-issue-patch 三元组</li><li>reward：edit similarity (oracle patch ↔ predicted patch) + test pass binary</li><li>算法：纯 GRPO + format reward</li></ul>
|
||
<p><strong>Result</strong>: Llama-3.3-70B + SWE-RL 在 SWE-bench Verified 上 41%（无 scaffold），证明 <strong>rule-based RL on real PR data 可以让 model 学到 emergent reasoning behavior</strong>——例如 file-level retrieval planning、root cause analysis、test self-validation。</p>
|
||
<h3 id="615-openvla-kim-et-al-2024-arxiv-240609246">6.15 OpenVLA (Kim et al. 2024 arXiv 2406.09246)</h3>
|
||
<p><strong>Setting</strong>: vision-language-action 7B 模型 for robot manipulation，970K 真实机器人 demonstrations 上 fine-tune (开源 base + LoRA fine-tuning recipe)；论文重点是 imitation learning + parameter-efficient adaptation，<strong>不是 task-specific RL fine-tune</strong>。</p>
|
||
<p><strong>Key</strong>: 是 robot agent 而非 LLM agent；常被作为"LLM agent vs robot agent"对比讨论。后续工作（如 OpenVLA-OFT、π-RL 系列）才在 OpenVLA base 上做 RL，但 OpenVLA 论文本身没做 RL。</p>
|
||
<h3 id="616-anthropic-computer-use公开知识--训练细节未披露">6.16 Anthropic Computer-Use（公开知识 — 训练细节未披露）</h3>
|
||
<p>Anthropic Claude 3.5 Sonnet (new) 2024-10-22 起支持 Computer-Use；3.7 / 4.0 / 4.5 / Opus 4.x 持续迭代。</p>
|
||
<p><strong>公开资料只描述了能力 + 安全护栏，训练算法 / reward 形式未披露</strong>：</p>
|
||
<ul><li>action space = 屏幕坐标 + 键盘事件 + 鼠标事件（截图作为 observation）</li><li>安全性：constitutional AI 风格的护栏 + 红队 + prompt-injection 防御</li><li>system card 提到训练涉及人工演示 + 合成数据，但<strong>没有公开</strong> RLHF / GRPO / verifier reward 等具体细节</li></ul>
|
||
<p>社区<strong>推测</strong>（<strong>仅推测</strong>，非官方）：可能用 GRPO/PPO + verifier-based reward + Constitutional AI 风格的 AI feedback，但无任何官方确认。面试时讨论 Computer-Use 训练应明确区分 "公开能力" vs "推测内部细节"。</p>
|
||
<h2 id="7-code-patternspytorch--伪代码">§7 Code patterns（PyTorch / 伪代码）</h2>
|
||
<p>实现 Agentic RL 时最容易写错的几段。每段独立可读。</p>
|
||
<h3 id="71-agent-rolloutstate-action-reward-收集">7.1 Agent rollout（state, action, reward 收集）</h3>
|
||
<pre><code class="language-python">import torch
|
||
from dataclasses import dataclass
|
||
|
||
@dataclass
|
||
class Step:
|
||
obs: str # 上一步 observation 或 prompt
|
||
thought_tokens: list[int] # agent 生成的 thought
|
||
action_tokens: list[int] # agent 生成的 tool_call JSON
|
||
tool_name: str
|
||
tool_args: dict
|
||
observation: str # tool 返回结果
|
||
done: bool
|
||
|
||
def rollout(policy, env, prompt, max_steps=20, max_tokens_per_step=512):
|
||
"""
|
||
一条 agent trajectory rollout
|
||
返回: trajectory (list of Step), final reward
|
||
"""
|
||
trajectory = []
|
||
history = prompt
|
||
for step_idx in range(max_steps):
|
||
# ── agent 生成 (thought, action) ──
|
||
agent_output = policy.generate(
|
||
prompt=history,
|
||
stop_tokens=["</action>"],
|
||
max_new_tokens=max_tokens_per_step,
|
||
temperature=0.7,
|
||
)
|
||
thought, action_json = parse_thought_action(agent_output)
|
||
|
||
# ── 调用 tool ──
|
||
tool_name = action_json["tool"]
|
||
tool_args = action_json["args"]
|
||
if tool_name == "final_answer":
|
||
obs = action_json["answer"]
|
||
done = True
|
||
else:
|
||
obs = env.call(tool_name, tool_args)
|
||
done = False
|
||
|
||
# 更新 history（拼回 prompt）
|
||
history = history + agent_output + f"\n<obs>{obs}</obs>\n"
|
||
|
||
trajectory.append(Step(
|
||
obs=history, # 前置 history
|
||
thought_tokens=tokenize(thought),
|
||
action_tokens=tokenize(action_json),
|
||
tool_name=tool_name,
|
||
tool_args=tool_args,
|
||
observation=obs,
|
||
done=done,
|
||
))
|
||
|
||
if done or step_idx == max_steps - 1:
|
||
break
|
||
|
||
# ── 终点 reward ──
|
||
final_reward = env.compute_reward(trajectory)
|
||
return trajectory, final_reward</code></pre>
|
||
<h3 id="72-trajectory-level-gae-advantage">7.2 Trajectory-level GAE advantage</h3>
|
||
<pre><code class="language-python">import torch
|
||
|
||
def trajectory_gae(rewards, values, dones, gamma=0.99, lam=0.95):
|
||
"""
|
||
计算 step-level GAE advantage
|
||
rewards: [T] per-step reward (多数 = 0, 终点 = R)
|
||
values: [T+1] V(s_0)...V(s_T), V(s_T) 应为 0 (terminal)
|
||
dones: [T] 1 if terminal else 0
|
||
返回: advantages [T], returns [T]
|
||
"""
|
||
T = rewards.shape[0]
|
||
advantages = torch.zeros_like(rewards)
|
||
gae = 0.0
|
||
for t in reversed(range(T)):
|
||
non_term = 1.0 - dones[t]
|
||
delta = rewards[t] + gamma * values[t + 1] * non_term - values[t]
|
||
gae = delta + gamma * lam * non_term * gae
|
||
advantages[t] = gae
|
||
returns = advantages + values[:T]
|
||
return advantages, returns</code></pre>
|
||
<h3 id="73-ppo-loss-adapted-to-agentwith-actionmask">7.3 PPO loss adapted to agent（with action_mask）</h3>
|
||
<p><strong>这是 Agentic RL 最重要的一段代码</strong>。区别 RLHF 的 PPO：必须显式 mask out observation token。</p>
|
||
<pre><code class="language-python">import torch
|
||
import torch.nn.functional as F
|
||
|
||
def ppo_agent_step(policy, value, batch, eps_clip=0.2, c_v=0.5, c_e=0.01):
|
||
"""
|
||
batch:
|
||
input_ids: [B, L] full trajectory tokens (prompt + thought + action + obs ...)
|
||
action_mask: [B, L] 1 = agent-generated token, 0 = prompt/observation/pad
|
||
old_log_probs: [B, L] log π_θ_old at sample time, 0 at masked positions
|
||
advantages: [B, L] step-level GAE advantages (broadcast to all agent tokens of that step)
|
||
returns: [B, L] GAE returns for value loss
|
||
"""
|
||
logits = policy(batch["input_ids"]).logits # [B, L, V]
|
||
log_probs = F.log_softmax(logits[:, :-1], dim=-1) # [B, L-1, V]
|
||
targets = batch["input_ids"][:, 1:].unsqueeze(-1)
|
||
new_log_probs = log_probs.gather(-1, targets).squeeze(-1) # [B, L-1]
|
||
new_log_probs = F.pad(new_log_probs, (1, 0)) # 对齐 [B, L]
|
||
|
||
# ── 关键: action_mask ──
|
||
mask = batch["action_mask"].float()
|
||
# 只对 agent 自己生成的 token 算 ratio / loss
|
||
ratio = torch.exp((new_log_probs - batch["old_log_probs"]) * mask)
|
||
# observation 位置: ratio = exp(0) = 1, 不影响 surr1/surr2
|
||
|
||
A = batch["advantages"]
|
||
surr1 = ratio * A
|
||
surr2 = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip) * A
|
||
# mask 后求均值（避免 observation token 拉低 loss scale）
|
||
policy_loss = -((torch.min(surr1, surr2) * mask).sum() / mask.sum().clamp_min(1.0))
|
||
|
||
# value loss: 也只在 agent token 上算（observation 的 V 没意义）
|
||
V = value(batch["input_ids"]).squeeze(-1) # [B, L]
|
||
value_loss = (((V - batch["returns"]) ** 2) * mask).sum() / mask.sum().clamp_min(1.0)
|
||
|
||
# entropy bonus: 只在 agent token 上
|
||
probs = log_probs.exp()
|
||
entropy = -(probs * log_probs).sum(-1) # [B, L-1]
|
||
entropy = F.pad(entropy, (1, 0))
|
||
entropy_bonus = (entropy * mask).sum() / mask.sum().clamp_min(1.0)
|
||
|
||
loss = policy_loss + c_v * value_loss - c_e * entropy_bonus
|
||
|
||
# 监控
|
||
with torch.no_grad():
|
||
approx_kl = ((ratio - 1) - torch.log(ratio.clamp_min(1e-8))) * mask
|
||
approx_kl = approx_kl.sum() / mask.sum().clamp_min(1.0)
|
||
return loss, {
|
||
"policy": policy_loss.item(),
|
||
"value": value_loss.item(),
|
||
"entropy": entropy_bonus.item(),
|
||
"approx_kl": approx_kl.item(),
|
||
}</code></pre>
|
||
<div class="callout callout-warn"><div class="callout-title">agent_mask 的 5 个易错点</div></div>
|
||
<ul><li>必须 cover <strong>prompt tokens</strong>：prompt 不是 agent generate 的，mask = 0</li><li>必须 cover <strong>all observation tokens</strong>：tool 返回的每一个 token（甚至包括 <code><obs></code> 标签本身）mask = 0</li><li>必须 cover <strong>all padding</strong>：右 pad 的位置 mask = 0</li><li><strong>separator token</strong>（如 <code><action></code>, <code></thought></code>）算 agent token，mask = 1</li><li>multi-turn batch 里<strong>不同 trajectory 的 mask pattern 不同</strong>，必须 per-sample 算</li></ul>
|
||
<h3 id="74-grpo-group-relative-reward-on-agent-trajectories">7.4 GRPO group-relative reward on agent trajectories</h3>
|
||
<pre><code class="language-python">import torch
|
||
import torch.nn.functional as F
|
||
|
||
def grpo_agent_loss(policy, ref_policy, batch, eps_clip=0.2, beta=0.04):
|
||
"""
|
||
batch:
|
||
input_ids: [N, L] N = sum_b G samples in batch
|
||
action_mask: [N, L] agent-generated token mask
|
||
old_log_probs: [N, L] detached log probs at rollout time
|
||
rewards: [N] trajectory-level outcome reward
|
||
group_id: [N] same prompt → same group_id
|
||
"""
|
||
rewards = batch["rewards"]
|
||
gid = batch["group_id"].long()
|
||
|
||
# ── 组内归一化 ──
|
||
num_groups = int(gid.max().item()) + 1
|
||
counts = torch.zeros(num_groups, device=rewards.device).scatter_add_(
|
||
0, gid, torch.ones_like(rewards))
|
||
sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
|
||
0, gid, rewards)
|
||
group_mean = sums / counts.clamp_min(1.0)
|
||
diff_sq = (rewards - group_mean[gid]) ** 2
|
||
sq_sums = torch.zeros(num_groups, device=rewards.device).scatter_add_(
|
||
0, gid, diff_sq)
|
||
group_std = (sq_sums / counts.clamp_min(1.0)).sqrt()
|
||
|
||
A = (rewards - group_mean[gid]) / (group_std[gid] + 1e-8) # [N]
|
||
A = A.unsqueeze(-1) # [N, 1] 整段共享
|
||
|
||
# ── log-prob ratio ──
|
||
logits = policy(batch["input_ids"]).logits[:, :-1]
|
||
log_probs = F.log_softmax(logits, dim=-1)
|
||
tgt = batch["input_ids"][:, 1:].unsqueeze(-1)
|
||
new_log_probs = log_probs.gather(-1, tgt).squeeze(-1)
|
||
new_log_probs = F.pad(new_log_probs, (1, 0)) # [N, L]
|
||
mask = batch["action_mask"].float()
|
||
|
||
ratio = torch.exp((new_log_probs - batch["old_log_probs"]) * mask)
|
||
|
||
# ── PPO-Clip surrogate (advantage broadcast 到整段) ──
|
||
surr1 = ratio * A
|
||
surr2 = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip) * A
|
||
|
||
# ── KL with K3 estimator (Schulman 2020 blog) ──
|
||
with torch.no_grad():
|
||
ref_logits = ref_policy(batch["input_ids"]).logits[:, :-1]
|
||
ref_log_probs = F.log_softmax(ref_logits, dim=-1)
|
||
ref_token_lp = ref_log_probs.gather(-1, tgt).squeeze(-1)
|
||
ref_token_lp = F.pad(ref_token_lp, (1, 0))
|
||
delta = ref_token_lp - new_log_probs # log(π_ref / π_θ)
|
||
kl_per_token = torch.exp(delta) - delta - 1.0 # K3, non-negative
|
||
|
||
token_obj = torch.min(surr1, surr2) - beta * kl_per_token # [N, L]
|
||
seq_len = mask.sum(dim=-1).clamp_min(1.0) # [N]
|
||
per_seq = (token_obj * mask).sum(dim=-1) / seq_len # [N]
|
||
loss = -per_seq.mean()
|
||
|
||
return loss, {
|
||
"reward_mean": rewards.mean().item(),
|
||
"advantage_std": A.squeeze(-1).std().item(),
|
||
"kl": (kl_per_token * mask).sum().item() / mask.sum().clamp_min(1.0).item(),
|
||
}</code></pre>
|
||
<h3 id="75-outcome--step-reward-combination">7.5 Outcome + step reward combination</h3>
|
||
<pre><code class="language-python">def composite_reward(trajectory, weights=None):
|
||
"""
|
||
把 outcome / format / shaping reward 合成最终 reward
|
||
weights: dict[str, float]
|
||
"""
|
||
if weights is None:
|
||
weights = {"outcome": 1.0, "format": 0.2, "tool_eff": 0.1, "length": -0.05}
|
||
|
||
r = {}
|
||
r["outcome"] = outcome_verifier(trajectory) # in {0, 1}
|
||
r["format"] = format_score(trajectory) # in [0, 1]
|
||
r["tool_eff"] = tool_efficiency_score(trajectory) # in [-1, 1]
|
||
r["length"] = max(0, len(trajectory.steps) - target_len) # excess steps
|
||
|
||
total = sum(weights[k] * r[k] for k in r)
|
||
return total, r</code></pre>
|
||
<div class="callout callout-info"><div class="callout-title">process-then-outcome 复合形式</div><p>若有 PRM，可以：先 PRM 给 step-level shaping，但<strong>最终轨迹 reward 至少 50% 由 outcome 决定</strong>，避免 agent 偷只关心 PRM score 的局部最优。</p></div>
|
||
<h3 id="76-verifier-based-rewardcode-test--math-match">7.6 Verifier-based reward（code test / math match）</h3>
|
||
<pre><code class="language-python">import re
|
||
import subprocess
|
||
|
||
def verifier_reward(trajectory, task_type, ground_truth):
|
||
"""
|
||
rule-based outcome reward
|
||
"""
|
||
final = trajectory.final_answer
|
||
|
||
if task_type == "math":
|
||
return float(extract_boxed_answer(final) == normalize(ground_truth))
|
||
|
||
if task_type == "code":
|
||
code = extract_python_code(final)
|
||
if code is None:
|
||
return 0.0
|
||
passed = 0
|
||
for test in ground_truth["tests"]:
|
||
try:
|
||
result = subprocess.run(
|
||
["python", "-c", code + "\n" + test],
|
||
capture_output=True, timeout=5,
|
||
)
|
||
if result.returncode == 0:
|
||
passed += 1
|
||
except subprocess.TimeoutExpired:
|
||
continue
|
||
return passed / len(ground_truth["tests"])
|
||
|
||
if task_type == "swe":
|
||
patch = extract_unified_diff(final)
|
||
if patch is None:
|
||
return 0.0
|
||
success = apply_patch_and_run_test(
|
||
repo=ground_truth["repo"],
|
||
patch=patch,
|
||
test=ground_truth["test"],
|
||
)
|
||
return float(success)
|
||
|
||
if task_type == "webshop":
|
||
return webshop_grader(trajectory.final_state, ground_truth)
|
||
|
||
raise ValueError(f"Unknown task type: {task_type}")</code></pre>
|
||
<h2 id="8-frontier-2024-2026-关键趋势">§8 Frontier (2024-2026 关键趋势)</h2>
|
||
<h3 id="81-critic-free-rl-is-the-new-default">8.1 Critic-free RL is the new default</h3>
|
||
<p>DeepSeek-R1 (Jan 2025) 之后，<strong>critic-free RL（GRPO 系）成为开源 agent RL 主流</strong>。原因：</p>
|
||
<ul><li>value model 在长 horizon agent 上几乎学不动</li><li>一份模型显存省下来可以加大 batch / group size</li><li>调参简单（不用调 $c_v$, value lr）</li></ul>
|
||
<p>verl (ByteDance 2024+)、OpenRLHF、TRL 都已把 GRPO / RLOO / ReMax 当 first-class trainer。</p>
|
||
<h3 id="82-rule-based-reward-在-agent-上的胜利">8.2 Rule-based reward 在 agent 上的胜利</h3>
|
||
<p>WebRL / ReSearch / SWE-RL / RAGEN 共同点：<strong>outcome reward + format reward，rule-based</strong>，避开 learned RM 的 reward hacking 主 failure mode。</p>
|
||
<p><strong>为什么 rule-based 突然变可行</strong>：</p>
|
||
<ul><li>Agent task 比 alignment task 更"可程序化验证"——test pass / answer match / task complete</li><li>DeepSeek-R1 证明 rule-based 在 LLM RL 上 stable 且 scalable</li><li>learned RM 在 multi-turn 上更容易 hack（轨迹空间大）</li></ul>
|
||
<h3 id="83-long-horizon-training-的-infra-challenge">8.3 Long-horizon training 的 infra challenge</h3>
|
||
<p>agent RL rollout 慢，因为每个 trajectory 包含 tool I/O 延迟。infra 趋势：</p>
|
||
<ul><li><strong>vLLM / SGL 异步 rollout</strong>：把 generation 与 training 解耦，rollout 池化</li><li><strong>Sandboxed execution</strong>：tool execution 在 isolated container，并行化</li><li><strong>Trajectory queue</strong>：rollout worker / training worker 异步，trajectory 通过 message queue 流转</li><li><strong>off-policy correction</strong>：rollout 与 update 之间有 lag，用 IS clip 或 V-trace 校正</li></ul>
|
||
<p>代表实现：<strong>verl</strong> (ByteDance Seed open-source), <strong>AReaL</strong> (Ant Group + Tsinghua, async RL system arXiv 2505.24298), <strong>OpenRLHF v0.5+</strong>.</p>
|
||
<h3 id="84-tool-rl-on-tau-bench--swe-bench--osworld">8.4 Tool-RL on TAU-bench / SWE-bench / OSWorld</h3>
|
||
<p>工业 benchmark 上的 SOTA 趋势（2025-2026 公开数字）：</p>
|
||
<table><thead><tr><th>Benchmark</th><th>Task</th><th>2024 SOTA</th><th>2025-2026 SOTA</th><th>Key approach</th></tr></thead><tbody><tr><td><strong>TAU-bench (retail)</strong></td><td>customer service multi-turn</td><td>~50% (GPT-4)</td><td>70-80% (Claude 4.x, GPT-5)</td><td>RLHF + agent SFT</td></tr><tr><td><strong>SWE-bench Verified</strong></td><td>GitHub PR fix</td><td>~25% (Claude 3.5)</td><td><strong>70-80%+</strong> (Claude 4.x, o3)</td><td>Agent scaffold + RL</td></tr><tr><td><strong>OSWorld</strong></td><td>OS GUI task</td><td>~12% (GPT-4V)</td><td>~50-60% (Claude 4.x, Operator)</td><td>Computer-Use RL</td></tr><tr><td><strong>WebArena</strong></td><td>web nav</td><td>14.4% (GPT-4)</td><td>43% (WebRL Llama-8B)</td><td>curriculum + RL</td></tr><tr><td><strong>GAIA</strong></td><td>general assistant</td><td>15% (GPT-4)</td><td>60-70% (Claude 4.x, o3)</td><td>Agent + tool RL</td></tr></tbody></table>
|
||
<p>注意：benchmark contamination 风险大，<strong>2026Q1 OpenAI 已弃用 SWE-bench Verified</strong>（原因：contamination + test flaw）；当前更可信 benchmark 是 SWE-Lancer、SWE-bench Multilingual、private holdout。</p>
|
||
<h3 id="85-anthropic-computer-useclaude-35--45--opus-4x">8.5 Anthropic Computer-Use（Claude 3.5 → 4.5 → Opus 4.x）</h3>
|
||
<p><strong>公开知识</strong>：</p>
|
||
<ul><li>2024-10-22 Claude 3.5 Sonnet (new) 首发 Computer-Use beta</li><li>2025: Claude 3.7 / 4.0 / 4.5 持续迭代，速度 + 准确率提升</li><li>训练涉及大量人工演示 + AI-generated rollout + RLHF + 安全红队</li><li>Action space = 屏幕截图 + 鼠标 + 键盘 + 文件系统访问</li><li>公开 system card 提及 Constitutional AI + Computer-Use specific safety filter</li></ul>
|
||
<p><strong>算法层面的 inferred 信息</strong>（学术界推测，未官方确认）:</p>
|
||
<ul><li>训练大概率涉及 RLHF on screenshot trajectories</li><li>reward 含 task completion grader (LLM-as-judge) + safety classifier</li><li>可能用 GRPO / RLOO style critic-free RL（公开 paper 多次提及 critic-free）</li></ul>
|
||
<h3 id="86-2025-2026-papers-群像">8.6 2025-2026 papers 群像</h3>
|
||
<table><thead><tr><th>论文</th><th>方向</th><th>核心贡献</th></tr></thead><tbody><tr><td><strong>KodCode</strong> (Xu et al. 2025)</td><td>代码 agent RL</td><td>高质量代码 RL 数据集 + GRPO baseline</td></tr><tr><td><strong>DAPO</strong> (Yu et al. 2025 ByteDance)</td><td>GRPO 改进</td><td>clip higher / dynamic sampling / token loss / overlong shaping</td></tr><tr><td><strong>VAPO</strong> (字节跳动 2025)</td><td>GRPO 加 lightweight critic</td><td>trace-level credit dilution 缓解</td></tr><tr><td><strong>CISPO</strong> (MiniMax 2025)</td><td>importance sampling 改进</td><td>解决 negative advantage 大 ratio 失稳</td></tr><tr><td><strong>R-Zero</strong></td><td>Self-Play RL</td><td>Challenger-Solver self-play，learnability reward</td></tr><tr><td><strong>Absolute Zero</strong></td><td>Self-Play RL</td><td>完全无外部 task，code executor 当 verifier</td></tr><tr><td><strong>Search-R1</strong></td><td>search agent RL</td><td>search 当 first-class action + rule reward</td></tr><tr><td><strong>Light-R1</strong> / <strong>Sky-T1</strong></td><td>reasoning + tool RL</td><td>开源 reproduction R1 + agent extension</td></tr><tr><td><strong>OpenAgent</strong> / <strong>Llama-Agent</strong></td><td>数据 + 框架</td><td>大规模 agent SFT + RL pipeline</td></tr></tbody></table>
|
||
<div class="callout callout-info"><div class="callout-title">2026Q1-Q2 趋势</div><p>agent RL on real environment（OS, browser, IDE）成为开源主线。<strong>simulator → 真实环境 → 部署 RL</strong> 的 sim-to-real pipeline 是下一个 frontier。Anthropic / OpenAI / DeepSeek / Meta 等都在做但细节未公开。</p></div>
|
||
<h2 id="9-failure-modes--工程经验">§9 Failure modes & 工程经验</h2>
|
||
<h3 id="91-agentic-rl-的七宗罪">9.1 Agentic RL 的"七宗罪"</h3>
|
||
<table><thead><tr><th>失败模式</th><th>症状</th><th>根因</th><th>缓解</th></tr></thead><tbody><tr><td><strong>Token mask 漏 observation</strong></td><td>reward 上不去 / agent 学到怪行为</td><td>gradient 流到 environment token</td><td>严格 per-sample action_mask</td></tr><tr><td><strong>Reward hacking on grader</strong></td><td>benchmark 涨但人评下降</td><td>grader 有漏洞</td><td>grader ensemble + holdout test</td></tr><tr><td><strong>Length-explosion</strong></td><td>agent 拖到 context 上限</td><td>reward 与 length 正相关</td><td>length penalty + max_steps cap</td></tr><tr><td><strong>Tool-call hallucination</strong></td><td>agent 调不存在的 tool</td><td>base model SFT 不够</td><td>Agent-FLAN-style negative SFT</td></tr><tr><td><strong>Loop / repetition</strong></td><td>agent 反复调同样 tool</td><td>exploration 不够</td><td>tool-call diversity bonus + ε-tool</td></tr><tr><td><strong>Group $\sigma = 0$ collapse</strong></td><td>advantage = NaN / 0</td><td>全 success 或全 fail</td><td>data filter + std clamp</td></tr><tr><td><strong>KL collapse</strong></td><td>policy entropy → 0</td><td>β 太小</td><td>per-step KL + entropy bonus</td></tr></tbody></table>
|
||
<h3 id="92-online-vs-offline-rl-trade-off">9.2 Online vs offline RL trade-off</h3>
|
||
<table><thead><tr><th>维度</th><th>Online RL (PPO/GRPO)</th><th>Offline RL (DPO/RFT)</th></tr></thead><tbody><tr><td>数据效率</td><td>低（每轮新 rollout）</td><td>高（一次 dataset 多次训）</td></tr><tr><td>训练速度</td><td>慢（rollout 含 tool I/O）</td><td>快（pure SFT-style）</td></tr><tr><td>性能上限</td><td>高（持续学习新分布）</td><td>中（受限于 dataset 分布）</td></tr><tr><td>实现复杂度</td><td>高（trajectory queue + verifier server）</td><td>低</td></tr><tr><td>适合</td><td>长期投入 + 真实环境</td><td>资源受限 + 已有 demo data</td></tr></tbody></table>
|
||
<p><strong>实践建议</strong>：从 offline RL（agent SFT + DPO）起手，建立 baseline；有算力后迁移到 online RL（PPO/GRPO + verifier）。</p>
|
||
<h3 id="93-rollout-优化经验">9.3 Rollout 优化经验</h3>
|
||
<ul><li><strong>vLLM PagedAttention</strong> 比 HF generate 快 5-10×，是 agent rollout 的必装</li><li><strong>Tool sandbox</strong> 用 Docker + gVisor，单机并行 64-128 个 trajectory 没问题</li><li><strong>Async rollout pipeline</strong>：rollout worker 不阻塞 trainer worker</li><li><strong>Trajectory replay buffer</strong>：FIFO + priority 混合，replay 时按 reward 加权</li><li><strong>Batch size = group_size × prompt_per_batch</strong>：典型 $G = 16$, prompt = 32 → 512 trajectory / batch</li></ul>
|
||
<h3 id="94-debug-checklist">9.4 Debug checklist</h3>
|
||
<p>agent RL 跑崩了，按顺序排查：</p>
|
||
<ol><li><strong>看 reward</strong>：是否多数 trajectory reward = 0？rule 写错了？</li><li><strong>看 length 分布</strong>：是否大多数 trajectory 在 max_steps 截断？说明 agent 不会 finish</li><li><strong>看 action_mask</strong>：是否正确 cover prompt + observation + padding？</li><li><strong>看 KL</strong>：approx_kl 是否爆？β 调大；entropy 是否塌？entropy_bonus 调大</li><li><strong>看 group_std</strong>：是否大量 group 全 0/全 1？需要 data filter</li><li><strong>看 tool call distribution</strong>：是否过度依赖某一个 tool？引入 diversity bonus</li><li><strong>看 sample efficiency</strong>：单 prompt 多 rollout reward 方差是否合理？太大说明 base 太弱</li></ol>
|
||
<h2 id="10-25-高频面试题l1-必会--l2-进阶--l3-顶级-lab">§10 25 高频面试题（L1 必会 / L2 进阶 / L3 顶级 lab）</h2>
|
||
<p>按难度分 3 档：L1 = 任何 agent / LLM RL 岗会问；L2 = research / alignment 团队会问；L3 = 顶级 lab 硬核题。每题点开看答案要点 + 易踩坑。</p>
|
||
<h3 id="l1-必会题10-题">L1 必会题（10 题）</h3>
|
||
<details>
|
||
|
||
<summary>Q1. Agentic RL 和 RLHF 的本质区别？</summary>
|
||
<ul><li><strong>RLHF</strong>: single-turn alignment，state = prompt，action = 整段 response，reward 来自 RM 对偏好的打分，horizon = 1</li><li><strong>Agentic RL</strong>: multi-turn decision-making，state = (obs, history)，action = (thought, tool_call)，reward 来自外部环境（test pass / task success），horizon = 10-200</li><li>算法层面都用 PPO/GRPO，但 Agentic 必须加 <strong>action_mask</strong>（只对 agent token 算 loss）</li><li>reward 形式：RLHF 偏好打分（subjective），Agentic 客观 outcome（objective）</li></ul>
|
||
<p>把它们当成同一件事；或不知道 action_mask 的必要性。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q2. 为什么 Agentic RL 必须用 action_mask？</summary>
|
||
<ul><li>agent trajectory 含两类 token：agent 自己 generate 的 + tool 返回的 observation</li><li>如果不 mask，PPO/GRPO 的 ratio 和 loss 会流到 observation token 上</li><li>后果：(a) policy 试图"教 tool 怎么回答"，无意义且会 reward hack；(b) gradient 被 low-information observation 稀释；(c) KL penalty 错误地把 environment text 当自己分布估计</li><li>实现：<code>action_mask: [B, L]</code>，1 = agent token，0 = prompt/obs/pad；loss 计算时除以 <code>mask.sum()</code> 标准化</li></ul>
|
||
<p>说"只在 response 上算 loss"不够具体（agent task 没有"response"这个明确边界）；或漏了 mask 必须 cover all observation tokens。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q3. GRPO 在 agent 上比 PPO 好的核心原因？</summary>
|
||
<ul><li><strong>省 critic</strong>：agent value 在长 horizon + sparse reward 下几乎学不动</li><li><strong>trace-level reward 直接匹配 trace-level credit</strong>：不需要 per-token V，避开 value 学不动的痛</li><li><strong>组内归一化</strong> 自动做 variance reduction，比 raw advantage 稳</li><li><strong>省一份显存</strong>：可以扩大 batch / group size</li><li>限制：长 trajectory 上 advantage 太粗（整段共享），credit dilution 仍存在</li></ul>
|
||
<p>只说"省 critic"不全面；或不知道 trace-level reward 与 trace-level credit 的匹配关系。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q4. Outcome reward 和 process reward 的区别？agent 上常用哪个？</summary>
|
||
<ul><li><strong>Outcome reward</strong>: trajectory 终点 1 个 reward（test pass / answer match），极稀疏，credit assignment 难，但 reward hacking 风险低</li><li><strong>Process reward</strong>: 每步打分，dense，credit assignment 易，但 reward hacking 风险高（PRM 可被 hack）</li><li><strong>Agent 上主流: outcome reward</strong>——agent task ground truth 明确（test/grader）；R1, SWE-RL, ReSearch, RAGEN 都用 outcome-only</li><li>Process reward 主要用于 reasoning-heavy 任务（PRM 在数学 step 上），agent 上少用</li></ul>
|
||
<p>以为 process reward 总是好（实际 agent 上 outcome 更稳）；或不知道 reward hacking 风险差异。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q5. Verifier-based reward 比 learned RM 好在哪？</summary>
|
||
<ul><li><strong>接近 ground truth</strong>：避开 learned RM 漂出训练分布的主要 failure mode</li><li><strong>可重复</strong>：同一 trajectory reward 一致（learned RM 输出有噪声）</li><li><strong>显存便宜</strong>：执行 verifier 比一次 LLM forward 便宜数百倍</li><li><strong>可解释</strong>：reward 来自客观 rule，可 trace 失败原因</li><li><strong>限制</strong>：只能用于可验证任务（math/code/grader-able web）</li></ul>
|
||
<p>说 verifier-based "完全没有 hacking"——错，仍可能被正则漏洞 / format trick / test 泄露 hack，但比 RM hacking 容易堵。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q6. R1 / R1-Zero 的方法能直接用在 agent 上吗？</summary>
|
||
<ul><li>算法可以直接搬：GRPO + rule-based reward + per-step KL + token mask</li><li><p>但需要补：</p>
|
||
<ul><li><strong>action_mask</strong>: agent 有 observation token，R1 数学任务没有</li><li><strong>trajectory rollout infrastructure</strong>: 含 tool I/O，比纯 generation 复杂</li><li><strong>format reward 调整</strong>: agent task 的 format 是 JSON tool call，不只是 <code><think></code></li></ul></li><li>代表性工作 ReSearch / RAGEN / ToolRL / SWE-RL 都是 R1 算法 + 上述改造</li></ul>
|
||
<p>说"R1 不能直接搬"——其实可以但要改 wrapper；或不知道 ReSearch / RAGEN / ToolRL 这些工作。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q7. Agent RL 中为什么需要 SFT warm-start？</summary>
|
||
<ul><li>base model 不知道怎么 emit 合法 tool call schema（JSON 格式 / argument 名称）</li><li>直接 from-scratch RL 几乎不可能 explore 到合法 tool call → reward 全 0 → 学不动</li><li>SFT warm-start（AgentTuning / Agent-FLAN 数据）让 model "知道动作空间长什么样"</li><li>之后 RL 在合法 action 子空间内 optimize</li></ul>
|
||
<p>只说"RL 慢，SFT 加速"——不够；核心是 action space exploration 困难，SFT 解决"知道动作空间"。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q8. Length penalty 在 agent RL 中起什么作用？</summary>
|
||
<ul><li>agent 容易学到"拖长 trajectory 拿对答案"的捷径（reward hacking）</li><li>length penalty 给超出 budget 的 trajectory 减分：$r = r_\text{outcome} - \lambda \max(0, T - T_\text{target})$</li><li>DAPO 的 "overlong shaping" 是工业级实现（指数衰减）</li><li>限制：penalty 过大会让 agent 不敢探索；要 cap</li></ul>
|
||
<p>不知道 length-explosion 是 agent RL 常见 failure mode；或不会写 length penalty 公式。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q9. Group std = 0 的情况怎么处理？</summary>
|
||
<ul><li>当 group 内 G 个 rollout reward 全相同（全 success / 全 fail）→ $\sigma = 0$</li><li>加 $\epsilon$ 时 advantage = 0，policy gradient 项归零，<strong>但 KL 项仍存在</strong>（policy 仍被拉回 reference）</li><li>不加 $\epsilon$ 时是 NaN</li><li><p>实践：</p>
|
||
<ul><li><strong>Skip 该 prompt</strong>（data filter）：常见做法，避免无信号更新</li><li><strong>Clamp σ 下限</strong>（如 0.1）：保留少量信号</li><li><strong>DAPO dynamic sampling</strong>：丢全对全错 group</li></ul></li></ul>
|
||
<p>说一定会 NaN（不对，看实现）；或不知道这种 prompt 暗示 task 过易/过难。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q10. SWE-RL 是怎么训的？</summary>
|
||
<ul><li>数据：GitHub PR commit 历史，构造 ~76M context-issue-patch 三元组（Meta 公开）</li><li>Reward: rule-based = patch similarity (oracle ↔ pred) + test pass binary</li><li>算法: 纯 GRPO + format reward</li><li>Model: Llama-3.3-70B</li><li>Result: SWE-bench Verified 上 41%（无 scaffold），证明 rule-based RL 让 model 学到 emergent reasoning：file retrieval / root cause / test self-validation</li></ul>
|
||
<p>不知道 SWE-RL 的 reward 设计；或以为是 SFT 而非 RL。</p>
|
||
</details>
|
||
<h3 id="l2-进阶题10-题">L2 进阶题（10 题）</h3>
|
||
<details>
|
||
|
||
<summary>Q11. 推导 PPO loss 在 agent 上加 action_mask 后的形式。</summary>
|
||
<ol><li>标准 PPO-Clip：$L = \mathbb{E}[\min(\rho_t A_t, \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon) A_t)]$</li><li>agent 有 $m_t \in \{0, 1\}$，1 = agent token，0 = obs/prompt</li><li>ratio 计算时 $\rho_t = \exp((\log\pi_\theta - \log\pi_\text{old}) \cdot m_t)$，observation 位置 $\rho = e^0 = 1$，不影响 surr1/surr2</li><li>loss 归一化：$L^\text{agent} = -\sum_t m_t \cdot \min(\rho_t A_t, \text{clip}) / \sum_t m_t$</li><li>KL 项也只在 agent token：$\text{KL}_\text{total} = \sum_t m_t \cdot \text{KL}_t / \sum_t m_t$</li></ol>
|
||
<p>只写公式不解释 mask 的作用；或忘了 normalization 用 mask.sum() 而非 batch size。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q12. RAGEN / StarPO 的关键贡献？</summary>
|
||
<ul><li><strong>StAble multi-tuRn Policy Optimization</strong>: critic-free GRPO 变种，整段 trajectory 共享 advantage</li><li><strong>严格 token mask</strong>：observation 位置 mask = 0，loss 只在 agent token</li><li><strong>rollout 多样性 = collapse 防火墙</strong>：group_size $G = 16$ 比 $G = 4$ 显著更稳</li><li><strong>trajectory length 信号</strong>：失败 trajectory length 大时加 length penalty</li><li>适用：multi-turn agent task（与 single-turn alignment 区分）</li></ul>
|
||
<p>只说"GRPO 变种"——不够；要说出 multi-turn 上的 stability 贡献。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q13. WebRL 的 self-evolving curriculum 是怎么工作的？</summary>
|
||
<ul><li>初始 task set $\mathcal{T}_0$（small），训 policy 跑 trajectory</li><li>失败 trajectory 进入 buffer，加到下一轮 curriculum $\mathcal{T}_{k+1}$</li><li>retrospective rollout：失败 trajectory 用 LLM 改造成"正确 trajectory"（hindsight relabel），再做 SFT</li><li>ORM (Outcome Reward Model) 训自 task success → 在线给 RL reward</li><li>Result: Llama-3.1-8B 在 WebArena 上 43%（vs GPT-4 14.4%）</li></ul>
|
||
<p>不知道 self-evolving 的循环结构；或把 WebRL 当作纯 SFT。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q14. 为什么 self-rewarding LM 在 agent 上比在 alignment 上风险更高？</summary>
|
||
<ul><li>alignment 任务有"客观偏好分布"，LLM judge 与人评有较高相关</li><li>agent 任务有<strong>客观 ground truth</strong>（test pass / task success）—— judge 自己可能错（认错对错）</li><li>iterative drift：每轮把"自评对"的轨迹强化 → 远离 ground truth</li><li>探索退化：自评偏好已知 pattern → 抑制探索新 tool</li><li>主流做法：agent RL 优先 rule-based ground truth，self-rewarding 仅作开放式任务 fallback</li></ul>
|
||
<p>说 self-rewarding 总是危险（alignment 上仍可用）；或不知道 agent 上 ground truth 客观性是关键</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q15. 推 outcome-reward sparsity 对 critic learning 的影响。</summary>
|
||
<ul><li>value $V_\phi(s_t) = \mathbb{E}_\pi[\sum_{l \ge 0} \gamma^l r_{t+l} \mid s_t]$</li><li>sparse terminal reward → $V(s_t) \approx \gamma^{T-t} \cdot P(\text{success} \mid s_t)$</li><li>中间状态 $s_t$ 的 value 几乎只取决于"未来是否成功"——这是隐含 long-horizon 预测</li><li>value MSE loss $(V_\phi - V_\text{target})^2$ 在多数 step 上 target 接近 0，gradient 极小</li><li>等价于"几乎没有 supervision"——value 学不动是 sparse reward 的必然结果</li><li>这也是 GRPO 省 critic 的理论基础：critic 本来就学不动，省了反而省去 noise</li></ul>
|
||
<p>只说"critic 学不动"；不会推 $V \approx \gamma^{T-t} P(\text{success})$；或不知道这是 GRPO 设计动机。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q16. ToolRL 的 reward 设计有什么 nontrivial 之处？</summary>
|
||
<ul><li>composite: $r = r_\text{correct} + \alpha r_\text{format} + \gamma r_\text{tool-eff}$</li><li>$r_\text{tool-eff}$ 惩罚冗余 tool call（重复调同样工具 / 调用无效 tool）</li><li>这是典型的 shaping reward：缓解 length-explosion + tool-overuse 两个 failure mode</li><li>weight 平衡：outcome ≫ format > tool-eff，避免 shaping override outcome</li><li>BFCL benchmark 上 7B 接近 GPT-4</li></ul>
|
||
<p>只说"加 tool call 奖励"；不知道 shaping reward weight 平衡是关键。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q17. Hindsight relabeling 在 agent RL 中怎么用？</summary>
|
||
<ul><li>失败 trajectory 不丢，改造为 "alternative task" 的 successful trajectory</li><li>例：agent 想买 A 商品但停在 B → 改造为"找到 B 商品"，reward = 1</li><li>实现：<code>alt_task = describe(trajectory.final_state)</code>，relabel reward = 1</li><li>适用：开放 web 环境、navigation；不适用：math/code（错答案不能改成对答案）</li><li>起源：HER (Andrychowicz 2017 NeurIPS) for robot manipulation</li></ul>
|
||
<p>不知道 HER 起源；或不知道适用边界（开放环境 vs 答题任务）。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q18. Per-step KL penalty 和 trajectory KL penalty 的区别？</summary>
|
||
<ul><li><strong>per-step KL</strong>: 每个 agent token 算 KL($\pi_\theta(\cdot \mid s_t) \| \pi_\text{ref}(\cdot \mid s_t)$)；加进 reward 或 loss</li><li><strong>trajectory KL</strong>: 整段 trajectory 一个 KL；加进 loss</li><li>per-step 更精细，能控制每步漂移；trajectory 简单但缺乏 token-level resolution</li><li>GRPO 用 per-step + K3 estimator（数值稳定）</li><li>agent RL 上 per-step 更主流（trajectory 太长，单 KL 数值不稳）</li></ul>
|
||
<p>混淆两者；或不知道 K3 estimator 解决数值问题（K3: $\text{KL} \approx \exp(\Delta) - \Delta - 1$ 非负）。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q19. Subgoal decomposition + process reward 怎么做？什么时候用？</summary>
|
||
<ul><li>长 trajectory 切 subgoal：100 步 trajectory → 5 个 subgoal × 20 步</li><li>每个 subgoal 终点给 process reward（subgoal 是否完成）</li><li><p>实现路径：</p>
|
||
<ul><li>hand-crafted: 人写 subgoal 判据</li><li>LLM planner: planner LLM 拆 subgoal，verifier 判</li><li>PRM-style: PRM 评每步</li></ul></li><li>适用：long-horizon agent + 可以 hand-craft subgoal 的任务</li><li>风险：subgoal boundary 错画 → agent 学到"刻意触发 subgoal reward 而不真正完成 task"</li></ul>
|
||
<p>说 process reward 总是好——错；要说出风险与限制。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q20. Online RL vs Offline RL 在 agent 上的 trade-off？</summary>
|
||
<ul><li><strong>Online RL (PPO/GRPO)</strong>: 数据效率低（每轮新 rollout），但持续学新分布</li><li><strong>Offline RL (DPO/RFT)</strong>: 数据效率高，但受限于 dataset 分布</li><li>agent rollout 慢（含 tool I/O），online RL 训练吞吐低</li><li>实践：先 offline 起手（SFT + DPO），再 online refinement</li><li>代表：AgentQ 是 offline（MCTS + DPO）；WebRL 是 online；SWE-RL 是 online</li></ul>
|
||
<p>只说"online 慢"；不知道 agent rollout 含 tool I/O 是主要瓶颈。</p>
|
||
</details>
|
||
<h3 id="l3-顶级-lab-题5-题">L3 顶级 lab 题（5 题）</h3>
|
||
<details>
|
||
|
||
<summary>Q21. 推导 GRPO advantage 公式 + token mask 的完整 loss，并解释 agent token mask 的两种等价放置方式。</summary>
|
||
<ol><li><p><strong>Group-relative advantage</strong>:</p>
|
||
<ul><li>rollout group $\{r_1, ..., r_G\}$ per prompt</li><li>$\mu = \frac{1}{G}\sum r_i$, $\sigma = \sqrt{\frac{1}{G}\sum (r_i - \mu)^2}$</li><li>$\hat{A}_i = (r_i - \mu) / (\sigma + \epsilon)$</li></ul></li><li><strong>Trajectory-level broadcast</strong>: $\hat{A}_{i,t} = \hat{A}_i$（所有 agent token 共享）</li><li><p><strong>Token-masked ratio + loss</strong>：</p>
|
||
<ul><li>朴素 $\rho_{i,t} = \exp(\log\pi_\theta(a_{i,t} \mid s_{i,t}) - \log\pi_\text{old}(a_{i,t} \mid s_{i,t}))$ —— 在 observation 位置也会有数值（model 估算 env text 的概率）</li><li><p><strong>要点</strong>：只要最终 objective / gradient 只覆盖 agent token，mask 放 ratio 内还是 loss 外都<strong>数学等价</strong>：</p>
|
||
<ul><li><strong>Inside-ratio</strong>：$\rho_{i,t} = \exp((\log\pi_\theta - \log\pi_\text{old}) \cdot m_{i,t})$ → obs 位置 $\rho=1$，进 clip 后 $\min(\cdot)$ 项 = $A_{i,t}$ 但乘以 $m_{i,t}=0$ 后归零（在 loss 外的 sum 中）</li><li><strong>Outside-ratio (mask loss only)</strong>：保留 obs 位 $\rho_{i,t}$ 数值；最终 loss = $-\sum_t m_t \cdot \min(...)$，obs 位贡献 $m_t = 0$ 直接归零</li></ul></li><li><strong>两者梯度都只覆盖 agent token</strong>（mask 是乘法，梯度对 obs 位都是 0）</li><li>但实践上 <strong>Inside-ratio 更安全</strong>：避免 obs 位 $\rho$ 数值参与 clip 触发判断或被日志 / 监控（如 mean ratio）误读为异常。生产实现（verl / OpenRLHF）多用 inside-ratio</li></ul></li><li><strong>Full loss</strong>: $$L = -\frac{1}{G} \sum_i \frac{1}{\sum_t m_{i,t}} \sum_t m_{i,t} \cdot \Big(\min(\rho_{i,t} \hat{A}_i, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_i) - \beta \cdot \text{KL}_{i,t}\Big)$$</li></ol>
|
||
<p>不会推 step 4 (mask 位置影响 ratio 数值)；或公式背得对但不解释 mask 设计哲学。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q22. GRPO 在 long-horizon agent 上比 PPO sample efficient 的根本原因？</summary>
|
||
<p><strong>Trace-level reward 与 trace-level credit 的自然对齐</strong>（不止是"省 critic"）：</p>
|
||
<ol><li><strong>Sparse terminal reward 下 critic 学不动</strong>：$V(s_t) \approx \gamma^{T-t} P(\text{success})$，gradient 极小；PPO 的 GAE-advantage 受 noisy critic 拖累</li><li><strong>GRPO 直接用 trace-level reward 当 advantage</strong>：等价 sequence-level MC return，在 sparse reward 下是 unbiased estimator</li><li><strong>Group baseline 比 critic baseline 更稳</strong>：同 prompt G rollouts → group mean 自动反映该 prompt 的难度，variance reduction 更精准</li><li><strong>PPO clipping + group size 联合限制更新幅度</strong>：避免单 outlier reward 推飞 policy</li><li><strong>省 value model 显存</strong> 是 secondary benefit，不是 primary reason</li><li><strong>Rule-based outcome reward 难被 RM hacked</strong>：在 agent 上比 learned RM 稳</li></ol>
|
||
<p>只说"省 critic"——不够；要说出 sparse reward 下 critic learn 不动是根因。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q23. 如何设计一个 RL framework 同时支持 reasoning RL（R1）和 Agentic RL（ReSearch / WebRL）？</summary>
|
||
<p>抽象出五层：</p>
|
||
<ol><li><p><strong>Data layer</strong>:</p>
|
||
<ul><li>reasoning: (prompt, ground_truth) tuples</li><li>agent: (task, env_spec, reward_fn) 三元组</li><li>统一为 <code>Task(prompt, verifier)</code>，verifier 是 callable</li></ul></li><li><p><strong>Rollout layer</strong>:</p>
|
||
<ul><li>reasoning: 直接 generate</li><li>agent: 含 tool I/O 的 multi-step rollout (vLLM + sandboxed tool executor)</li><li>统一为 <code>Trajectory(tokens, action_mask, reward)</code> 接口</li></ul></li><li><p><strong>Reward layer</strong>:</p>
|
||
<ul><li>reasoning: rule-based (answer match / test pass)</li><li>agent: composite (outcome + format + tool_eff + length penalty)</li><li>统一为 <code>Reward(traj) -> float</code></li></ul></li><li><p><strong>Loss layer</strong>:</p>
|
||
<ul><li>PPO with action_mask</li><li>GRPO with group_id + action_mask</li><li>DPO with chosen_mask / rejected_mask</li><li>通过 <code>loss_fn(batch, model, ref_model) -> loss</code> interface</li></ul></li><li><p><strong>Infra layer</strong>:</p>
|
||
<ul><li>vLLM rollout pool</li><li>Sandboxed tool executor (Docker + gVisor)</li><li>Trajectory replay buffer (FIFO + priority)</li><li>Async trainer / rollout</li></ul></li></ol>
|
||
<p>代表实现: <strong>verl</strong> (字节) 已支持 reasoning + agent；<strong>OpenRLHF</strong> 部分支持。</p>
|
||
<p>只列 PPO 不考虑 agent rollout infra；或不知道 verl / OpenRLHF 的当前支持范围。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q24. Anthropic Computer-Use 训练方法 — 已知 vs 推测的清晰边界</summary>
|
||
<p><strong>官方公开（system card / blog）</strong>：</p>
|
||
<ul><li>action space = 屏幕截图 (vision observation) + 鼠标 + 键盘 events</li><li>能力持续迭代 Claude 3.5 (new) 2024-10-22 → 3.7 / 4.0 / 4.5 / Opus 4.x</li><li>安全机制：constitutional AI 风格的护栏 + 红队 + prompt-injection 防御</li><li>训练涉及人工演示 + 合成数据（system card 一般性陈述）</li></ul>
|
||
<p><strong>未公开 / 完全保密</strong>：</p>
|
||
<ul><li>具体 RL 算法（PPO? GRPO? Critic-free? 都没说）</li><li>Reward signal 形式（task completion grader? Pair-wise preference? Safety classifier 权重?）</li><li>Train data 规模 / 来源 / 演示 vs 合成比例</li><li>是否有专门的 screenshot RM / VLM-as-judge</li></ul>
|
||
<p><strong>社区合理推测</strong>（<strong>仅推测，不要在面试中说成事实</strong>）：</p>
|
||
<ul><li>可能是 RLHF on screenshot trajectories（pair-wise 偏好 + task success outcome 混合）</li><li>可能 critic-free（呼应 DeepSeek-R1 GRPO 等开源趋势）</li><li>可能用 VLM-as-judge for screenshot 理解</li><li>可能 curriculum 简单 → 复杂</li></ul>
|
||
<p><strong>面试时务必区分 "公开能力" vs "推测内部"</strong>：说"Anthropic 用 GRPO + screenshot RM"是错的（无证据）；说"我推测可能用了 critic-free RL，因为 Anthropic 在其他场景倾向 GRPO/RLHF 风格"才是诚实的表述。这种区分能力是高级面试的加分项。</p>
|
||
</details>
|
||
<details>
|
||
|
||
<summary>Q25. 如果让你设计 next-gen Agentic RL 算法，会怎么改进？</summary>
|
||
<p>可能方向（任答 3-4 个，每个要有 trade-off 讨论）：</p>
|
||
<ol><li><strong>Lightweight critic for long-horizon</strong>: 不用 full-size value model，但用小型 step-level critic 缓解 trace-level credit dilution。VAPO 已尝试。Trade-off: 加显存 vs 缓解 long-trajectory 信号稀释</li><li><strong>Hierarchical reward</strong>: subgoal-level reward + outcome reward 组合。Trade-off: 需要 subgoal definition（人工 or planner LLM），boundary 错画风险</li><li><strong>Off-policy correction with V-trace / Retrace</strong>: rollout 慢，让 stale samples 也能用。Trade-off: IS bias vs sample efficiency</li><li><strong>Trajectory hindsight relabeling + RL</strong>: 失败 trajectory 自动改造为 alternative task 的成功 trajectory，扩 data。Trade-off: 适用 open-ended task，不适用 closed-form answer</li><li><strong>Multi-task reward normalization</strong>: 每个 task domain (math/code/web) 独立归一化，避免 reward scale 不平衡</li><li><strong>Reward model uncertainty</strong>: 多 RM ensemble，min/mean-std 防 over-optimization。Trade-off: 算力</li><li><strong>Async distributed rollout</strong>: rollout 与 train 完全异步，trajectory queue + worker pool。已是 industry default (verl, OpenRLHF v0.5+)</li><li><strong>Self-curriculum + adaptive difficulty</strong>: WebRL 思路 + R-Zero 的 learnability reward 结合，自动找 model success rate ~50% 的任务</li><li><strong>Multi-objective Pareto optimization</strong>: 不再单一 scalar reward，task success + safety + efficiency 同时优化，输出 Pareto front</li></ol>
|
||
<p>只罗列 "加 attention / 加更多模型" 没 trade-off；或不知道 DAPO / VAPO / CISPO 等近期工作；或忽略 infra 层面 (async rollout) 的重要性。</p>
|
||
</details>
|
||
<h2 id="a-附录参考文献清单">§A 附录：参考文献清单</h2>
|
||
<p>按方向分组，论文经 web 检索 + arXiv 验证作者 / 年份 / 会议。少数 2025-2026 会议归属未定的论文以 arXiv 记。</p>
|
||
<p><strong>Agent SFT / 基础</strong></p>
|
||
<ul><li>Zeng et al. 2023 arXiv 2310.12823 <em>AgentTuning: Enabling Generalized Agent Abilities for LLMs</em> (THU)</li><li>Chen et al. 2024 ACL Findings arXiv 2403.12881 <em>Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for LLMs</em></li><li>Trung et al. 2024 ACL arXiv 2401.08967 <em>ReFT: Reasoning with Reinforced Fine-Tuning</em></li></ul>
|
||
<p><strong>RL on agent / reasoning（基础算法）</strong></p>
|
||
<ul><li>Schulman et al. 2017 arXiv 1707.06347 <em>Proximal Policy Optimization Algorithms</em></li><li>Schulman et al. 2016 ICLR <em>High-Dimensional Continuous Control Using GAE</em></li><li>Shao et al. 2024 arXiv 2402.03300 <em>DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models</em>（GRPO 提出）</li><li>DeepSeek-AI 2025 arXiv 2501.12948 <em>DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning</em></li><li>Yu et al. 2025 ByteDance arXiv 2503.14476 <em>DAPO: An Open-Source LLM Reinforcement Learning System at Scale</em></li></ul>
|
||
<p><strong>Tool-augmented RL</strong></p>
|
||
<ul><li>Qian et al. 2025 arXiv 2504.13958 <em>ToolRL: Reward is All Tool Learning Needs</em> (arXiv preprint; no formal venue as of 2026-05)</li><li>Chen et al. 2025 arXiv 2503.19470 <em>ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning</em> (accepted to <strong>NeurIPS 2025</strong>)</li><li>Wang et al. 2025 arXiv 2504.20073 <em>Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning</em> (RAGEN; StarPO = <strong>S</strong>tate-<strong>T</strong>hinking-<strong>A</strong>ctions-<strong>R</strong>eward Policy Optimization)</li></ul>
|
||
<p><strong>Web / GUI agent RL</strong></p>
|
||
<ul><li>Qi et al. 2024 ICLR-25 arXiv 2411.02337 <em>WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning</em></li><li>Putta et al. 2024 arXiv 2408.07199 <em>Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents</em></li><li>Furuta et al. 2024 ICLR arXiv 2305.11854 <em>Multimodal Web Navigation with Instruction-Finetuned Foundation Models</em> (WebGUM)</li></ul>
|
||
<p><strong>Code agent RL</strong></p>
|
||
<ul><li>Le et al. 2022 NeurIPS arXiv 2207.01780 <em>CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning</em></li><li>Shojaee et al. 2023 arXiv 2301.13816 <em>Execution-Based Code Generation Using Deep Reinforcement Learning</em> (PPOCoder)</li><li>Wei et al. 2025 Meta FAIR arXiv 2502.18449 <em>SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution</em></li></ul>
|
||
<p><strong>Embodied / robot agent</strong></p>
|
||
<ul><li>Baker et al. 2022 NeurIPS arXiv 2206.11795 <em>Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos</em></li><li>Kim et al. 2024 arXiv 2406.09246 <em>OpenVLA: An Open-Source Vision-Language-Action Model</em></li></ul>
|
||
<p><strong>Self-rewarding / exploration</strong></p>
|
||
<ul><li>Yuan et al. 2024 ICML arXiv 2401.10020 <em>Self-Rewarding Language Models</em></li><li>Andrychowicz et al. 2017 NeurIPS arXiv 1707.01495 <em>Hindsight Experience Replay</em></li></ul>
|
||
<p><strong>RLHF / DPO 基础（cross-reference）</strong></p>
|
||
<ul><li>Ouyang et al. 2022 NeurIPS <em>Training Language Models to Follow Instructions with Human Feedback</em></li><li>Rafailov et al. 2023 NeurIPS <em>Direct Preference Optimization</em></li><li>Bai et al. 2022 Anthropic arXiv 2212.08073 <em>Constitutional AI</em></li><li>Lee et al. 2023 Google arXiv 2309.00267 <em>RLAIF: Scaling RLHF with AI Feedback</em></li></ul>
|
||
<p><strong>Reward model / verification</strong></p>
|
||
<ul><li>Lightman et al. 2024 ICLR arXiv 2305.20050 (OpenAI 2023) <em>Let's Verify Step by Step</em> (PRM800K)</li><li>Wang et al. 2024 ACL arXiv 2312.08935 <em>Math-Shepherd: Verify and Reinforce LLMs Step-by-Step without Human Annotations</em></li><li>Coste et al. 2024 ICLR <em>Reward Model Ensembles Help Mitigate Overoptimization</em></li></ul>
|
||
<p><strong>Infrastructure / framework</strong></p>
|
||
<ul><li>TRL (HuggingFace): https://github.com/huggingface/trl —— 标准 PPO / DPO / GRPO trainer</li><li>OpenRLHF: https://github.com/OpenRLHF/OpenRLHF —— PPO / GRPO / RLOO 工业化实现</li><li>verl (ByteDance): https://github.com/volcengine/verl —— GRPO / DAPO / agent RL 主流框架</li><li>ReaLHF / AReaL (Ant Group + Tsinghua, async RL system, arXiv 2505.24298)</li></ul>
|
||
<p><strong>SOTA benchmarks (2024-2026)</strong></p>
|
||
<ul><li>Jimenez et al. 2024 ICLR arXiv 2310.06770 <em>SWE-bench: Can Language Models Resolve Real-World GitHub Issues?</em></li><li>Yao et al. 2024 arXiv 2406.12045 <em>τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains</em></li><li>Xie et al. 2024 NeurIPS arXiv 2404.07972 <em>OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments</em></li><li>Zhou et al. 2024 ICLR arXiv 2307.13854 <em>WebArena: A Realistic Web Environment for Building Autonomous Agents</em></li><li>Mialon et al. 2024 ICLR <em>GAIA: A Benchmark for General AI Assistants</em></li><li>Chan et al. 2024 arXiv 2410.07095 <em>MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering</em> (OpenAI)</li></ul>
|
||
<p><strong>Anthropic Computer-Use（公开知识）</strong></p>
|
||
<ul><li>Claude 3.5 Sonnet (new) 2024-10-22 Computer Use beta launch (Anthropic blog + system card)</li><li>Claude 3.7 / 4.0 / 4.5 / Opus 4.x system cards (Anthropic 公开)</li></ul>
|
||
<p>代码框架建议：</p>
|
||
<ul><li>起步用 TRL（HF）的 GRPOTrainer + 自写 verifier</li><li>工业化用 verl（GRPO/RLOO/DAPO 都支持，含 agent rollout）</li><li>自研 multi-turn 用 OpenRLHF v0.5+ 的 agent example + 加 tool sandbox</li></ul>
|
||
|
||
<footer class="aris-footer">
|
||
Generated by <a href="https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep/blob/main/skills/render-html/SKILL.md">ARIS <code>/render-html</code></a> ·
|
||
source path <code>docs/tutorials/agentic_rl_tutorial.md</code> ·
|
||
SHA256 <code>d3d476658bea</code> ·
|
||
generated at 2026-05-19 14:03 UTC.
|
||
This is a generated view — edit the source Markdown, then re-render.
|
||
</footer>
|
||
</main>
|
||
|
||
</div>
|
||
</body>
|
||
</html>
|