wehub-resource-sync
75 lines
5.7 KiB
JSON
75 lines
5.7 KiB
JSON
{
|
||
"skill": "interview-cheatsheet",
|
||
"source": "docs/tutorials/agentic_rl_tutorial.md",
|
||
"output": "docs/tutorials/agentic_rl_tutorial.html",
|
||
"topic": "Agentic RL — RL for LLM agents (AgentTuning / ToolRL / RAGEN / WebRL / SWE-RL / CodeRL / VPT / OpenVLA / Self-Rewarding LM / HER)",
|
||
"effort": "max",
|
||
"byline": "Ruofeng Yang (杨若峰), Shanghai Jiao Tong University",
|
||
"reviewer": "codex gpt-5.5 xhigh, fresh thread per round",
|
||
"math_code_review": {
|
||
"verdict": "PASS (after main-session DIY substantive fixes)",
|
||
"rounds": [
|
||
{
|
||
"run": "0 (subagent — draft only)",
|
||
"verdict": "draft delivered",
|
||
"notes": "Solo subagent (low codex contention) wrote 1545-line draft in ~13 min; 3 [needs-verify] markers left for main-session resolution. Subagent intentionally skipped Steps 3-6 to avoid codex MCP concurrency hang."
|
||
},
|
||
{
|
||
"run": 1,
|
||
"verdict": "FAIL → substantive fixes applied",
|
||
"thread_id": "019e4083-56ce-75b0-ac40-0bb7a8c22e15",
|
||
"reviewer": "main-session DIY (strictest mode)",
|
||
"real_issues_caught": [
|
||
"§2.3 PPO公式 outer $\\mathbb{E}_t$ 与内层 $\\sum_t$ 同索引 → notational collision",
|
||
"Q21 mask-inside-ratio 'necessity' explanation is wrong: inside-ratio vs outside-ratio mask are mathematically equivalent (both zero gradient on obs tokens). Inside-ratio is just safer for monitoring/logging.",
|
||
"Q24 Anthropic Computer-Use 训练 detail 超出公开资料 — 'critic-free / LLM-judge / Curriculum' 都是推测说成事实",
|
||
"§6.1 VPT 误标 'DeepMind+OpenAI' → 应是仅 OpenAI",
|
||
"§9 / appendix AReaL 误标 'Anthropic-related' → 应是 Ant Group + Tsinghua (arXiv 2505.24298)",
|
||
"§6.11 WebGUM 误标为 'offline SFT + RL fine-tune' → 论文实际是 imitation/SFT only",
|
||
"§6.15 OpenVLA 误标 'task-specific RL fine-tune' → 论文是 970K demonstrations imitation learning (后续工作才在 OpenVLA base 上做 RL)",
|
||
"§6.16 + Q24 Anthropic Computer-Use 'reward / RL 细节' 描述超出官方资料",
|
||
"§6.8 RAGEN/StarPO 标题错（不是 \"Multi-Turn Reinforcement Learning of Reasoning-Trained Language Agents\"，是 \"Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning\"）",
|
||
"§6.8 StarPO 展开错（不是 \"StAble multi-tuRn\"，是 \"State-Thinking-Actions-Reward Policy Optimization\"），且 StarPO-S 提到 critic incorporation，不能笼统说 critic-free",
|
||
"Length 1545 lines exceeds 800-1500 target by 45 (content-dense, accepted WARN)"
|
||
],
|
||
"needs_verify_resolved": [
|
||
"ToolRL Qian 2025 arXiv 2504.13958: VERIFIED (arXiv preprint, no formal venue)",
|
||
"ReSearch Chen 2025 arXiv 2503.19470: VERIFIED + NeurIPS 2025 accepted",
|
||
"RAGEN/StarPO Wang 2025 arXiv 2504.20073: PARTIAL FAIL — arXiv ID + Wang authorship OK, but title + StarPO expansion wrong (now fixed)"
|
||
],
|
||
"fixes_applied": [
|
||
"§2.3 PPO formula: outer $\\mathbb{E}_{\\tau \\sim \\pi_\\text{old}}$ (trajectory), inner $\\sum_{t=1}^{T}$ (token); added clarifying note about index distinction",
|
||
"Q21 rewritten: explicitly stated inside-ratio vs outside-ratio mask are mathematically equivalent (mask is multiplication → gradient = 0 on obs both ways); inside-ratio preferred only for safer monitoring",
|
||
"Q24 rewritten with explicit '官方公开 / 未公开 / 推测' three-bucket framing; emphasizes interviewers value the distinction",
|
||
"§6.1 VPT: 'DeepMind+OpenAI' → 'OpenAI'",
|
||
"§9 + appendix AReaL: 'Anthropic-related' → 'Ant Group + Tsinghua, async RL system, arXiv 2505.24298'",
|
||
"§6.11 WebGUM: 'offline SFT + RL fine-tune' → 'offline SFT from demonstrations (not RL fine-tune); base for later web agent RL (WebRL/AgentQ)'",
|
||
"§6.15 OpenVLA: 'task-specific RL fine-tune' → '970K demonstrations imitation learning + LoRA fine-tuning; not RL (后续 OpenVLA-OFT / π-RL 才做 RL)'",
|
||
"§6.16 Anthropic Computer-Use rewritten with strict '公开 vs 推测' separation; reward/algorithm details explicitly marked as 'not disclosed'",
|
||
"§6.8 RAGEN: corrected title to 'Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning'; StarPO = State-Thinking-Actions-Reward Policy Optimization; StarPO-S notes critic incorporation",
|
||
"Appendix RAGEN citation line: removed [needs-verify], added correct full title + StarPO expansion"
|
||
],
|
||
"warnings_deferred_as_low": [
|
||
"Length 1545 over target by 45 lines — content-dense WARN",
|
||
"§9 SWE-bench Pro / SWE-Lancer / SWE-bench Multilingual mention lacks inline citation — to add footnote in follow-up",
|
||
"TL;DR over-generalization of ToolRL/ReSearch/RAGEN as 'outcome-only + format shaping + token-mask loss + GRPO' is broadly directional but RAGEN explicitly has finer-grained reasoning-aware reward (StarPO-S)"
|
||
]
|
||
}
|
||
]
|
||
},
|
||
"render_review": {
|
||
"verdict": "PASS",
|
||
"rounds": [
|
||
{
|
||
"run": 1,
|
||
"verdict": "PASS",
|
||
"thread_id": "019e408c-b822-7323-9c31-3db5e73ab0ce",
|
||
"reviewer": "codex gpt-5.5 xhigh, fresh thread (main session)",
|
||
"notes": "13/13 functional checks pass. 12 h2 + 57 h3 (69 TOC links all resolve). 25 details blocks. 16 display equations. 13 code blocks. 8 tables. 13 callouts."
|
||
}
|
||
]
|
||
},
|
||
"summary": "Agentic RL tutorial: solo subagent draft (1545 lines, 13 min) → main-session DIY strict review caught 11 substantive issues (PPO formula index collision, Q21 mask explanation, Q24 Anthropic CU overclaim, VPT/AReaL/WebGUM/OpenVLA attribution errors, RAGEN title + StarPO expansion). All fixed. Render review 13/13 PASS.",
|
||
"rendered_at": "2026-05-19"
|
||
}
|