Group-in-Group Policy Optimization for LLM Agent Training

arxiv:2505.10978

Original source · Paper JSON

Post-training · 2025-05-16

Cites (3) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Group-in-Group Policy Optimization for LLM Agent Training
• Abstract
• 1 Introduction
• 2 Related Work
• 3 Preliminaries
• 4 Training LLM Agents with GiGPO
• 4.1 Episode Relative Advantages
• 4.2 Step Relative Advantages
• 4.3 Group-in-Group Policy Optimization
• 5 Experiment
• 5.1 Experiment Setup
• 5.2 Performance on ALFWorld and WebShop
• 5.3 Performance on QA tasks
• 5.4 Ablation Study
• 5.5 Dynamics of Step-Level Group
• 5.6 Computational Budget
• 6 Conclusions and Limitations
• Acknowledgements
• Appendix A Open Source Codebase: verl-agent
• Appendix B Broader Impacts
• Appendix C Unbiasedness
• Appendix D Pseudo Code
• Appendix E Experiment Details
• E.1 Details of Training
• E.2 Prompts
• E.3 Performance on Vision-Language Agents
• E.4 Orthogonality to Single-Turn Group-Based RL
• E.5 Sensitivity Analysis on ω \omega
• Appendix F Reasoning Behavior
• F.1 ALFWorld
• F.2 WebShop
• F.3 Search
• Instructions for reporting errors