DAPO: An Open-Source LLM Reinforcement Learning System at Scale

arxiv:2503.14476

Original source · Paper JSON

Post-training · 2025-03-18

Cites (2) · Cited by (47)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Abstract
• 1 Introduction
• 2 Preliminary
• 2.1 Proximal Policy Optimization (PPO)
• 2.2 Group Relative Policy Optimization (GRPO)
• 2.3 Removing KL Divergence
• 2.4 Rule-based Reward Modeling
• 3 DAPO
• 3.1 Raise the Ceiling: Clip-Higher
• 3.2 The More the Merrier: Dynamic Sampling
• 3.3 Rebalancing Act: Token-Level Policy Gradient Loss
• 3.4 Hide and Seek: Overlong Reward Shaping
• 3.5 Dataset Transformation
• 4 Experiments
• 4.1 Training Details
• 4.2 Main Results
• 4.3 Training Dynamics
• 4.4 Case Study
• 5 Conclusion
• Contributions
• Acknowledgments
• Appendix A Dataset Transformation
• Original Problem
• Transformed Problem
• Model’s output (Markdown Text)
• Transformation Thought Process
• Answer Format
• Transformed Problem
• Answer Calculation for the Transformed Problem
• Final Answer
• Appendix B Supplementary Case
• Instructions for reporting errors