Understanding R1-Zero-Like Training: A Critical Perspective

arxiv:2503.20783

Original source · Paper JSON

Post-training · 2025-03-26

Cites (3) · Cited by (29)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Understanding R1-Zero-Like Training: A Critical Perspective
• Abstract
• 1 Introduction
• 2 Analysis on Base Models
• 2.1 R1-Zero Trainability: Templates Construct Exploratory Base Policies
• 2.2 Qwen-2.5 Models Unlock the Best Performance When Discarding Template
• 2.3 Aha Moment Already Appears in Base Models Including DeepSeek-V3-Base
• 3 Analysis on Reinforcement Learning
• 3.1 GRPO Leads to Biased Optimization
• 3.2 Dr. GRPO: Group Relative Policy Optimization Done Right
• 3.3 A Duet of Template and Question Set Coverage in RL dynamics
• 3.4 Domain-Specific Pretraining Improves RL Ceiling
• 4 Closing Remarks
• Appendix A Policy Gradient Derivations
• Appendix B Detailed Benchmark Results
• Appendix C Extended Empirical Results
• Appendix D Keyword-based Detection and LLM-Based Identification of Self-Reflection Behaviors
• Appendix E Examples of Aha Moment in DeepSeek-V3-Base
• Appendix F Comparison Between DeepSeek-V3-Base and DeepSeek-R1-Zero
• Appendix G Detailed Experimental Settings
• Appendix H Prompts Used for GPT-As-A-Judge
• Instructions for reporting errors