It Takes Two: Your GRPO Is Secretly DPO

arxiv:2510.00977

Original source · Paper JSON

Post-training · 2025-10-01

Cites (4) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• It Takes Two: Your GRPO Is Secretly DPO
• Abstract
• 1 Introduction
• 2 Preliminary
• 2.1 Problem Setting and Notation
• 2.2 The Story of Variance Reduction: VPG, PPO, and GRPO
• 3 A Tale of Two Algorithms: GRPO and DPO
• 3.1 Contrastive Objective for Sequences
• Definition 3.1 (Contrastive Loss for Sequences) .
• DPO is a 1-vs-1 contrastive learning
• 3.2 GRPO: N-vs-M Contrastive Learning
• Proposition 3.2 .
• 3.3 Echoes of Contrastiveness: GRPO and DPO
• 4 Why Viewing GRPO From Contrastive Learning?
• 4.1 Variance Reduction via Contrastive Objective
• Proposition 4.1 .
• 4.2 GRPO with Small Group Size: It Should Fail, But Doesn’t
• 4.3 Introducing 2-GRPO
• 5 Experiments
• 5.1 Math Reasoning
• 5.2 Ablation Study: The Effect of Group Size
• 6 Conclusion
• Appendix
• Appendix A Related Work
• A.1 Contrastive Learning and LLM Alignment
• A.2 Adaptive Rollouts in RLVR
• Appendix B Theorems
• B.1 Variance Reduction of Policy Gradient Estimate
• B.2 Mean Estimation with Samples n = 2 {n=2}
• B.3 Reveal GRPO as Contrastive Learning
• Proof of Proposition 3.2 .
• B.4 Further Discussion on Importance Sampling and the Log-likelihood Term
• B.5 Proof of Lemma B.1 : DPO is 1-vs-1 contrastive learning
• Lemma B.1 .
• Proof of Lemma B.1 .
• B.6 GRPO v.s. DPO from Contrastive Learning
• B.7 Proof of Proposition 4.1
• Proof.
• B.8 The Correlation between The Positive and The Negative
• B.9 Proof of Proposition C.1
• Proof.
• B.10 Proof of Lemma C.3
• Proof of Lemma C.3 .
• Appendix C Theoretical Analysis of 2-GRPO
• C.1 Implicit Weighting in Stochastic Optimization
• Proposition C.1 .
• C.2 Key of Variance Reduction: the Training Batch Size, not the Group Size
• Definition C.2 (Variance of Gradient Estimate in Mini-Batch) .
• Lemma C.3 .
• C.3 Exploration on Hard Questions
• Proposition C.4 .
• Appendix D Experiments
• D.1 Experiment Details
• Dataset and Baselines
• Hyper-parameters
• D.2 The Connection Between Training Rollouts and Computational Cost
• D.3 Sample Discard Rate of 2-GRPO
• D.4 Ablation Study on Group Size
• Appendix E Limitation
• Appendix F The Use of Large Language Models (LLMs)
• Instructions for reporting errors