CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

arxiv:2509.09675

Original source · Paper JSON

Reinforcement learning · 2025-09-11

Cites (4) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models
• Abstract
• 1 Introduction
• 2 Preliminaries: RLVR, GRPO and PPO
• 2.1 Group Relative Policy Optimization (GRPO, Shao et al. 2024 )
• 2.2 Proximal Policy Optimization (PPO, Schulman et al. 2017 )
• 3 CDE: Curiosity-Driven Exploration
• 3.1 Challenge of Count-based exploration for RLVR
• 3.2 Exploration Guided by Actor Curiosity
• Intuitions and Theoretical Foundation
• Theorem 3.1 .
• 3.3 Exploration Guided by Critic Curiosity
• Theorem 3.2 .
• 4 Experiments
• 4.1 Dataset and Model
• 4.2 Main Results
• 4.3 Understanding the Effect of the PPL Bonus
• 4.4 Further Analysis of the Multi-Head Critic
• 5 Related Work
• 5.1 Reinforcement Learning (RL) for LLM reasoning
• 5.2 Efficient exploration
• 6 Conclusion and Future Work
• Appendix A Training Details
• Appendix B Prompt
• Appendix C Details on Hash-based pseudo count
• Appendix D Proof for Calibration Theorem
• Appendix E Proof for Consistency of Multi-head Critic Bonus
• Assumption E.1 (Linear MDP) .
• Lemma E.2 (Proposition 2.3 in Jin et al. (2020) ) .
• Formal Version and Proof of Theorem 3.2
• Theorem E.3 .
• Proof.
• Instructions for reporting errors