Learning to Reason without External Rewards

arxiv:2505.19590

Original source · Paper JSON

Post-training · 2025-05-26

Cites (11) · Cited by (6)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Learning to Reason without External Rewards
• Abstract
• 1 Introduction
• 2 Related Work
• Reinforcement Learning from Human Feedback (RLHF).
• Reinforcement Learning with Verifiable Rewards (RLVR).
• Intrinsic Signals and Self-Play in LLM Optimization.
• 3 Method
• 3.1 Reinforcement Learning from Internal Feedback (RLIF)
• 3.2 Intuitor : Policy Optimization with Self-Certainty
• 4 Experimental Setup
• 5 Results and Analysis
• 5.1 Learning to Follow Instructions
• 5.2 Fostering Structured Reasoning
• 5.3 Understanding Emergent Long-Form Reasoning
• 5.4 Online Self-Certainty Prevents Reward Exploitation
• 5.5 Ablation Studies
• 6 Discussion and Future Research
• 7 Conclusion
• Ethics Statement
• Reproducibility Statement
• Appendix A Additional Background
• A.1 From External Supervision to Internal Feedback
• Appendix B Additional Experimental Details
• B.1 Influence of the KL Penalty
• B.2 Scaling to Larger Models
• B.3 Generalization Across Model Families
• B.4 Comparison with Alternative Reward Signals
• B.5 Ablation on Optimization Strategy: Policy Gradient vs. Direct Optimization
• B.6 The Effect of Rollout Size
• B.7 Attempts at Combining Golden Answers and Self-Certainty
• B.8 Sequential Training Across Domains
• B.9 Standard Deviation of Response Correctness
• B.10 Failure Case Analysis
• B.11 Training Hyperparameters
• Appendix C Prompts and Model Completions
• C.1 Training Prompts
• C.2 Example from LiveCodeBench Code Generation
• C.3 Example from LiveCodeBench Code Generation
• C.4 Example from CRUXEval-O
• C.5 Rollout Example of Intuitor During Training on Code Corpus
• C.6 Evolution of Qwen2.5-3B’s Responses on LiveCodeBench Code Generation trained with Intuitor on MATH
• Instructions for reporting errors