Reinforcing General Reasoning without Verifiers

arxiv:2505.21493

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-05-27

Cites (7) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Reinforcing General Reasoning without Verifiers
• Abstract
• 1 Introduction
• 2 Methodology
• 2.1 Preliminaries: Verifier-Based Reinforcement Learning
• 2.2 VeriFree Policy Optimization
• Theorem 1 .
• 2.3 Comparison to Existing Approaches
• 2.4 How to handle the tokenization at patching point?
• 3 Experiments
• 3.1 Setup
• 3.2 Main Results
• 3.3 Ablation Study
• 4 Related Works
• 5 Conclusions
• Appendix
• Table of Contents
• Appendix A Theoretical Analysis
• A.1 Derivation of Gradient Estimators
• Proof.
• A.2 Proof of Lower Variance
• Proof.
• Appendix B Off-policy Gradient Estimators
• Appendix C Dataset Details
• Appendix D Extended Empirical Results
• Appendix E Case Study
• Appendix F Broader Impact
• Instructions for reporting errors