Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

arxiv:2605.06241

Original source · Paper JSON

Post-training · 2026-05-07

Cites (6) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning
• Abstract
• 1 Introduction
• 2 Background and Experimental Setup
• 2.1 Reinforcement Learning with Verifiable Rewards
• 2.2 Token-Level Entropy and Decision Points
• 2.3 Models and Baselines
• 3 What RL Actually Changes: Sparse Corrections at Decision Points
• 3.1 Disagreement Is Rare, Conservative, and Concentrated at Decision Points
• 3.2 Correcting Only the Disagreements Recovers RL Performance
• 3.3 Entropy Alone Identifies the Critical Positions
• 4 The Correction Is Low-Dimensional
• 4.1 Distilling RL into a Low‑Rank Adapter
• 4.2 A Small Adapter Captures RL’s Full Correction
• From Representability to Learnability
• 5 ReasonMaxxer – Entropy‑Gated Contrastive Fine‑Tuning
• 5.1 Problem Selection: Exploiting the Edge of Competence
• 5.2 Decision‑Point Identification via Entropy
• 5.3 Advantage‑Weighted Contrastive Loss with Base Anchoring
• 6 Experiments
• 6.1 Experimental Setup
• Benchmarks and evaluation protocol.
• Training configuration for ReasonMaxxer .
• Cost estimation.
• 6.2 Results and Analysis
• ReasonMaxxer matches full RL on clean comparisons.
• Performance generalizes beyond pure RL settings.
• Efficiency and scalability.
• 6.3 Ablation Studies
• Sensitivity to τ \tau .
• Necessity of the contrastive term.
• 7 Related Work
• What RL does for reasoning.
• Entropy and decision points in LLM reasoning.
• RL post‑training baselines.
• RL‑free alternatives for reasoning.
• Efficiency in reasoning.
• 8 Conclusion
• Appendix A Detailed KL‑LoRA Compression Ablations
• Appendix B Implementation Details
• KL‑LoRA Distillation
• ReasonMaxxer
• Appendix C Prompting and Answer Extraction
• Prompt Templates
• Qwen2.5 and Qwen3.
• Mistral‑7B.
• DeepSeek‑R1‑Distill.
• Appendix D Cost Estimation Details
• ReasonMaxxer.
• SimpleRL‑Zoo.
• Open‑Reasoner‑Zero.
• PRIME‑Zero.
• General‑Reasoner (Qwen3‑4B).
• DeepSeek‑based baselines.
• Mistral‑7B.
• Instructions for reporting errors