{"id":"arxiv:2605.06241","title":"Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning","date":"2026-05-07","section":"Post-training","quality":0.489291,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/","source_url":"https://arxiv.org/abs/2605.06241","text_available":true,"authors":["Ömer Faruk Akgül","Rajgopal Kannan","Willie Neiswanger"],"outgoing":[{"source":"arxiv:2605.06241","target":"arxiv:2402.03300","evidence":"id","id":"arxiv:2402.03300","title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/636fb871904bc06f742cfdea069796f9e9665f95bd58dc63260eaff5154ef985/"},{"source":"arxiv:2605.06241","target":"arxiv:2504.13837","evidence":"id","id":"arxiv:2504.13837","title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b90f70269aeb02311d74ffaadf47e139683424a47118dc18619000ba72174d17/"},{"source":"arxiv:2605.06241","target":"arxiv:2504.20571","evidence":"id","id":"arxiv:2504.20571","title":"Reinforcement Learning for Reasoning in Large Language Models with One Training Example","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b2b21c3df49619588c371411dba9467a41f81b3fd4da5441898b5145bb0de120/"},{"source":"arxiv:2605.06241","target":"arxiv:2505.15134","evidence":"id","id":"arxiv:2505.15134","title":"The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/12b01b403e280a32d41fd991a27fa30f33315e09f0f42a4f43e9bece5ad9c53d/"},{"source":"arxiv:2605.06241","target":"arxiv:2506.01939","evidence":"id","id":"arxiv:2506.01939","title":"Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/e0f4a55bca612bce5895b6b98f89b0efc14cbd8cf45946a7743347cf6daee1a1/"},{"source":"arxiv:2605.06241","target":"arxiv:2510.14901","evidence":"id","id":"arxiv:2510.14901","title":"Reasoning with Sampling: Your Base Model is Smarter Than You Think","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/46373e6f386387299bbade0a302219a611c3497c5e5b5970e3502f7a5f189687/"}],"incoming":[],"text_parts":[{"start":0,"end":15771,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html"},{"start":15771,"end":31213,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html"},{"start":31213,"end":46867,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html"},{"start":46867,"end":48712,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-4.html"}],"contents":[{"title":"Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"Abstract","offset":90,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"1 Introduction","offset":1757,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"2 Background and Experimental Setup","offset":7608,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"2.1 Reinforcement Learning with Verifiable Rewards","offset":7648,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"2.2 Token-Level Entropy and Decision Points","offset":9414,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"2.3 Models and Baselines","offset":10513,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"3 What RL Actually Changes: Sparse Corrections at Decision Points","offset":10884,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"3.1 Disagreement Is Rare, Conservative, and Concentrated at Decision Points","offset":11811,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"3.2 Correcting Only the Disagreements Recovers RL Performance","offset":13802,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"3.3 Entropy Alone Identifies the Critical Positions","offset":15078,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-1.html#text"},{"title":"4 The Correction Is Low-Dimensional","offset":16439,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"4.1 Distilling RL into a Low‑Rank Adapter","offset":17222,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"4.2 A Small Adapter Captures RL’s Full Correction","offset":18084,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"From Representability to Learnability","offset":19547,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"5 ReasonMaxxer – Entropy‑Gated Contrastive Fine‑Tuning","offset":20120,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"5.1 Problem Selection: Exploiting the Edge of Competence","offset":20670,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"5.2 Decision‑Point Identification via Entropy","offset":21920,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"5.3 Advantage‑Weighted Contrastive Loss with Base Anchoring","offset":22975,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"6 Experiments","offset":25537,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"6.1 Experimental Setup","offset":25798,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"Benchmarks and evaluation protocol.","offset":25826,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"Training configuration for ReasonMaxxer .","offset":26370,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"Cost estimation.","offset":27110,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"6.2 Results and Analysis","offset":27801,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"ReasonMaxxer matches full RL on clean comparisons.","offset":27929,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"Performance generalizes beyond pure RL settings.","offset":28957,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"Efficiency and scalability.","offset":29685,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"6.3 Ablation Studies","offset":31157,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"Sensitivity to τ \\tau .","offset":31183,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-2.html#text"},{"title":"Necessity of the contrastive term.","offset":32122,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"7 Related Work","offset":33153,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"What RL does for reasoning.","offset":33380,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Entropy and decision points in LLM reasoning.","offset":34353,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"RL post‑training baselines.","offset":35111,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"RL‑free alternatives for reasoning.","offset":36159,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Efficiency in reasoning.","offset":36836,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"8 Conclusion","offset":38307,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Appendix A Detailed KL‑LoRA Compression Ablations","offset":39337,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Appendix B Implementation Details","offset":40539,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"KL‑LoRA Distillation","offset":40731,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"ReasonMaxxer","offset":41392,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Appendix C Prompting and Answer Extraction","offset":42753,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Prompt Templates","offset":43060,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Qwen2.5 and Qwen3.","offset":43082,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Mistral‑7B.","offset":43287,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"DeepSeek‑R1‑Distill.","offset":43441,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Appendix D Cost Estimation Details","offset":43663,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"ReasonMaxxer.","offset":44351,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"SimpleRL‑Zoo.","offset":44766,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Open‑Reasoner‑Zero.","offset":45371,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"PRIME‑Zero.","offset":46275,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"General‑Reasoner (Qwen3‑4B).","offset":46620,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"DeepSeek‑based baselines.","offset":46835,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-3.html#text"},{"title":"Mistral‑7B.","offset":47294,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-4.html#text"},{"title":"Instructions for reporting errors","offset":47748,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/part-4.html#text"}],"text_code_points":48712,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/c9e4fdd6e0c24f5a635ff70c2bca19da19c4012a05f69ceb20fdc0a86fd8b775/source.md"}