Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning

arxiv:2509.03646

Original source · Paper JSON

Post-training · 2025-09-03

Cites (5) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
• Abstract
• 1 Introduction
• 2 The Emergent Reasoning Hierarchy
• 2.1 A Functional Proxy for the Reasoning Hierarchy
• 2.2 Emergence of the Reasoning Hierarchy
• 2.2.1 Forging Reliable Low-level Skills
• 2.2.2 Steering the Skills with Strategic Planning
• 3 HICRA: Hierarchy-Aware Credit Assignment
• 4 Experiments
• 4.1 Main Results
• 4.2 Analysis of RL’s Impact on Reasoning
• 4.2.1 Mastery of Strategic Planning Unlocks Improved Reasoning during RL
• 4.2.2 Justifying HICRA: Targeted vs. Indiscriminate Exploration
• 4.2.3 Semantic Entropy: A Compass for Strategic Exploration
• 4.2.4 Planning Tokens vs. High-Entropy ”Fork” Tokens
• 5 Related Work
• 6 Conclusions
• Appendix A SG Construction
• Appendix B Sensitivity Analysis of SGs
• Appendix C Full Training Dynamics
• Appendix D The Distribution Matching Perspective of Policy Gradients
• Appendix E Extended Materials of Experiments
• E.1 Experimental Setups
• Training Datasets and Benchmarks
• Baselines and Implementation.
• E.2 HICRA Presumes a Dependency on a Procedural Foundation
• Instructions for reporting errors