Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning — part 3/4
Cleaned-text code points [31582, 47576). Bibliography omitted here; source.md includes it.
Figure 10 and Figure 9 reveal a crucial asymmetry. While a majority of planning tokens exhibit high entropy (aligning with their role as points of strategic choice), the reverse is not true: most high-entropy tokens are not planning tokens. This finding highlights the limitations of using high entropy as a standalone proxy for strategic function. High token-level entropy ensures sampling diversity, but it does not guarantee semantic function. Many high-entropy tokens may correspond to variations in phrasing or calculation that do not alter the high-level reasoning path. In contrast, our approach identifies tokens based on their functional role in orchestrating the solution, providing a more direct and reliable signal for strategic credit assignment.
## 5 Related Work
Reinforcement Learning for LLM Reasoning. The application of Reinforcement Learning (RL) has been pivotal in enhancing the complex reasoning abilities of Large Language Models (LLMs). Seminal work by Ouyang et al. demonstrated the effectiveness of learning from human feedback to align models with user instructions. More recently, algorithms like Group Reward Policy Optimization ( Guo et al., 2025 ) have been developed to specifically incentivize reasoning capabilities in LLMs,VLMs, Agents ( Liu et al., 2025b ; Yu et al., 2025 ; Team et al., 2025 ; Liu et al., 2025a ; Wang et al., 2025c ; Wang et al., 2025b ; Su et al., 2025 ; Dai et al., 2025 ; Zheng et al., 2025 ) , leading to significant performance gains on downstream performance. While these methods have proven empirically successful, they typically apply optimization pressure agnostically across all generated tokens, without distinguishing between different functional roles within the reasoning process. Our work builds on this foundation but introduces a more targeted approach by focusing on the emergent reasoning hierarchy.
Analysis of RL Dynamics and Exploration in LLMs . A growing body of research seeks to understand the complex learning dynamics that occur during the RL fine-tuning of LLMs. Several studies have investigated the role of token-level entropy, observing intricate patterns and its connection to model exploration and uncertainty ( Cui et al., 2025 ; Chen et al., 2025 ) . Concurrently, phenomena such as sudden ”aha moments” and performance improvements from longer outputs (”length-scaling”) have been noted as characteristic but poorly understood outcomes of RL training ( Guo et al., 2025 ; Liu et al., 2025b ) .Our paper provides a unifying framework, interpreting these phenomena as evidence of a shift from procedural learning to strategic planning.
Furthermore, recent work has identified high-entropy ”fork tokens” as potential proxies for critical decision points in reasoning ( Wang et al., 2025d ) . Our work distinguishes itself by defining planning tokens based on their semantic function. We also validate the limitation of identifying crucial tokens solely based on entropy.
Exploration-Exploitation trade-off has become a long-standing research problem in classical RL literature. Among the vast literature, Entropy Regularization or Maximum-Entropy RL ( Levine, 2018 ; Haarnoja et al., 2017 ; Wang et al., 2023 ) is a standard technique to encourage exploration that can be seamlessly integrated with LLM RL training.
Hierarchical Reasoning and Cognition. The concept of hierarchical processing is a cornerstone of cognitive neuroscience, which posits that the human brain separates high-level, abstract planning from low-level motor or procedural execution ( Huntenburg et al., 2018 ; Murray et al., 2014 ; Zeraati et al., 2023 ; Zhu et al., 2025 ; Xiong et al., 2025 ) . HRM ( Wang et al., 2025a ) is inspired this cognitive architecture to design a specific neural architecture for hierarchical reasoning. Concurrently, this cognitive model provides a compelling parallel to the functional hierarchy we identify in RL-tuned LLMs, proposing that LLMs similarly develop a functional separation between strategic planning and procedural execution.
## 6 Conclusions
Our work establishes that reinforcement learning uncovers an emergent functional reasoning hierarchy in language models, demonstrating the critical performance bottleneck shifting from procedural skill to strategic exploration. This insight leads to our approach, HICRA, which demonstrates that specialized credit assignment targeting this strategic bottleneck yields more effective training. Extensive experiments validate the effectiveness of HICRA and offer deep insights into advanced reasoning through strategic exploration.
Our work opens several future research directions. First, it suggests a paradigm shift away from treating all tokens equally and prompts a rethinking of the action space away from individual tokens toward semantic, strategic units. Second, it calls for developing process-oriented approaches capable of valuing correct strategic choice even if the final answer is flawed. Finally, the likely universality of this reasoning hierarchy in complex reasoning tasks suggests that applying these principles to domains like code generation and agentic tool-use is a valuable path forward.
## Appendix A SG Construction
This pipeline result in the following collection of SGs.
Strategic Grams
## Appendix B Sensitivity Analysis of SGs
This automated procedure is designed to yield a high-precision functional proxy for strategic planning, not an exhaustive lexicon of all possible SGs. We set reasonable hyper-parameters for identifying SGs, and we contend that the resulting SG collection is sufficiently representative to reveal the core learning dynamics. To validate this claim, we conduct a sensitivity analysis by randomly removing 30% of the identified SGs and re-running our main analysis. As shown in Figure 11 and Figure 12 , the semantic entropy curves remain qualitatively identical, and the curves for perplexity and token entropy only slightly change. Semantic entropy here calculates the entropy of frequency distribution, which itself is not sensitive to slight changes in the frequency mass, as long as there are sufficient numbers of bins. This demonstrates the robustness of our SG identification and the findings derived from it.
## Appendix C Full Training Dynamics
Following the discussion in the main paper, we make the following further observations based on the provided training charts: • The initial skill-consolidation phase might be brief or absent for some models. In the cases of the Vision-Language Models (Qwen2.5 VL-Instruct and MiMO VL-Instruct), Qwen3 4B-Instruct, Deepseek-Distill-Llama-8B, the exploration of strategic planning begins almost immediately at the start of training. This is evidenced by a significant and immediate rise in the semantic entropy of strategic grams, which occurs in tandem with a rapid boost in validation accuracy. We conjecture this is because: (a) for the VL scenarios, publicly available datasets is learned quickly by state-of-the-art models ( Wang et al., 2025c ) ; (b) strong base models like Qwen3 4B-Instruct already possess a solid foundation of low-level skills and primarily need to adapt to formatting before focusing on higher-level strategic planning.
• Token-level entropy does not directly correlate with model accuracy. This is strongly supported across multiple experiments. For instance, with Llama3.1 8B, Qwen3 4B-Instruct, and the VL models, token-level entropy either remains flat or decreases throughout training. During the same period, however, validation accuracy shows a steady and significant increase. This demonstrates a clear disconnect between next-token uncertainty and overall task performance.
• Token-level entropy is misleading for policy exploration. This observation holds true across all experiments. The Qwen3 4B-Instruct model offers a particularly stark example: its token-level entropy remains almost perfectly flat, while its semantic entropy (diversity of strategic grams) consistently increases throughout training. This contrast highlights that the variety of semantic structures a model learns is completely different from the statistical uncertainty of its next-token predictions. Figure 4 illustrates the differences of the two entropy. The core difference is about scale: token-level entropy measures the uncertainty of every next-token, including the vast amount of low-level tokens such as formatting, executions that are doomed to become confident throughout training. In contrast, semantic entropy measures the diversity of the overall meanings being expressed. A model can be very predictable in its next-token choice under a given context but still create a wide variety of different arguments or structures.
• Lack of Strategic Exploration Hinders Sustained Improvement in Llama Models. We observe that the Llama-3.1-8B-Base model initially focuses almost exclusively on consolidating low-level procedural skills, a phase marked by decreasing perplexity and token entropy on execution tokens. However, once the performance gains from this procedural refinement diminish, the model fails to pivot towards exploring high-level planning strategies. This leads to performance stagnation and, eventually, degradation. This behavior stands in stark contrast to the more successful Deepseek Distilled Llama model, which engages in high-level strategic exploration from the very beginning of training, bypassing a distinct procedural consolidation phase. We hypothesize that for the standard Llama models, the intense initial focus on procedural correctness prematurely collapses the diversity of high-level reasoning strategies. By the time low-level skills are mastered, the model has likely converged on simpler reasoning patterns, which inhibits its ability to subsequently discover and adopt more complex and effective problem-solving approaches.
## Appendix D The Distribution Matching Perspective of Policy Gradients
Imagine an ideal, or ”target,” policy, π ∗ ( a | s ) \pi^{*}(a|s) , that we want our current policy, π θ ( a | s ) \pi_{\theta}(a|s) , to emulate. We can conceptualize this target distribution as being proportional to the exponentiated advantage of the actions: π ∗ ( a | s ) ∝ π θ o l d ( a | s ) exp ( A ^ ( a , s ) ) \pi^{*}(a|s)\propto\pi_{\theta_{old}}(a|s)\exp(\hat{A}(a,s)) (1)
Or more concretely, π ∗ ( a | s ) = 1 Z ( s ) π θ o l d ( a | s ) exp ( A ^ ( a , s ) β ) \pi^{*}(a|s)=\frac{1}{Z(s)}\pi_{\theta_{old}}(a|s)\exp\left(\frac{\hat{A}(a,s)}{\beta}\right) (2) This target policy, π ∗ ( a | s ) \pi^{*}(a|s) , re-weights the old policy based on the advantage of each action. Here, actions with a positive advantage ( A ^ > 0 \hat{A}>0 ) get their probability boosted exponentially, while actions with a negative advantage ( A ^ < 0 \hat{A}<0 ) get their probability suppressed. The term β \beta acts as a ”temperature” parameter.
The goal is to find a new policy, π θ \pi_{\theta} , that is as close as possible to this ideal target distribution, π ∗ \pi^{*} . This is equivalent to minimizing the KL divergence: min θ KL ( π ∗ ( a | s ) | | π θ ( a | s ) ) \min_{\theta}\text{KL}(\pi^{*}(a|s)||\pi_{\theta}(a|s)) (3) Expanding this KL divergence term: KL ( π θ | | π ∗ ) \displaystyle\text{KL}(\pi_{\theta}||\pi^{*}) = 𝔼 a ∼ π θ [ log π θ ( a | s ) π ∗ ( a | s ) ] \displaystyle=\mathbb{E}_{a\sim\pi_{\theta}}\left[\log\frac{\pi_{\theta}(a|s)}{\pi^{*}(a|s)}\right] = 𝔼 a ∼ π θ [ log π θ ( a | s ) − log π ∗ ( a | s ) ] \displaystyle=\mathbb{E}_{a\sim\pi_{\theta}}[\log\pi_{\theta}(a|s)-\log\pi^{*}(a|s)] Substitute our definition of log π ∗ ( a | s ) = log π θ o l d ( a | s ) + A ^ ( a , s ) β − log Z ( s ) \log\pi^{*}(a|s)=\log\pi_{\theta_{old}}(a|s)+\frac{\hat{A}(a,s)}{\beta}-\log Z(s) : = 𝔼 a ∼ π θ [ log π θ ( a | s ) − ( log π θ o l d ( a | s ) + A ^ ( a , s ) β − log Z ( s ) ) ] \displaystyle=\mathbb{E}_{a\sim\pi_{\theta}}\left[\log\pi_{\theta}(a|s)-\left(\log\pi_{\theta_{old}}(a|s)+\frac{\hat{A}(a,s)}{\beta}-\log Z(s)\right)\right] ∝ 𝔼 a ∼ π θ [ ( log π θ ( a | s ) − log π θ o l d ( a | s ) ) − A ^ ( a , s ) β ] \displaystyle\propto\mathbb{E}_{a\sim\pi_{\theta}}\left[\left(\log\pi_{\theta}(a|s)-\log\pi_{\theta_{old}}(a|s)\right)-\frac{\hat{A}(a,s)}{\beta}\right] = 1 β 𝔼 a ∼ π θ [ β KL ( π θ | | π θ o l d ) − A ^ ( a , s ) ] \displaystyle=\frac{1}{\beta}\mathbb{E}_{a\sim\pi_{\theta}}\left[\beta\text{KL}(\pi_{\theta}||\pi_{\theta_{old}})-\hat{A}(a,s)\right] Minimizing this is equivalent to maximizing its negative: max θ 𝔼 a ∼ π θ [ A ^ ( a , s ) − β KL ( π θ | | π θ o l d ) ] \max_{\theta}\mathbb{E}_{a\sim\pi_{\theta}}\left[\hat{A}(a,s)-\beta\text{KL}(\pi_{\theta}||\pi_{\theta_{old}})\right] (4)
This expression is nearly identical to the PPO-KL objective, where the policy update is constrained using a KL divergence regularizer ( Schulman et al., 2017 ) .
Therefore, the PPO objective is essentially solving a distribution matching problem toward a target distribution shaped by the advantage function. It follows that a standard policy gradient ( Williams, 2004 ) update nudges the policy π θ o l d \pi_{\theta_{old}} toward an implicit target distribution π ∗ \pi^{\ast} defined by the advantage function. After the gradient update, the target distribution becomes the new policy sampled for exploration. Therefore, adjusting the advantage function (or credit assignment) essentially shapes the exploration policy during RL training .
## Appendix E Extended Materials of Experiments
### E.1 Experimental Setups
##### Training Datasets and Benchmarks
The training dataset for LLM reasoning is sourced from DAPO ( Yu et al., 2025 ) and DeepScaleR ( Luo et al., 2025 ) . The dataset for training VLM is sourced from ViRL39K ( Wang et al., 2025c ) . We evaluate all models on a diverse set of challenging mathematical reasoning benchmarks to rigorously test their complex reasoning capabilities. The text-only benchmarks include AIME24, AIME25 ( Mathematical Association of America, 2024 ) , Math500 ( Lightman et al., 2023 ) , AMC23, Minerva ( Lewkowycz et al., 2022 ) , and Olympiad ( He et al., 2024 ) . We follow Deepseek R1 ( Guo et al., 2025 ) ’s evaluation protocol, where the performance is measured by Pass@1 Accuracy with temperature 0.6 0.6 sampling. For benchmarks with less than 100 queries, we use average accuracy of 32 samplings on AIME24/25 and 4 samplings on AMC23 ( Yu et al., 2025 ) . Following VL-Rethinker ( Wang et al., 2025c ) , we evaluate on MathVista ( Lu et al., 2023 ) , MathVerse ( Zhang et al., 2024 ) , MathVision ( Wang et al., 2024 ) , and EMMA ( Hao et al., 2025 ) for assessing multimodal reasoning across domains and disciplines. For all evaluation, we adopt strict answer matching that relies on the \boxed format.
##### Baselines and Implementation.
We compare HICRA against three primary baselines: the Base model (before RL), the widely adopted GRPO baseline with clip-higher ( Yu et al., 2025 ) by default, and Entropy Regularization : GRPO with an additional regularization loss on token-level entropy ( Cheng et al., 2025 ) . For HICRA, we set the amplification hyperparameter α \alpha to 0.2 0.2 and identify planning tokens using the Strategic Grams (SGs) methodology detailed in Section 2.1. For all experiments, we increase the training context length from 16K to 32K when the response clip rate exceeds 20% ( Luo et al., 2025 ) . For the specific experiments on Llama-3.1-Instruct, we add a dynamic filtering mechanism ( Yu et al., 2025 ) based on GRPO Clip-Higher due to significant vanishing advantanges ( Wang et al., 2025c ) . We use two to four sets of eight A100 (80G) for training all models, and we stop the experiments if performance continues to degrade during extended training.
Prompt Used for Error Analysis. Listing shows the prompt for identifying error types in our experiments.
Instruction for Identifying Error Types