Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning — part 2/4

Cleaned-text code points [15906, 31582). Bibliography omitted here; source.md includes it.

This strategic diversification provides the most direct evidence for our thesis: the model isn’t just getting better at executing plans; it’s getting better at planning itself. While the model explores new high-level strategic moves, the conditional entropy of procedural grams (grey line) remains stable. This suggests that once a procedural skill like arithmetic is mastered, there is little incentive to find diverse ways to perform it. The improved reasoning performance comes from discovering new ways to combine these established skills, which is the core function of strategic planning.

Crucially, this expansion of the strategic playbook directly correlates with tangible performance gains . The fourth column shows that the rise in strategic diversity is accompanied by a parallel increase in the length of reasoning chains and a sustained boost in overall accuracy. This demonstrates that after procedural skills are consolidated, the development of strategic planning becomes the primary bottleneck and driver for advanced reasoning performance.

Explaining Puzzling Phenomena. This emergent reasoning hierarchy provides a unified explanation for previously observed behaviors. • “Aha moments” are the behavioral signature of the model discovering, mastering, and reinforcing a new, powerful strategy or set of strategic constructs.

• “Length-scaling” is highly consistent with increase in strategic diversity. As Figure 3 shows, the rise in semantic entropy of planning tokens is strongly correlated with an increase in average sequence length. More sophisticated strategies – involving planning, case analysis, and self-reflections – are mediated by planning tokens and naturally produce longer, more successful reasoning traces.

Semantic Entropy: A Good Compass for Exploration. The trends in Figure 3 also highlight a critical flaw in using aggregate token-level entropy (column 2) to track exploration.

Unluckily, the decrease in token-level entropy sometimes mislead practitioners into the conception of declined exploration. This is incorrect, however, as it contradicts the fact of increasing exploration in strategic plans (semantic entropy of planning tokens) and the improving reasoning performance. Figure 4 visualize the difference between token-level entropy and semantic entropy, demonstrating that, a model can be very predictable in its next-token choice (token entropy) under a given context but still create diverse semantic structures and arguments.

In Section 4.2.3 , we compare semantic entropy with token entropy and Pass@K. Our results show that semantic entropy avoids the flaws in token entropy by directly measuring diversity at the semantic level of meaningful strategic units. Its trend accurately reflects the expansion of the model’s strategic playbook, making it a more reliable diagnostic tool for tracking genuine exploration and predicting sustained performance improvements. It also complements Pass@K metric with further benefits. We refer interested readers to the appendix for a full analysis of training dynamics across eight LLM and VLMs and the deeper insights into RL training and exploration.

## 3 HICRA: Hierarchy-Aware Credit Assignment

Our empirical analysis reveals a fundamental insight: RL improves reasoning by rediscovering and operationalizing the strategic layer of reasoning inherited from the model’s pre-training priors. The learning process is characterized by a dynamic shift in its learning frontiers. Initially, the model is constrained by procedural correctness, but as it masters these foundational skills, the frontier for performance improvement shifts to the exploration and mastery of high-level strategic planning.

This observation exposes a core inefficiency in prevailing RL algorithms like GRPO, which apply optimization pressure agnostically across all tokens. Such methods fail to concentrate learning where it matters most – on the emergent strategic bottleneck. To address this, we propose an algorithm designed to focus the model’s learning capacity on the sparse, high-impact planning tokens that orchestrate a successful reasoning trace.

Formulation. We introduce Hierarchy-Aware Credit Assignment (HICRA) , an algorithm that builds upon the GRPO framework to allocate credit based on the reasoning hierarchy. In GRPO, given a query 𝐪 \mathbf{q} from a dataset 𝒟 \mathcal{D} , the policy π θ \pi_{\theta} generates a set of G G output trajectories { 𝐨 1 , … , 𝐨 G } \{\mathbf{o}_{1},\dots,\mathbf{o}_{G}\} . The advantage for a token o i , t o_{i,t} at timestep t t in trajectory 𝐨 i \mathbf{o}_{i} is the group-normalized reward: A ^ i , t = R ⁡ ( 𝐪 , 𝐨 i ) − 1 G ​ ∑ j = 1 G R ⁡ ( 𝐪 , 𝐨 j ) \hat{A}_{i,t}=R(\mathbf{q},\mathbf{o}_{i})-\frac{1}{G}\sum_{j=1}^{G}R(\mathbf{q},\mathbf{o}_{j})\vskip-8.5359pt

HICRA, pronounced “high-krah”, modifies this advantage to prioritize planning tokens. Let 𝒮 i \mathcal{S}_{i} be the set of indices corresponding to planning tokens within trajectory 𝐨 i \mathbf{o}_{i} , identified using the method in Section 2.1. We define the HICRA advantage as: A ^ i , t HICRA = { A ^ i , t + α ⋅ | A ^ i , t | if ​ t ∈ 𝒮 i A ^ i , t if ​ t ∉ 𝒮 i \hat{A}_{i,t}^{\text{HICRA}}=\begin{cases}\hat{A}_{i,t}+\alpha\cdot|\hat{A}_{i,t}|&\text{if }t\in\mathcal{S}_{i}\\ \hat{A}_{i,t}&\text{if }t\notin\mathcal{S}_{i}\end{cases} where α ∈ ( 0 , 1 ) \alpha\in(0,1) is a hyperparameter controlling the amplification intensity (we use α = 0.2 \alpha=0.2 in our experiments). This formulation creates a clear learning hierarchy: for successful trajectories ( A ^ i , t > 0 \hat{A}_{i,t}>0 ), it amplifies the credits for planning tokens, while for unsuccessful ones ( A ^ i , t < 0 \hat{A}_{i,t}<0 ), it dampens their penalty. The resulting RL objective and its policy gradient (simplified without PPO clipping) are: 𝒥 ⁡ ( θ ) = 𝔼 𝐪 ∼ 𝒟 , 𝐨 i ∼ π θ ​ [ A ^ i , t HICRA ] , ∇ 𝒥 ​ ( θ ) = 𝔼 ⁡ [ A ^ i , t HICRA ⋅ ∇ log ⁡ π θ ​ ( o i , t | 𝐪 , 𝐨 i , < t ) ] \mathcal{J}(\theta)=\mathbb{E}_{\mathbf{q}\sim\mathcal{D},\mathbf{o}_{i}\sim\pi_{\theta}}\left[\hat{A}_{i,t}^{\text{HICRA}}\right],\quad\nabla\mathcal{J}(\theta)=\mathbb{E}\left[\hat{A}_{i,t}^{\text{HICRA}}\cdot\nabla\log\pi_{\theta}(o_{i,t}|\mathbf{q},\mathbf{o}_{i,<t})\right] By translating the amplified advantage into a stronger policy gradient, HICRA directly focuses the model’s optimization on the strategic elements of its reasoning process.

Connection to Strategic Exploration. The core mechanism of HICRA engineers more effective exploration by reshaping the policy update’s target distribution. A standard policy gradient ( Williams, 2004 ) update nudges the policy π θ o ​ l ​ d \pi_{\theta_{old}} toward an implicit target distribution π ∗ \pi^{\ast} defined by the advantage function described as follows (the derivation is included in the appendix): π ∗ ​ ( o i , t | 𝐪 , 𝐨 i , < t ) ∝ π θ o ​ l ​ d ​ ( o i , t | 𝐪 , 𝐨 i , < t ) ​ exp ⁡ ( A ^ i , t ) \pi^{\ast}(o_{i,t}|\mathbf{q},\mathbf{o}_{i,<t})\propto\pi_{\theta_{old}}(o_{i,t}|\mathbf{q},\mathbf{o}_{i,<t})\exp(\hat{A}_{i,t}) Typically, this update pressure is applied isotropically, affecting all token types uniformly. HICRA breaks this symmetry. By using the modified advantage A ^ HICRA \hat{A}^{\text{HICRA}} , it creates a new target distribution, π HICRA ∗ \pi^{*}_{\text{HICRA}} , that is anisotropically stretched toward the strategic dimensions of the action space. This new target distribution places significantly greater probability mass on planning tokens (through the term exp ​ ( A ^ i , t ) \text{exp}(\hat{A}_{i,t}) ), particularly those within high-reward trajectories.

This anisotropic reshaping fosters a potent virtuous feedback loop: (a) the policy is incentivized to explore the subspace of strategic plans more thoroughly; (b) this leads to the faster discovery of effective reasoning patterns; and (c) when these strategies yield high rewards, the amplified advantage ensures they are strongly reinforced, cementing the model’s planning capabilities far more efficiently. We also validate the effects of HICRA in exploration through experiments in Section 4 .

## 4 Experiments

Models and Datasets. Our experiments use open-source models including Qwen2.5-7B ( Yang et al., 2024 ) , Qwen3-4B ( Yang et al., 2025 ) , LLama-3.1-8B ( Grattafiori et al., 2024 ) , and VLMs like Qwen2.5-VL-7b ( Yang et al., 2024 ) and MiMO-VL-7B ( Xiaomi, 2025 ) , covering both base and instruction-tuned variants. We train on established reasoning datasets DAPO ( Yu et al., 2025 ) , DeepScaleR ( Luo et al., 2025 ) and ViRL39K ( Wang et al., 2025c ) for VLMs.

Benchmarks and Baselines. We evaluate on a suite of challenging text-only (e.g., AIME24, AIME25 ( Mathematical Association of America, 2024 ) , Math500 ( Lightman et al., 2023 ) , AMC23, Minerva ( Lewkowycz et al., 2022 ) , and Olympiad ( He et al., 2024 ) ) and multimodal (e.g., MathVista ( Lu et al., 2023 ) , MathVerse ( Zhang et al., 2024 ) , MathVision ( Wang et al., 2024 ) , EMMA ( Hao et al., 2025 ) ) benchmarks. We adopt the evaluation protocols of Deepseek R1, using Pass@1 with random samplings. We compare HICRA against three primary baselines: the Base model (before RL), the widely adopted GRPO baseline with clip-higher ( Yu et al., 2025 ) by default, and Entropy Regularization : GRPO with an additional regularization loss on token-level entropy ( Cheng et al., 2025 ) . A comprehensive description of our evaluation protocol, training implementation, and additional model-specific details can be found in the Appendix.

### 4.1 Main Results

Our primary results, summarized in Table 1 and Table 2 , show that HICRA consistently and outperforms both the GRPO baselines across text-only models and vision-language models on various benchmarks. On the strongest base model, Qwen3-4B-Instruct, HICRA’s gains demonstrate that even on highly capable models, selectively amplifying the learning signal for strategic reasoning yields substantial improvements. This trend holds for non-instruct-tuned models as well, providing strong empirical evidence for our central claim: by identifying and focusing on the emergent strategic bottleneck, HICRA accelerates the development of advanced reasoning abilities more efficiently than agnostic methods.

### 4.2 Analysis of RL’s Impact on Reasoning

We conduct a series of analyses to dissect how RL improves reasoning. First, we have linked strategic planning to reasoning through analyses of the training dynamics in Section 2.2 ; Second, we verify the key effects of RL by showing the frequency dynamics of different errors throughout training (Section 4.2.1 ); we then justify the effectiveness of HICRA in exploration by comparing with standard entropy-regularized baselines. Finally, we compare different ways of tracking exploration for RL performance and identifying critical tokens for RL.

#### 4.2.1 Mastery of Strategic Planning Unlocks Improved Reasoning during RL

To understand where RL applies the most leverage, we analyzed the evolution of error types in failed rollouts. We first manually reviewed failures and nominated four distinct error causes. GPT-4o was then prompted to classify each failure into one of these causes via a multiple-choice question. Finally, we parsed these classifications into two broader categories: “Planning & Strategy” (e.g., flawed logic, incorrect high-level plan) and “Others” (e.g., calculation mistakes, fact-retrieval errors). The prompt used is included in the appendix.

Figure 5 reveals a consistent pattern: the primary benefit of RL stems from fixing high-level strategic faults. Across all models, the reduction in strategic errors is more pronounced than the reduction in other errors. This pattern is especially illuminating for Qwen2.5-7B-Base, where non-planning errors does not decrease. We conjecture that while the model may be improving its procedural reliability, these low-level enhancements do not translate to correct answers because the high-level strategy remains the limiting factor. A perfectly executed incorrect plan will still result in failure.

This evidence strongly supports our claim that the strategic bottleneck is the key to unlocking advanced reasoning . RL preferentially corrects these high-level faults over low-level execution mistakes, as improving strategic planning provides the most direct path to solving complex problems.

#### 4.2.2 Justifying HICRA: Targeted vs. Indiscriminate Exploration

Our findings suggest that performance gains are driven by mastering high-level strategic planning, which motivates HICRA’s design to concentrate learning on planning tokens. As shown in Figure 6 , HICRA’s success is linked to its ability to sustain a higher level of semantic entropy than GRPO. This heightened diversity in high-level strategies directly correlates with stronger and more stable validation accuracy, confirming that focused strategic exploration is a primary driver of reasoning improvements.

To further validate this, we compared HICRA against an entropy-regularized baseline. This baseline adds (upon GRPO) an entropy regularization loss applied to all tokens uniformly. The results in Figure 7 show that promoting token-level entropy for sampling diverse tokens is counterproductive.

• The entropy regularization baseline successfully increases Token Entropy , but this fails to translate into performance gains; its Validation Accuracy stagnates and is the lowest of the three methods. This is because indiscriminately promoting token-level diversity only encourages non-productive verbosity on the vast majority of low-level tokens.

• In contrast, HICRA achieves a significantly higher Semantic Entropy , a targeted boost in the diversity of strategic plans that strongly correlates with its superior validation accuracy. This demonstrates that the key to enhanced reasoning is not just to explore, but to focus exploration on the strategic portion of the action space.

#### 4.2.3 Semantic Entropy: A Compass for Strategic Exploration

Given the crucial role of strategic exploration in unlocking reasoning performance during RL, effectively measuring it accurately is paramount. We find that semantic entropy offers distinctive benefits than common alternatives such as token-level entropy or Pass@K ( Chen et al., 2021 ) .

Limitations of Token Entropy and Pass@K. As shown in Figure 8 for MiMO-VL-7B, token-level entropy “collapses” for both HICRA and GRPO, simply because the vast majority of low-level tokens are doomed to become certain, thus pulling the average token entropy down. However, this decrease in token entropy might mislead researchers to suggest that exploration has ceased. Similarly, the Pass@8 (Training) metric quickly saturates, rendering it useless for distinguishing the ongoing learning dynamics.

Semantic Entropy as the Differentiator. In the same experiment, semantic entropy tells a more accurate story. It remains high, indicating continued exploration of diverse reasoning strategies. Crucially, HICRA consistently maintains a higher semantic entropy than GRPO, and this advantage directly correlates with its superior final validation accuracy. This also demonstrates the generality of our approach, extending effectively to multimodal reasoning tasks on vision-language models like MiMO-VL-7B.

#### 4.2.4 Planning Tokens vs. High-Entropy ”Fork” Tokens

Recent work has proposed high-entropy tokens, sometimes called “fork tokens,” to imply its role as proxies for decision points in a reasoning trace ( Wang et al., 2025d ) . Our analysis investigates the relationship between our functionally-defined planning tokens and this entropy-based definition.