Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning — part 1/4

Cleaned-text code points [0, 15906). Bibliography omitted here; source.md includes it.

# Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning

###### Abstract

Reinforcement Learning (RL) has proven highly effective at enhancing the complex reasoning abilities of Large Language Models (LLMs), yet underlying mechanisms driving this success remain largely opaque. Our analysis reveals that puzzling phenomena like “aha moments”, “length-scaling” and entropy dynamics are not disparate occurrences but hallmarks of an emergent reasoning hierarchy, akin to the separation of high-level strategic planning from low-level procedural execution in human cognition. We uncover a compelling two-phase dynamic: initially, a model is constrained by procedural correctness and must improve its low-level skills. The learning bottleneck then decisively shifts, with performance gains being driven by the exploration and mastery of high-level strategic planning. This insight exposes a core inefficiency in prevailing RL algorithms like GRPO, which apply optimization pressure agnostically and dilute the learning signal across all tokens. To address this, we propose Hierarchy-Aware Credit Assignment (HICRA), an algorithm that concentrates optimization efforts on high-impact planning tokens. Our extensive experiments validate that HICRA significantly outperforms strong baselines, and offer deep insights into how reasoning advances through the lens of strategic exploration.

## 1 Introduction

Reinforcement Learning (RL) has become instrumental in advancing the complex reasoning capabilities of Large Language Models (LLMs) across diverse domains ( Ouyang et al., 2022 ; Jaech et al., 2024 ; Yang et al., 2024 ; Guo et al., 2025 ; Team et al., 2025 ) . However, this empirical success is accompanied by a significant gap in our understanding of the underlying learning dynamics. The training process often yields phenomena that are as effective as they are poorly understood: models can experience sudden ‘aha moments’, where they seemingly acquire new emergent skills ( Guo et al., 2025 ) ; they exhibit ‘length-scaling’ effects, where reasoning performance improves with longer, more detailed outputs ( Guo et al., 2025 ; Team et al., 2025 ) ; and they display complex dynamics in token-level entropy ( Yu et al., 2025 ; Cui et al., 2025 ) . This gap motivates a fundamental question:

What unlocks enhanced reasoning in LLMs during RL, and how should we leverage this understanding to design more principled and efficient RL algorithms?

Our investigation is guided by a key insight: RL does not train models de novo. It fine-tunes base models already imbued with priors from pre-training on vast corpora of human-written solutions. These solutions inherently encodes the hierarchical structure of human reasoning – a highly efficient cognitive strategy evolved under biological constraints. This prompts us to ask: does RL unlock advanced reasoning by (re-)discovering this hierarchical structure as a promising pathway for solving math problems?

To test this hypothesis, we analyze the RL training process through the lens of hierarchical reasoning. Drawing a parallel to the cognitive architecture of the human brain (Fig. 1 ), which separates high-level, deliberate strategic planning from the rapid execution of learned procedures ( Murray et al., 2014 ; Zeraati et al., 2023 ; Huntenburg et al., 2018 ) , we propose a decomposition of model-generated tokens into two functional hierarchy : • High-level Planning Tokens: The high-level strategic moves that orchestrate the reasoning process. These tokens manifest as logical maneuvers, including deduction (e.g., ”we can use the fact that”), branching (e.g., ”let’s try a different approach”), and backtracing (e.g., ”but the problem mentions that”).

• Low-level Execution Tokens: The operational building blocks of a solution. These comprise concrete, low-level steps such as arithmetic calculations, variable substitutions, and the direct application of known formulas.

Our analysis across eight text-only and vision-language models confirms this hypothesis, revealing a consistent two-phase dynamic that explains the emergence of this reasoning hierarchy in LMs. We find the optimization pressure of RL is not static; instead, its learning frontier shifts. Initially, the process is constrained by procedural correctness . A single calculation error can invalidate an entire solution, creating a powerful learning signal that compels the model to first master low-level execution tokens. Once proficiency in these foundational skills is achieved, the learning bottleneck shifts to strategic planning . We find that exploring and mastering the use of planning tokens is what unlocks significant and sustained improvements in reasoning ability.

This emergent two-phase mechanism provides a unifying framework for the puzzling phenomena observed in RL training. It explains ”aha moments” as the discovery and internalization of high-level strategic reasoning strategies, such as self-reflections. It also accounts for the ”length-scaling” effect, as employing more sophisticated strategies – involving thorough planning and logical backtracing – naturally elongates the reasoning trace with structured, strategic deliberation. Notably, it provides a unified perspective to understand the complex token entropy dynamics across different models, through the lens of high-impact planning tokens and gradually confident execution tokens.

This discovery – that the learning frontiers dynamically shifts to strategic planning – is more than an academic curiosity; it provides a clear blueprint for a more effective RL algorithm. If the primary driver for advanced reasoning is the mastery of high-level strategic planning, then current agnostic credit assignment methods used in the prevailing GRPO ( Guo et al., 2025 ) and its variants ( Yu et al., 2025 ; Liu et al., 2025b ; Wang et al., 2025c ) are fundamentally inefficient, as they dilute optimization pressure across all tokens rather than concentrating it where it matters most.

Based on this insight, we propose Hierarchy-aware Credit Assignment (HICRA) , a novel algorithm designed to focus optimization pressure directly on this emergent strategic bottleneck. By selectively amplifying the learning signal for planning tokens, HICRA accelerates the exploration and reinforcement of effective high-level reasoning, leading to significant performance gains as demonstrated in our experiments.

Contributions. In this work, we advance the understanding of how LLMs learn to reason via RL. We demonstrate that the learning process is not monolithic but an emergent two-phase learning dynamic driven by the hierarchical priors in base models and solution structure of the reasoning tasks. This insight reveals that the true bottleneck for advanced reasoning is the mastery of high-level strategic planning, which current agnostic credit assignment methods neglect. To bridge this gap, we pioneer with an original and simple solution, HICRA. Through extensive experiments across LLMs and VLMs, we not only validate the effectiveness of HICRA, but also offer deep insights into how HICRA works through the lens of strategic exploration.

## 2 The Emergent Reasoning Hierarchy

Guiding Insight: The pre-training priors and the inherent structure of reasoning tasks create a strong inductive bias. For the task of math problem-solving, hierarchical reasoning proves an efficient and prominent strategy, which is discovered through RL training and unlocks advanced reasoning.

### 2.1 A Functional Proxy for the Reasoning Hierarchy

To analyze this reasoning hierarchy, we must first distinguish high-level strategic planning from low-level procedural execution within the model’s generated tokens. This is challenging because a token’s function is defined by its context, not its intrinsic meaning.

To address this gap, we draw inspiration from human cognition. When a person reasons through a problem, we easily identify their strategic thinking by its function. A phrase like, “Let’s try a different approach,” functions as a high-level strategic maneuver that guides the problem-solving direction. In contrast, a phrase like, “so we add 5 to both sides,” is a low-level procedural step. Inspired by this functional distinction, we introduce Strategic Grams as a functional proxy to circumvent the difficulty of formally defining what is a “planning token”.

Strategic Grams (SGs) are defined as n n -grams that function as a single semantic unit to guide the logical flow. We use n-grams because they capture the phrasal nature of strategic language (e.g., ”let’s consider the case”) which is lost at the single-token level. These SGs facilitate three main types of logical moves: (a) deduction, (b) branching, and (c) backtracing, as we show in Figure 2 .

Given a collection of SGs, our classification heuristic is straightforward: A token is classified as a strategic planning token if it is part of a Strategic Gram in the current context. All other tokens are classified as procedural execution tokens .

For simplicity, we use “execution tokens” to encompass all non-planning tokens, including concrete calculations, formatting, and other procedural language.

A key challenge is identifying a set of SGs in a principled and automated manner, avoiding the subjectivity of manual annotation or reliance on proprietary models. Our approach is based on a key insight: SGs function as the reusable scaffolding of a reasoning process (Fig. 2 ). This functional role imparts a distinct statistical signature: SGs should appear frequently across a wide range of different solutions but be used sparingly within any single solution. However, linguistic diversity of strategic language presents another hurdle: a single strategic intent can be expressed in numerous ways.

Our pipeline is meticulously designed to overcome these challenges. It first groups semantically equivalent n-grams to consolidate diverse phrasing and then identifies which of these consolidated concepts exhibit the statistical signature of strategic scaffolding. Specifically, we construct the SG set via the following three-step procedure:

1. Semantic Clustering: We first extract all n n -grams (where n ∈ [ 3 , 5 ] n\in[3,5] ) from a large corpus of successful reasoning solutions. Using the sentence_transformers package, we project each n n -gram into a semantic embedding space using a pre-trained sentence transformer, and then apply a clustering algorithm to this embedding space. This step groups lexically diverse but semantically equivalent n n -grams into a single cluster, directly addressing the challenge of linguistic diversity.

2. Identification by Frequency: To identify the clusters that represent reusable reasoning patterns, we analyze their frequency at the corpus level. For each semantic cluster, we compute its Cluster Document Frequency (Cluster DF) : the frequency of unique solutions that contain at least one n n -gram from that cluster.

3. SG Construction: We filter for clusters with top 20% Cluster DF, implying that these SGs are common across many problems. The union of all n n -grams within these high-frequency clusters constitutes our final set of Strategic Grams.

This data-driven procedure yields a high-precision functional proxy, not an exhaustive lexicon of SGs. To validate the robustness of this approach, we conducted a sensitivity analysis by randomly removing 30% of the identified SGs. The resulting learning dynamic curves remained qualitatively identical (see Appendix), confirming that our SG set is sufficiently representative to reveal the core learning dynamics.

### 2.2 Emergence of the Reasoning Hierarchy

Building on our functional proxy for reasoning, we examine the learning dynamics of RL for LLM reasoning and finds an intriguing parallel with human-like hierarchical reasoning. Our empirical analysis – conducted consistently across different model families, Qwen2.5-7B ( Yang et al., 2024 ) , Qwen3-4B ( Yang et al., 2025 ) , Llama-3.1-8B ( Grattafiori et al., 2024 ) , Qwen2.5-VL-7B ( Bai et al., 2025 ) , MiMO-VL-7B ( Xiaomi, 2025 ) – reveals that enhanced reasoning is not a monolithic process, but driven by an evolution of the learning frontiers .

The learning process exhibit two overlapping phases: it often begins with a rapid consolidation of procedural reliability, conducive to the widespread low-level tokens. This is followed by a sustained period where the greatest potential for improvement shifts to the exploration of high-level strategic reasoning, which serves as the true engine of advanced performance.

#### 2.2.1 Forging Reliable Low-level Skills

The initial phase of RL training is dedicated to mastering the basics. The model must first build a reliable engine for low-level skills, e.g., formatting, performing calculations and other procedural steps. To observe this, we track two key metrics on the execution tokens:

• Relative Perplexity: Perplexity, the exponentiated average negative log-likelihood, measures model surprise. A lower value signifies higher confidence. We normalize the perplexity by its initial value to compare the rates of change in planning tokens and execution tokens.

• Token-Level Entropy: The Shannon entropy of the policy’s next-token distribution, H ( π ( ⋅ | x < t ) ) H(\pi(\cdot|x_{<t})) , measures its uncertainty. High entropy signals active exploration over the vocabulary at the next-token, while low entropy suggests confident exploitation.

The evidence for this phase is shown in the first two columns of Figure 3 , marked with ①. The Relative Perplexity of execution tokens (grey curves) plummets in the early stages of training before flattening (column 1). This shows the model rapidly becomes confidently correct in its procedural steps. This is reinforced by the Token Entropy graph (column 2), where entropy for execution tokens is consistently and significantly lower than for planning tokens. The model is not just confident; it actively reduces exploration of procedural alternatives to converge on reliable operations. This rapid mastery of the basics is the first learning frontier to be solved.

Notably, we find that this phase of low-level skill consolidation might be absent or shot in models with stronger capacity, as evident in MiMO-VL-Instruct and Qwen-4B-Instruct. This also supports the argument that the primary driver of RL is indeed the exploration of strategic planning. We refer the reader to check the full analysis of training dynamics across eight models in the appendix.

#### 2.2.2 Steering the Skills with Strategic Planning

Once the model becomes procedurally reliable, its performance gains are primarily driven by its ability to explore and deploy a diverse set of high-level strategies. To track this shift, we analyze the planning tokens using two key metrics. We compute the Semantic Entropy of strategic grams – the Shannon Entropy of the frequency distribution of strategic grams – to quantify the diversity of the model’s high-level strategic plans (illustrated in Fig. 4 ). To isolate procedural variety, we compute the conditional entropy of subsequent procedural n-grams given a preceding strategic gram. This second metric shows how varied is the subsequent procedural steps for a preceding strategic move.

The third column of Figure 3 provides clear evidence of this strategic exploration phase. The semantic entropy of strategic grams (red line, marked with ②) shows a distinct and steady increase. This indicates that the model is not converging on a single optimal strategy but is instead actively expanding its repertoire of strategic plans. This observation is critical: mastery in reasoning, in this context, is achieved by developing a rich and varied strategic playbook, which contrasts sharply with the sharp decrease in token-level entropy seen during the initial procedural consolidation phase.