Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning — part 1/4
Cleaned-text code points [0, 14943). Bibliography omitted here; source.md includes it.
# Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
###### Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs), while its mechanisms are not yet well understood. In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence reasoning performance. By examining token entropy patterns in Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of tokens exhibit high entropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways. Furthermore, studying how entropy patterns evolve during RLVR training reveals that RLVR largely adheres to the base model’s entropy patterns, primarily adjusting the entropy of high-entropy tokens. These findings highlight the significance of high-entropy tokens (i.e., forking tokens) to RLVR. We ultimately improve RLVR by restricting policy gradient updates to forking tokens and uncover a finding even beyond the 80/20 rule: utilizing only 20% of the tokens while maintaining performance comparable to full-gradient updates on the Qwen3-8B base model and significantly surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME’25 and +7.71 on AIME’24) and Qwen3-14B (+4.79 on AIME’25 and +5.21 on AIME’24) base models, highlighting a strong scaling trend. In contrast, training exclusively on the 80% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that decide reasoning directions. Collectively, our results highlight the potential to understand RLVR through a token-entropy perspective and optimize RLVR by leveraging high-entropy minority tokens to further improve LLM reasoning.
Project Page: https://shenzhi-wang.github.io/high-entropy-minority-tokens-rlvr
## 1 Introduction
The reasoning capabilities of Large Language Models (LLMs) have advanced substantially in domains like mathematics and programming, propelled by test-time scaling methodologies employed in OpenAI o1 ( OpenAI, 2024 ) , Claude 3.7 ( Anthropic, 2025 ) , DeepSeek R1 ( DeepSeek-AI et al., 2025 ) , Kimi K1.5 ( Team et al., 2025 ) , and Qwen3 ( Yang et al., 2025 ) . A pivotal technique driving these improvements is Reinforcement Learning with Verifiable Rewards (RLVR) ( Lambert et al., 2025 ; DeepSeek-AI et al., 2025 ; Yang et al., 2025 ) , where models optimize outputs through RL objectives tied to automated correctness verification. While recent RLVR advancements have stemmed from algorithmic innovations ( Yu et al., 2025 ; Yue et al., 2025b ; Guan et al., 2025 ) , cross-domain applications ( Xue et al., 2025 ; Liu et al., 2025 ; Pan et al., 2025 ) , and counterintuitive empirical insights ( Wang et al., 2025 ; Yue et al., 2025a ; Zhao et al., 2025 ) , existing implementations directly train over all tokens with limited understanding of which tokens actually facilitate reasoning. These approaches neglect the heterogeneous functional roles tokens play in reasoning processes, potentially hindering further performance gains by failing to prioritize critical decision points in sequential reasoning trajectories.
In this paper, we analyze the underlying mechanisms of RLVR through an innovative lens of token entropy patterns, investigating how tokens with varying entropy impact reasoning performance. We first point out that in the Chain-of-Thought (CoT) processes of LLMs, the entropy distribution exhibits a distinct pattern where the majority of tokens are generated with low entropy, while a critical minority of tokens emerge with high entropy. Through comparing the textual meanings of these two parts of tokens, we observe that the tokens with lowest average entropy primarily complete the ongoing linguistic structures, while the tokens with highest average entropy function as pivotal decision points that determine the trajectory of reasoning among multiple potential pathways (referred to as forks ), as depicted in Figure 1 (a). In addition to qualitatively anslysis, we conduct controlled experiments by manually modulating the entropy of forking tokens during decoding. Quantitative results reveal that moderately increasing the entropy of these high-entropy forking tokens leads to measurable improvements in reasoning performance, while artificially reducing their entropy results in performance degradation, confirming the importance of maintaining high entropy and the role as "forks" for these high-entropy tokens. Furthermore, by analyzing the evolution of token entropy during RLVR training, we find that the reasoning model largely retains the entropy patterns of the base model, exhibiting only gradual and relatively minor changes as training progresses. Additionally, RLVR primarily changes the entropy of high-entropy tokens, while the entropy of low-entropy tokens varies only within a small range. The above observations highlight the critical role high-entropy minority tokens may play in CoTs and RLVR training.
Building upon the discovery of forking tokens, we further refine RLVR by retaining policy gradient updates for only 20% of tokens with the highest entropy and masking gradients of the remaining 80%. We observe that although solely utilizing 20% of tokens, our approach can still achieve competitive reasoning performance on Qwen3-8B base model compared to full-gradient updates. Moreover, its effectiveness increases with model size, yielding reasoning improvements of +11.04 on AIME’25 and +7.71 on AIME’24 for the Qwen3-32B base model, and +4.79 on AIME’25 and +5.21 on AIME’24 for the Qwen3-14B base model, as shown in Figure 1 (b). Notably, the 32B model trained with only 20% high-entropy tokens attains scores of 63.5 on AIME’24 and 56.7 on AIME’25, setting a new state-of-the-art (SoTA) for reasoning models trained directly from base models with fewer than 600B parameters. Extending the maximum response length from 20k to 29k further elevates our 32B model’s AIME’24 score from 63.5 to 68.1. Conversely, training exclusively on the 80% lowest-entropy tokens results in severe performance degradation. These observations show that 20% of tokens achieve performance comparable to or exceeding 100%, even surpassing the 80/20 rule. The results demonstrate that the high-entropy minority tokens, functioning as critical decision points in reasoning trajectories, account for nearly all performance gains in RLVR.
Finally, we explore why retaining a small fraction of the highest-entropy tokens leads to strong performance in RLVR via a series of ablation studies. We adjust the chosen fraction of forking tokens, either by decreasing it from 20% to 10% or increasing it to 50% or 100%, and report the corresponding reasoning metrics and overall entropy during the training process. Experimental results demonstrate that retaining approximately 20% of the highest-entropy tokens optimally balances exploration and performance, while deviating from 20% reduces the overall entropy with diminished exploration and incurs worse performance. This suggests that only a critical subset of high-entropy tokens meaningfully contributes to the exploration during RL while others may be neutral or even detrimental. Reducing the proportion to 10% removes certain useful tokens, which weakens exploration. Increasing the proportion to 50% or 100% adds low-entropy tokens, which also reduces the effectiveness of exploration. Last but not least, retaining the top 20% of high-entropy tokens results in the largest performance gains for the 32B model, followed by the 14B model, and the smallest gains for the 8B model. This may be due to the insufficient capacity of the smaller model, which restricts its ability to benefit from increased exploration. These findings highlight the importance of preserving an appropriate proportion of high-entropy tokens in RLVR. As model size increases, the strategy of selecting high-entropy tokens appears to scale effectively.
In summary, our findings emphasize the pivotal role of high-entropy minority tokens in shaping the reasoning abilities of LLMs. We hope this inspires further analyses from the perspective of token entropy and informs more effective RLVR algorithms that strategically leverage these tokens to enhance reasoning performance. The key takeaways of our paper are as follows: • In CoTs, the majority of tokens are generated with low entropy, while only a small subset exhibits high entropy. These high-entropy minority tokens often act as "forks" in the reasoning process, guiding the model toward diverse reasoning paths. Maintaining high entropy at these critical forking tokens is beneficial for reasoning performance. (§ 3 )
• During RLVR training, the reasoning model largely preserves the base model’s entropy patterns, showing only gradual and minor changes. RLVR primarily adjusts the entropy of high-entropy tokens, while the entropy of low-entropy tokens fluctuates only within a narrow range. (§ 4 )
• High-entropy minority tokens drive nearly all reasoning performance gains during RLVR, whereas low-entropy majority tokens contribute little or may even hinder performance. One possible explanation is that, prior to performance convergence, a subset ( ∼ 20 % \sim 20\% in our experiments) of high-entropy tokens facilitates exploration, while low-entropy tokens offer minimal benefit or may even impede it. (§ 5 )
• Based on the insights above, we further discuss (i) high-entropy minority tokens as a potential reason why supervised fine-tuning (SFT) memorizes but RL generalizes, (ii) how prior knowledge and readability requirements shape the different entropy patterns seen in LLM CoTs compared to traditional RL trajectories, and (iii) the advantage of clip-higher over entropy bonus for RLVR. (§ 6 )
## 2 Preliminaries
### 2.1 Token entropy calculation
The token-level generation entropy (referred to as token entropy for brevity) for token t t is defined as H t := − ∑ j = 1 V p t , j log p t , j , where ( p t , 1 , ⋯ , p t , V ) = 𝒑 t = π 𝜽 ( ⋅ ∣ 𝒒 , 𝒐 < t ) = Softmax ( 𝒛 t T ) . H_{t}:=-\sum_{j=1}^{V}p_{t,j}\log p_{t,j},\ \ \text{where}\ \ (p_{t,1},\cdots,p_{t,V})=\boldsymbol{p}_{t}=\pi_{\boldsymbol{\theta}}(\cdot\mid\boldsymbol{q},\boldsymbol{o}_{<t})=\text{Softmax}\left(\frac{\boldsymbol{z}_{t}}{T}\right). (1) Here, π 𝜽 \pi_{\boldsymbol{\theta}} denotes an LLM parameterized by 𝜽 \boldsymbol{\theta} , 𝒒 \boldsymbol{q} is the input query, and 𝒐 < t = ( o 1 , o 2 , ⋯ , o t − 1 ) \boldsymbol{o}_{<t}=(o_{1},o_{2},\cdots,o_{t-1}) represents the previously generated tokens. V V is the vocabulary size, 𝒛 t ∈ ℝ V \boldsymbol{z}_{t}\in\mathbb{R}^{V} denotes the pre-softmax logits at time step t t , 𝒑 t ∈ ℝ V \boldsymbol{p}_{t}\in\mathbb{R}^{V} is the corresponding probability distribution over the vocabulary, and T ∈ ℝ T\in\mathbb{R} is the decoding temperature. In off-policy settings, sequences are generated by a rollout policy π ϕ \pi_{\boldsymbol{\phi}} while the training policy is π 𝜽 \pi_{\boldsymbol{\theta}} , with ϕ ≠ 𝜽 \boldsymbol{\phi}\neq\boldsymbol{\theta} . The entropy is still calculated using π 𝜽 \pi_{\boldsymbol{\theta}} , as defined in Equation 1 , to measure the uncertainty of the training policy in the given sequence.
#### "Token entropy" corresponds to the token generation distribution, not a specific token.
Throughout our paper, we clarify that the token entropy H t H_{t} refers to the entropy at index t t , which is determined by the token generation distribution 𝒑 t \boldsymbol{p}_{t} rather than by any specific token o t o_{t} sampled from 𝒑 t \boldsymbol{p}_{t} . For brevity, when discussing the token o t o_{t} sampled from the distribution 𝒑 t \boldsymbol{p}_{t} , we describe its associated entropy as H t H_{t} and refer to H t H_{t} as the token entropy of o t o_{t} . However, if there exists another index t ′ ≠ t t^{\prime}\neq t such that o t ′ = o t o_{t^{\prime}}=o_{t} , the token entropy of o t ′ o_{t^{\prime}} is not necessarily equal to H t H_{t} .
### 2.2 RLVR Algorithms
#### Proximal Policy Optimization (PPO)
PPO ( Schulman et al., 2017 ) is a widely adopted policy gradient algorithm in RLVR. To stabilize training, PPO restricts policy updates to remain within a proximal region of the old policy π 𝜽 old \pi_{\boldsymbol{\theta}_{\text{old}}} using the following clipped surrogate to maximize the objective: J PPO ( θ ) = \displaystyle J_{\text{PPO}}(\theta)= 𝔼 ℬ ∼ 𝒟 , ( 𝒒 , 𝒂 ) ∼ ℬ , 𝒐 ∼ π θ old ( ⋅ ∣ 𝒒 ) [ min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) ] , \displaystyle\mathbb{E}_{\mathcal{B}\sim\mathcal{D},(\boldsymbol{q},\boldsymbol{a})\sim\mathcal{B},\boldsymbol{o}\sim\pi_{\theta_{\text{old}}}(\cdot\mid\boldsymbol{q})}\left[\min\left(r_{t}(\theta)\hat{A}_{t},\,\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right)\right], with r t ( 𝜽 ) = π 𝜽 ( o t | 𝒒 , 𝒐 < t ) π 𝜽 old ( o t | 𝒒 , 𝒐 < t ) . \displaystyle\text{with}\ \ r_{t}(\boldsymbol{\theta})=\frac{\pi_{\boldsymbol{\theta}}(o_{t}|\boldsymbol{q},\boldsymbol{o}_{<t})}{\pi_{\boldsymbol{\theta}_{\text{old}}}(o_{t}|\boldsymbol{q},\boldsymbol{o}_{<t})}. (2) Here, 𝒟 \mathcal{D} is a dataset of queries 𝒒 \boldsymbol{q} and corresponding ground-truth answers 𝒂 \boldsymbol{a} , ℬ \mathcal{B} is a batch sampled from 𝒟 \mathcal{D} , ϵ ∈ ℝ \epsilon\in\mathbb{R} is a hyperparameter typically set to 0.2 0.2 , and A ^ t \hat{A}_{t} is the estimated advantage computed using a value network.
#### Group Relative Policy Optimization (GRPO)
Building on the clipped objective in Equation 2 , GRPO ( Shao et al., 2024 ) discards the value network by estimating advantages using the average reward within a group of sampled responses. Specifically, for each query 𝒒 \boldsymbol{q} and its ground-truth answer 𝒂 \boldsymbol{a} , the rollout policy π 𝜽 old \pi_{\boldsymbol{\theta}{\text{old}}} generates a group of responses { 𝒐 i } i = 1 G \{\boldsymbol{o}^{i}\}_{i=1}^{G} with corresponding outcome rewards { R i } i = 1 G \{R^{i}\}_{i=1}^{G} , where G ∈ ℝ G\in\mathbb{R} is the group size. The estimated advantage A ^ t i \hat{A}^{i}_{t} is then computed as: A ^ t i = r i − mean ( { R i } i = 1 G ) std ( { R i } i = 1 G ) , where R i = { 1.0 if is_equivalent ( 𝒂 , 𝒐 i ) , 0.0 otherwise . \hat{A}^{i}_{t}=\frac{r^{i}-\text{mean}(\{R^{i}\}_{i=1}^{G})}{\text{std}(\{R^{i}\}_{i=1}^{G})},\quad\text{where}\ \ R^{i}=\begin{cases}1.0&\quad\text{if }\ \ \texttt{is\_equivalent}(\boldsymbol{a},\boldsymbol{o}^{i}),\\ 0.0&\quad\text{otherwise}.\end{cases} (3)
In addition to this modified advantage estimation, GRPO adds a KL penalty term to the clipped objective in Equation 2 .
#### Dynamic sAmpling Policy Optimization (DAPO)