# Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

###### Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs), while its mechanisms are not yet well understood. In this work, we undertake a pioneering exploration of RLVR through the novel perspective of token entropy patterns, comprehensively analyzing how different tokens influence reasoning performance. By examining token entropy patterns in Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of tokens exhibit high entropy, and these tokens act as critical forks that steer the model toward diverse reasoning pathways. Furthermore, studying how entropy patterns evolve during RLVR training reveals that RLVR largely adheres to the base model’s entropy patterns, primarily adjusting the entropy of high-entropy tokens. These findings highlight the significance of high-entropy tokens (i.e., forking tokens) to RLVR. We ultimately improve RLVR by restricting policy gradient updates to forking tokens and uncover a finding even beyond the 80/20 rule: utilizing only 20% of the tokens while maintaining performance comparable to full-gradient updates on the Qwen3-8B base model and significantly surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME’25 and +7.71 on AIME’24) and Qwen3-14B (+4.79 on AIME’25 and +5.21 on AIME’24) base models, highlighting a strong scaling trend. In contrast, training exclusively on the 80% lowest-entropy tokens leads to a marked decline in performance. These findings indicate that the efficacy of RLVR primarily arises from optimizing the high-entropy tokens that decide reasoning directions. Collectively, our results highlight the potential to understand RLVR through a token-entropy perspective and optimize RLVR by leveraging high-entropy minority tokens to further improve LLM reasoning.

Project Page: https://shenzhi-wang.github.io/high-entropy-minority-tokens-rlvr

## 1 Introduction

The reasoning capabilities of Large Language Models (LLMs) have advanced substantially in domains like mathematics and programming, propelled by test-time scaling methodologies employed in OpenAI o1 ( OpenAI, 2024 ) , Claude 3.7 ( Anthropic, 2025 ) , DeepSeek R1 ( DeepSeek-AI et al., 2025 ) , Kimi K1.5 ( Team et al., 2025 ) , and Qwen3 ( Yang et al., 2025 ) . A pivotal technique driving these improvements is Reinforcement Learning with Verifiable Rewards (RLVR) ( Lambert et al., 2025 ; DeepSeek-AI et al., 2025 ; Yang et al., 2025 ) , where models optimize outputs through RL objectives tied to automated correctness verification. While recent RLVR advancements have stemmed from algorithmic innovations ( Yu et al., 2025 ; Yue et al., 2025b ; Guan et al., 2025 ) , cross-domain applications ( Xue et al., 2025 ; Liu et al., 2025 ; Pan et al., 2025 ) , and counterintuitive empirical insights ( Wang et al., 2025 ; Yue et al., 2025a ; Zhao et al., 2025 ) , existing implementations directly train over all tokens with limited understanding of which tokens actually facilitate reasoning. These approaches neglect the heterogeneous functional roles tokens play in reasoning processes, potentially hindering further performance gains by failing to prioritize critical decision points in sequential reasoning trajectories.

In this paper, we analyze the underlying mechanisms of RLVR through an innovative lens of token entropy patterns, investigating how tokens with varying entropy impact reasoning performance. We first point out that in the Chain-of-Thought (CoT) processes of LLMs, the entropy distribution exhibits a distinct pattern where the majority of tokens are generated with low entropy, while a critical minority of tokens emerge with high entropy. Through comparing the textual meanings of these two parts of tokens, we observe that the tokens with lowest average entropy primarily complete the ongoing linguistic structures, while the tokens with highest average entropy function as pivotal decision points that determine the trajectory of reasoning among multiple potential pathways (referred to as forks ), as depicted in Figure 1 (a). In addition to qualitatively anslysis, we conduct controlled experiments by manually modulating the entropy of forking tokens during decoding. Quantitative results reveal that moderately increasing the entropy of these high-entropy forking tokens leads to measurable improvements in reasoning performance, while artificially reducing their entropy results in performance degradation, confirming the importance of maintaining high entropy and the role as "forks" for these high-entropy tokens. Furthermore, by analyzing the evolution of token entropy during RLVR training, we find that the reasoning model largely retains the entropy patterns of the base model, exhibiting only gradual and relatively minor changes as training progresses. Additionally, RLVR primarily changes the entropy of high-entropy tokens, while the entropy of low-entropy tokens varies only within a small range. The above observations highlight the critical role high-entropy minority tokens may play in CoTs and RLVR training.

Building upon the discovery of forking tokens, we further refine RLVR by retaining policy gradient updates for only 20% of tokens with the highest entropy and masking gradients of the remaining 80%. We observe that although solely utilizing 20% of tokens, our approach can still achieve competitive reasoning performance on Qwen3-8B base model compared to full-gradient updates. Moreover, its effectiveness increases with model size, yielding reasoning improvements of +11.04 on AIME’25 and +7.71 on AIME’24 for the Qwen3-32B base model, and +4.79 on AIME’25 and +5.21 on AIME’24 for the Qwen3-14B base model, as shown in Figure 1 (b). Notably, the 32B model trained with only 20% high-entropy tokens attains scores of 63.5 on AIME’24 and 56.7 on AIME’25, setting a new state-of-the-art (SoTA) for reasoning models trained directly from base models with fewer than 600B parameters. Extending the maximum response length from 20k to 29k further elevates our 32B model’s AIME’24 score from 63.5 to 68.1. Conversely, training exclusively on the 80% lowest-entropy tokens results in severe performance degradation. These observations show that 20% of tokens achieve performance comparable to or exceeding 100%, even surpassing the 80/20 rule. The results demonstrate that the high-entropy minority tokens, functioning as critical decision points in reasoning trajectories, account for nearly all performance gains in RLVR.

Finally, we explore why retaining a small fraction of the highest-entropy tokens leads to strong performance in RLVR via a series of ablation studies. We adjust the chosen fraction of forking tokens, either by decreasing it from 20% to 10% or increasing it to 50% or 100%, and report the corresponding reasoning metrics and overall entropy during the training process. Experimental results demonstrate that retaining approximately 20% of the highest-entropy tokens optimally balances exploration and performance, while deviating from 20% reduces the overall entropy with diminished exploration and incurs worse performance. This suggests that only a critical subset of high-entropy tokens meaningfully contributes to the exploration during RL while others may be neutral or even detrimental. Reducing the proportion to 10% removes certain useful tokens, which weakens exploration. Increasing the proportion to 50% or 100% adds low-entropy tokens, which also reduces the effectiveness of exploration. Last but not least, retaining the top 20% of high-entropy tokens results in the largest performance gains for the 32B model, followed by the 14B model, and the smallest gains for the 8B model. This may be due to the insufficient capacity of the smaller model, which restricts its ability to benefit from increased exploration. These findings highlight the importance of preserving an appropriate proportion of high-entropy tokens in RLVR. As model size increases, the strategy of selecting high-entropy tokens appears to scale effectively.

In summary, our findings emphasize the pivotal role of high-entropy minority tokens in shaping the reasoning abilities of LLMs. We hope this inspires further analyses from the perspective of token entropy and informs more effective RLVR algorithms that strategically leverage these tokens to enhance reasoning performance. The key takeaways of our paper are as follows: • In CoTs, the majority of tokens are generated with low entropy, while only a small subset exhibits high entropy. These high-entropy minority tokens often act as "forks" in the reasoning process, guiding the model toward diverse reasoning paths. Maintaining high entropy at these critical forking tokens is beneficial for reasoning performance. (§ 3 )

• During RLVR training, the reasoning model largely preserves the base model’s entropy patterns, showing only gradual and minor changes. RLVR primarily adjusts the entropy of high-entropy tokens, while the entropy of low-entropy tokens fluctuates only within a narrow range. (§ 4 )

• High-entropy minority tokens drive nearly all reasoning performance gains during RLVR, whereas low-entropy majority tokens contribute little or may even hinder performance. One possible explanation is that, prior to performance convergence, a subset ( ∼ 20 % \sim 20\% in our experiments) of high-entropy tokens facilitates exploration, while low-entropy tokens offer minimal benefit or may even impede it. (§ 5 )

• Based on the insights above, we further discuss (i) high-entropy minority tokens as a potential reason why supervised fine-tuning (SFT) memorizes but RL generalizes, (ii) how prior knowledge and readability requirements shape the different entropy patterns seen in LLM CoTs compared to traditional RL trajectories, and (iii) the advantage of clip-higher over entropy bonus for RLVR. (§ 6 )

## 2 Preliminaries

### 2.1 Token entropy calculation

The token-level generation entropy (referred to as token entropy for brevity) for token t t is defined as H t := − ∑ j = 1 V p t , j log p t , j , where ( p t , 1 , ⋯ , p t , V ) = 𝒑 t = π 𝜽 ( ⋅ ∣ 𝒒 , 𝒐 < t ) = Softmax ( 𝒛 t T ) . H_{t}:=-\sum_{j=1}^{V}p_{t,j}\log p_{t,j},\ \ \text{where}\ \ (p_{t,1},\cdots,p_{t,V})=\boldsymbol{p}_{t}=\pi_{\boldsymbol{\theta}}(\cdot\mid\boldsymbol{q},\boldsymbol{o}_{<t})=\text{Softmax}\left(\frac{\boldsymbol{z}_{t}}{T}\right). (1) Here, π 𝜽 \pi_{\boldsymbol{\theta}} denotes an LLM parameterized by 𝜽 \boldsymbol{\theta} , 𝒒 \boldsymbol{q} is the input query, and 𝒐 < t = ( o 1 , o 2 , ⋯ , o t − 1 ) \boldsymbol{o}_{<t}=(o_{1},o_{2},\cdots,o_{t-1}) represents the previously generated tokens. V V is the vocabulary size, 𝒛 t ∈ ℝ V \boldsymbol{z}_{t}\in\mathbb{R}^{V} denotes the pre-softmax logits at time step t t , 𝒑 t ∈ ℝ V \boldsymbol{p}_{t}\in\mathbb{R}^{V} is the corresponding probability distribution over the vocabulary, and T ∈ ℝ T\in\mathbb{R} is the decoding temperature. In off-policy settings, sequences are generated by a rollout policy π ϕ \pi_{\boldsymbol{\phi}} while the training policy is π 𝜽 \pi_{\boldsymbol{\theta}} , with ϕ ≠ 𝜽 \boldsymbol{\phi}\neq\boldsymbol{\theta} . The entropy is still calculated using π 𝜽 \pi_{\boldsymbol{\theta}} , as defined in Equation 1 , to measure the uncertainty of the training policy in the given sequence.

#### "Token entropy" corresponds to the token generation distribution, not a specific token.

Throughout our paper, we clarify that the token entropy H t H_{t} refers to the entropy at index t t , which is determined by the token generation distribution 𝒑 t \boldsymbol{p}_{t} rather than by any specific token o t o_{t} sampled from 𝒑 t \boldsymbol{p}_{t} . For brevity, when discussing the token o t o_{t} sampled from the distribution 𝒑 t \boldsymbol{p}_{t} , we describe its associated entropy as H t H_{t} and refer to H t H_{t} as the token entropy of o t o_{t} . However, if there exists another index t ′ ≠ t t^{\prime}\neq t such that o t ′ = o t o_{t^{\prime}}=o_{t} , the token entropy of o t ′ o_{t^{\prime}} is not necessarily equal to H t H_{t} .

### 2.2 RLVR Algorithms

#### Proximal Policy Optimization (PPO)

PPO ( Schulman et al., 2017 ) is a widely adopted policy gradient algorithm in RLVR. To stabilize training, PPO restricts policy updates to remain within a proximal region of the old policy π 𝜽 old \pi_{\boldsymbol{\theta}_{\text{old}}} using the following clipped surrogate to maximize the objective: J PPO ​ ( θ ) = \displaystyle J_{\text{PPO}}(\theta)= 𝔼 ℬ ∼ 𝒟 , ( 𝒒 , 𝒂 ) ∼ ℬ , 𝒐 ∼ π θ old ( ⋅ ∣ 𝒒 ) [ min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) ] , \displaystyle\mathbb{E}_{\mathcal{B}\sim\mathcal{D},(\boldsymbol{q},\boldsymbol{a})\sim\mathcal{B},\boldsymbol{o}\sim\pi_{\theta_{\text{old}}}(\cdot\mid\boldsymbol{q})}\left[\min\left(r_{t}(\theta)\hat{A}_{t},\,\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right)\right], with r t ​ ( 𝜽 ) = π 𝜽 ​ ( o t | 𝒒 , 𝒐 < t ) π 𝜽 old ​ ( o t | 𝒒 , 𝒐 < t ) . \displaystyle\text{with}\ \ r_{t}(\boldsymbol{\theta})=\frac{\pi_{\boldsymbol{\theta}}(o_{t}|\boldsymbol{q},\boldsymbol{o}_{<t})}{\pi_{\boldsymbol{\theta}_{\text{old}}}(o_{t}|\boldsymbol{q},\boldsymbol{o}_{<t})}. (2) Here, 𝒟 \mathcal{D} is a dataset of queries 𝒒 \boldsymbol{q} and corresponding ground-truth answers 𝒂 \boldsymbol{a} , ℬ \mathcal{B} is a batch sampled from 𝒟 \mathcal{D} , ϵ ∈ ℝ \epsilon\in\mathbb{R} is a hyperparameter typically set to 0.2 0.2 , and A ^ t \hat{A}_{t} is the estimated advantage computed using a value network.

#### Group Relative Policy Optimization (GRPO)

Building on the clipped objective in Equation 2 , GRPO ( Shao et al., 2024 ) discards the value network by estimating advantages using the average reward within a group of sampled responses. Specifically, for each query 𝒒 \boldsymbol{q} and its ground-truth answer 𝒂 \boldsymbol{a} , the rollout policy π 𝜽 ​ old \pi_{\boldsymbol{\theta}{\text{old}}} generates a group of responses { 𝒐 i } i = 1 G \{\boldsymbol{o}^{i}\}_{i=1}^{G} with corresponding outcome rewards { R i } i = 1 G \{R^{i}\}_{i=1}^{G} , where G ∈ ℝ G\in\mathbb{R} is the group size. The estimated advantage A ^ t i \hat{A}^{i}_{t} is then computed as: A ^ t i = r i − mean ​ ( { R i } i = 1 G ) std ​ ( { R i } i = 1 G ) , where R i = { 1.0 if is_equivalent ​ ( 𝒂 , 𝒐 i ) , 0.0 otherwise . \hat{A}^{i}_{t}=\frac{r^{i}-\text{mean}(\{R^{i}\}_{i=1}^{G})}{\text{std}(\{R^{i}\}_{i=1}^{G})},\quad\text{where}\ \ R^{i}=\begin{cases}1.0&\quad\text{if }\ \ \texttt{is\_equivalent}(\boldsymbol{a},\boldsymbol{o}^{i}),\\ 0.0&\quad\text{otherwise}.\end{cases} (3)

In addition to this modified advantage estimation, GRPO adds a KL penalty term to the clipped objective in Equation 2 .

#### Dynamic sAmpling Policy Optimization (DAPO)

Building on GRPO, DAPO ( Yu et al., 2025 ) removes the KL penalty, introduces a clip-higher mechanism, incorporates dynamic sampling, applies a token-level policy gradient loss, and adopts overlong reward shaping, leading to the following maximization objective, where r t i ​ ( 𝜽 ) r_{t}^{i}(\boldsymbol{\theta}) is defined as in Equation 2 , and A ^ t i \hat{A}_{t}^{i} is computed as in Equation 3 : 𝒥 DAPO ​ ( θ ) = \displaystyle\mathcal{J}_{\text{DAPO}}(\theta)= 𝔼 ℬ ∼ 𝒟 , ( 𝒒 , 𝒂 ) ∼ ℬ , { 𝒐 i } i = 1 G ∼ π θ old ( ⋅ ∣ 𝒒 ) [ 1 ∑ i = 1 G | 𝒐 i | ∑ i = 1 G ∑ t = 1 | 𝒐 i | min ( r t i ( θ ) A ^ t i , \displaystyle\mathbb{E}_{\mathcal{B}\sim\mathcal{D},(\boldsymbol{q},\boldsymbol{a})\sim\mathcal{B},\{\boldsymbol{o}^{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid\boldsymbol{q})}\Bigg[\frac{1}{\sum_{i=1}^{G}|\boldsymbol{o}^{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}\min\Big(r_{t}^{i}(\theta)\hat{A}_{t}^{i},\, clip ( r t i ( θ ) , 1 − ϵ low , 1 + ϵ high ) A ^ t i ) ] , s.t. 0 < | { 𝒐 i ∣ is_equivalent ( 𝒂 , 𝒐 i ) } | < G . \displaystyle\text{clip}\big(r_{t}^{i}(\theta),1-\epsilon_{\text{low}},1+\epsilon_{\text{high}}\big)\hat{A}_{t}^{i}\Big)\Bigg],\quad\text{s.t.}\quad 0<\left|\left\{\boldsymbol{o}^{i}\mid\texttt{is\_equivalent}(\boldsymbol{a},\boldsymbol{o}^{i})\right\}\right|<G. (4)

DAPO is one of the state-of-the-art RLVR algorithms without a value network. In this work, we use DAPO as the baseline for our RLVR experiments.

## 3 Analyzing Token Entropy in Chain-of-Thought Reasoning

Although prior works ( Yang et al., 2025 ; Yu et al., 2025 ; Yue et al., 2025b ) have highlighted the importance of generation entropy in chain-of-thought reasoning, they typically analyze the entropy of all tokens collectively. In this section, we take a closer look at generation entropy in chain-of-thought reasoning by examining it at the token level. To this end, we use Qwen3-8B ( Yang et al., 2025 ) , one of the most recent and capable reasoning models within a comparable parameter scale, to generate responses for queries from AIME’24 and AIME’25, using a decoding temperature of T = 1.0 T=1.0 . We enforce the use of the thinking mode for every question and collect over 10 6 10^{6} response tokens. For each token, the entropy is computed according to the formulation in Equation 1 . The statistical analysis of the entropy values of these 10 6 10^{6} tokens is presented in Figure 2 . Furthermore, a visualization of token entropy for an entire long CoT response is provided in Figures 12 to 17 in the Appendix. From these analyses, we identify the following entropy patterns:

#### Entropy Pattern 1 in CoTs: Typically, only a minority of tokens are generated with high entropy, while a majority of tokens are outputted with low entropy.

We can observe in Figure 2 (a) that the entropy of a large amount of tokens are quite small, and only a small amount of tokens have high entropy. Specifically, the entropy of over half the tokens (approximately 50.64%) is below 10 − 2 10^{-2} , while only 20% of tokens have entropy greater than 0.672.

#### Entropy Pattern 2 in CoTs: Tokens with the highest entropy typically serve to bridge the logical connection between two consecutive parts of reasoning, while tokens with the lowest entropy tend to complete the current part of a sentence or finish constructing a word. Other tokens combine these two functions to varying degrees.

In Figure 2 (b) and (c), we select the 100 tokens generated with the highest average entropy and the lowest average entropy from a total of 10 6 10^{6} tokens, respectively. To mitigate the impact of noise on the average entropy, we only consider tokens with frequencies above 100. High-entropy tokens often act as logical connectors within and across sentences, such as "wait," "however," and "unless" (indicating contrasts or shifts), "thus" and "also" (showing progression or addition), or "since" and "because" (expressing causality). Similarly, tokens like "suppose," "assume," "given," and "define" frequently appear in mathematical derivations to introduce assumptions, known conditions, or definitions. Conversely, low-entropy tokens are often word suffixes, source code fragments, or mathematical expression components, all of which exhibit high determinism. Additionally, Figures 12 to 17 provide detailed visualization of token entropy in a long CoT, showing that most tokens outside the highest-entropy or lowest-entropy groups blend bridging and continuation functions to varying degrees.

#### High-entropy tokens as "forks" in chain-of-thoughts

Based on the two observed patterns above, we refer to high-entropy tokens as "forking tokens" , as they often lead to different potential branches with high uncertainty in the reasoning process. To further confirm the role of forking tokens in a quantitative way, using Qwen3-8B ( Yang et al., 2025 ) with a maximum response length of 28,672, we assign different decoding temperatures to the forking tokens and the other tokens in the evaluation on AIME 2024 and AIME 2025. Specifically, to analyze the effects of varying combinations of these temperatures on their behavior, we adjust probability distribution 𝒑 t ′ ∈ ℝ V \boldsymbol{p}^{\prime}_{t}\in\mathbb{R}^{V} for each token t t as follows: 𝒑 t ′ = Softmax ​ ( 𝒛 t T t ′ ) , where ​ T t ′ = { T high if ​ H t > h threshold , T low otherwise . \boldsymbol{p}^{\prime}_{t}=\text{Softmax}\left(\frac{\boldsymbol{z}_{t}}{T^{\prime}_{t}}\right),\qquad\text{where\quad}T^{\prime}_{t}=\begin{cases}T_{\text{high}}&\quad\text{if }H_{t}>h_{\text{threshold}},\\ T_{\text{low}}&\quad\text{otherwise}.\end{cases} (5)

Here, 𝒛 t ∈ ℝ V \boldsymbol{z}_{t}\in\mathbb{R}^{V} denotes the pre-softmax logits for token t t , and T t ′ ∈ ℝ T_{t}^{\prime}\in\mathbb{R} represents the adjusted temperature for token t t ; h threshold = 0.672 h_{\text{threshold}}=0.672 is the entropy threshold used to distinguish forking tokens from the other tokens, and is estimated by calculating the 80th percentile among the sampled 10 6 10^{6} tokens above; T high ∈ ℝ T_{\text{high}}\in\mathbb{R} and T low ∈ ℝ T_{\text{low}}\in\mathbb{R} correspond to the temperatures for forking tokens and the other tokens, respectively.

The effects of varying T high T_{\text{high}} and T low T_{\text{low}} are presented in Figure 3 . It can be seen that lowering T high T_{\text{high}} significantly degrades performance compared to lowering T low T_{\text{low}} . In contrast, increasing T high T_{\text{high}} results in substantially better performance than increasing T low T_{\text{low}} , which can even cause LLMs generating nonsensical outputs. These results suggest that forking tokens benefit from being assigned a relatively higher temperature compared to other tokens. Given that forking tokens naturally exhibit higher entropy than other tokens, this further supports the need for them to operate at an even higher entropy level. This observation aligns with their role as "forks," where high entropy enables them to branch into diverse reasoning directions.

## 4 RLVR Preserves and Reinforces Base Model Entropy Patterns

In this section, building on the observations of entropy patterns in CoTs discussed in Section 3 , we further investigate how these patterns evolve throughout RLVR training.

#### RLVR primarily preserves the existing entropy patterns of the base models

To analyze the evolution of entropy patterns during RLVR training, we apply DAPO ( Yu et al., 2025 ) to the Qwen3-14B base model (details in Section 5 ). Using the reasoning model after RLVR, we generate 16 responses per question across the six benchmarks in Table 2 . For each token in these responses, we compute logits using reasoning models from various RLVR stages and identify those in the top 20% entropy. We then calculate the overlap ratio (i.e., the fraction of shared top 20% high-entropy positions) between each intermediate model and both the base and final RLVR models. As shown in Table 1 , although overlap with the base model gradually decreases and overlap with the final RLVR model increases, the base model’s overlap still remains above 86% at convergence (step 1360), suggesting that RLVR largely retains the base model’s entropy patterns regarding which tokens exhibit high or low uncertainty.

#### RLVR predominantly alters the entropy of high-entropy tokens, whereas the entropy of low-entropy tokens remains comparatively stable with minimal variations.

Using the same setup as Table 1 , we compute the average entropy change after RLVR for each 5% entropy percentile range of the base model. It is observed that tokens with higher initial entropy in the base model tend to exhibit larger increases in entropy after RLVR. This observation could also further reinforce that RLVR primarily preserves the entropy patterns of the base model. Moreover, Figure 5 illustrates the evolution of entropy percentiles during RLVR training using the Qwen3-14B base model. The figure reveals that as we move from the 0th to the 100th percentile, the range of fluctuations during the whole RLVR training steadily diminishes. Thus, these observations suggest that throughout the whole training process, RLVR primarily adjusts the entropy of high-entropy tokens, while the entropy of low-entropy tokens exhibits minor variation and remains relatively stable.

## 5 High-Entropy Minority Tokens Drive Effective RLVR

Reinforcement learning with verifiable rewards (RLVR) has become one of the most widely used approaches for training reasoning models ( Yang et al., 2025 ; DeepSeek-AI et al., 2025 ; Yu et al., 2025 ; Yue et al., 2025b ) . However, there is a lack of research on which types of tokens contribute the most to the learning of reasoning models. As highlighted in Section 3 and Section 4 , high-entropy minority tokens are particularly important. In this section, we investigate the contribution of these high-entropy minority tokens, also referred to as forking tokens, on the development of reasoning capabilities during RLVR.

### 5.1 Formulation of RLVR Using Only Policy Gradients of the Highest-Entropy Tokens

Building on DAPO’s objective in Equation 4 , we discard the policy gradients of low-entropy tokens and train the model using only the policy gradients of high-entropy tokens. For each batch ℬ \mathcal{B} sampled from the dataset 𝒟 \mathcal{D} , we calculate the maximum objective as: 𝒥 HighEnt ℬ ( θ ) = 𝔼 ℬ ∼ 𝒟 , ( 𝒒 , 𝒂 ) ∼ ℬ , { 𝒐 i } i = 1 G ∼ π θ old ( ⋅ ∣ 𝒒 ) [ 1 ∑ i = 1 G ∑ t = 1 | 𝒐 i | 𝕀 [ H t i ≥ τ ρ ℬ ] ∑ i = 1 G ∑ t = 1 | 𝒐 i | 𝕀 [ H t i ≥ τ ρ ℬ ] \displaystyle\mathcal{J}_{\text{HighEnt}}^{\mathcal{B}}(\theta)=\mathbb{E}_{{\mathcal{B}\sim\mathcal{D},(\boldsymbol{q},\boldsymbol{a})\sim\mathcal{B}},\{\boldsymbol{o}^{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid\boldsymbol{q})}\Bigg[\frac{1}{{\color[rgb]{1,0,0}\sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}\mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right]}}\sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}{\color[rgb]{1,0,0}\mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right]} (6) ⋅ min ( r t i ( θ ) A ^ t i , clip ( r t i ( θ ) , 1 − ϵ low , 1 + ϵ high ) A ^ t i ) ] , s.t. 0 < | { 𝒐 i ∣ is_equivalent ( 𝒂 , 𝒐 i ) } | < G . \displaystyle\cdot\min\Big(r_{t}^{i}(\theta)\hat{A}_{t}^{i},\,\text{clip}\big(r_{t}^{i}(\theta),1-\epsilon_{\text{low}},1+\epsilon_{\text{high}}\big)\hat{A}_{t}^{i}\Big)\Bigg],\ \text{s.t.}\ 0<\left|\left\{\boldsymbol{o}^{i}\mid\texttt{is\_equivalent}(\boldsymbol{a},\boldsymbol{o}^{i})\right\}\right|<G. Here, H t i H_{t}^{i} denotes the entropy of token t t in response i i , 𝕀 ⁡ [ ⋅ ] \mathbb{I}[\cdot] is the indicator function that evaluates to 1 1 if the condition inside holds and 0 0 otherwise, ρ ∈ ( 0 , 1 ] \rho\in(0,1] is a predefined ratio specifying the top proportion of high-entropy tokens to be selected within a batch, and τ ρ ℬ \tau_{\rho}^{\mathcal{B}} is the corresponding entropy threshold within the batch ℬ \mathcal{B} such that only tokens with H t i ≥ τ ρ ℬ H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}} , comprising the top- ρ \rho fraction of all tokens in the batch, are used to compute the gradient.

Comparing Equation 6 with Equation 4 , there are only two differences, as highlighted in red in Equation 6 : (i) The advantage term is multiplied by 𝕀 [ H t i ≥ τ ρ ℬ ] \mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right] , ensuring that within each (micro-)batch ℬ \mathcal{B} , only tokens o t i o_{t}^{i} whose corresponding entropy H t i ≥ τ ρ ℬ H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}} are involved in the policy gradient loss calculation; (ii) The normalization term for the number of tokens is adjusted to ∑ i = 1 G ∑ t = 1 | 𝒐 i | 𝕀 [ H t i ≥ τ ρ ℬ ] \sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}\mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right] , meaning that only tokens whose entropy is at least the threshold τ ρ ℬ \tau_{\rho}^{\mathcal{B}} are considered.

### 5.2 Experimental Setup

#### Training details

We adapt our training codebase from verl ( Sheng et al., 2024 ) and follow the training recipe of DAPO ( Yu et al., 2025 ) , one of the state-of-the-art RL algorithms for LLMs. Both configurations, RLVR with full gradients (vanilla DAPO depicted in Equation 4 ) and RLVR with only policy gradients on forking tokens (described in Equation 6 ), employ techniques such as clip-higher, dynamic sampling, token-level policy gradient loss, and overlong reward shaping ( Yu et al., 2025 ) . For fair comparisons, we apply the same hyperparameters as recommended by DAPO: for clip-higher, ϵ high = 0.28 , ϵ low = 0.2 \epsilon_{\text{high}}=0.28,\epsilon_{\text{low}}=0.2 ; for overlong reward shaping, the maximum response length is 20480 and the cache length is 4096. Furthermore, we use a training batch size of 512 and a mini-batch size of 32 in verl’s configuration, resulting in 16 gradient steps per training batch, with a learning rate of 10 − 6 10^{-6} and no learning rate warmup or scheduling. Importantly, the training process excludes both KL divergence loss and entropy loss. To evaluate the scaling ability of these methods, we perform RLVR experiments across the Qwen3-32B base and Qwen3-8B base models, using DAPO-Math-17K ( Yu et al., 2025 ) as the training dataset. For main results, we set ρ = 20 % \rho=20\% in Equation 6 , meaning that the policy is updated using only the gradients of the top 20% highest-entropy tokens within each batch. The chat template we use for Qwen3 models is "User:\n[question]\nPlease reason step by step, and put your final answer within \boxed{}.\n\nAssistant:\n" with "<|endoftext|>" serving as the EOS token, where "[question]" should be replaced by the specific question.

#### Evaluation

We evaluate our models on 6 standard mathematical reasoning benchmarks commonly used for assessing reasoning capabilities: AIME’24, AIME’25, AMC’23, MATH500 ( Hendrycks et al., 2021 ) , Minerva, and OlympiadBench ( He et al., 2024 ) . All evaluations are conducted in a zero-shot setting. For each question, we generate 16 independent responses under a decoding temperature T = 1.0 T=1.0 , and report the average accuracy and the average number of tokens per response.

### 5.3 Main Results

#### High-entropy tokens drive reinforcement learning for LLM reasoning.

Figure 6 and Table 2 compare vanilla DAPO which uses all tokens, and our approach that retains only the top 20% high-entropy tokens in the policy gradient loss. Surprisingly, discarding the bottom 80% low-entropy tokens does not degrade reasoning performance and can even lead to improvements across six benchmarks. On the Qwen3-32B base model, this approach delivers gains of 7.71 points on AIME’24 and 11.04 points on AIME’25. Similarly, the Qwen3-14B base model shows improvements of 5.21 points on AIME’24 and 4.79 points on AIME’25. For the Qwen3-8B base model, performance remains unaffected. These findings suggest that the gains in reasoning ability during RLVR are driven primarily by high-entropy tokens, while low-entropy tokens may have little effect on or could even hinder reasoning performance, particularly on the Qwen3-32B and Qwen3-14B base models.

To conduct a deeper analysis, we vary the proportion, denoted as ρ \rho in Equation 6 , for experiments, as shown in Figure 7 (a). The results show that the performance of the Qwen3-8B base model remains relatively consistent across different proportions, such as 10%, 20%, and 50%. For the Qwen3-14B and Qwen3-32B base model, Figure 7 (c) and (e) reveal that reducing ρ \rho from 20% to 10% leads to a slight drop in performance, while increasing it sharply to 100% results in a notable decline. These observations indicate that within a reasonable range, reasoning performance is largely insensitive to the exact value of ρ \rho . More importantly, they suggest that focusing on high-entropy tokens, rather than using all tokens, generally preserves performance, and could even offer substantial gains in larger models.

#### Low-entropy tokens contribute minimally to reasoning performance.

As illustrated in Figure 7 (a) and (b), retaining only the bottom 80% of tokens with low entropy during RLVR leads to a substantial decline in performance, even though these tokens account for 80% of the total token count used in training. This finding indicates that low-entropy tokens contribute minimally to enhancing reasoning capabilities, highlighting the greater importance of high-entropy tokens for effective model training.

#### The effectiveness of high-entropy tokens may lie in their ability to enhance exploration.

Our analysis reveals that focusing on a subset of high-entropy tokens, approximately 20% in our experiments, strikes an effective balance between exploration and training stability in RLVR. As illustrated in Figure 7 (b), (d) and (f), adjusting the ratio ρ \rho from 20% to either 10%, 50%, or 100% leads to persistently lower overall entropy starting from the early training phase and continuing up to the point where performance begins to converge. Moreover, training with the bottom 80% of low-entropy tokens results in significantly reduced overall entropy. These findings indicate that retaining a certain proportion of high-entropy tokens may facilitate effective exploration. Tokens outside this range could be less helpful or possibly even detrimental to exploration, particularly during the critical phase before performance convergence. This might explain why, on the Qwen3-32B base model, DAPO using only the top 20% high-entropy tokens outperforms vanilla DAPO, as shown in Figure 6 (a). However, on the Qwen3-8B base model, probably due to the model’s lower capacity, the benefits of enhanced exploration appear limited.

#### Focusing on forking tokens in the policy gradient loss benefits larger reasoning models.

We present the scaling trend when utilizing only forking tokens in Figure 8 . On the AIME’24 and AIME’25 benchmarks, we observe that as the model size increases, the performance gain over vanilla DAPO becomes increasingly significant. This suggests a promising conclusion: focusing solely on forking tokens in the policy gradient loss could offer greater advantages in larger reasoning models.

### 5.4 Analysis

#### Generalization ability to other domains

As outlined in Section 5.2 , we used the DAPO-Math-17K dataset, which primarily consists of mathematical data, for our RLVR experiments. Here, we test whether DAPO, when trained on a math dataset and using only a small fraction of high-entropy tokens in the policy gradient loss, can still surpass vanilla DAPO on out-of-distribution test sets, such as LiveCodeBench ( Jain et al., 2024 ) . The results comparing DAPO with only top 10 % 10\% or 20 % 20\% tokens with highest entropy to vanilla DAPO (which uses 100 % 100\% tokens) on the Qwen3-32B base are illustrated in Figure 9 , using the same setup described in Section 5.2 . From these results, we observe that even when retaining only top 10 % 10\% or 20 % 20\% tokens with highest entropy, DAPO still significantly outperforms vanilla DAPO on the out-of-distribution test dataset LiveCodeBench. This finding suggests that high-entropy tokens may be associated with the generalization capabilities of reasoning models. Retaining only a small subset of tokens with the highest entropy could potentially enhance the generalization ability of reasoning models.

#### Unlocking more potential of RLVR with only forking tokens

In the experiments described in Section 5.3 , we set a maximum response length of 20480. As shown in Figure 10 , we increased the maximum response length for DAPO w/ only forking tokens (depicted in Figure 6 (a) and (b))—trained from the Qwen3-32B base model—to 29696. This adjustment resulted in an improvement of the already SoTA performance on AIME’24, increasing from 63.54 to 68.12. These findings suggest that the full potential of our approach may not yet be realized, and with a longer context length or potentially more challenging training data, even greater performance gains could be achieved.

#### Results on models other than Qwen

We compare DAPO using only forking tokens (i.e., the top 20% tokens with the highest entropy) against vanilla DAPO on models other than the Qwen series, specifically the Llama-3.1-8B model. When DAPO is applied to the Llama-3.1-8B base model, we observe that it achieves very low accuracy (approximately 1%) on the training dataset (i.e., DAPO-MATH-17K ( Yu et al., 2025 ) ) and often generates responses with repetitive words early in the RL training process. To address this, we use the Qwen3-32B model ( Yang et al., 2025 ) as a teacher to generate responses for DAPO-MATH-17K queries. From the generated queries, we randomly sample 10,000 with correct answers to serve as cold-start data and perform supervised fine-tuning (SFT) on the Llama-3.1-8B base model ( Grattafiori et al., 2024 ) . The remaining 7,398 queries are reserved for RL after the cold-start phase. The AIME’24 score and response length during RL training are plotted in Figure 11 . The results indicate that DAPO with only forking tokens still surpasses vanilla DAPO, while also producing longer responses on average. However, given the relatively low performance of both configurations on AIME’24, we believe the results on Llama-3.1-8B are less convincing compared to those observed on the Qwen3 models.

## 6 Discussions

#### Discussion 1: High-entropy minority tokens (i.e., forking tokens) could play a key role in explaining why RL generalizes while SFT memorizes.

Chu et al. (2025) demonstrated empirically that RL, particularly with outcome-based rewards, exhibits strong generalization to unseen, rule-based tasks, whereas supervised fine-tuning (SFT) is prone to memorizing training data and struggles with generalization outside the training distribution. We hypothesize that one critical factor underlying the differing generalization capabilities of RL and SFT may be related to entropy in forking tokens. Our experiments (e.g., Figure 5 and Figure 7 ) suggest that RL tends to preserve or even increase the entropy of forking tokens, maintaining the flexibility of reasoning paths. In contrast, SFT pushes output logits towards one-hot distributions, leading to reduced entropy in forking tokens and, consequently, a loss of reasoning path flexibility. This flexibility may be a crucial determinant of a reasoning model’s ability to generalize effectively to unseen tasks.

#### Discussion 2: Unlike traditional RL, LLM reasoning integrates prior knowledge and must produce readable output. Consequently, LLM CoTs contain a mix of low-entropy majority tokens and high-entropy minority tokens, whereas traditional RL can assume uniform action entropy throughout a trajectory.

As shown in Figure 2 (a), most LLM CoT tokens have low entropy, with only a small fraction exhibiting high entropy. In contrast, traditional RL typically formulates each action distribution as Gaussian with a predefined standard deviation ( Schulman et al., 2017 ; Raffin et al., 2021 ; Weng et al., 2022 ) , resulting in uniform entropy across actions. We attribute this distinct entropy pattern in LLM CoTs to their pretraining on large-scale prior knowledge and the need for language fluency. This forces most tokens to align with memorized linguistic structures, yielding low entropy. Only a small set of tokens that are inherently uncertain in the pretraining corpus allows for exploration, and thus exhibits high entropy. This deduction is consistent with our results in Table 1 .

#### Discussion 3: In RLVR, entropy bonus may be suboptimal, as it increases the entropy of low-entropy majority tokens. In contrast, clip-higher effectively promotes entropy in high-entropy minority tokens.

In RL, entropy bonus is commonly added to the training loss to encourage exploration by increasing the entropy of actions—a well-established practice in traditional tasks ( Schulman et al., 2017 ; Williams, 1992 ; Mnih et al., 2016 ) , and recently applied to LLM reasoning ( Sheng et al., 2024 ; Hu et al., 2024 ) . However, as discussed above, unlike typical RL trajectories, LLM CoTs display distinct entropy patterns. Increasing entropy across all tokens can degrade performance by disrupting the low-entropy majority, while selectively increasing the entropy of high-entropy minority tokens improves performance ( Figure 3 ). Thus, uniformly applied entropy bonuses are suboptimal for CoT reasoning. Instead, clip-higher ( Yu et al., 2025 ) , which moderately raises ϵ high \epsilon_{\text{high}} in Equation 4 , better targets high-entropy tokens. Empirically, we observe that tokens with high importance ratios r t ​ ( θ ) r_{t}(\theta) (as defined in Equation 2 ) tend to have higher entropy. By including more of these tokens in training, clip-higher increases overall entropy without significantly affecting low-entropy tokens, as supported by Yu et al. (2025) and illustrated in Figure 5 .

## 7 Related Work

#### Reinforcement learning for LLM

Before the advent of reasoning-capable models like OpenAI’s o1 ( OpenAI, 2024 ) , reinforcement learning (RL) was widely used in reinforcement learning from human feedback (RLHF) to improve large language models’ (LLMs) instruction-following and alignment with human preferences ( Ouyang et al., 2022 ) . RLHF methods are broadly categorized into online and offline preference optimization. Online methods, such as PPO ( Schulman et al., 2017 ) , GRPO ( Shao et al., 2024 ) , and REINFORCE ( Williams, 1992 ) , generate responses during training and receive real-time feedback. Offline methods like DPO ( Rafailov et al., 2023 ) , SimPO ( Meng et al., 2024 ) , and KTO ( Ethayarajh et al., 2024 ) optimize policies using pre-collected preferences, typically from human annotators or LLMs. While offline methods are more training-efficient, they often underperform compared to online approaches ( Tang et al., 2024 ) . Recently, RL with verifiable rewards (RLVR) ( Lambert et al., 2025 ) has emerged as a promising approach for enhancing reasoning in LLMs, particularly in domains like mathematics and programming ( Shao et al., 2024 ; DeepSeek-AI et al., 2025 ; Yang et al., 2025 ; Lambert et al., 2025 ) . OpenAI o1 ( OpenAI, 2024 ) was the first to show that RL can effectively incentivize reasoning at scale. Building on o1, models such as DeepSeek R1 ( DeepSeek-AI et al., 2025 ) , QwQ ( Team, 2025 ) , Kimi k1.5 ( Team et al., 2025 ) , and Qwen3 ( Yang et al., 2025 ) have aimed to match or exceed its performance. DeepSeek R1 stands out for showing that strong reasoning can emerge through outcome-based optimization using the online RL algorithm GRPO ( Shao et al., 2024 ) . It also introduced a “zero RL” paradigm, where reasoning abilities are elicited from the base model without conventional RL fine-tuning. Inspired by these results, subsequent methods such as DAPO ( Yu et al., 2025 ) , VAPO ( Yue et al., 2025b ) , SimpleRLZoo ( Zeng et al., 2025 ) , and Open-Reasoner-Zero ( Hu et al., 2025 ) have further explored RL-based reasoning. In this work, we use DAPO as our baseline to investigate key aspects of RL applied to LLMs.

#### Analysis on reinforcement learning with verifiable rewards

Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a prevalent approach to enhance the reasoning capabilities of large language models (LLMs). Several studies have analyzed the characteristics of RLVR and its related concepts. Gandhi et al. (2025) find that the presence of reasoning behaviors, rather than the correctness of answers, is the key factor driving performance improvements in reinforcement learning. Similarly, Li et al. (2025) show that the structure of long chains of thought (CoT) is critical to the learning process, while the content of individual reasoning steps has minimal impact. Vassoyan et al. (2025) identify "critical tokens" in CoTs, which are decision points where models are prone to errors, and propose encouraging exploration around these tokens by modifying the KL penalty. Lin et al. (2024) also identify critical tokens that significantly influence incorrect outcomes and demonstrate that identifying and replacing these tokens can alter model behavior. Our finding that RLVR primarily focuses on forking tokens in reasoning paths may share some common ground with the observations of Gandhi et al. (2025) and Li et al. (2025) , who suggest that RLVR primarily learns the format rather than the content. However, our analysis goes further by identifying the finding at the token level. Moreover, the concept of critical tokens in Vassoyan et al. (2025) and Lin et al. (2024) is closely related to the high-entropy minority tokens we introduce. In contrast to prior work, which judges token importance based on correctness of the output, we propose token entropy as a criterion that may more accurately reflect the underlying mechanisms of LLMs.

## 8 Conclusion

In this work, we analyze RLVR through a novel perspective of token entropy, providing fresh insights into the mechanisms of reasoning in LLMs. Our study of CoT reasoning shows that only a small subset of tokens exhibit high entropy and serve as forks in reasoning paths that influence reasoning directions. Additionally, our analysis of entropy dynamics during RLVR training reveals that the reasoning model largely retains the base model’s entropy patterns, with RLVR mainly modifying the entropy of already high-entropy tokens. Building on these findings, which underscore the significance of high-entropy minority tokens, we restrict policy gradient updates in RLVR to the top 20% highest-entropy tokens. This approach achieves performance comparable to, or even surpassing, full-token RLVR training, while exhibiting a strong scaling trend with model size. In contrast, directing optimization toward the low-entropy majority results in a significant decline in performance. These findings indicate that RLVR’s effectiveness stems primarily from optimizing this high-entropy subset, suggesting more focused and efficient strategies for improving LLM reasoning capabilities.

#### Limitations

We believe there is still room for improvement in our work. First, our experiments could be extended to models beyond the Qwen family. Although we attempted to evaluate our approach on LLaMA models, they struggled to achieve meaningful performance on the AIME benchmarks. Additionally, the scope of our dataset could be expanded to encompass domains beyond mathematics, such as programming or more complex tasks like ARC-AGI ( Chollet et al., 2025 ; Chollet, 2019 ) . Furthermore, our findings are based on specific experimental settings, and it is possible that the observations and conclusions presented in this paper may not generalize to all RLVR scenarios. For instance, in a different RLVR setting, the effective proportion of 20% observed in our experiments may need to be adjusted to a different value to achieve optimal results.

#### Future Directions

Future directions involve developing new RLVR algorithms to better leverage high-entropy minority tokens and exploring how these insights can enhance not only RLVR but also other approaches, such as supervised fine-tuning (SFT), distillation, inference, and multi-modal training.