Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning β€” part 2/4

Cleaned-text code points [14943, 30257). Bibliography omitted here; source.md includes it.

Building on GRPO, DAPO ( Yu et al., 2025 ) removes the KL penalty, introduces a clip-higher mechanism, incorporates dynamic sampling, applies a token-level policy gradient loss, and adopts overlong reward shaping, leading to the following maximization objective, where r t i ​ ( 𝜽 ) r_{t}^{i}(\boldsymbol{\theta}) is defined as in Equation 2 , and A ^ t i \hat{A}_{t}^{i} is computed as in Equation 3 : π’₯ DAPO ​ ( ΞΈ ) = \displaystyle\mathcal{J}_{\text{DAPO}}(\theta)= 𝔼 ℬ ∼ π’Ÿ , ( 𝒒 , 𝒂 ) ∼ ℬ , { 𝒐 i } i = 1 G ∼ Ο€ ΞΈ old ( β‹… ∣ 𝒒 ) [ 1 βˆ‘ i = 1 G | 𝒐 i | βˆ‘ i = 1 G βˆ‘ t = 1 | 𝒐 i | min ( r t i ( ΞΈ ) A ^ t i , \displaystyle\mathbb{E}_{\mathcal{B}\sim\mathcal{D},(\boldsymbol{q},\boldsymbol{a})\sim\mathcal{B},\{\boldsymbol{o}^{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid\boldsymbol{q})}\Bigg[\frac{1}{\sum_{i=1}^{G}|\boldsymbol{o}^{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}\min\Big(r_{t}^{i}(\theta)\hat{A}_{t}^{i},\, clip ( r t i ( ΞΈ ) , 1 βˆ’ Ο΅ low , 1 + Ο΅ high ) A ^ t i ) ] , s.t. 0 < | { 𝒐 i ∣ is_equivalent ( 𝒂 , 𝒐 i ) } | < G . \displaystyle\text{clip}\big(r_{t}^{i}(\theta),1-\epsilon_{\text{low}},1+\epsilon_{\text{high}}\big)\hat{A}_{t}^{i}\Big)\Bigg],\quad\text{s.t.}\quad 0<\left|\left\{\boldsymbol{o}^{i}\mid\texttt{is\_equivalent}(\boldsymbol{a},\boldsymbol{o}^{i})\right\}\right|<G. (4)

DAPO is one of the state-of-the-art RLVR algorithms without a value network. In this work, we use DAPO as the baseline for our RLVR experiments.

## 3 Analyzing Token Entropy in Chain-of-Thought Reasoning

Although prior works ( Yang et al., 2025 ; Yu et al., 2025 ; Yue et al., 2025b ) have highlighted the importance of generation entropy in chain-of-thought reasoning, they typically analyze the entropy of all tokens collectively. In this section, we take a closer look at generation entropy in chain-of-thought reasoning by examining it at the token level. To this end, we use Qwen3-8B ( Yang et al., 2025 ) , one of the most recent and capable reasoning models within a comparable parameter scale, to generate responses for queries from AIME’24 and AIME’25, using a decoding temperature of T = 1.0 T=1.0 . We enforce the use of the thinking mode for every question and collect over 10 6 10^{6} response tokens. For each token, the entropy is computed according to the formulation in Equation 1 . The statistical analysis of the entropy values of these 10 6 10^{6} tokens is presented in Figure 2 . Furthermore, a visualization of token entropy for an entire long CoT response is provided in Figures 12 to 17 in the Appendix. From these analyses, we identify the following entropy patterns:

#### Entropy Pattern 1 in CoTs: Typically, only a minority of tokens are generated with high entropy, while a majority of tokens are outputted with low entropy.

We can observe in Figure 2 (a) that the entropy of a large amount of tokens are quite small, and only a small amount of tokens have high entropy. Specifically, the entropy of over half the tokens (approximately 50.64%) is below 10 βˆ’ 2 10^{-2} , while only 20% of tokens have entropy greater than 0.672.

#### Entropy Pattern 2 in CoTs: Tokens with the highest entropy typically serve to bridge the logical connection between two consecutive parts of reasoning, while tokens with the lowest entropy tend to complete the current part of a sentence or finish constructing a word. Other tokens combine these two functions to varying degrees.

In Figure 2 (b) and (c), we select the 100 tokens generated with the highest average entropy and the lowest average entropy from a total of 10 6 10^{6} tokens, respectively. To mitigate the impact of noise on the average entropy, we only consider tokens with frequencies above 100. High-entropy tokens often act as logical connectors within and across sentences, such as "wait," "however," and "unless" (indicating contrasts or shifts), "thus" and "also" (showing progression or addition), or "since" and "because" (expressing causality). Similarly, tokens like "suppose," "assume," "given," and "define" frequently appear in mathematical derivations to introduce assumptions, known conditions, or definitions. Conversely, low-entropy tokens are often word suffixes, source code fragments, or mathematical expression components, all of which exhibit high determinism. Additionally, Figures 12 to 17 provide detailed visualization of token entropy in a long CoT, showing that most tokens outside the highest-entropy or lowest-entropy groups blend bridging and continuation functions to varying degrees.

#### High-entropy tokens as "forks" in chain-of-thoughts

Based on the two observed patterns above, we refer to high-entropy tokens as "forking tokens" , as they often lead to different potential branches with high uncertainty in the reasoning process. To further confirm the role of forking tokens in a quantitative way, using Qwen3-8B ( Yang et al., 2025 ) with a maximum response length of 28,672, we assign different decoding temperatures to the forking tokens and the other tokens in the evaluation on AIME 2024 and AIME 2025. Specifically, to analyze the effects of varying combinations of these temperatures on their behavior, we adjust probability distribution 𝒑 t β€² ∈ ℝ V \boldsymbol{p}^{\prime}_{t}\in\mathbb{R}^{V} for each token t t as follows: 𝒑 t β€² = Softmax ​ ( 𝒛 t T t β€² ) , where ​ T t β€² = { T high if ​ H t > h threshold , T low otherwise . \boldsymbol{p}^{\prime}_{t}=\text{Softmax}\left(\frac{\boldsymbol{z}_{t}}{T^{\prime}_{t}}\right),\qquad\text{where\quad}T^{\prime}_{t}=\begin{cases}T_{\text{high}}&\quad\text{if }H_{t}>h_{\text{threshold}},\\ T_{\text{low}}&\quad\text{otherwise}.\end{cases} (5)

Here, 𝒛 t ∈ ℝ V \boldsymbol{z}_{t}\in\mathbb{R}^{V} denotes the pre-softmax logits for token t t , and T t β€² ∈ ℝ T_{t}^{\prime}\in\mathbb{R} represents the adjusted temperature for token t t ; h threshold = 0.672 h_{\text{threshold}}=0.672 is the entropy threshold used to distinguish forking tokens from the other tokens, and is estimated by calculating the 80th percentile among the sampled 10 6 10^{6} tokens above; T high ∈ ℝ T_{\text{high}}\in\mathbb{R} and T low ∈ ℝ T_{\text{low}}\in\mathbb{R} correspond to the temperatures for forking tokens and the other tokens, respectively.

The effects of varying T high T_{\text{high}} and T low T_{\text{low}} are presented in Figure 3 . It can be seen that lowering T high T_{\text{high}} significantly degrades performance compared to lowering T low T_{\text{low}} . In contrast, increasing T high T_{\text{high}} results in substantially better performance than increasing T low T_{\text{low}} , which can even cause LLMs generating nonsensical outputs. These results suggest that forking tokens benefit from being assigned a relatively higher temperature compared to other tokens. Given that forking tokens naturally exhibit higher entropy than other tokens, this further supports the need for them to operate at an even higher entropy level. This observation aligns with their role as "forks," where high entropy enables them to branch into diverse reasoning directions.

## 4 RLVR Preserves and Reinforces Base Model Entropy Patterns

In this section, building on the observations of entropy patterns in CoTs discussed in Section 3 , we further investigate how these patterns evolve throughout RLVR training.

#### RLVR primarily preserves the existing entropy patterns of the base models

To analyze the evolution of entropy patterns during RLVR training, we apply DAPO ( Yu et al., 2025 ) to the Qwen3-14B base model (details in Section 5 ). Using the reasoning model after RLVR, we generate 16 responses per question across the six benchmarks in Table 2 . For each token in these responses, we compute logits using reasoning models from various RLVR stages and identify those in the top 20% entropy. We then calculate the overlap ratio (i.e., the fraction of shared top 20% high-entropy positions) between each intermediate model and both the base and final RLVR models. As shown in Table 1 , although overlap with the base model gradually decreases and overlap with the final RLVR model increases, the base model’s overlap still remains above 86% at convergence (step 1360), suggesting that RLVR largely retains the base model’s entropy patterns regarding which tokens exhibit high or low uncertainty.

#### RLVR predominantly alters the entropy of high-entropy tokens, whereas the entropy of low-entropy tokens remains comparatively stable with minimal variations.

Using the same setup as Table 1 , we compute the average entropy change after RLVR for each 5% entropy percentile range of the base model. It is observed that tokens with higher initial entropy in the base model tend to exhibit larger increases in entropy after RLVR. This observation could also further reinforce that RLVR primarily preserves the entropy patterns of the base model. Moreover, Figure 5 illustrates the evolution of entropy percentiles during RLVR training using the Qwen3-14B base model. The figure reveals that as we move from the 0th to the 100th percentile, the range of fluctuations during the whole RLVR training steadily diminishes. Thus, these observations suggest that throughout the whole training process, RLVR primarily adjusts the entropy of high-entropy tokens, while the entropy of low-entropy tokens exhibits minor variation and remains relatively stable.

## 5 High-Entropy Minority Tokens Drive Effective RLVR

Reinforcement learning with verifiable rewards (RLVR) has become one of the most widely used approaches for training reasoning models ( Yang et al., 2025 ; DeepSeek-AI et al., 2025 ; Yu et al., 2025 ; Yue et al., 2025b ) . However, there is a lack of research on which types of tokens contribute the most to the learning of reasoning models. As highlighted in Section 3 and Section 4 , high-entropy minority tokens are particularly important. In this section, we investigate the contribution of these high-entropy minority tokens, also referred to as forking tokens, on the development of reasoning capabilities during RLVR.

### 5.1 Formulation of RLVR Using Only Policy Gradients of the Highest-Entropy Tokens

Building on DAPO’s objective in Equation 4 , we discard the policy gradients of low-entropy tokens and train the model using only the policy gradients of high-entropy tokens. For each batch ℬ \mathcal{B} sampled from the dataset π’Ÿ \mathcal{D} , we calculate the maximum objective as: π’₯ HighEnt ℬ ( ΞΈ ) = 𝔼 ℬ ∼ π’Ÿ , ( 𝒒 , 𝒂 ) ∼ ℬ , { 𝒐 i } i = 1 G ∼ Ο€ ΞΈ old ( β‹… ∣ 𝒒 ) [ 1 βˆ‘ i = 1 G βˆ‘ t = 1 | 𝒐 i | 𝕀 [ H t i β‰₯ Ο„ ρ ℬ ] βˆ‘ i = 1 G βˆ‘ t = 1 | 𝒐 i | 𝕀 [ H t i β‰₯ Ο„ ρ ℬ ] \displaystyle\mathcal{J}_{\text{HighEnt}}^{\mathcal{B}}(\theta)=\mathbb{E}_{{\mathcal{B}\sim\mathcal{D},(\boldsymbol{q},\boldsymbol{a})\sim\mathcal{B}},\{\boldsymbol{o}^{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid\boldsymbol{q})}\Bigg[\frac{1}{{\color[rgb]{1,0,0}\sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}\mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right]}}\sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}{\color[rgb]{1,0,0}\mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right]} (6) β‹… min ( r t i ( ΞΈ ) A ^ t i , clip ( r t i ( ΞΈ ) , 1 βˆ’ Ο΅ low , 1 + Ο΅ high ) A ^ t i ) ] , s.t. 0 < | { 𝒐 i ∣ is_equivalent ( 𝒂 , 𝒐 i ) } | < G . \displaystyle\cdot\min\Big(r_{t}^{i}(\theta)\hat{A}_{t}^{i},\,\text{clip}\big(r_{t}^{i}(\theta),1-\epsilon_{\text{low}},1+\epsilon_{\text{high}}\big)\hat{A}_{t}^{i}\Big)\Bigg],\ \text{s.t.}\ 0<\left|\left\{\boldsymbol{o}^{i}\mid\texttt{is\_equivalent}(\boldsymbol{a},\boldsymbol{o}^{i})\right\}\right|<G. Here, H t i H_{t}^{i} denotes the entropy of token t t in response i i , 𝕀 ⁑ [ β‹… ] \mathbb{I}[\cdot] is the indicator function that evaluates to 1 1 if the condition inside holds and 0 0 otherwise, ρ ∈ ( 0 , 1 ] \rho\in(0,1] is a predefined ratio specifying the top proportion of high-entropy tokens to be selected within a batch, and Ο„ ρ ℬ \tau_{\rho}^{\mathcal{B}} is the corresponding entropy threshold within the batch ℬ \mathcal{B} such that only tokens with H t i β‰₯ Ο„ ρ ℬ H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}} , comprising the top- ρ \rho fraction of all tokens in the batch, are used to compute the gradient.

Comparing Equation 6 with Equation 4 , there are only two differences, as highlighted in red in Equation 6 : (i) The advantage term is multiplied by 𝕀 [ H t i β‰₯ Ο„ ρ ℬ ] \mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right] , ensuring that within each (micro-)batch ℬ \mathcal{B} , only tokens o t i o_{t}^{i} whose corresponding entropy H t i β‰₯ Ο„ ρ ℬ H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}} are involved in the policy gradient loss calculation; (ii) The normalization term for the number of tokens is adjusted to βˆ‘ i = 1 G βˆ‘ t = 1 | 𝒐 i | 𝕀 [ H t i β‰₯ Ο„ ρ ℬ ] \sum_{i=1}^{G}\sum_{t=1}^{|\boldsymbol{o}^{i}|}\mathbb{I}\left[H_{t}^{i}\geq\tau_{\rho}^{\mathcal{B}}\right] , meaning that only tokens whose entropy is at least the threshold Ο„ ρ ℬ \tau_{\rho}^{\mathcal{B}} are considered.

### 5.2 Experimental Setup

#### Training details

We adapt our training codebase from verl ( Sheng et al., 2024 ) and follow the training recipe of DAPO ( Yu et al., 2025 ) , one of the state-of-the-art RL algorithms for LLMs. Both configurations, RLVR with full gradients (vanilla DAPO depicted in Equation 4 ) and RLVR with only policy gradients on forking tokens (described in Equation 6 ), employ techniques such as clip-higher, dynamic sampling, token-level policy gradient loss, and overlong reward shaping ( Yu et al., 2025 ) . For fair comparisons, we apply the same hyperparameters as recommended by DAPO: for clip-higher, Ο΅ high = 0.28 , Ο΅ low = 0.2 \epsilon_{\text{high}}=0.28,\epsilon_{\text{low}}=0.2 ; for overlong reward shaping, the maximum response length is 20480 and the cache length is 4096. Furthermore, we use a training batch size of 512 and a mini-batch size of 32 in verl’s configuration, resulting in 16 gradient steps per training batch, with a learning rate of 10 βˆ’ 6 10^{-6} and no learning rate warmup or scheduling. Importantly, the training process excludes both KL divergence loss and entropy loss. To evaluate the scaling ability of these methods, we perform RLVR experiments across the Qwen3-32B base and Qwen3-8B base models, using DAPO-Math-17K ( Yu et al., 2025 ) as the training dataset. For main results, we set ρ = 20 % \rho=20\% in Equation 6 , meaning that the policy is updated using only the gradients of the top 20% highest-entropy tokens within each batch. The chat template we use for Qwen3 models is "User:\n[question]\nPlease reason step by step, and put your final answer within \boxed{}.\n\nAssistant:\n" with "<|endoftext|>" serving as the EOS token, where "[question]" should be replaced by the specific question.

#### Evaluation

We evaluate our models on 6 standard mathematical reasoning benchmarks commonly used for assessing reasoning capabilities: AIME’24, AIME’25, AMC’23, MATH500 ( Hendrycks et al., 2021 ) , Minerva, and OlympiadBench ( He et al., 2024 ) . All evaluations are conducted in a zero-shot setting. For each question, we generate 16 independent responses under a decoding temperature T = 1.0 T=1.0 , and report the average accuracy and the average number of tokens per response.

### 5.3 Main Results

#### High-entropy tokens drive reinforcement learning for LLM reasoning.