Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning — part 3/4

Cleaned-text code points [30257, 46116). Bibliography omitted here; source.md includes it.

Figure 6 and Table 2 compare vanilla DAPO which uses all tokens, and our approach that retains only the top 20% high-entropy tokens in the policy gradient loss. Surprisingly, discarding the bottom 80% low-entropy tokens does not degrade reasoning performance and can even lead to improvements across six benchmarks. On the Qwen3-32B base model, this approach delivers gains of 7.71 points on AIME’24 and 11.04 points on AIME’25. Similarly, the Qwen3-14B base model shows improvements of 5.21 points on AIME’24 and 4.79 points on AIME’25. For the Qwen3-8B base model, performance remains unaffected. These findings suggest that the gains in reasoning ability during RLVR are driven primarily by high-entropy tokens, while low-entropy tokens may have little effect on or could even hinder reasoning performance, particularly on the Qwen3-32B and Qwen3-14B base models.

To conduct a deeper analysis, we vary the proportion, denoted as ρ \rho in Equation 6 , for experiments, as shown in Figure 7 (a). The results show that the performance of the Qwen3-8B base model remains relatively consistent across different proportions, such as 10%, 20%, and 50%. For the Qwen3-14B and Qwen3-32B base model, Figure 7 (c) and (e) reveal that reducing ρ \rho from 20% to 10% leads to a slight drop in performance, while increasing it sharply to 100% results in a notable decline. These observations indicate that within a reasonable range, reasoning performance is largely insensitive to the exact value of ρ \rho . More importantly, they suggest that focusing on high-entropy tokens, rather than using all tokens, generally preserves performance, and could even offer substantial gains in larger models.

#### Low-entropy tokens contribute minimally to reasoning performance.

As illustrated in Figure 7 (a) and (b), retaining only the bottom 80% of tokens with low entropy during RLVR leads to a substantial decline in performance, even though these tokens account for 80% of the total token count used in training. This finding indicates that low-entropy tokens contribute minimally to enhancing reasoning capabilities, highlighting the greater importance of high-entropy tokens for effective model training.

#### The effectiveness of high-entropy tokens may lie in their ability to enhance exploration.

Our analysis reveals that focusing on a subset of high-entropy tokens, approximately 20% in our experiments, strikes an effective balance between exploration and training stability in RLVR. As illustrated in Figure 7 (b), (d) and (f), adjusting the ratio ρ \rho from 20% to either 10%, 50%, or 100% leads to persistently lower overall entropy starting from the early training phase and continuing up to the point where performance begins to converge. Moreover, training with the bottom 80% of low-entropy tokens results in significantly reduced overall entropy. These findings indicate that retaining a certain proportion of high-entropy tokens may facilitate effective exploration. Tokens outside this range could be less helpful or possibly even detrimental to exploration, particularly during the critical phase before performance convergence. This might explain why, on the Qwen3-32B base model, DAPO using only the top 20% high-entropy tokens outperforms vanilla DAPO, as shown in Figure 6 (a). However, on the Qwen3-8B base model, probably due to the model’s lower capacity, the benefits of enhanced exploration appear limited.

#### Focusing on forking tokens in the policy gradient loss benefits larger reasoning models.

We present the scaling trend when utilizing only forking tokens in Figure 8 . On the AIME’24 and AIME’25 benchmarks, we observe that as the model size increases, the performance gain over vanilla DAPO becomes increasingly significant. This suggests a promising conclusion: focusing solely on forking tokens in the policy gradient loss could offer greater advantages in larger reasoning models.

### 5.4 Analysis

#### Generalization ability to other domains

As outlined in Section 5.2 , we used the DAPO-Math-17K dataset, which primarily consists of mathematical data, for our RLVR experiments. Here, we test whether DAPO, when trained on a math dataset and using only a small fraction of high-entropy tokens in the policy gradient loss, can still surpass vanilla DAPO on out-of-distribution test sets, such as LiveCodeBench ( Jain et al., 2024 ) . The results comparing DAPO with only top 10 % 10\% or 20 % 20\% tokens with highest entropy to vanilla DAPO (which uses 100 % 100\% tokens) on the Qwen3-32B base are illustrated in Figure 9 , using the same setup described in Section 5.2 . From these results, we observe that even when retaining only top 10 % 10\% or 20 % 20\% tokens with highest entropy, DAPO still significantly outperforms vanilla DAPO on the out-of-distribution test dataset LiveCodeBench. This finding suggests that high-entropy tokens may be associated with the generalization capabilities of reasoning models. Retaining only a small subset of tokens with the highest entropy could potentially enhance the generalization ability of reasoning models.

#### Unlocking more potential of RLVR with only forking tokens

In the experiments described in Section 5.3 , we set a maximum response length of 20480. As shown in Figure 10 , we increased the maximum response length for DAPO w/ only forking tokens (depicted in Figure 6 (a) and (b))—trained from the Qwen3-32B base model—to 29696. This adjustment resulted in an improvement of the already SoTA performance on AIME’24, increasing from 63.54 to 68.12. These findings suggest that the full potential of our approach may not yet be realized, and with a longer context length or potentially more challenging training data, even greater performance gains could be achieved.

#### Results on models other than Qwen

We compare DAPO using only forking tokens (i.e., the top 20% tokens with the highest entropy) against vanilla DAPO on models other than the Qwen series, specifically the Llama-3.1-8B model. When DAPO is applied to the Llama-3.1-8B base model, we observe that it achieves very low accuracy (approximately 1%) on the training dataset (i.e., DAPO-MATH-17K ( Yu et al., 2025 ) ) and often generates responses with repetitive words early in the RL training process. To address this, we use the Qwen3-32B model ( Yang et al., 2025 ) as a teacher to generate responses for DAPO-MATH-17K queries. From the generated queries, we randomly sample 10,000 with correct answers to serve as cold-start data and perform supervised fine-tuning (SFT) on the Llama-3.1-8B base model ( Grattafiori et al., 2024 ) . The remaining 7,398 queries are reserved for RL after the cold-start phase. The AIME’24 score and response length during RL training are plotted in Figure 11 . The results indicate that DAPO with only forking tokens still surpasses vanilla DAPO, while also producing longer responses on average. However, given the relatively low performance of both configurations on AIME’24, we believe the results on Llama-3.1-8B are less convincing compared to those observed on the Qwen3 models.

## 6 Discussions

#### Discussion 1: High-entropy minority tokens (i.e., forking tokens) could play a key role in explaining why RL generalizes while SFT memorizes.

Chu et al. (2025) demonstrated empirically that RL, particularly with outcome-based rewards, exhibits strong generalization to unseen, rule-based tasks, whereas supervised fine-tuning (SFT) is prone to memorizing training data and struggles with generalization outside the training distribution. We hypothesize that one critical factor underlying the differing generalization capabilities of RL and SFT may be related to entropy in forking tokens. Our experiments (e.g., Figure 5 and Figure 7 ) suggest that RL tends to preserve or even increase the entropy of forking tokens, maintaining the flexibility of reasoning paths. In contrast, SFT pushes output logits towards one-hot distributions, leading to reduced entropy in forking tokens and, consequently, a loss of reasoning path flexibility. This flexibility may be a crucial determinant of a reasoning model’s ability to generalize effectively to unseen tasks.

#### Discussion 2: Unlike traditional RL, LLM reasoning integrates prior knowledge and must produce readable output. Consequently, LLM CoTs contain a mix of low-entropy majority tokens and high-entropy minority tokens, whereas traditional RL can assume uniform action entropy throughout a trajectory.

As shown in Figure 2 (a), most LLM CoT tokens have low entropy, with only a small fraction exhibiting high entropy. In contrast, traditional RL typically formulates each action distribution as Gaussian with a predefined standard deviation ( Schulman et al., 2017 ; Raffin et al., 2021 ; Weng et al., 2022 ) , resulting in uniform entropy across actions. We attribute this distinct entropy pattern in LLM CoTs to their pretraining on large-scale prior knowledge and the need for language fluency. This forces most tokens to align with memorized linguistic structures, yielding low entropy. Only a small set of tokens that are inherently uncertain in the pretraining corpus allows for exploration, and thus exhibits high entropy. This deduction is consistent with our results in Table 1 .

#### Discussion 3: In RLVR, entropy bonus may be suboptimal, as it increases the entropy of low-entropy majority tokens. In contrast, clip-higher effectively promotes entropy in high-entropy minority tokens.

In RL, entropy bonus is commonly added to the training loss to encourage exploration by increasing the entropy of actions—a well-established practice in traditional tasks ( Schulman et al., 2017 ; Williams, 1992 ; Mnih et al., 2016 ) , and recently applied to LLM reasoning ( Sheng et al., 2024 ; Hu et al., 2024 ) . However, as discussed above, unlike typical RL trajectories, LLM CoTs display distinct entropy patterns. Increasing entropy across all tokens can degrade performance by disrupting the low-entropy majority, while selectively increasing the entropy of high-entropy minority tokens improves performance ( Figure 3 ). Thus, uniformly applied entropy bonuses are suboptimal for CoT reasoning. Instead, clip-higher ( Yu et al., 2025 ) , which moderately raises ϵ high \epsilon_{\text{high}} in Equation 4 , better targets high-entropy tokens. Empirically, we observe that tokens with high importance ratios r t ​ ( θ ) r_{t}(\theta) (as defined in Equation 2 ) tend to have higher entropy. By including more of these tokens in training, clip-higher increases overall entropy without significantly affecting low-entropy tokens, as supported by Yu et al. (2025) and illustrated in Figure 5 .

## 7 Related Work

#### Reinforcement learning for LLM

Before the advent of reasoning-capable models like OpenAI’s o1 ( OpenAI, 2024 ) , reinforcement learning (RL) was widely used in reinforcement learning from human feedback (RLHF) to improve large language models’ (LLMs) instruction-following and alignment with human preferences ( Ouyang et al., 2022 ) . RLHF methods are broadly categorized into online and offline preference optimization. Online methods, such as PPO ( Schulman et al., 2017 ) , GRPO ( Shao et al., 2024 ) , and REINFORCE ( Williams, 1992 ) , generate responses during training and receive real-time feedback. Offline methods like DPO ( Rafailov et al., 2023 ) , SimPO ( Meng et al., 2024 ) , and KTO ( Ethayarajh et al., 2024 ) optimize policies using pre-collected preferences, typically from human annotators or LLMs. While offline methods are more training-efficient, they often underperform compared to online approaches ( Tang et al., 2024 ) . Recently, RL with verifiable rewards (RLVR) ( Lambert et al., 2025 ) has emerged as a promising approach for enhancing reasoning in LLMs, particularly in domains like mathematics and programming ( Shao et al., 2024 ; DeepSeek-AI et al., 2025 ; Yang et al., 2025 ; Lambert et al., 2025 ) . OpenAI o1 ( OpenAI, 2024 ) was the first to show that RL can effectively incentivize reasoning at scale. Building on o1, models such as DeepSeek R1 ( DeepSeek-AI et al., 2025 ) , QwQ ( Team, 2025 ) , Kimi k1.5 ( Team et al., 2025 ) , and Qwen3 ( Yang et al., 2025 ) have aimed to match or exceed its performance. DeepSeek R1 stands out for showing that strong reasoning can emerge through outcome-based optimization using the online RL algorithm GRPO ( Shao et al., 2024 ) . It also introduced a “zero RL” paradigm, where reasoning abilities are elicited from the base model without conventional RL fine-tuning. Inspired by these results, subsequent methods such as DAPO ( Yu et al., 2025 ) , VAPO ( Yue et al., 2025b ) , SimpleRLZoo ( Zeng et al., 2025 ) , and Open-Reasoner-Zero ( Hu et al., 2025 ) have further explored RL-based reasoning. In this work, we use DAPO as our baseline to investigate key aspects of RL applied to LLMs.

#### Analysis on reinforcement learning with verifiable rewards

Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a prevalent approach to enhance the reasoning capabilities of large language models (LLMs). Several studies have analyzed the characteristics of RLVR and its related concepts. Gandhi et al. (2025) find that the presence of reasoning behaviors, rather than the correctness of answers, is the key factor driving performance improvements in reinforcement learning. Similarly, Li et al. (2025) show that the structure of long chains of thought (CoT) is critical to the learning process, while the content of individual reasoning steps has minimal impact. Vassoyan et al. (2025) identify "critical tokens" in CoTs, which are decision points where models are prone to errors, and propose encouraging exploration around these tokens by modifying the KL penalty. Lin et al. (2024) also identify critical tokens that significantly influence incorrect outcomes and demonstrate that identifying and replacing these tokens can alter model behavior. Our finding that RLVR primarily focuses on forking tokens in reasoning paths may share some common ground with the observations of Gandhi et al. (2025) and Li et al. (2025) , who suggest that RLVR primarily learns the format rather than the content. However, our analysis goes further by identifying the finding at the token level. Moreover, the concept of critical tokens in Vassoyan et al. (2025) and Lin et al. (2024) is closely related to the high-entropy minority tokens we introduce. In contrast to prior work, which judges token importance based on correctness of the output, we propose token entropy as a criterion that may more accurately reflect the underlying mechanisms of LLMs.

## 8 Conclusion

In this work, we analyze RLVR through a novel perspective of token entropy, providing fresh insights into the mechanisms of reasoning in LLMs. Our study of CoT reasoning shows that only a small subset of tokens exhibit high entropy and serve as forks in reasoning paths that influence reasoning directions. Additionally, our analysis of entropy dynamics during RLVR training reveals that the reasoning model largely retains the base model’s entropy patterns, with RLVR mainly modifying the entropy of already high-entropy tokens. Building on these findings, which underscore the significance of high-entropy minority tokens, we restrict policy gradient updates in RLVR to the top 20% highest-entropy tokens. This approach achieves performance comparable to, or even surpassing, full-token RLVR training, while exhibiting a strong scaling trend with model size. In contrast, directing optimization toward the low-entropy majority results in a significant decline in performance. These findings indicate that RLVR’s effectiveness stems primarily from optimizing this high-entropy subset, suggesting more focused and efficient strategies for improving LLM reasoning capabilities.

#### Limitations