Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

arxiv:2506.01939

Original source · Paper JSON

Post-training · 2025-06-02

Cites (6) · Cited by (11)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• Abstract
• 1 Introduction
• 2 Preliminaries
• 2.1 Token entropy calculation
• "Token entropy" corresponds to the token generation distribution, not a specific token.
• 2.2 RLVR Algorithms
• Proximal Policy Optimization (PPO)
• Group Relative Policy Optimization (GRPO)
• Dynamic sAmpling Policy Optimization (DAPO)
• 3 Analyzing Token Entropy in Chain-of-Thought Reasoning
• Entropy Pattern 1 in CoTs: Typically, only a minority of tokens are generated with high entropy, while a majority of tokens are outputted with low entropy.
• Entropy Pattern 2 in CoTs: Tokens with the highest entropy typically serve to bridge the logical connection between two consecutive parts of reasoning, while tokens with the lowest entropy tend to complete the current part of a sentence or finish constructing a word. Other tokens combine these two functions to varying degrees.
• High-entropy tokens as "forks" in chain-of-thoughts
• 4 RLVR Preserves and Reinforces Base Model Entropy Patterns
• RLVR primarily preserves the existing entropy patterns of the base models
• RLVR predominantly alters the entropy of high-entropy tokens, whereas the entropy of low-entropy tokens remains comparatively stable with minimal variations.
• 5 High-Entropy Minority Tokens Drive Effective RLVR
• 5.1 Formulation of RLVR Using Only Policy Gradients of the Highest-Entropy Tokens
• 5.2 Experimental Setup
• Training details
• Evaluation
• 5.3 Main Results
• High-entropy tokens drive reinforcement learning for LLM reasoning.
• Low-entropy tokens contribute minimally to reasoning performance.
• The effectiveness of high-entropy tokens may lie in their ability to enhance exploration.
• Focusing on forking tokens in the policy gradient loss benefits larger reasoning models.
• 5.4 Analysis
• Generalization ability to other domains
• Unlocking more potential of RLVR with only forking tokens
• Results on models other than Qwen
• 6 Discussions
• Discussion 1: High-entropy minority tokens (i.e., forking tokens) could play a key role in explaining why RL generalizes while SFT memorizes.
• Discussion 2: Unlike traditional RL, LLM reasoning integrates prior knowledge and must produce readable output. Consequently, LLM CoTs contain a mix of low-entropy majority tokens and high-entropy minority tokens, whereas traditional RL can assume uniform action entropy throughout a trajectory.
• Discussion 3: In RLVR, entropy bonus may be suboptimal, as it increases the entropy of low-entropy majority tokens. In contrast, clip-higher effectively promotes entropy in high-entropy minority tokens.
• 7 Related Work
• Reinforcement learning for LLM
• Analysis on reinforcement learning with verifiable rewards
• 8 Conclusion
• Limitations
• Future Directions