ESPO: Entropy Importance Sampling Policy Optimization — part 2/3

Cleaned-text code points [15220, 30964). Bibliography omitted here; source.md includes it.

In ESPO, we extend this signal to the fine-grained level. Tokens are grouped into separation { τ } \{\tau\} according to their entropy values, so that tokens with similar uncertainty belong to the same separation: τ k = { t ∣ e t i ∈ ℛ k } , { τ k } k = 1 K i = 𝒮 ⁡ ( y i ) , \displaystyle\tau_{k}=\{\,t\mid e_{t}^{i}\in\mathcal{R}_{k}\,\},\qquad\{\tau_{k}\}_{k=1}^{K_{i}}=\mathcal{S}(y_{i}), (7) where ℛ k \mathcal{R}_{k} denotes the entropy range corresponding to the k k -th group and 𝒮 \mathcal{S} represents the whole response sentence. In contrast to previous methods that apply fixed ranges, ESPO uses entropy statistics to dynamically compute each group’s clipping bound: ϵ τ = α log ⁡ | 𝒱 | ⋅ e t i | τ | , \displaystyle\epsilon_{\tau}=\frac{\alpha}{\log|\mathcal{V}|}\cdot\frac{e_{t}^{i}}{|\tau|}, (8)

where α \alpha is a global scaling factor and the denominator log ⁡ | 𝒱 | \log|\mathcal{V}| serves as the theoretical upper bound of the token entropy, normalizing the clipping coefficient across different vocabulary scales. Consequently, groups with higher average normalized entropy obtain larger clipping ranges, allowing more freedom for policy updates in uncertain regions, while low-entropy, confident groups are constrained by tighter clipping bounds to maintain stability.

This entropy-normalized adaptive design enables ESPO to dynamically balance exploration and stability, allocating larger optimization flexibility to uncertain regions while preserving GSPO’s stable training behavior across the entire sequence.

### 3.3 Entropy Grouping Importance Sampling

In the same way as GSPO-token decomposes the importance ratio to align optimization with the reward unit, we also decompose the ratio to allow token-wise advantage customization . Specifically, we define the entropy grouping importance ratio as follows: s τ i , t ​ ( θ ) := s ​ g ​ [ s τ i ​ ( θ ) ] ⋅ π θ ​ ( y i , t | x , y i , < t ) s ​ g ​ [ π θ old ​ ( y i , t | x , y i , < t ) ] , \displaystyle s_{\tau}^{i,t}(\theta):=sg\!\left[s_{\tau}^{i}(\theta)\right]\cdot\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{sg\!\left[\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})\right]}, (9) where s ​ g ​ [ ⋅ ] sg[\cdot] denotes the stop-gradient operator that blocks backpropagation through the detached component. The scaling term at the token grouping level s τ i ​ ( θ ) s_{\tau}^{i}(\theta) is defined as the length-normalized sequence ratio: s τ i ​ ( θ ) = ( π θ ​ ( y τ | x ) π θ old ​ ( y τ | x ) ) 1 | y τ | \displaystyle s_{\tau}^{i}(\theta)=\left(\frac{\pi_{\theta}(y_{\tau}|x)}{\pi_{\theta_{\text{old}}}(y_{\tau}|x)}\right)^{\!\tfrac{1}{|y_{\tau}|}} (10)

Intuitively, s τ i ​ ( θ ) s_{\tau}^{i}(\theta) captures global consistency throughout the group, while the token-level ratio π θ / π θ old \pi_{\theta}/\pi_{\theta_{\text{old}}} provides fine-grained local sensitivity. This hierarchical construction allows ESPO to adaptively modulate gradients within each group, preserving stable updates for globally consistent spans and emphasizing local corrections where necessary. As shown in Figure 2 , ESPO introduce entropy-based decomposition, enabling adaptive gradient modulation within each group.

### 3.4 Token Grouping Advantage Normalization

To address the long-standing issue of sparse reward signals and coarse credit assignment in reinforcement learning for large language models, many new works are emerging. For example, [ 5 ] , [ 26 ] , and [ 8 ] achieve fine-grained credit assignment by using the language model itself to perform a simple value function estimation, resulting in better performance than the baseline. This demonstrates the importance of fine-grained credit assignment in the online reinforcement learning (RL) training of reasoning models.

In this paper, We refine the advantage estimation at a finer granularity and compute token-wise normalized advantages within each group to better reflect local reward contributions. Formally, the token-level normalized advantage is defined as

A ^ τ i , t \displaystyle\hat{A}_{\tau}^{i,t} = 𝕀 [ i ∈ 𝒢 ] ⋅ R ⁡ ( y i , a ) − μ 𝒢 σ 𝒢 , \displaystyle=\mathbb{I}[i\in\mathcal{G}]\cdot\frac{R(y_{i},a)-\mu_{\mathcal{G}}}{\sigma_{\mathcal{G}}}, (11) where 𝒢 \displaystyle\text{where}\quad\mathcal{G} : = { i ∣ verifier succeeds } , \displaystyle:=\{\,i\mid\text{verifier succeeds}\,\}, μ 𝒢 \displaystyle\mu_{\mathcal{G}} : = mean ⁡ ( { R ⁡ ( y i , a ) } i ∈ 𝒢 ) , \displaystyle:=\operatorname{mean}\!\left(\{R(y_{i},a)\}_{i\in\mathcal{G}}\right), σ 𝒢 \displaystyle\sigma_{\mathcal{G}} : = std ⁡ ( { R ⁡ ( y i , a ) } i ∈ 𝒢 ) . \displaystyle:=\operatorname{std}\!\left(\{R(y_{i},a)\}_{i\in\mathcal{G}}\right).

Here R ⁡ ( y i , a ) R(y_{i},a) denotes the group-aware reward assigned by the verifier. Unlike GSPO, where the same advantage is propagated to all tokens in a sequence, ESPO allows each token (or group) to be weighted according to its relative contribution to the final reward. This design alleviates the sparsity of the reward signal by distributing gradients more precisely to informative regions, ensuring that learning is concentrated on tokens that are most responsible for success while suppressing noise from irrelevant or failed samples. As a result, ESPO enhances both the stability and efficiency of credit assignment across long and complex sequences.

## 4 Experiments

### 4.1 Experimental Setup

#### Baselines.

We compare ESPO with two classes of policy optimization baselines: (1) sequence-level methods (GSPO) and (2) token-level methods (DAPO, GMPO, CISPO). This covers the spectrum of granularity choices in contemporary LLM fine-tuning.

#### Configuration.

All experiments use the VERL framework [ 13 ] with vLLM [ 6 ] for rollouts. We train on SimpleRL [ 20 ] (8,192 prompts) for 3 epochs (200 steps) with a fixed learning rate of 1 × 10 − 6 1\times 10^{-6} . Each prompt generates G = 8 G=8 responses (temperature 1.0, max length 16,384) with batch size 128. Training runs on a 32-GPU H100 cluster.

#### Evaluation.

We report accuracy on MATH500 [ 7 ] and average@32 on AIME 2024/2025 [ 22 , 23 ] and HMMT [ 1 ] . All methods are evaluated on Qwen3 base models of varying scales and architectures: dense (1.7B, 4B, 14B) and MoE (30B-A3B), under identical initialization and hyperparameters.

### 4.2 Overall Performance Comparison

#### Comprehensive Benchmark Results.

Table 1 presents the performance of ESPO and baseline methods across four mathematical reasoning benchmarks and multiple model scales. ESPO achieves the highest average performance in all settings, demonstrating its consistent superiority.

On the smallest dense model (Qwen3-1.7B), ESPO matches or exceeds the best token-level baselines on AIME24 and MATH500 while providing a noticeable 3.1-point improvement on AIME25. This indicates that ESPO’s fine-grained yet stable optimization is particularly beneficial when model capacity is limited.

With medium-scale dense models (Qwen3-4B and 14B), ESPO maintains a narrow but consistent lead over strong token-level competitors like CISPO. While the absolute margins decrease as models become more capable, ESPO still attains the best average scores, suggesting it offers a robust trade-off between granularity and stability.

The advantage becomes most pronounced on the large MoE model (Qwen3-30B-A3B). Here, ESPO outperforms both sequence-level and token-level baselines by substantial margins—achieving up to 12.1-point gains over GSPO on AIME24 and 1.4-point gains over CISPO on AIME25. This highlights ESPO’s effectiveness in heterogeneous architectures where fixed-granularity methods struggle to adapt.

Across all model sizes and architectures, ESPO delivers balanced improvements without task-specific tuning, confirming its value as a general-purpose optimization strategy.

#### Training Dynamics.

Figure 3 compares the training behavior of ESPO with representative sequence-level (GSPO) and token-level (CISPO) methods on Qwen3-30B-A3B. Three key observations emerge:

First, ESPO converges at a comparable or faster rate than baselines in terms of training accuracy (Fig. 3 a), indicating more efficient optimization. Second, while baseline methods stabilize or only slowly increase their response lengths, ESPO sustains a steady growth in generated sequence length throughout training (Fig. 3 b), demonstrating continued exploration of longer reasoning trajectories. Third, ESPO demonstrates more stable entropy reduction than CISPO while starting from a more moderate initial value than both CISPO and GSPO (Fig. 3 c). Its smooth descent leads to a final policy that is marginally more confident than GSPO’s, indicating effective exploration without excessive randomness.

Collectively, these dynamics suggest that ESPO optimizes more efficiently (faster accuracy rise), explores more thoroughly (longer trajectories), and refines its policy more stably (smooth entropy decay) than fixed-granularity baselines.

### 4.3 Analysis of ESPO’s Mechanisms

#### Addressing Gradient Underutilization.

The performance advantages of ESPO stem from its ability to overcome the core limitations of fixed-granularity optimization. Figure 4 compares the clipping fractions for both upper and lower bounds across methods, providing insights into their update efficiency and stability.

Sequence-level methods exhibit the highest clipping fractions, indicating that a large portion of samples are truncated during updates. This occurs because a single sequence-level importance ratio is shared across all tokens; once this ratio exceeds the clipping threshold, the entire trajectory is discarded. Consequently, many informative gradients are suppressed, leading to the gradient underutilization problem highlighted in our introduction.

Token-level methods alleviate excessive truncation by using wider clipping ranges and finer-grained importance ratios. However, this relaxed constraint permits updates with highly variable token-wise ratios to pass through, introducing substantial noise into gradient estimates. Such instability is particularly problematic when importance ratios vary sharply across tokens, undermining training robustness.

ESPO maintains low clipping fractions without resorting to overly permissive ranges. By grouping tokens based on entropy and applying group-aware clipping, ESPO selectively constrains high-variance updates while preserving informative gradients. This approach enables more efficient use of training samples, avoiding both the wastefulness of sequence-level methods and the instability of token-level alternatives.

#### Validating Adaptive Optimization.

ESPO achieves this balanced behavior through its entropy-adaptive clipping mechanism. Figure 5 validates that this mechanism operates as designed, dynamically aligning optimization strength with model uncertainty.

During early training stages, the policy exhibits high predictive entropy (orange dashed line), reflecting broad uncertainty about token predictions. In this phase, ESPO assigns wider clipping ranges (lower clipping fractions), allowing greater exploration and policy deviation. As learning progresses, entropy steadily decreases as the model gains confidence, and ESPO automatically tightens its clipping constraints—indicated by the rising clipping fractions (blue and green lines). This adaptive behavior ensures that more samples fall within the trusted region once the policy stabilizes.

The inverse relationship between entropy and clipping fraction confirms that ESPO successfully modulates exploration intensity according to the model’s own uncertainty: high entropy encourages learning diversity, while low entropy enforces conservative updates. This principled adaptation explains how ESPO maintains stable optimization throughout training while still exploring effectively.

### 4.4 Ablation Studies

Having established that ESPO’s adaptive mechanism effectively addresses the limitations of fixed-granularity methods, we now validate the necessity of its individual design choices through ablation experiments. All ablations use the Qwen3-30B-A3B model under identical training configurations.

#### Core Components.

We first examine the contribution of ESPO’s two key innovations: adaptive clipping and entropy grouping importance sampling. Table 2 shows that removing either component significantly degrades performance.

Removing adaptive clipping —by replacing it with fixed clipping schemes—consistently harms results. We test two variants: (1) aggressive sequence-level clipping (0.0003, 0.0004) and (2) conservative token-level clipping (0.2, 0.28). Both cause comparable performance drops despite their different granularities, confirming that static clipping cannot adapt to evolving training dynamics and that ESPO’s adaptive mechanism is essential.

Removing entropy grouping causes the most severe degradation across all benchmarks. Without structured grouping based on entropy, clipping operates in an uncoordinated token-wise manner, significantly increasing gradient variance. This demonstrates that merely adjusting clipping ranges is insufficient; aligning clipping behavior with the model’s entropy structure is crucial for stable optimization.

The substantial performance gap between the full ESPO and these ablated versions validates that both components are necessary and work synergistically.

#### Grouping Proportion.

Given the importance of token grouping, we analyze the optimal proportion of high-entropy tokens to include. Figure 6 reveals a clear trade-off.

Selecting too few tokens (e.g., 10%) leads to weaker exploration. As observed in prior work [ 16 ] , overly aggressive filtering can remove certain useful tokens that still correspond to meaningful decision points, resulting in faster entropy decay and reduced diversity early in training. Consequently, the model may converge prematurely before sufficiently exploring the reasoning space.

Conversely, selecting too many tokens (e.g., 40%) introduces a substantial number of low-entropy tokens. These tokens are often highly certain and knowledge-driven, and contribute limited information for improving reasoning behavior. Including them in large proportions dilutes the influence of informative gradients and reduces exploration efficiency.

Across experiments, selecting approximately 20% of the highest-entropy tokens provides the optimal balance. This intermediate choice preserves enough informative decision points to sustain exploration while avoiding excessive inclusion of low-entropy tokens that can hinder optimization, validating our grouping strategy.

#### Hyperparameter Sensitivity.

Finally, we examine the sensitivity of ESPO to α \alpha , the global scaling factor controlling update strength (Table 3 ). Performance peaks at α = 0.02 \alpha=0.02 , with both smaller and larger values yielding suboptimal results.

When α \alpha is too small (0.01), clipping becomes overly aggressive, limiting exploration. When α \alpha is too large (0.05, 0.10), clipping becomes too permissive, reducing stability. This narrow optimal range underscores the importance of balanced exploration and confirms that our chosen α = 0.02 \alpha=0.02 represents an effective default setting.

## 5 Conclusion

We introduce Entropy Importance Sampling Policy Optimization (ESPO), an entropy-aware RL framework that groups tokens by uncertainty and clips adaptively. On challenging mathematical benchmarks, ESPO exhibits accelerated convergence, reduced clipping, and superior sample-efficiency versus strong baselines under equal compute. Future directions include alternative grouping criteria, learnable grouping mechanisms, and extensions to multi-turn dialogue and program synthesis.

## Ethical Statement