ESPO: Entropy Importance Sampling Policy Optimization — part 1/3
Cleaned-text code points [0, 15220). Bibliography omitted here; source.md includes it.
# ESPO: Entropy Importance Sampling Policy Optimization
###### Abstract
Reinforcement learning (RL) has become a central component of post-training for large language models (LLMs), particularly for complex reasoning tasks that require stable optimization over long generation horizons. However, achieving performance at scale often introduces a fundamental trade-off between training stability and training efficiency. Token-level optimization applies fine-grained updates at the individual units, but is prone to high variance in gradient estimation, which can result in unstable training dynamics. In contrast, Sequence-level optimization often relies on aggressive clipping mechanisms to ensure stable updates. However, such design may discard a large fraction of valid training samples, leading to inefficient gradient utilization and reduced training efficiency. We refer to this phenomenon as gradient underutilization . In this work, we propose E ntropy Importance S ampling P olicy O ptimization (ESPO), a novel framework that aims to combine fine-grained updates with stable training. ESPO decomposes sequences into groups based on predictive entropy, enabling (1) Entropy Grouping Importance Sampling to capture intra-sequence heterogeneity, and (2) Entropy Adaptive Clipping to dynamically allocate trust regions based on model uncertainty. Extensive experiments on mathematical reasoning benchmarks demonstrate that ESPO not only accelerates convergence but also achieves state-of-the-art performance, notably improving accuracy on the challenging mathematical benchmarks. 1 1 1 Code is available at: https://github.com/ShopeeLLM/ESPO
## 1 Introduction
Recent progress in LLM has increasingly emphasized their ability to perform complex reasoning, where correct outputs depend on multi-step logical inference rather than surface-level pattern matching. In such settings, post-training objectives extend beyond instruction following and require aligning the model’s generation process with verifiable reasoning correctness. Reinforcement learning has therefore become a key component of modern reasoning-oriented training pipelines. By allowing models to explore diverse reasoning trajectories and optimize towards task-level correctness, RL provides a natural framework for improving decision-making over long sequences.
However, current RL strategies face a fundamental trade-off induced by optimization granularity , which directly affects the balance between training stability and training efficiency. Early approaches like Proximal Policy Optimization (PPO) [ 9 ] operate at the token level, offering fine-grained control but suffering from high variance and the heavy computational burden of maintaining value networks. To address this, a series of variants have been proposed and can be broadly categorized into two classes: token-level (e.g., GRPO [ 12 ] , DAPO [ 19 ] , GMPO [ 24 ] , CISPO [ 2 ] , etc.) and sequence-level (e.g., GSPO [ 25 ] , etc.) optimization. In practice, token-level variants are prone to unstable training dynamics, while sequence-level approaches tend to exhibit low data utilization due to aggressive clipping.
To resolve these limitations, we introduce E ntropy Importance S ampling P olicy O ptimization (ESPO), a framework that restores fine-grained optimization without sacrificing the stability benefits of group-based methods. The core insight of ESPO is that policy entropy [ 10 ] serves as a natural indicator for optimization granularity . Previous work [ 16 ] proposes using entropy as the criterion for credit assignment by computing gradients only for the most uncertain tokens. This selective update improves efficiency by concentrating the learning on regions where the model exhibits greater uncertainty. Another line of work [ 15 ] categorizes tokens into distinct entropy types, and assigns each a fixed clipping range. These designs are motivated by the role of entropy, as illustrated in Figure 1 , high-entropy tokens correspond to pivotal decision points that steer the reasoning trajectory across alternative branches.
ESPO operates on token groups —dynamically grouped by their entropy profile. Specifically, it employs Entropy Grouping Importance Sampling to decompose the global sequence ratio into local group ratios, allowing gradients to flow freely in consistent segments while being carefully modulated in divergent ones. Furthermore, it introduces Entropy Adaptive Clipping , which assigns wider clipping bounds to high-entropy (uncertain) groups to encourage exploration, and tighter bounds to low-entropy (confident) groups to enforce stability.
The contributions of this work are summarized as follows:
• We identify limitations in existing methods with respect to training efficiency and training stability, and categorize these issues into token-level and sequence-level policy optimization.
• We propose E ntropy Importance S ampling P olicy O ptimization (ESPO), an entropy-based algorithm that control optimization granularity and improve data utilization.
• We provide detailed empirical analyses and ablation studies to examine the effects of core components, offering competitive performance on mathematical reasoning benchmarks.
## 2 Background
### 2.1 Related Works
Reinforcement learning has emerged as a central paradigm for optimizing LLMs, particularly in tasks requiring verifiable reasoning such as mathematics, coding, and question answering. Early RL approaches were largely based on Proximal Policy Optimization (PPO) [ 9 ] , where policy updates are guided by reward models or rule-based verifiers. To reduce the computational cost and instability of value-function training in PPO, Group Relative Policy Optimization (GRPO) [ 11 ] eliminates the critic and estimates advantages through group-wise relative ranking, achieving strong performance without relying on explicit value models. Building on GRPO, a series of recent methods have further improved stability, credit assignment, and rollout efficiency. DAPO [ 19 ] introduces clip-higher and dynamic sampling to counter entropy collapse and filter zero-advantage groups, while GSPO [ 25 ] elevates optimization granularity to the sequence level, aligning the update unit with reward signals. The geometric average of importance sampling ratios is used in GMPO [ 24 ] and employs a narrower clip range to reduce the optimization variance. GTPO [ 14 ] and GRPO-S leverages entropy-weighted rewards to better handle outliers and sparse credit. DCPO [ 18 ] dynamically adapts clipping ranges by token to enhance exploration and make better use of otherwise degenerate rollouts. CISPO [ 2 ] further refines clipping behavior by applying clipping only to the importance sampling ratios, while leaving gradient magnitudes unclipped. Complementary efforts focus on reward shaping, advantage normalization, and data-centric strategies: EMPO [ 21 ] incorporates semantic entropy, BNPO [ 17 ] normalizes rewards via Beta distributions, and large-scale datasets with curriculum learning, such as Open-Reasoner-Zero [ 4 ] , have proven crucial for high-quality signal alignment. Beyond this, SPO [ 3 ] incorporates segment-level optimization as a principled design to address the credit assignment challenge, thereby achieving more accurate reward attribution. Together, these developments reflect a broader trend toward efficient, stable, and reasoning-aligned RL methods for LLMs.
### 2.2 Preliminary
Token-level policy optimization performs policy updates at the granularity of individual tokens. Representative approaches include GRPO, DAPO, GMPO and CISPO, which differ in their data utilization method, importance sampling computation, clipping strategies. We introduce GRPO as a concrete example in the following.
GRPO [ 11 ] is a reinforcement learning algorithm that evaluates candidate responses purely through relative preferences within a group. For a given query q q , GRPO generates a set of responses { y i } i = 1 G \{y_{i}\}_{i=1}^{G} and assigns a sequence-level advantage A i A_{i} to each response. The policy is updated using a PPO-inspired objective applied at the token level: 𝒥 GRPO ( θ ) = 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | min ( r i , t ( θ ) A i , clip ( r i , t ( θ ) , 1 − ϵ low token , 1 + ϵ high token ) A i ) ] \mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\\ \min\Big(r_{i,t}(\theta)A_{i},\ \operatorname{clip}(r_{i,t}(\theta),1-\epsilon_{\text{low}}^{\text{token}},1+\epsilon_{\text{high}}^{\text{token}})A_{i}\Big)\Bigg] (1) where r i , t ( θ ) = π θ ( y i , t ∣ q , y i , < t ) π θ old ( y i , t ∣ q , y i , < t ) r_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}\mid q,y_{i,<t})} represents the token-level importance ratio.
Despite its simplicity, GRPO faces significant challenges. Token-level ratios are inherently noisy, and this noise compounds across long sequences, leading to unstable gradients [ 24 ] . Clipping can exacerbate this instability by unevenly scaling token contributions, sometimes resulting in divergence or training collapse. Furthermore, since each token ratio is estimated from a single sample, it provides only a weak approximation of the underlying policy distribution, misaligning the optimization with actual reward signals [ 25 ] . Lastly, token-focused credit assignment ignores global sequence patterns, making it difficult to guide coherent long-horizon generation.
Sequence-level policy optimization aims to better align the optimization objective with sequence-level importance sampling by treating entire trajectories as optimization units. GSPO [ 25 ] is a representative approach in this category.
Instead of treating tokens independently, GSPO computes an importance ratio at the sequence level: s i ( θ ) = ( π θ ( y i ∣ q ) π θ old ( y i ∣ q ) ) 1 | y i | s_{i}(\theta)=\Big(\frac{\pi_{\theta}(y_{i}\mid q)}{\pi_{\theta_{\text{old}}}(y_{i}\mid q)}\Big)^{\frac{1}{|y_{i}|}} (2) and projects it to individual tokens using a stop-gradient operation: s i , t ( θ ) = sg [ s i ( θ ) ] ⋅ π θ ( y i , t ∣ q , y i , < t ) sg [ π θ ( y i , t ∣ q , y i , < t ) ] s_{i,t}(\theta)=\text{sg}[s_{i}(\theta)]\cdot\frac{\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})}{\text{sg}[\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})]} (3)
The training objective combines sequence-level ratios with clipped advantages: 𝒥 GSPO ( θ ) = 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | min ( s i , t ( θ ) A i , clip ( s i , t ( θ ) , 1 − ϵ low seq , 1 + ϵ high seq ) A i ) ] \mathcal{J}_{\text{GSPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\\ \min\Big(s_{i,t}(\theta)A_{i},\,\operatorname{clip}\!\big(s_{i,t}(\theta),1-\epsilon_{\text{low}}^{\text{seq}},1+\epsilon_{\text{high}}^{\text{seq}}\big)A_{i}\Big)\Bigg] (4)
This sequence-level approach stabilizes training by reducing sensitivity to outlier tokens. However, all tokens within a sequence share the same ratio and advantage, making it difficult to attribute credit to specific positions. Additionally, conservative clipping thresholds, required to maintain stability, result in many samples being truncated and slow policy improvement. Relaxing the thresholds could improve learning but risks reintroducing the instability GSPO seeks to eliminate.
## 3 Entropy Importance Sampling Policy Optimization
ESPO is designed to improve data efficiency and training stability by structuring policy updates according to model uncertainty. At a high level, ESPO consists of two stages. First, ESPO performs Entropy Grouping Importance Sampling , where tokens within each sequence are grouped according to their entropy and the sequence-level importance ratio is decomposed into group-level ratios. This enables more fine-grained credit assignment by allowing updates to focus on high-uncertainty decision regions while maintaining coherence within low-uncertainty segments. Second, ESPO applies Entropy Adaptive Clipping , which assigns different clipping ranges to different entropy groups, encouraging exploration on uncertain tokens while enforcing stability on confident ones.
We first formulate the complete ESPO objective (Section 3.1), then elaborate on each component: entropy-adaptive clipping (Section 3.2), token grouping by entropy (Section 3.3), and token-level advantage normalization (Section 3.4).
### 3.1 Objective Formulation
Formally, given a query x ∼ 𝒟 x\sim\mathcal{D} and a group of G G sampled responses { y i } i = 1 G ∼ π θ old ( ⋅ | x ) \{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x) , we partition each sequence y i y_{i} into a collection of groups { τ } \{\tau\} , where each groups y τ y_{\tau} is not necessarily contiguous in the token space. Instead of relying on textual or syntactic boundaries, we group samples based on the model’s internal features that reflect uncertainty. Such uncertainty-related features can be defined at different orders, and ESPO specifically leverage second-order signals(e.g., policy entropy) for grouping. These features reflect the model’s internal uncertainty and confidence structure, allowing the groups to capture regions of similar policy behavior rather than continuous spans in surface form. ESPO defines the following objective:
𝒥 ESPO ( θ ) = 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | τ | ∑ τ 1 | y τ | ∑ t = 1 | y τ | min ( s τ i , t ( θ ) A ^ τ i , t , clip ( s τ i , t ( θ ) , 1 − ϵ τ , 1 + ϵ τ ) A ^ τ i , t ) ] \mathcal{J}_{\mathrm{ESPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\bigg[\frac{1}{G}\sum_{i=1}^{G}{\color[rgb]{1,0,0}\frac{1}{|\tau|}\sum_{\tau}\frac{1}{|y_{\tau}|}}\\ \sum_{t=1}^{|y_{\tau}|}\min\big(s_{\tau}^{i,t}(\theta)\hat{A}_{\tau}^{i,t},\,\operatorname{clip}(s_{\tau}^{i,t}(\theta),1-{\color[rgb]{1,0,0}\epsilon_{\tau}},1+{\color[rgb]{1,0,0}\epsilon_{\tau}})\hat{A}_{\tau}^{i,t}\big)\bigg] (5) where s τ i , t ( θ ) s_{\tau}^{i,t}(\theta) denotes the importance ratio at the token-level within groups τ \tau , ϵ τ \epsilon_{\tau} is the entropy adaptive clipping threshold, and A ^ i , t \hat{A}_{i,t} is the normalized token-level advantage.
### 3.2 Entropy Adaptive Clipping
Entropy provides a natural and model-intrinsic signal for measure token-level uncertainty. Following prior studies, we leverage predictive entropy to both token groups and determine their clipping range.
Specifically, we first compute the token-level entropy: e t i = − ∑ v ∈ 𝒱 π θ old ( v ∣ x , y i , < t ) log π θ old ( v ∣ x , y i , < t ) \displaystyle e_{t}^{i}=-\sum_{v\in\mathcal{V}}\pi_{\theta_{\text{old}}}(v\mid x,y_{i,<t})\log\pi_{\theta_{\text{old}}}(v\mid x,y_{i,<t}) (6) where 𝒱 \mathcal{V} denotes the vocabulary space. The entropy reflects how uncertain the model is about the next-token distribution, with higher values indicating less confidence.