Soft Adaptive Policy Optimization — part 2/3
Cleaned-text code points [14124, 29869). Bibliography omitted here; source.md includes it.
The hyperparameter τ \tau controls the rate of attenuation: larger values produce faster decay. Although negative tokens are crucial for exploration and for preventing overfitting, they typically introduce greater instability than positive tokens. We justify this claim by analyzing how token-level gradients propagate through the logits. Let z = [ z 1 , z 2 , … , z | 𝒱 | ] z=[z_{1},z_{2},...,z_{|\mathcal{V}|}] denote the logits (with vocabulary size | 𝒱 | |\mathcal{V}| ), let v v denote a token, and compute output probabilities via a softmax operation, i.e., π θ ( v ∣ q , y i , < t ) = exp ( z v ) / ∑ v ′ exp ( z v ′ ) \pi_{\theta}(v\mid q,y_{i,<t})=\exp(z_{v})/\sum_{v^{\prime}}\exp(z_{v^{\prime}}) . We have ∂ log π θ ( y i , t ∣ q , y i , < t ) A ^ i , t ∂ z v \displaystyle\frac{\partial\log\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})\,\widehat{A}_{i,t}}{\partial z_{v}} = ∂ π θ ( y i , t ∣ q , y i , < t ) ∂ z v ⋅ A ^ i , t π θ ( y i , t ∣ q , y i , < t ) \displaystyle=\frac{\partial\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})}{\partial z_{v}}\cdot\frac{\widehat{A}_{i,t}}{\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})} (9) = 𝟙 ( v = y i , t ) exp ( z y i , t ) ∑ v ′ ∈ 𝒱 exp ( z v ′ ) − exp ( z y i , t ) exp ( z v ) ( ∑ v ′ ∈ 𝒱 exp ( z v ′ ) ) 2 ⋅ A ^ i , t π θ ( y i , t ∣ q , y i , < t ) \displaystyle=\frac{\mathbbm{1}(v=y_{i,t})\exp(z_{y_{i,t}})\sum_{v^{\prime}\in\mathcal{V}}\exp(z_{v^{\prime}})-\exp(z_{y_{i,t}})\exp(z_{v})}{\left(\sum_{v^{\prime}\in\mathcal{V}}\exp(z_{v^{\prime}})\right)^{2}}\cdot\frac{\widehat{A}_{i,t}}{\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})} = { ( 1 − π θ ( y i , t ∣ q , y i , < t ) ) ⋅ A ^ i , t if v = y i , t (sampled token) − π θ ( v ∣ q , y i , < t ) ⋅ A ^ i , t otherwise (unsampled token) \displaystyle=\begin{cases}\bigl(1-\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})\bigr)\cdot\widehat{A}_{i,t}&\text{if }v=y_{i,t}\quad\text{(sampled token)}\\ -\pi_{\theta}(v\mid q,y_{i,<t})\cdot\widehat{A}_{i,t}&\text{otherwise}\quad\text{(unsampled token)}\end{cases}
Positive advantages increase the sampled token’s logit and decrease all unsampled logits; negative advantages do the opposite, raising the logits of many unsampled tokens. In RL fine-tuning for LLMs, the action space is a large vocabulary (often hundreds of thousands of tokens), whereas the number of desirable actions in a given state is small. Consequently, negative gradients diffuse to numerous irrelevant tokens—providing some regularization but also inducing instability, especially in off-policy scenarios. Accordingly, we use distinct temperatures for positive and negative tokens and set τ neg > τ pos \tau_{\text{neg}}>\tau_{\text{pos}} so that gradients on negative tokens decay more rapidly, thereby improving training stability and performance.
## 4 A Gating‑Function Perspective on SAPO’s Connections to GRPO and GSPO
#### Unified surrogate
We consider a unified surrogate of the form 𝒥 ( θ ) = 𝔼 q ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | f i , t ( r i , t ( θ ) ) A ^ i , t ] , \mathcal{J}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}f_{i,t}\big(r_{i,t}(\theta)\big)\;\widehat{A}_{i,t}\right], (10) where f i , t ( ⋅ ) f_{i,t}(\cdot) is an algorithm-specific gating function. We further define the length-normalized sequence-level ratio as the geometric mean of token ratios: s i ( θ ) = ( π θ ( y i ∣ q ) π θ old ( y i ∣ q ) ) 1 | y i | = exp ( 1 | y i | ∑ t = 1 | y i | log r i , t ( θ ) ) , s i , t ( θ ) = sg [ s i ( θ ) ] ⋅ π θ ( y i , t | q , y i , < t ) sg [ π θ ( y i , t | q , y i , < t ) ] , \displaystyle s_{i}(\theta)=\left(\frac{\pi_{\theta}(y_{i}\mid q)}{\pi_{\theta_{\text{old}}}(y_{i}\mid q)}\right)^{\!\frac{1}{|y_{i}|}}=\exp\!\left(\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\log r_{i,t}(\theta)\right),\quad s_{i,t}(\theta)=\mathrm{sg}\left[s_{i}(\theta)\right]\cdot\frac{\pi_{\theta}(y_{i,t}|q,y_{i,<t})}{\mathrm{sg}\left[\pi_{\theta}(y_{i,t}|q,y_{i,<t})\right]}, (11) where sg [ ⋅ ] \mathrm{sg}[\cdot] denotes the stop gradient operation.
#### Algorithm-specific f i , t f_{i,t}
The algorithms differ in the choice of f i , t f_{i,t} : SAPO: f i , t SAPO ( r i , t ( θ ) ) = 4 τ i σ ( τ i ( r i , t ( θ ) − 1 ) ) , τ i = { τ pos , A ^ i > 0 , τ neg , A ^ i ≤ 0 , \displaystyle f_{i,t}^{\mathrm{SAPO}}(r_{i,t}(\theta))=\frac{4}{\tau_{i}}\;\sigma\!\big(\tau_{i}\,(r_{i,t}(\theta)-1)\big),\qquad\tau_{i}=\begin{cases}\tau_{\mathrm{pos}},&\widehat{A}_{i}>0,\\ \tau_{\mathrm{neg}},&\widehat{A}_{i}\leq 0,\end{cases} (12) GRPO: f i , t GRPO ( r i , t ( θ ) , A ^ i ) = { min ( r i , t ( θ ) , 1 + ε ) , A ^ i > 0 , max ( r i , t ( θ ) , 1 − ε ) , A ^ i ≤ 0 , \displaystyle f_{i,t}^{\mathrm{GRPO}}(r_{i,t}(\theta);\widehat{A}_{i})=\begin{cases}\min\!\big(r_{i,t}(\theta),\;1+\varepsilon\big),&\widehat{A}_{i}>0,\\ \max\!\big(r_{i,t}(\theta),\;1-\varepsilon\big),&\widehat{A}_{i}\leq 0,\end{cases} (13) GSPO: f i , t GSPO ( r i , t ( θ ) , A ^ i ) ≡ f i , t seq ( s i , t ( θ ) , A ^ i ) = { min ( s i , t ( θ ) , 1 + ε ) , A ^ i > 0 , max ( s i , t ( θ ) , 1 − ε ) , A ^ i ≤ 0 . \displaystyle f_{i,t}^{\mathrm{GSPO}}(r_{i,t}(\theta);\widehat{A}_{i})\equiv f_{i,t}^{\mathrm{seq}}\!\big(s_{i,t}(\theta);\widehat{A}_{i}\big)=\begin{cases}\min\!\big(s_{i,t}(\theta),\;1+\varepsilon\big),&\widehat{A}_{i}>0,\\ \max\!\big(s_{i,t}(\theta),\;1-\varepsilon\big),&\widehat{A}_{i}\leq 0.\end{cases} (14) Note that GSPO’s f i , t f_{i,t} is token-invariant within a sequence, while SAPO and GRPO are token-dependent.
#### Gradient form for SAPO/GRPO
Differentiating ( 10 ) and using ∇ θ r i , t ( θ ) = r i , t ( θ ) ∇ θ log π θ ( y i , t ∣ q , y i , < t ) \nabla_{\theta}r_{i,t}(\theta)=r_{i,t}(\theta)\nabla_{\theta}\log\pi_{\theta}(y_{i,t}\mid q,y_{i,<t}) , we obtain ∇ θ 𝒥 ( θ ) = 𝔼 [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | f i , t ′ ( r i , t ( θ ) ) r i , t ( θ ) ∇ θ log π θ ( y i , t ∣ q , y i , < t ) A ^ i ] . \nabla_{\theta}\mathcal{J}(\theta)=\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}f_{i,t}^{\prime}\big(r_{i,t}(\theta)\big)\;r_{i,t}(\theta)\;\nabla_{\theta}\log\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})\;\widehat{A}_{i}\right]. (15)
### 4.1 SAPO–GSPO Connection: Reduction to a Sequence-Level Soft Gate
We show that SAPO reduces to a GSPO-like sequence-level formulation under mild conditions, while retaining token-level adaptivity in heterogeneous sequences.
#### SAPO’s token-level soft gate
Using σ ( x ) ( 1 − σ ( x ) ) = 1 ( e x / 2 + e − x / 2 ) 2 = 1 4 sech 2 ( x / 2 ) \sigma(x)(1-\sigma(x))=\frac{1}{(e^{x/2}+e^{-x/2})^{2}}=\tfrac{1}{4}\mathrm{sech}^{2}(x/2) , we have f i , t SAPO ′ ( r i , t ( θ ) ) = 4 σ ( τ i ( r i , t ( θ ) − 1 ) ) ( 1 − σ ( τ i ( r i , t ( θ ) − 1 ) ) ) = sech 2 ( τ i 2 ( r i , t ( θ ) − 1 ) ) , f_{i,t}^{\mathrm{SAPO}\,^{\prime}}(r_{i,t}(\theta))=4\,\sigma\!\big(\tau_{i}\,(r_{i,t}(\theta)-1)\big)\Big(1-\sigma\!\big(\tau_{i}\,(r_{i,t}(\theta)-1)\big)\Big)=\mathrm{sech}^{2}\!\left(\frac{\tau_{i}}{2}\,(r_{i,t}(\theta)-1)\right), (16)
#### Assumptions
We invoke two common assumptions: (A1) Small-step/on-policy: r i , t ( θ ) ≈ 1 r_{i,t}(\theta)\approx 1 . Thus, log r i , t ( θ ) ≈ r i , t ( θ ) − 1 \log r_{i,t}(\theta)\approx r_{i,t}(\theta)-1 .
(A2) Low intra-sequence dispersion: letting z i , t ( θ ) := log r i , t ( θ ) z_{i,t}(\theta):=\log r_{i,t}(\theta) and μ i ( θ ) := 1 | y i | ∑ t z i , t ( θ ) = log s i ( θ ) \mu_{i}(\theta):=\tfrac{1}{|y_{i}|}\sum_{t}z_{i,t}(\theta)=\log s_{i}(\theta) , the variance Var i ( θ ) := 1 | y i | ∑ t ( z i , t ( θ ) − μ i ( θ ) ) 2 \mathrm{Var}_{i}(\theta):=\tfrac{1}{|y_{i}|}\sum_{t}(z_{i,t}(\theta)-\mu_{i}(\theta))^{2} is small for most sequences.
Under (A1), we have f i , t SAPO ′ ( r i , t ( θ ) ) = sech 2 ( τ i 2 ( r i , t ( θ ) − 1 ) ) ≈ sech 2 ( τ i 2 log r i , t ( θ ) ) = : g τ i ( z i , t ( θ ) ) . f_{i,t}^{\mathrm{SAPO}\,^{\prime}}(r_{i,t}(\theta))=\mathrm{sech}^{2}\!\left(\frac{\tau_{i}}{2}\,(r_{i,t}(\theta)-1)\right)\;\approx\;\mathrm{sech}^{2}\!\left(\frac{\tau_{i}}{2}\,\log r_{i,t}(\theta)\right)=:g_{\tau_{i}}\!\big(z_{i,t}(\theta)\big). (17)
#### Average token gates ⇒ \Rightarrow sequence gate
By a second-order Taylor expansion of the smooth function g τ ( z ) = sech 2 ( τ 2 z ) g_{\tau}(z)=\mathrm{sech}^{2}(\tfrac{\tau}{2}z) around μ i ( θ ) = log s i ( θ ) \mu_{i}(\theta)=\log s_{i}(\theta) , g τ i ( z i , t ( θ ) ) = g τ i ( μ i ( θ ) ) + g τ i ′ ( μ i ( θ ) ) ( z i , t ( θ ) − μ i ( θ ) ) + 1 2 g τ i ′′ ( ξ i , t ( θ ) ) ( z i , t ( θ ) − μ i ( θ ) ) 2 , g_{\tau_{i}}(z_{i,t}(\theta))=g_{\tau_{i}}(\mu_{i}(\theta))+g^{\prime}_{\tau_{i}}(\mu_{i}(\theta))\,(z_{i,t}(\theta)-\mu_{i}(\theta))+\frac{1}{2}\,g^{\prime\prime}_{\tau_{i}}(\xi_{i,t}(\theta))\,(z_{i,t}(\theta)-\mu_{i}(\theta))^{2}, (18) for some ξ i , t ( θ ) \xi_{i,t}(\theta) between z i , t ( θ ) z_{i,t}(\theta) and μ i ( θ ) \mu_{i}(\theta) . Averaging over tokens cancels the linear term: 1 | y i | ∑ t = 1 | y i | g τ i ( z i , t ( θ ) ) = g τ i ( μ i ( θ ) ) + 1 2 ( 1 | y i | ∑ t = 1 | y i | g τ i ′′ ( ξ i , t ( θ ) ) ( z i , t ( θ ) − μ i ( θ ) ) 2 ) . \frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}g_{\tau_{i}}(z_{i,t}(\theta))=g_{\tau_{i}}(\mu_{i}(\theta))+\frac{1}{2}\left(\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}g^{\prime\prime}_{\tau_{i}}(\xi_{i,t}(\theta))\,(z_{i,t}(\theta)-\mu_{i}(\theta))^{2}\right). (19) For g τ ( z ) = sech 2 ( α z ) g_{\tau}(z)=\mathrm{sech}^{2}(\alpha z) with α = τ 2 \alpha=\tfrac{\tau}{2} , a direct calculation gives g τ ′′ ( z ) = α 2 ( 4 sech 2 ( α z ) − 6 sech 4 ( α z ) ) , sup z | g τ ′′ ( z ) | = 2 α 2 = τ 2 2 . g^{\prime\prime}_{\tau}(z)=\alpha^{2}\Big(4\,\mathrm{sech}^{2}(\alpha z)-6\,\mathrm{sech}^{4}(\alpha z)\Big),\quad\sup_{z}|g^{\prime\prime}_{\tau}(z)|=2\alpha^{2}=\frac{\tau^{2}}{2}. (20) Hence, the average token gate is well-approximated by the sequence gate with a uniform bound: D i ( θ ) = | 1 | y i | ∑ t g τ i ( z i , t ( θ ) ) − g τ i ( μ i ( θ ) ) | ≤ 1 2 sup z | g τ i ′′ ( z ) | Var i ( θ ) = τ i 2 4 Var i ( θ ) . D_{i}(\theta)=\left|\frac{1}{|y_{i}|}\sum_{t}g_{\tau_{i}}(z_{i,t}(\theta))-g_{\tau_{i}}(\mu_{i}(\theta))\right|\leq\;\frac{1}{2}\,\sup_{z}|g_{\tau_{i}}^{\prime\prime}(z)|\;\mathrm{Var}_{i}(\theta)=\frac{\tau_{i}^{2}}{4}\;\mathrm{Var}_{i}(\theta). (21)
Starting from ( 15 ) and applying r i , t ( θ ) ≈ 1 r_{i,t}(\theta)\approx 1 (A1), we have ∇ θ 𝒥 SAPO ≈ 𝔼 [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | g τ i ( z i , t ( θ ) ) ∇ θ log π θ ( y i , t ∣ q , y i , < t ) A ^ i ] . \nabla_{\theta}\mathcal{J}_{\mathrm{SAPO}}\approx\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}g_{\tau_{i}}(z_{i,t}(\theta))\;\nabla_{\theta}\log\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})\;\widehat{A}_{i}\right]. (22) Using ( 21 ), we have ∇ θ 𝒥 SAPO \displaystyle\nabla_{\theta}\mathcal{J}_{\mathrm{SAPO}} ≈ 𝔼 [ 1 G ∑ i = 1 G g τ i ( log s i ( θ ) ) ( 1 | y i | ∑ t = 1 | y i | ∇ θ log π θ ( y i , t ∣ q , y i , < t ) ) A ^ i ] \displaystyle\approx\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}g_{\tau_{i}}\!\big(\log s_{i}(\theta)\big)\;\left(\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\nabla_{\theta}\log\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})\right)\;\widehat{A}_{i}\right] (23) = 𝔼 [ 1 G ∑ i = 1 G g τ i ( log s i ( θ ) ) ∇ θ log s i ( θ ) A ^ i ] . \displaystyle=\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}g_{\tau_{i}}\!\big(\log s_{i}(\theta)\big)\;\nabla_{\theta}\log s_{i}(\theta)\;\widehat{A}_{i}\right]. Thus, under (A1)–(A2), SAPO reduces to a sequence-level update structurally similar to GSPO with a smooth gate g τ i ( log s i ( θ ) ) = sech 2 ( τ i 2 log s i ( θ ) ) g_{\tau_{i}}(\log s_{i}(\theta))=\mathrm{sech}^{2}(\tfrac{\tau_{i}}{2}\log s_{i}(\theta)) .
#### Do the two assumptions (A1) and (A2) hold?
We empirically assess the small-step assumption (A1) and the low intra-sequence dispersion assumption (A2) by plotting the histograms of token ratios r i , t ( θ ) r_{i,t}(\theta) and per-sequence log-ratio variance Var i ( θ ) \mathrm{Var}_{i}(\theta) in Figures 2 and 3 for both MoE and dense models. The MoE model is a cold-start checkpoint of Qwen3-30B-A3B, and the dense model is a cold-start checkpoint of Qwen3-4B. The statistics are computed over more than 10 5 10^{5} sequences and 10 9 10^{9} tokens drawn from off-policy mini-batches. We observe that r i , t ( θ ) r_{i,t}(\theta) is sharply concentrated around 1 1 , and Var i ( θ ) \mathrm{Var}_{i}(\theta) typically remains below 0.02 0.02 , with a relatively wider distribution for the MoE model (likely reflecting heterogeneity induced by expert routing) and a tighter concentration for the dense model. These distributions indicate that (A1) and (A2) hold in the majority of cases, especially for dense architectures. Moreover, small D i ( θ ) D_{i}(\theta) directly implies that the average token gate is well-approximated by the sequence-level gate, supporting our reduction.
#### Advantages over GSPO
Compared with GSPO, SAPO has the following advantages: (1) Smoothness and stability. The soft gate varies continuously with the sequence deviation, avoiding the discontinuities of hard clipping and reducing optimization noise. (2) Token-level adaptivity with sequence-level coherence. Under (A1) and (A2), SAPO acts like a sequence-level method; when these conditions are violated (heterogeneous tokens or outliers), SAPO defaults to its token-level gating, selectively down-weighting outliers while preserving informative tokens—an ability GSPO lacks.
### 4.2 SAPO–GRPO Connection: Smooth Token Gates vs. Hard Token Clipping
#### GRPO’s piecewise-hard token gate
For GRPO, f i , t GRPO ( r i , t ( θ ) , A ^ i ) f_{i,t}^{\mathrm{GRPO}}(r_{i,t}(\theta);\widehat{A}_{i}) is piecewise constant with respect to the clipping band. Differentiating yields f i , t GRPO ′ ( r i , t ( θ ) , A ^ i ) = { 1 , A ^ i > 0 and r i , t ( θ ) ≤ 1 + ε , 0 , A ^ i > 0 and r i , t ( θ ) > 1 + ε , 1 , A ^ i ≤ 0 and r i , t ( θ ) ≥ 1 − ε , 0 , A ^ i ≤ 0 and r i , t ( θ ) < 1 − ε . f_{i,t}^{\mathrm{GRPO}\,^{\prime}}(r_{i,t}(\theta);\widehat{A}_{i})=\begin{cases}1,&\widehat{A}_{i}>0\text{ and }r_{i,t}(\theta)\leq 1+\varepsilon,\\ 0,&\widehat{A}_{i}>0\text{ and }r_{i,t}(\theta)>1+\varepsilon,\\ 1,&\widehat{A}_{i}\leq 0\text{ and }r_{i,t}(\theta)\geq 1-\varepsilon,\\ 0,&\widehat{A}_{i}\leq 0\text{ and }r_{i,t}(\theta)<1-\varepsilon.\end{cases} (24)
Hence, GRPO employs a binary trust region: tokens inside receive the same gradient as that of the unclipped objective; tokens outside receive zero gradient.
#### Advantages over GRPO
Compared with GRPO, SAPO replaces the hard indicator in ( 24 ) with the smooth kernel f i , t SAPO ′ ( r i , t ( θ ) ) = sech 2 ( τ i 2 ( r i , t ( θ ) − 1 ) ) f_{i,t}^{\mathrm{SAPO}\,^{\prime}}(r_{i,t}(\theta))=\mathrm{sech}^{2}(\tfrac{\tau_{i}}{2}(r_{i,t}(\theta)-1)) , which avoids gradient vanishing and enables more stable update dynamics. When the policy change is small, gradients remain responsive and permit larger parameter updates; as the deviation grows, gradients shrink smoothly, resulting in more conservative adjustments. In contrast, GRPO’s hard token clipping yields an all‑or‑nothing gate, often leading to brittle and unstable optimization behavior.
### 4.3 Summary