Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing — part 1/3

Cleaned-text code points [0, 15705). Bibliography omitted here; source.md includes it.

# Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

###### Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address specific deviations. Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision that facilitates rapid early improvement, yet it frequently collapses during prolonged training. We trace this late-stage instability to two intrinsic flaws: self-distillation on already-correct samples introduces optimization ambiguity, and the self-teacher’s signal reliability progressively degrades. To resolve these issues, we propose Sample-Routed Policy Optimization (SRPO) , a unified on-policy framework that routes correct samples to GRPO’s reward-aligned reinforcement and failed samples to SDPO’s targeted logit-level correction. SRPO further incorporates an entropy-aware dynamic weighting mechanism to suppress high-entropy, unreliable distillation targets while emphasizing confident ones. Evaluated across five benchmarks and two model scales, SRPO achieves both the rapid early improvement of SDPO and the long-horizon stability of GRPO. It consistently surpasses the peak performance of both baselines, raising the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO, while simultaneously yielding moderate response lengths and lowering per-step compute cost by up to 17.2%.

## 1 Introduction

Post-training large language models through reinforcement learning with verifiable rewards (RLVR) has emerged as a standard approach for improving reasoning and problem-solving capabilities ( Jaech et al., 2024 ; Guo et al., 2025 ; Team et al., 2025 ; Yang et al., 2025 ) . Among RLVR methods, Group Relative Policy Optimization ( Shao et al., 2024 , GRPO;) is widely adopted for its simplicity and stability. GRPO estimates advantages by normalizing outcome rewards across a group of rollouts, producing a single scalar advantage that is applied uniformly to every token in a rollout. For successful rollouts, this uniform assignment is generally appropriate, as most intermediate steps support the correct outcome. Conversely, for failed rollouts, this coarse token credit assignment distributes a uniform penalty across the entire sequence. Consequently, the policy update lacks the focus needed to address specific deviations, which ultimately diminishes sample efficiency and slows convergence ( Khandoga et al., 2026 ; Kumar et al., 2026 ; Parthasarathi et al., 2025 ) .

To overcome this sparsity in credit assignment, recent work has turned to on-policy distillation ( Agarwal et al., 2024 ; Lu and Lab, 2025 ) and self-distillation ( Hübotter et al., 2026 ; Zhao et al., 2026 ; Ye et al., 2026 ; Song et al., 2026 ) , which provide dense logit-level guidance for more precise optimization. Self-distillation removes the need for an external teacher by conditioning the model on privileged context (e.g., the correct solution) to supervise its own generated trajectories. A prominent example, Self-Distillation Policy Optimization ( Hübotter et al., 2026 , SDPO;) , often achieves much faster early convergence in complex domains such as scientific reasoning and agentic tool use. However, as shown in Figure 1 (a), this early advantage is not sustained: under prolonged training, SDPO is consistently surpassed by GRPO and often suffers catastrophic collapse. While recent work Kim et al. (2026) attributes similar instability in math domains to the suppression of epistemic verbalization, we provide a complementary diagnosis from the perspective of the distillation signal and attribute this instability to two intrinsic causes within the self-distillation mechanism.

First, self-distillation on already-correct samples introduces optimization ambiguity. In SDPO, the self-teacher is conditioned on a successful sibling rollout to provide dense, logit-level targets. While this is effective for correcting failed samples, it can be counterproductive for already-correct ones: forcing a successful rollout to match a different successful sibling imposes arbitrary logit-level preferences between reward-equivalent reasoning paths. Figure 1 (b) supports this view: restricting SDPO updates to failed samples retains most of its benefit, whereas applying it only to correct samples degrades performance and accelerates collapse.

Second, the quality of the self-teacher’s distillation signal degrades as training progresses. As the gap between the self-teacher and student narrows during training ( Hübotter et al., 2026 ) , the distillation signal becomes less informative, while the self-teacher’s token-level entropy rises (Figure 1 (c)), indicating increasingly uncertain predictions. This degradation in informativeness and reliability contributes directly to the late-stage instability of SDPO.

These observations suggest that GRPO and SDPO have complementary optimization properties. For correct samples, the sequence-level credit assignment of GRPO is usually sufficient, and its Monte Carlo advantages robustly anchor the policy update toward expected reward maximization ( Zhang et al., 2024 ; Hübotter et al., 2026 ) . But for failed samples with localized reasoning errors, dense logit-level correction of SDPO is more effective and avoids the ambiguity above when restricted to failed trajectories. Based on this insight, we introduce Sample-Routed Policy Optimization (SRPO) , a unified on-policy framework that routes correct samples to a GRPO branch and failed samples with available teacher information to an SDPO branch. To mitigate late-stage signal degradation, we further equip the SDPO branch with an entropy-aware dynamic weighting mechanism that downweights uncertain distillation targets and emphasizes reliable corrections. This design enables rapid correction early in training while increasingly relying on reward-aligned reinforcement as more rollouts become correct, thereby stabilizing late-stage optimization.

Evaluated across five benchmarks following the protocol of Hübotter et al. (2026) and two Qwen3 model scales ( Yang et al., 2025 ) , SRPO consistently achieves the highest peak performance. Specifically, it raises the five-benchmark average on Qwen3-8B to 77.4% (+3.4 over GRPO, +6.3 over SDPO) and on Qwen3-4B to 74.2% (+4.5 over GRPO, +7.5 over SDPO). Furthermore, SRPO maintains a moderate response length, avoiding both the verbosity of GRPO and the excessive brevity of pure SDPO, a phenomenon recently linked to degraded epistemic reasoning ( Kim et al., 2026 ) . It also reduces per-step compute cost by up to 17.2% over long training horizons. Our contributions are threefold:

• We identify two intrinsic causes of late-stage instability in SDPO: self-distillation on already-correct samples introduces optimization ambiguity, and the quality of the self-teacher’s distillation signal progressively degrades.

• We propose SRPO, a unified framework that bridges group-relative and self-distillation policy optimization by routing each sample to the optimization signal best suited to its learning status, augmented by entropy-aware dynamic weighting to suppress unreliable distillation targets and emphasize reliable ones.

• We demonstrate across five benchmarks and two model scales that SRPO improves early training efficiency, long-horizon stability, and peak accuracy, while simultaneously yielding moderate response lengths and lower per-step compute time.

## 2 Preliminaries

We review the two optimization paradigms unified by SRPO. Throughout, let x x denote a prompt, { y i } i = 1 G \{y_{i}\}_{i=1}^{G} a group of G G on-policy rollouts sampled from the current policy π θ \pi_{\theta} , and { r i } i = 1 G \{r_{i}\}_{i=1}^{G} the corresponding scalar rewards.

### 2.1 Group Relative Policy Optimization

GRPO is a policy-gradient method for post-training with verifiable rewards that eliminates the need for a learned critic. For each prompt x x , the policy generates a group of G G rollouts and obtains a scalar reward for each. The advantage of rollout i i is estimated by normalizing its reward relative to the group:

A i GRPO = r i − r ¯ σ r + ϵ , A_{i}^{\mathrm{GRPO}}=\frac{r_{i}-\bar{r}}{\sigma_{r}+\epsilon},

where r ¯ \bar{r} and σ r \sigma_{r} are the mean and standard deviation of { r i } i = 1 G \{r_{i}\}_{i=1}^{G} . The policy is updated via a clipped surrogate objective:

ℒ GRPO ​ ( θ ) = 𝔼 ⁡ [ min ⁡ ( ρ i , t ​ ( θ ) ​ A i GRPO , clip ⁡ ( ρ i , t ​ ( θ ) , 1 − ε , 1 + ε ) ​ A i GRPO ) ] , \mathcal{L}_{\mathrm{GRPO}}(\theta)=\mathbb{E}\!\left[\min\!\left(\rho_{i,t}(\theta)\,A_{i}^{\mathrm{GRPO}},\;\operatorname{clip}(\rho_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\,A_{i}^{\mathrm{GRPO}}\right)\right],

where ρ i , t ​ ( θ ) = π θ ​ ( y i , t ∣ x , y i , < t ) / π θ old ​ ( y i , t ∣ x , y i , < t ) \rho_{i,t}(\theta)=\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})\,/\,\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid x,y_{i,<t}) is the importance-sampling ratio at token position t t of rollout i i . Because A i GRPO A_{i}^{\mathrm{GRPO}} is a sequence-level quantity assigned uniformly to every token in a rollout, GRPO delivers reward-aligned yet coarse-grained credit assignment: it reliably reinforces or suppresses entire rollouts, but cannot identify which individual tokens are responsible for the outcome.

### 2.2 Self-Distillation Policy Optimization

SDPO augments the reward signal with dense logit-level supervision derived from self-distillation. Rather than relying solely on scalar rewards, it constructs a feedback-conditioned self-teacher from the same model. The student distribution is π θ ( ⋅ ∣ x ) \pi_{\theta}(\cdot\mid x) , while the self-teacher distribution is π θ ( ⋅ ∣ x , f ) \pi_{\theta}(\cdot\mid x,f) , where f f denotes auxiliary information obtained during the rollout process (e.g., a successful sibling rollout from the same group or environment feedback such as execution traces).

Given a rollout y i y_{i} , SDPO trains the student to match the self-teacher’s distribution along the original trajectory by minimizing a logit-level divergence. Using KL divergence as an illustration:

ℒ SDPO ( θ ) = ∑ t KL ( π θ ( ⋅ ∣ x , y i , < t ) ∥ stopgrad [ π θ ( ⋅ ∣ x , f , y i , < t ) ] ) , \mathcal{L}_{\mathrm{SDPO}}(\theta)=\sum_{t}\mathrm{KL}\!\Big(\pi_{\theta}(\cdot\mid x,y_{i,<t})\;\Big\|\;\operatorname{stopgrad}\big[\pi_{\theta}(\cdot\mid x,f,y_{i,<t})\big]\Big),

where the specific divergence may also be instantiated as the reverse KL or Jensen–Shannon divergence, and the self-teacher parameters are maintained as an exponential moving average (EMA) of the student ( Hübotter et al., 2026 ) .

The self-teacher does not generate a new trajectory; it re-scores the student’s own rollout under the enriched context ( x , f ) (x,f) , so the entire procedure remains on-policy while providing dense logit-level guidance on the model’s own rollouts.

The two methods differ fundamentally in their supervision signals. GRPO is reward-driven: its advantage is derived from outcome rewards via group normalization, producing updates that are directly aligned with expected return but uniformly distributed across tokens. SDPO is teacher-driven: its advantage is induced by the discrepancy between the self-teacher and student distributions, yielding dense logit-level guidance whose quality depends on the self-teacher. The complementarity between coarse, reward-aligned updates and dense, teacher-dependent guidance motivates SRPO, which routes each sample to the supervision signal best suited to its learning needs.

## 3 The SRPO

SRPO is a unified on-policy framework that routes each rollout to the supervision signal best suited to its learning status. Correct rollouts are optimized with GRPO for reward-aligned reinforcement; incorrect rollouts with available teacher information are optimized with SDPO for dense logit-level correction. An entropy-aware dynamic weighting mechanism further modulates token-level contributions on the SDPO branch, suppressing unreliable distillation targets while emphasizing confident ones. Figure 2 illustrates the overall framework.

### 3.1 Sample-Level Routing

For each rollout y i y_{i} , we define two binary indicators: a correctness flag c i = 𝟏 ​ [ y i ​ is correct ] c_{i}=\mathbf{1}[y_{i}\text{ is correct}] and a teacher-availability flag m i = 𝟏 ​ [ teacher information is available for ​ y i ] m_{i}=\mathbf{1}[\text{teacher information is available for }y_{i}] . The routing mask is then

z i SDPO = ( 1 − c i ) ​ m i , z i GRPO = 1 − z i SDPO . z_{i}^{\mathrm{SDPO}}=(1-c_{i})\,m_{i},\qquad z_{i}^{\mathrm{GRPO}}=1-z_{i}^{\mathrm{SDPO}}.

That is, only incorrect rollouts with available teacher information are routed to the SDPO branch; all remaining rollouts are optimized with GRPO.

This routing does not alter the underlying policy-gradient structure, because both branches update the same policy on the same on-policy trajectories, with only the form of the advantage estimator differing. For GRPO, the gradient takes the standard policy-gradient form

∇ θ ℒ GRPO = − 𝔼 ⁡ [ ∑ t ∇ θ ​ log ​ π θ ​ ( y t ∣ x , y < t ) ⋅ A i GRPO ] , \nabla_{\theta}\mathcal{L}_{\mathrm{GRPO}}=-\mathbb{E}\!\left[\sum_{t}\nabla_{\theta}\log\pi_{\theta}(y_{t}\mid x,y_{<t})\cdot A_{i}^{\mathrm{GRPO}}\right],

where the sequence-level advantage A i GRPO A_{i}^{\mathrm{GRPO}} is shared across all tokens in rollout i i . For SDPO, prior work ( Hübotter et al., 2026 ) shows that distillation gradient admits an analogous form

− ∇ θ ℒ SDPO = 𝔼 ⁡ [ ∑ t ∑ v ∈ 𝒱 ∇ θ ​ log ​ π θ ​ ( v ∣ x , y < t ) ⋅ A t SDPO ​ ( v ) ] , -\nabla_{\theta}\mathcal{L}_{\mathrm{SDPO}}=\mathbb{E}\!\left[\sum_{t}\sum_{v\in\mathcal{V}}\nabla_{\theta}\log\pi_{\theta}(v\mid x,y_{<t})\cdot A_{t}^{\mathrm{SDPO}}(v)\right],

where the logit-level advantage A t SDPO ​ ( v ) A_{t}^{\mathrm{SDPO}}(v) is induced by the discrepancy between the self-teacher and student distributions. The two methods can thus be viewed as advantage estimators at different granularities (reward-derived and sequence-level versus teacher-derived and logit-level), and sample routing simply selects the more appropriate estimator for each sample.

### 3.2 Dynamic-Weighted SDPO

Even within the SDPO branch, teacher supervision is not equally reliable across tokens: low-entropy predictions typically provide clear corrective signals, whereas high-entropy predictions are more likely to introduce noise. We therefore introduce entropy-aware dynamic weighting, which reweights the SDPO loss at the token level according to teacher entropy. For brevity, we refer to this variant as Dynamic-Weighted SDPO (DW-SDPO) throughout this section.

Let q i , t ​ ( v ) = π θ ​ ( v ∣ x , f i , y i , < t ) q_{i,t}(v)=\pi_{\theta}(v\mid x,f_{i},y_{i,<t}) denote the self-teacher distribution at position t t of rollout i i , and let

H i , t = − ∑ v ∈ 𝒱 q i , t ( v ) log q i , t ( v ) H_{i,t}=-\sum_{v\in\mathcal{V}}q_{i,t}(v)\log q_{i,t}(v)

be its entropy. We define the unnormalized weight w ~ i , t = exp ⁡ ( − β ​ H i , t ) \tilde{w}_{i,t}=\exp(-\beta H_{i,t}) , where β > 0 \beta>0 controls sensitivity to entropy differences, and normalize over all valid SDPO tokens to preserve the overall loss scale:

w i , t = w ~ i , t 1 | Ω sdpo | ​ ∑ ( j , s ) ∈ Ω sdpo w ~ j , s , w_{i,t}=\frac{\tilde{w}_{i,t}}{\frac{1}{|\Omega_{\mathrm{sdpo}}|}\sum_{(j,s)\in\Omega_{\mathrm{sdpo}}}\tilde{w}_{j,s}},