Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing — part 2/3

Cleaned-text code points [15705, 31182). Bibliography omitted here; source.md includes it.

where Ω sdpo \Omega_{\mathrm{sdpo}} is the set of valid tokens routed to the SDPO branch. The weighted token loss is then ℓ i , t DW ​ - ​ SDPO = w i , t ​ ℓ i , t SDPO \ell_{i,t}^{\mathrm{DW\text{-}SDPO}}=w_{i,t}\,\ell_{i,t}^{\mathrm{SDPO}} , where ℓ i , t SDPO \ell_{i,t}^{\mathrm{SDPO}} is the base SDPO token loss. This reweighting does not alter the functional form of SDPO; it only modulates each token’s contribution according to teacher confidence, emphasizing reliable corrections while suppressing uncertain ones.

### 3.3 Training Objective

Let ℓ i , t GRPO \ell_{i,t}^{\mathrm{GRPO}} denote the token-level GRPO loss (the sequence-level advantage distributed over valid response tokens) and ℓ i , t DW ​ - ​ SDPO \ell_{i,t}^{\mathrm{DW\text{-}SDPO}} the weighted SDPO loss defined above. The combined objective is

ℒ final = ∑ i , t z i GRPO ​ ℓ i , t GRPO + ∑ i , t z i SDPO ​ ℓ i , t DW ​ - ​ SDPO ∑ i , t z i GRPO + ∑ i , t z i SDPO , \mathcal{L}_{\mathrm{final}}=\frac{\sum_{i,t}z_{i}^{\mathrm{GRPO}}\ell_{i,t}^{\mathrm{GRPO}}\;+\;\sum_{i,t}z_{i}^{\mathrm{SDPO}}\ell_{i,t}^{\mathrm{DW\text{-}SDPO}}}{\sum_{i,t}z_{i}^{\mathrm{GRPO}}\;+\;\sum_{i,t}z_{i}^{\mathrm{SDPO}}},

where all summations over t t are restricted to valid response tokens. The denominator normalizes by the total number of routed tokens, so each branch contributes in proportion to the tokens it covers. This avoids introducing an additional mixing hyperparameter and naturally adapts to the evolving sample composition: early in training, when failures are frequent, more tokens flow through the SDPO branch, giving dense correction a larger effective weight; as the policy improves and more rollouts succeed, the GRPO branch dominates, anchoring the update to the reward objective.

Algorithm 1 summarizes the full training procedure.

## 4 Experiments

### 4.1 Experimental Setup

Model We use instruct-tuned base models from the Qwen3 family ( Yang et al., 2025 ) at two scales: Qwen3-4B and Qwen3-8B. This setting allows us to examine whether the behavior of SRPO is consistent across model sizes. Unless otherwise noted, analyses other than the main performance comparison are conducted at the 8B scale.

Datasets We follow the evaluation setup of SDPO and consider five benchmarks: Chemistry, Physics, Biology, Materials, and Tool Use. The first four are science question-answering tasks built from the reasoning subsets of SciKnowEval ( Feng et al., 2024 ) and target undergraduate-level scientific reasoning in different domains. Tool Use evaluates whether the model can map a user request and a tool specification to the correct tool call, using ToolAlpaca ( Tang et al., 2023 ) . Following SDPO, we perform a train-test split on each benchmark to evaluate in-domain generalization.

Baselines We compare against two baselines: (1) GRPO , a strengthened implementation of GRPO ( Shao et al., 2024 ) following recent best practices ( Olmo et al., 2025 ; Khatri et al., 2025 ) , including asymmetric clipping ( Yu et al., 2025 ) , unbiased advantage normalization ( Liu et al., 2025 ) , and off-policy correction for distributed inference ( Yao et al., 2025 ) ; and (2) SDPO , which replaces reward-only supervision with self-distillation from a feedback-conditioned self-teacher and provides a finer-grained but potentially biased training signal. In our experiments, SDPO uses successful sibling rollouts within the same group as teacher information for failed samples.

Implementation Details For both GRPO and SDPO, we adopt the training setup and hyperparameters from the original SDPO paper, where each method’s configuration was selected via grid search over learning rates and mini-batch sizes to maximize the validation accuracy ( Hübotter et al., 2026 ) . Both methods use a training batch size of 32 and sample 8 rollouts per prompt; the main differences are the mini-batch size and learning rate: GRPO uses a mini-batch size of 8 and a learning rate of 1 × 10 − 6 1\times 10^{-6} , whereas SDPO uses 32 with 1 × 10 − 5 1\times 10^{-5} . For SRPO, we keep the training batch size, mini-batch size, and rollout number the same as in SDPO, set the learning rate to 5 × 10 − 6 5\times 10^{-6} to balance the reward-driven and self-distillation signals within a single objective, and use a dynamic-weighting temperature β \beta with default value 1. All experiments are conducted on 8 NVIDIA H20 GPUs.

### 4.2 Main Results

SRPO achieves early efficiency, long-horizon stability, and a higher performance ceiling. Table 1 reports the highest avg@16 achieved within each wall-clock budget, following the reporting protocol of SDPO. 1 1 1 We note that Qwen3-4B slightly outperforms Qwen3-8B on the base instruct checkpoints across all five benchmarks. These benchmarks were not explicitly targeted during Qwen3 fine-tuning ( Hübotter et al., 2026 ) , and such nonmonotonic scaling on out-of-distribution downstream tasks is a well-documented phenomenon ( McKenzie et al., 2023 ; Lourie et al., 2025 ) . Crucially, the larger 8B model still achieves higher post-training performance and larger total training gains despite starting from a lower base, consistent with the expected scaling behavior, indicating that our conclusions are not affected by this anomalous ordering of base-model performance. On Qwen3-8B, SRPO improves the 10h average from 71.1 (SDPO) and 74.0 (GRPO) to 77.4; on Qwen3-4B, the corresponding improvement is from 66.7 and 69.7 to 74.2. Across both scales, SDPO saturates early, as evidenced by its identical 5h and 10h averages, while GRPO improves more steadily before eventually plateauing. SRPO largely avoids both issues, matching the early training efficiency of SDPO while maintaining steady improvement over longer horizons and ultimately exceeding the peak performance of both baselines. Notably, at 10h on Qwen3-8B, SRPO improves over GRPO by +4.1 on Chemistry, +4.8 on Physics, +2.2 on Biology, +3.7 on Materials, and +2.2 on Tool Use. We attribute this to entropy-aware dynamic weighting on the SDPO branch: even when the self-teacher becomes noisier in later training, reweighting by teacher confidence preserves useful logit-level guidance while suppressing uncertain targets, enabling SRPO to continue improving beyond the point where pure GRPO plateaus.

To complement the tabular summary, Figure 3 plots representative learning curves on Qwen3-8B, which reveal two recurring patterns.

Pattern 1: When self-distillation is effective, SRPO extends the advantage. In Chemistry, SDPO leads at 1h (71.6 vs. 69.2 for SRPO), but SRPO overtakes it by 5h and reaches 83.0 at 10h, exceeding both SDPO (80.6) and GRPO (78.9). As Figure 3 (a) shows, SRPO tracks SDPO’s rapid early rise while avoiding its subsequent collapse. Biology follows a similar trajectory: SRPO achieves the best 1h result (55.8), and the gap widens as SDPO stalls at 58.5 while SRPO climbs to 72.8 at 10h (Figure 3 (b)).

Pattern 2: When self-distillation is ineffective, SRPO remains stable. As Figure 3 (c) shows, SDPO degrades substantially over time on Tool Use, whereas SRPO remains stable and tracks or exceeds GRPO throughout (65.2, 71.2, 71.2 vs. 64.3, 68.5, 69.0 for GRPO). Both patterns reflect the effectiveness of the sample-routing design: when self-distillation is useful, SRPO exploits it to accelerate learning; when it is not, the GRPO branch anchors optimization to the reward objective and prevents drift.

### 4.3 Ablation Study

Sample routing is more robust than advantage-level mixing over long horizons. To isolate the mixing strategy, we first compare SRPO w/o dynamic weighting against an Advantage Mix control that combines GRPO and SDPO at the advantage level:

A i , t Mix ​ ( v ) = λ ​ A i , t GRPO ​ ( v ) + ( 1 − λ ) ​ A i , t SDPO ​ ( v ) , λ ∈ [ 0 , 1 ] , A_{i,t}^{\mathrm{Mix}}(v)=\lambda A_{i,t}^{\mathrm{GRPO}}(v)+(1-\lambda)A_{i,t}^{\mathrm{SDPO}}(v),\qquad\lambda\in[0,1],

where the GRPO term is reward-derived and the SDPO term is feedback-derived. We set λ = 0.9 \lambda=0.9 to keep the two advantages on a comparable scale, consistent with the mixing ratio used in SDPO ( Hübotter et al., 2026 ) , and keep all other hyperparameters unchanged. Advantage Mix is slightly better at 1h (+0.7), but falls behind by 2.5 points at 5h and 3.3 points at 10h, with no further gain after 5h.

This pattern matches the changing roles of the two signals over training. Early on, when self-distillation remains high quality, mixing dense SDPO guidance with reward-aligned GRPO updates can help. Later, as the SDPO signal becomes less reliable, advantage-level mixing instead propagates this noise into the learning process, harming stability. By contrast, sample routing confines SDPO to failed samples and leaves correct samples under GRPO, reducing interference and yielding stronger long-term performance.

Dynamic weighting provides an additional late-stage gain on top of sample routing. We then compare SRPO against SRPO w/o dynamic weighting to isolate the effect of entropy-aware weighting. Adding dynamic weighting improves the average result by 0.4 at 1h, 0.7 at 5h, and 1.8 at 10h. The widening gain suggests that this component matters most when the self-teacher becomes less reliable and noisier. This is consistent with the role of entropy-aware weighting: it emphasizes high-confidence dense corrections while suppressing uncertain targets, further stabilizing the SDPO branch in later training.

Together, these ablations suggest that SRPO’s gains come from two complementary components: sample routing provides the stronger mixing strategy and the main source of long-horizon robustness, while dynamic weighting adds further late-stage improvement by improving the reliability of the SDPO branch.

### 4.4 Response Length and Compute Time

SRPO yields moderate response lengths between GRPO and SDPO. Figure 4 (a) shows response length during training of Qwen3-8B on Chemistry. The three methods exhibit different trends: GRPO produces the longest responses, SDPO the shortest, and SRPO settles between the two. The verbosity of GRPO inflates inference cost, while the excessive brevity of SDPO has been linked to degraded reasoning due to the suppression of epistemic verbalization ( Kim et al., 2026 ) . SRPO’s moderate response length suggests a balance between the two, potentially mitigating both issues.

SRPO achieves the lowest per-step compute time over long training horizons. Figure 4 (b) reports the average seconds per training step of Qwen3-8B, averaged over the five benchmarks. At 1h, SRPO incurs a 17.4% overhead relative to GRPO (83.4s vs. 71.0s per step), while being lower than SDPO (83.4s vs. 85.9s). As training proceeds, the cost advantage shifts in favor of SRPO. At 5h, it is 4.9% faster than GRPO and 6.7% faster than SDPO (78.3s vs. 82.4s and 83.9s). At 10h, the advantage widens further, reaching 17.2% over GRPO and 9.4% over SDPO (75.8s vs. 91.5s and 83.7s).

These results are consistent with the design of SRPO. Early in training, failed samples are more frequent, so the SDPO branch is activated more often and the additional self-teacher log-probs computation is more visible. Later in training, the fraction of failed samples decreases, reducing the self-teacher overhead. At the same time, SRPO produces shorter responses than GRPO, further lowering inference cost. Taken together, SRPO improves not only training efficiency and stability, but also computational efficiency in terms of response length and per-step compute time.

## 5 Conclusion

We revisit the trade-off between reward-driven reinforcement and self-distillation in LLM post-training and propose SRPO, a unified on-policy framework that routes successful samples to GRPO for reward-aligned reinforcement and failed samples with teacher information to SDPO for dense logit-level correction, together with entropy-aware dynamic weighting to suppress unreliable self-distillation signals and emphasize confident ones. Across five benchmarks and two model scales, SRPO consistently outperforms both pure GRPO and SDPO, demonstrating that sample-level routing can preserve the early efficiency of self-distillation while maintaining the long-horizon stability of reward-driven reinforcement. Moreover, SRPO yields moderate response lengths and lower per-step compute time over long training horizons. An important direction for future work is to extend this framework to environments with richer feedback, so that self-distillation branch can better leverage environment information.

## Ethics Statement

This work studies post-training optimization methods for large language models and does not introduce new capabilities targeted at harmful applications. However, improving reasoning quality may still increase dual-use risks (e.g., more effective generation of misleading or unsafe content). We therefore recommend deployment only under standard safety controls, including content moderation, policy-based filtering, and rate limiting.

Our experiments use publicly available benchmark datasets (SciKnowEval and ToolAlpaca-style tool-use tasks) and automatic verifiable rewards. We do not collect personal data, do not involve human subjects, and do not perform user profiling. The training objective does not use private annotations or sensitive metadata.

From an environmental perspective, SRPO is trained on GPU clusters and thus incurs non-trivial energy use. At the same time, our results show lower per-step compute time over long horizons compared with strong baselines, which may partially reduce the total compute required to reach a target performance level. We plan to release implementation details to support transparent evaluation and responsible reproduction.

## Appendix A Related Work

### A.1 Reinforcement Learning with Verifiable Rewards

Post-training with verifiable rewards has become a central paradigm for LLM alignment and adaptation, building on policy-gradient foundations such as REINFORCE and PPO ( Williams, 1992 ; Schulman et al., 2017 ) . A growing body of work applies these ideas to LLM post-training, where sequence-level outcome rewards guide optimization on model-sampled trajectories ( Guo et al., 2025 ; Shao et al., 2024 ; Yu et al., 2025 ; Liu et al., 2025 ; Zheng et al., 2025 ; Zhang et al., 2025 ; Zhoubian et al., 2025 ) . Among them, GRPO estimates advantages from group-relative rewards without requiring a separate critic, making it a strong and scalable baseline ( Shao et al., 2024 ) .

However, these methods typically assign a single scalar advantage uniformly to every token, so credit assignment remains coarse. Recent analyses have shown that this uniform assignment dilutes gradients across causally irrelevant tokens ( Khandoga et al., 2026 ) , hinders localization of semantic errors in near-correct programs ( Kumar et al., 2026 ) , and introduces bias that grows with sequence length ( Parthasarathi et al., 2025 ) . A complementary line of work seeks to improve credit assignment through process supervision or process reward models, which provide denser step-level signals derived from intermediate states or feedback ( Lightman et al., 2023 ; Setlur et al., 2025 ; Cui et al., 2025 ) . These approaches offer finer-grained guidance but usually require additional learned reward estimators. This trade-off motivates methods that provide denser supervision without introducing an additional reward model.

### A.2 On-Policy Distillation and Self-Distillation