Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing — part 3/3

Cleaned-text code points [31182, 42631). Bibliography omitted here; source.md includes it.

Distillation transfers behavior from a teacher to a student by matching output distributions or intermediate representations ( Hinton et al., 2015 ; Kim and Rush, 2016 ; Sanh et al., 2019 ) . More recent on-policy distillation methods reduce train-test mismatch by training the student on its own trajectories while receiving teacher guidance on those same trajectories ( Agarwal et al., 2024 ; Gu et al., 2023 ; Lu and Lab, 2025 ) . Relative to reward-only RL, these methods provide denser supervision, but they typically rely on a separate and often stronger external teacher.

Self-distillation removes the need for an external teacher by supervising the model with a conditioned version of itself. Context distillation first showed that a model can internalize behavior induced by privileged context into its parameters ( Snell et al., 2022 ) . More recent work extends this idea to self-improvement and on-policy self-distillation settings, including learning from self-generated trajectories or richer conditioning information ( Mitra and Ulukus, 2025 ; Hübotter et al., 2025 ; Hübotter et al., 2026 ; Shenfeld et al., 2026 ; Zhao et al., 2026 ; Buening et al., 2026 ; Wang et al., 2026 ) . A representative example is SDPO ( Hübotter et al., 2026 ) , which samples rollouts from the current policy and distills the logit-level distribution of a feedback-conditioned self-teacher back into the same policy. However, feedback-conditioned on-policy self-distillation can exhibit late-stage degradation: concurrent work by Kim et al. (2026) attributes this to the suppression of epistemic verbalization, while our analysis (Section 1) traces it to ambiguity on correct samples and progressive degradation of the self-teacher signal.

Overall, prior RL-based post-training methods provide reward alignment but rely on coarse sequence-level supervision. Distillation-based methods provide denser logit-level guidance, and self-distillation removes the need for an external teacher, but feedback-conditioned on-policy self-distillation may suffer from sample-dependent ambiguity and degraded signal quality in later training. To address this gap, our work studies how reward-driven and self-distillation-based supervision can be combined within a unified framework based on sample routing, thereby leveraging the strengths of both post-training paradigms.

## Appendix B Experimental Details

### B.1 Technical Setup

All experiments were conducted on a single node equipped with 8 NVIDIA H20 GPUs interconnected via NVLink, providing a total of 768 GB VRAM. Our software environment uses GPU driver version 550.144.03, CUDA 12.4, and PyTorch 2.8.0.

Our implementation is based on the verl library ( Sheng et al., 2025 ) . We use PyTorch Fully Sharded Data Parallel (FSDP2) for distributed training across GPUs. For rollout generation, we employ SGLang ( Zheng et al., 2024 ) instead of the vLLM backend ( Kwon et al., 2023 ) used in the original SDPO implementation, as SGLang provides better compatibility with our environment. Since both engines implement the same sampling algorithms and support identical temperature, top- p p , and other decoding parameters, the choice of inference backend affects only throughput and does not alter the sampling, preserving a fair comparison with SDPO.

### B.2 Hyperparameters

Table 3 summarizes the hyperparameters for all three methods. For the two baselines (GRPO and SDPO), we directly adopt the configurations selected via grid search in the original SDPO work ( Hübotter et al., 2026 ) ; see that paper for details on the search procedure. For SRPO, we inherit all non-learning-rate hyperparameters from SDPO and set the learning rate to 5 × 10 − 6 5\times 10^{-6} , halfway between the GRPO and SDPO rates, to balance the reward-driven and self-distillation signals within a unified framework. The GRPO branch within SRPO uses the same loss-specific parameters as the standalone GRPO baseline, and the SDPO branch uses the same loss-specific parameters as the standalone SDPO baseline. The only additional hyperparameter introduced by SRPO is the dynamic-weighting temperature β \beta , which we set to 1 as default.

### B.3 Prompt Templates

We use the same prompt templates as SDPO ( Hübotter et al., 2026 ) without any modification, ensuring a fair comparison across all methods. The Science Q&A benchmarks (Chemistry, Physics, Biology, Materials) share a common multiple-choice format, while Tool Use follows a separate tool-calling format. We reproduce both templates below.

### B.4 Benchmark Details

We use the exact train/test splits provided in the official SDPO github repository to ensure full comparability. Table 4 summarizes the dataset statistics.

The four Science Q&A benchmarks are formatted as four-option single-choice questions targeting undergraduate-level scientific reasoning. Each question presents a problem statement (often involving domain-specific notation such as SMILES strings in Chemistry, physical equations in Physics, protein sequences in Biology, or crystal lattice parameters in Materials) followed by four candidate answers. The Tool Use benchmark pairs a natural-language user request with a tool-API specification (including function names, parameter schemas, and output types); the model must produce the correct tool call in a structured Thought / Action / Action Input format.

Table 5 shows one representative example from each benchmark.

### B.5 Teacher Information Construction

As described in Section 3, the SDPO branch requires teacher information f i f_{i} for each rollout y i y_{i} to construct the feedback-conditioned self-teacher distribution π θ ( ⋅ ∣ x , f i , y i , < t ) \pi_{\theta}(\cdot\mid x,f_{i},y_{i,<t}) . Following SDPO ( Hübotter et al., 2026 ) , we use successful sibling rollouts within the same group as teacher information. Since our experimental setting does not include rich environment feedback (e.g., runtime errors in coding tasks), the only available source of teacher information is a correct sibling rollout from the same prompt.

#### Construction procedure.

For each prompt x x , the policy generates a group of G = 8 G=8 rollouts { y 1 , … , y G } \{y_{1},\ldots,y_{G}\} . We identify all correct rollouts in the group (those with reward r i ≥ 0.5 r_{i}\geq 0.5 ). For each rollout y i y_{i} , the teacher information f i f_{i} is constructed as follows: 1. Collect the indices of all correct rollouts for the same prompt, excluding rollout i i itself (to prevent a sample from serving as its own teacher).

2. If at least one correct sibling exists, select one and use its full response text as the teacher information. The teacher prompt is then formatted as:

The self-teacher processes this enriched prompt concatenated with the student’s own response tokens y i , < t y_{i,<t} , producing a logit-level distribution at each position that serves as the distillation target. Crucially, the self-teacher does not generate a new response; it re-scores the student’s existing trajectory under the enriched context.

#### Illustrative example.

Consider a prompt with G = 8 G=8 rollouts, of which rollouts y 2 y_{2} and y 5 y_{5} are correct (reward = 1.0 =1.0 ) and the remaining six are incorrect (reward = 0.0 =0.0 ). Table 6 shows the resulting routing decision for representative rollouts.

#### Fallback to GRPO when no teacher information is available.

When all G G rollouts for a prompt are incorrect, no correct sibling exists, so m i = 0 m_{i}=0 for every rollout. By the routing rule z i SDPO = ( 1 − c i ) ​ m i z_{i}^{\mathrm{SDPO}}=(1-c_{i})\,m_{i} , all rollouts are assigned to the GRPO branch despite being incorrect. Notably, when a rollout is the only correct one in its group, it is excluded from being its own teacher , so m i = 0 m_{i}=0 for that rollout. Since it is correct ( c i = 1 c_{i}=1 ), it is routed to GRPO regardless. Table 7 summarizes the complete decision logic.

This design ensures that the SDPO branch is activated only when dense logit-level correction is both needed (the rollout is incorrect) and feasible (a correct sibling provides informative teacher context). In all other cases, the update falls back to GRPO’s reward-aligned advantage signal.

## Appendix C Routing Statistics Over Training

Figure 5 visualizes how the sample-routing composition of SRPO evolves over the course of training. At the beginning of training, approximately 40% of samples are routed to the SDPO branch and 60% to the GRPO branch, reflecting the substantial fraction of incorrect rollouts that benefit from dense logit-level correction. As training progresses and the policy improves, the fraction of correct rollouts increases, causing more samples to be routed to the GRPO branch.

This dynamic shift has two important implications. First, it provides direct empirical support for the adaptive mixing behavior described in Section 3.3. SDPO branch contributes a substantial share in the early stage, providing meaningful dense logit-level correction when the policy is weaker and incorrect rollouts are frequent. As training proceeds and the policy improves, this contribution gradually diminishes while an increasing share of samples is handled by the GRPO branch, whose reward-aligned advantages provide a more stable and unbiased optimization signal for already-correct rollouts. The net effect is that SRPO automatically modulates the influence of self-distillation—leveraging it most when it is most beneficial and changing it to reward-aligned reinforcement for stability as the policy matures—without requiring any manual scheduling of the mixing ratio.

Second, the decreasing SDPO fraction directly explains the compute-time trend observed in Section 4.4 (Figure 4 (b)): since the self-teacher log-probability computation is only performed for samples on the SDPO branch, the per-step overhead of this additional forward pass diminishes as fewer samples require it. This accounts for why SRPO’s per-step compute time decreases steadily over training and eventually falls below that of both standalone GRPO and SDPO.

Figure 5 (c) further shows that the fraction of samples with constructable teacher information remains high throughout training. This indicates that the fallback to GRPO due to teacher unavailability ( m i = 0 m_{i}=0 ) is relatively infrequent; the primary driver of the routing shift is the increasing correctness of rollouts ( c i = 1 c_{i}=1 ), not the absence of teacher information.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .