Reinforcement Learning via Self-Distillation — part 2/6

Cleaned-text code points [15762, 31512). Bibliography omitted here; source.md includes it.

We evaluate tasks on which the model has not been explicitly fine-tuned: • Science Q&A (Chemistry, Physics, Biology, Materials science): Undergraduate-level scientific reasoning using reasoning subsets (L3) from SciKnowEval ( Feng et al., 2024a ) .

• Tool use : Mapping a tool-API specification and user request to the correct tool call, using ToolAlpaca ( Tang et al., 2023 ) .

We perform a train-test split to test in-domain generalization. We use Qwen3-8B ( Yang et al., 2025a ) and Olmo3-7B-Instruct ( Olmo et al., 2025 ) as initial checkpoints and report avg@16 relative to wall-clock training time, excluding initialization & validation.

##### Baselines.

We compare SDPO to an improved variant of GRPO ( Shao et al., 2024 ) , which incorporates several recent modifications ( Olmo et al., 2025 ; Khatri et al., 2026 ) such as asymmetric clipping ( Yu et al., 2025 ) , avoiding biased normalization ( Liu et al., 2025b ) , and correcting for off-policy data when using efficient inference frameworks ( Yao et al., 2025 ) . We integrate these modifications into a GRPO implementation that represents a strong baseline, as detailed in Equation 13 in Section A.4 . GRPO enables off-policy training through PPO’s clipped importance weighting ( Schulman et al., 2017 ) . We additionally report the special case of on-policy GRPO (matching the hyperparameters of vanilla SDPO). For both baselines, we perform a hyperparameter sweep and report results for the models that achieve the highest validation performance across all target tasks. Hyperparameters and training details are provided in Appendix E . We use the verl library ( Sheng et al., 2025 ) for fast multi-GPU training.

### 3.2 Results

Table 3 summarizes our results. We find that SDPO outperforms GRPO across almost all runs, often leading to substantial improvements. SDPO learns notably faster than GRPO, performing close to 5 hours of GRPO training after only 1 hour of training with SDPO in several cases. SDPO achieves a particularly substantial improvement over GRPO on the Chemistry task, as is displayed in Figure 6 (left) . With Olmo3-7B-Instruct, SDPO achieves the 5h GRPO accuracy in 50 minutes of wall-clock training time , a 6 × 6\times speedup. Moreover, SDPO’s 5h accuracy is more than 10 10 %-points higher than that of GRPO.

We remark that our results with SDPO use strictly on-policy training (i.e., one gradient step per generation batch). Given the known efficiency gains of off-policy methods that perform multiple gradient updates per generation batch, we believe that studying SDPO with off-policy updates is an exciting direction for future work.

### 3.3 Self-distillation learns to reason concisely

We consistently observe that SDPO produces substantially shorter generations than GRPO while achieving higher accuracy. SDPO’s responses are more than 3 × 3\times shorter on average across tasks (cf. Table 8 in Appendix D ). On Chemistry with Olmo3-7B-Instruct, SDPO even achieves an 11 × 11\times reduction in response length relative to GRPO while maintaining higher accuracy (Figure 6 (right) ). While recent progress in RLVR has demonstrated that scaling response length is a powerful driver of emergent reasoning capabilities ( Jaech et al., 2024 ; Guo et al., 2025 ; Muennighoff et al., 2025 ) , our results suggest that effective reasoning need not always be verbose. We find that SDPO improves the efficiency of reasoning.

Qualitatively, we observe that the longer responses from GRPO often stem from “superficial” reasoning rather than necessary analytical steps. GRPO frequently generates filler phrases like “Hmm” and “Wait” or enters circular logical loops that repeat previous steps verbatim. Figure 7 displays a representative example of this phenomenon. Remarkably, SDPO’s generations remain concise and avoid these superficial patterns. This may be explained by SDPO’s dense credit assignment, which assigns a specific advantage to each next-token prediction, leading to sparse advantages (cf. Figure 21 in Appendix F ). By improving the efficiency of reasoning, SDPO reduces inference generation time and demonstrates that reasoning performance can be improved by refining how the model reasons, not just how long it reasons.

## 4 Learning with Rich Environment Feedback

We next evaluate SDPO on coding tasks. Coding is a canonical example of an RL environment that provides rich feedback, such as runtime errors and failed unit tests. Learning to solve these coding problems requires strong credit assignment since the student must identify its precise mistakes to avoid repeating them in the future. LiveCodeBench ( Jain et al., 2025 , LCB;) provides a set of contest-style coding problems, ranging from simple to competition-level. We restrict our evaluation to the most recent LCBv6 subset of LCB, which contains 131 questions released between February and May 2025. We consider a setting with public and private unit tests, common for code contests and coding platforms like LeetCode, where the public tests are used for evaluation during training and the private tests are used for validation ( Chen et al., 2022 ; Le et al., 2022 ; El-Kishky et al., 2025 ; Samadi et al., 2025 ) . 5 5 5 We select public tests as a 50% random subset of private tests.

We use the Qwen3 ( Yang et al., 2025a ) model family for our experiments, with Qwen3-8B as default unless otherwise specified. We report the average accuracy over 4 rollouts and use the same GRPO baseline as outlined in Section 3.1 .

##### Results.

Figure 1 compares the learning curves of SDPO and GRPO on LCBv6. We find that SDPO achieves a substantially higher final accuracy (48.8%) than GRPO (41.2%) while also outperforming the strongest instruct models on the public LCBv6 leaderboard: 6 6 6 On the public leaderboard, the LCBv6 subset can be obtained by selecting February to May 2025. Claude Sonnet 4 (40.5%) and Claude Opus 4 (39.7%). Furthermore, SDPO reaches the final accuracy of GRPO in 4 × 4\times fewer generations. We include an extended comparison to other RLVR baselines that perform similarly to GRPO in Table 9 in the appendix. Differentiating between the easy, medium, and hard questions of LCB, we find that SDPO particularly improves over GRPO in solving medium and hard questions (cf. Figure 15 in the appendix).

### 4.1 Self-distillation benefits from stronger models

A central question for our work is whether SDPO is sensitive to the in-context learning ability of the base model. Intuitively, we expect that SDPO benefits from a strong in-context learner, since this enables the teacher to perform more accurate retrospection.

To answer this question, we perform a scaling study with different model sizes from the Qwen3 ( Yang et al., 2025a ) family. As shown by extensive prior work, the ability to learn in-context increases with model size ( Brown et al., 2020 , e.g.,) . As depicted in Figure 8 , SDPO significantly outperforms GRPO on larger models while only slightly improving over GRPO on smaller models. To determine whether SDPO can also underperform GRPO on a model weaker than Qwen3-0.6B, we performed an additional scaling study with Qwen2.5-Instruct ( Qwen et al., 2024 ) . While outperforming GRPO with Qwen2.5-7B and performing similarly with Qwen2.5-8B, we find that SDPO underperforms GRPO on Qwen2.5-1.5B, as seen in Figure 17 in Appendix D .

### 4.2 Self-distillation performs dense credit assignment

Whereas GRPO assigns a constant advantage to each generated token, SDPO assigns an individual advantage to each possible next token along the generated sequence based on the agreement of student and teacher. At each position t t in the generated sequence y y , there are | 𝒱 | |\mathcal{V}| possible next tokens where 𝒱 \mathcal{V} is the vocabulary. In distillation, this level is typically called the logit-level since it corresponds to the logits of the model. In practice, we approximate the full next-token distribution by the top- K K tokens plus the tail, and as such, SDPO assigns | y | ⋅ ( K + 1 ) |y|\cdot(K+1) unique advantages per sequence. This is illustrated in Figure 9 and allows SDPO to perform dense credit assignment.

A natural question is whether the performance gains of SDPO are due to leveraging rich feedback in RLRF or due to the dense credit assignment of SDPO. To answer this question, we ablate the performance of SDPO in three configurations: • Logit-level SDPO: credit assignment over the 100 most likely tokens (under the student) at each position.

• Token-level SDPO: credit assignment over the most likely token at each position.

• Sequence-level SDPO: We compute SDPO advantages for all generated tokens and average them to produce a single scalar advantage per sequence (as in GRPO). This does not perform denser credit assignment than GRPO but still leverages the rich feedback f f .

As shown in Figure 10 (left) , the dense credit assignment of logit-level SDPO leads to significant performance gains over token-level SDPO and sequence-level SDPO. Nevertheless, even sequence-level SDPO outperforms GRPO, indicating that leveraging rich feedback in RLRF can lead to substantial gains over RLVR methods even without dense credit assignment.

### 4.3 The self-teacher improves during training

Contrary to standard distillation, the self-teacher in SDPO is not frozen, but updated throughout training. This is a critical component of SDPO, since it enables the teacher to improve over time, which means that the student can learn from a stronger target. To investigate whether the self-teacher improves during training, we plot the average accuracy when generating using the self-teacher in Figure 10 (right) . We find that the self-teacher improves significantly during training. Most notably, the student’s accuracy surpasses the initial teacher’s accuracy in later stages of training. This demonstrates that SDPO enables true bootstrapping of a weak model to a strong model, without the initial self-teacher’s performance limiting the final student.

As described in Section 2.3 , SDPO uses a regularized teacher to stabilize training. As can be seen in Table 4 , a non-regularized teacher significantly underperforms the regularized teachers. Furthermore, trust-region and EMA teachers outperform the teacher frozen at the initial teacher’s parameters, showing that the teacher improves through parameter sharing with the student. Yet, SDPO performs well even with a frozen teacher.

### 4.4 On-policy self-distillation avoids catastrophic forgetting

Prior work has shown that a key benefit of on-policy algorithms, such as GRPO, is that models tend not to forget previously obtained capabilities ( Shenfeld et al., 2026b ; Chen et al., 2025b ; Lu & Thinking Machines Lab, 2025 ) . This is practically desirable since it enables continual training pipelines where a model is trained sequentially on diverse tasks without the need to retrain from scratch. To evaluate forgetting, we test the final checkpoints of GRPO and SDPO on diverse holdout tasks: IFEval ( Zhou et al., 2023 ) , which tests the ability of a model to follow precise format instructions; ArenaHard-v2 ( Li et al., 2025a ) , which is an LLM-judged benchmark of real-world instruction-following prompts derived from LMArena ( Chiang et al., 2024 ) ; and MMLU-Pro ( Wang et al., 2024b ) , which tests broad multi-task knowledge and reasoning. As displayed in Table 5 , SDPO learns the new task while mitigating degradation of initial capabilities, overall achieving a better performance–forgetting tradeoff than GRPO.

##### Off-policy self-distillation baseline.

As an additional baseline, we consider training the student via supervised fine-tuning (SFT) on successful generations from the self-teacher ( Scheurer et al., 2023 ; Dou et al., 2024 ; Zhou et al., 2025 ) . 7 7 7 SFT on a teacher’s predictions is a standard off-policy distillation approach ( Kim & Rush, 2016 ) . This requires 2 × 2\times the generations of SDPO for the same number of steps, since we have to generate from both the student and the teacher. We report SFT on the successes of the self-teacher, which achieves a higher accuracy than also including initial successes from the student in the SFT data. As shown in Table 5 , SFT on the self-teacher significantly underperforms SDPO on LCBv6, while leading to worse forgetting of prior capabilities. This mirrors prior findings on the instability of off-policy imitation ( Agarwal et al., 2024 , see, e.g.,) .

### 4.5 Can GRPO and SDPO be combined?

GRPO utilizes Monte Carlo advantages, which are unbiased with respect to the objective of maximizing expected reward J ( θ ) := 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ r ( y ∣ x ) ] J(\theta):=\smash{\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}{}\left[r(y\mid x)\right]} . In contrast, SDPO advantages are inherently biased with respect to J ⁡ ( θ ) J(\theta) due to being computed from rich feedback and a self-teacher. This dichotomy parallels the fundamental distinction between Monte Carlo and bootstrapped advantages in RL: while the latter are biased, they typically yield lower variance ( Sutton & Barto, 1998 ; Schulman et al., 2016 ) . This motivates a hybrid approach that combines reward-derived GRPO advantages with feedback-derived SDPO advantages: A i , t SDPO + GRPO ​ ( y ^ i , t ) := λ ​ A i , t GRPO ​ ( y ^ i , t ) + ( 1 − λ ) ​ A i , t SDPO ​ ( y ^ i , t ) , λ ∈ [ 0 , 1 ] . A_{i,t}^{\mathrm{SDPO+GRPO}}(\hat{y}_{i,t}):=\lambda A_{i,t}^{\mathrm{GRPO}}(\hat{y}_{i,t})+(1-\lambda)A_{i,t}^{\mathrm{SDPO}}(\hat{y}_{i,t}),\quad\lambda\in[0,1]. (3)

As shown in Figure 11 , SDPO+GRPO appears to be more robust to weaker models than SDPO. Intuitively, in a weaker model such as Qwen3-0.6B, the SDPO advantages are less reliable, and hence including the GRPO advantage helps to stabilize training. In contrast, we find that SDPO+GRPO slightly underperforms SDPO on stronger models such as Qwen3-8B. This suggests that the signal of GRPO, only informed by a scalar reward, can be actively harmful with a strong initial model.

### 4.6 Which feedback is most informative?

To understand which type of rich feedback is most informative, we ablate the three types of feedback present in a verifiable environment like code generation: the sample solution (if a successful rollout is available in the current rollout group), the environment output (such as runtime errors), and the student’s original attempt.

##### Sample solutions.

Including a sample solution from a failed attempt’s rollout group (if available) closely mirrors the group-relative advantages of GRPO. We emphasize that these sample solutions are always generated by the student, as in GRPO, and do not require an expert model. They allow for disincentivizing unsuccessful approaches if the model is already able to solve the question. However, unlike GRPO where all tokens receive the same negative advantage, the self-teacher can identify specific mistakes and provide feedback on how to fix them.

##### Environment output.

The environment output describes the state of the environment after the student’s attempt. This is complementary to sample solutions since it can provide useful signal even if the student has never solved the question before (a setting we explore extensively in Section 5 ). Leveraging environment output is a key differentiating factor between RLRF and RLVR settings.

##### Student’s original attempt.

The student’s original attempt y y does not have to be included in the reprompting template of the teacher. Indeed, we find that including it biases the teacher towards the student’s attempt (cf. Table 6 ). This reduces the entropy of the student’s distribution (particularly for initially uncertain tokens), thereby reducing exploration.