Reinforcement Learning via Self-Distillation — part 1/6
Cleaned-text code points [0, 15762). Bibliography omitted here; source.md includes it.
# Reinforcement Learning via Self-Distillation
###### Abstract
Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization ( SDPO ), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model’s ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of- k k sampling or multi-turn conversations with 3 × 3\times fewer attempts.
## 1 Introduction
Progress in deep reinforcement learning has shown that iterating on experience—acting, receiving feedback, and updating a policy—can unlock capabilities that are difficult to obtain from static supervision alone ( Mnih et al., 2015 ; Silver et al., 2016 ; Silver et al., 2017 ; Berner et al., 2019 ) . The same theme now appears in large language models (LLMs): large-scale post-training with reinforcement learning (RL) has substantially improved performance on reasoning-heavy tasks, especially in settings with programmatic or otherwise verifiable evaluation ( Jaech et al., 2024 ; Guo et al., 2025 ; Kimi et al., 2025 ; Olmo et al., 2025 ) .
Nevertheless, the dominant RL recipe for LLM post-training remains bottlenecked by credit assignment. Most current approaches operate in the setting of reinforcement learning with verifiable rewards (RLVR): given a question x x , the model samples an answer y ∼ π θ ( ⋅ ∣ x ) {y\sim\pi_{\theta}(\cdot\mid x)} and receives a scalar reward r ∈ ℝ r\in\mathbb{R} , often binary (e.g., unit-tests pass/fail in code generation). Modern policy gradient RLVR methods such as Group Relative Policy Optimization ( Shao et al., 2024 , GRPO;) estimate advantages from these sparse outcome rewards. Furthermore, when all rollouts in a group receive the same (often zero) reward, GRPO advantages collapse to zero and learning stalls. To overcome this sparsity, one might prefer distillation from a strong teacher ( Guo et al., 2025 ; Yang et al., 2025a ; Lu & Thinking Machines Lab, 2025 ; Guha et al., 2026 ) , which provides dense, token-level supervision. However, strong teachers are often unavailable in online learning, where the goal is to raise the capability ceiling beyond existing models.
In this work, we argue that the key limitation is not RL per se, but the information bottleneck imposed by scalar outcome rewards. Many verifiable environments expose rich tokenized feedback beyond scalar rewards r r , such as runtime errors, failing unit tests, or evaluations from an LLM judge. This feedback not only reveals whether a rollout was wrong, but also what went wrong. We formalize this more general setting as Reinforcement Learning with Rich Feedback ( RLRF ) and illustrate its difference to RLVR in Figure 2 . Here, feedback can be any tokenized representation of any state reached by an agentic system. The central question becomes: how can we convert rich feedback into effective credit assignment without requiring external supervision from a strong teacher?
Our starting point is the observation that LLMs already possess a powerful mechanism for using feedback: in-context learning ( Brown et al., 2020 ; Wei et al., 2022 ) . When conditioned on feedback, the same model can often identify plausible mistakes and propose a corrected approach. A common example of such feedback is the summary of failed test cases on coding platforms like LeetCode ( Figure 3 ). Many recent works leverage this capability to iteratively generate corrections ( Chen et al., 2021a ; Madaan et al., 2023 ; Shinn et al., 2023 ; Yao et al., 2024 ; Yuksekgonul et al., 2025 ; Lee et al., 2025 ) . In contrast, we use the current policy as a “self-teacher” that, rather than sampling a new response, re-evaluates the existing rollout after receiving rich feedback. Including the feedback in-context transforms the model’s next-token distribution, allowing the self-teacher to agree or disagree with the student’s original choices at specific tokens. This yields dense, logit-level credit assignment. For example, when provided with the feedback from Figure 3 , the self-teacher can identify how the initial attempt should be modified to avoid the runtime error. Crucially, this mechanism incurs no sampling overhead: we simply re-compute the log-probabilities of the original attempt under the self-teacher’s feedback-augmented context.
Building on this idea, we introduce Self-Distillation Policy Optimization ( SDPO ), an on-policy algorithm that performs RL via self-distillation. SDPO samples rollouts from the current policy, obtains rich environment feedback, and then minimizes a logit-level distillation loss that matches the current policy’s next-token distribution to that of the self-teacher. Conceptually, SDPO addresses the central limitation of applying distillation to online learning: the absence of a stronger external teacher. Instead of relying on a fixed teacher, SDPO leverages the model’s ability to recognize its own mistakes in hindsight. By conditioning the current policy on the rich feedback it just received, we construct a self-teacher that provides the dense supervision of distillation while retaining the exploration benefits of on-policy RL. Table 1 summarizes how this positions SDPO relative to RLVR and distillation baselines. We include a comprehensive summary of related work in Section 6 .
We show that SDPO is a policy gradient algorithm whose advantages are estimated using the self-teacher. This enables the implementation of SDPO with minor changes to standard RLVR pipelines, simply by swapping out the advantages.
##### Summary of evaluation results.
We evaluate SDPO in three online RL settings:
• Learning without rich feedback (§ 3 ): We evaluate standard RLVR environments that do not return any feedback beyond scalar rewards. Here, SDPO treats successful attempts sampled in the current batch as “feedback” for failed attempts on the same question. We perform training runs on scientific reasoning and tool use, starting with Qwen3-8B and Olmo3-7B-Instruct. We find that SDPO outperforms a strong GRPO baseline that integrates recent improvements: 70.2% vs. 66.6% final accuracy on aggregate. SDPO achieves higher accuracy with up to 11 × 11\times shorter generation lengths compared to GRPO, demonstrating that effective reasoning need not be verbose.
• Learning with rich feedback (§ 4 ): We evaluate competitive programming problems from LiveCodeBench v6 with LeetCode-style feedback. As shown in Figure 1 , SDPO substantially improves over GRPO, reaching a higher final accuracy (48.8% vs. 41.2%) and achieving GRPO’s final accuracy in 4 × 4\times fewer generations. SDPO’s gains grow with model scale, suggesting that the ability for self-teaching emerges as models become stronger in-context learners.
• Discovering novel solutions to hard tasks at test-time (§ 5 ): Finally, we demonstrate that SDPO can accelerate the discovery of solutions to difficult binary-reward questions. This contrasts with RLVR methods, which only begin learning once the first solution has been found. We leverage SDPO for Test-Time Self-Distillation , a form of test-time training where the model specializes to an individual test question. We consider very difficult LiveCodeBench questions, for which the base model’s pass@ 64 64 is below 0.03, and show that SDPO accelerates the discovery of solutions by 3 × 3\times .
## 2 SDPO: Self-Distillation Policy Optimization
We propose an algorithm that uses the in-context learning ability of the current policy for assigning credit. Our key object is the self-teacher , π θ ( ⋅ ∣ x , f ) \pi_{\theta}(\cdot\mid x,f) , which refers to the current policy (the “student”) prompted with the question x x and the rich feedback f f . Next to the students’ original attempt y y , f f may incorporate two key kinds of feedback: any environment output (such as runtime errors from a code environment) and a sample solution if x x was already solved with another attempt in the rollout group. 1 1 1 In standard RLVR implementations a rollout group contains multiple simultaneous attempts for x x . As discussed before, the self-teacher π θ ( ⋅ ∣ x , f ) \pi_{\theta}(\cdot\mid x,f) should have a higher accuracy than the student π θ ( ⋅ ∣ x ) \pi_{\theta}(\cdot\mid x) since it sees additional information in-context. This leads us to observe:
We can use the same policy in two different roles: As the student for the initial attempt and as the teacher to determine the value of actions in hindsight.
We introduce Self-Distillation Policy Optimization ( SDPO ) which repeatedly distills the self-teacher into the student. Given a question x x , we first sample rollouts from the student π θ \pi_{\theta} and obtain corresponding environment feedback. We then use the KL-divergence, KL ( p ∥ q ) = ∑ i p ( i ) log p ( i ) / q ( i ) \smash{\mathrm{KL}\left(p\|q\right)=\sum_{i}p(i)\log\nicefrac{{p(i)}}{{q(i)}}} , as a distance measure for the next-token distributions of student and teacher, and optimize a standard logit distillation loss: ℒ SDPO ( θ ) := ∑ t KL ( π θ ( ⋅ ∣ x , y < t ) ∥ stopgrad ( π θ ( ⋅ ∣ x , f , y < t ) ) ) \mathcal{L}_{\mathrm{SDPO}}(\theta):=\sum_{t}\mathrm{KL}(\pi_{\theta}(\cdot\mid x,y_{<t})\|\mathrm{stopgrad}(\pi_{\theta}(\cdot\mid x,f,y_{<t}))) (1)
where the stopgrad operator blocks gradients from flowing through the teacher, and thus prevents it from regressing towards the student and ignoring f f . The intuitive role of the teacher is to determine where and how the students’ original attempt y y was wrong through retrospection based on the feedback f f . Figure 4 shows an example of self-teaching with Qwen3-8B as student and self-teacher. We summarize SDPO in Algorithm 1 and display the teachers’ reprompt template in Table 2 .
We can derive the SDPO gradient as follows (see Section A.1 for details):
###### Proposition 2.1 .
The gradient of ℒ SDPO \mathcal{L}_{\mathrm{SDPO}} is ∇ ℒ SDPO ( θ ) = 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ ∑ t = 1 | y | 𝔼 y ^ t ∼ π θ ( ⋅ ∣ x , y < t ) [ log π θ ( y ^ t ∣ x , y < t ) π θ ( y ^ t ∣ x , f , y < t ) ⋅ ∇ θ log π θ ( y ^ t ∣ x , y < t ) ] ] . \boldsymbol{\nabla}\mathcal{L}_{\mathrm{SDPO}}(\theta)=\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\!\!\left[\sum_{t=1}^{|y|}\mathbb{E}_{\hat{y}_{t}\sim\pi_{\theta}(\cdot\mid x,y_{<t})}\!\!\left[\log\frac{\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\pi_{\theta}(\hat{y}_{t}\mid x,f,y_{<t})}\cdot\boldsymbol{\nabla}_{\!\!\theta}\,\log\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\right]\right]. (2)
### 2.1 Comparison to RLVR
Note that the SDPO gradient is a (negated) logit-level policy gradient where the advantages are estimated using the self-teacher. 2 2 2 See Section A.4 for a detailed comparison of the SDPO gradient to the standard policy gradient. We can therefore reuse standard RLVR implementations and simply swap out the advantages. Let y i y_{i} be the i i -th rollout from a rollout group of size G G for question x x , then comparing GRPO and SDPO we have: A i , t GRPO := r i − mean { r i } i = 1 G (constant in t ) , A i , t SDPO ( y ^ i , t ) = log π θ ( y ^ i , t ∣ x , f i , y i , < t ) π θ ( y ^ i , t ∣ x , y i , < t ) . A_{i,t}^{\mathrm{GRPO}}:=r_{i}-\mathrm{mean}\{r_{i}\}_{i=1}^{G}\;\text{{\color[rgb]{0.5,0.5,0.5}(constant in $t$)}},\quad A_{i,t}^{\mathrm{SDPO}}(\hat{y}_{i,t})=\log\frac{\pi_{\theta}(\hat{y}_{i,t}\mid x,f_{i},y_{i,<t})}{\pi_{\theta}(\hat{y}_{i,t}\mid x,y_{i,<t})}. The GRPO advantages are applied only to the sampled token y i , t y_{i,t} and are constant within a rollout y i y_{i} . 3 3 3 We use the GRPO ( Shao et al., 2024 ) advantage without normalization ( Liu et al., 2025b ) . In contrast, the SDPO advantages are zero only for tokens where student and teacher perfectly agree. The SDPO advantage is positive for tokens which are more likely under the teacher while being negative for tokens which are less likely under the teacher. Thus, SDPO can be seen as a direct extension of standard RLVR methods in two ways: 1. from 1-bit feedback to allowing arbitrary sequences of tokens as feedback , and
2. leveraging this rich feedback to estimate dense logit-level advantages .
This tight connection to RLVR methods also enables a straightforward extension of the SDPO gradient from Equation 2 to off-policy data via PPO-style clipped importance sampling ( Schulman et al., 2017 ) , see Section A.4 .
### 2.2 Compute time & memory
The only computational overhead of SDPO compared to GRPO is the additional computation of log-probs from the self-teacher, which can be effectively parallelized and is substantially faster than sequential generation. Figure 5 compares the compute time of SDPO and GRPO. As expected, the compute overhead of SDPO is relatively small. Here, we use a micro batch size of 2; 4 4 4 The micro batch size corresponds to # rollouts we train on at a time while accumulating gradients. compute time can be further reduced by using larger micro batch sizes.
Naively computing the KL divergence between student and teacher requires holding full logits of both models in memory. To avoid this, we approximate the KL divergence in the SDPO loss by performing top- K K distillation (i.e., only computing the top- K K logits of the student and the corresponding logits of the teacher alongside a term capturing the tail probability; cf. Section A.3 ). With a reasonable choice of K K (e.g., K = 100 {K=100} ), this avoids virtually any memory overhead while capturing most of the information.
### 2.3 Stability improvements
We find that two practical modifications significantly enhance the training stability of SDPO. First, we employ a regularized self-teacher, implemented either via an exponential moving average (EMA) of the student parameters or by interpolating the current teacher with the initial teacher (cf. Section A.2 ). As detailed later, both strategies effectively stabilize learning. Second, we adopt the symmetric Jensen-Shannon divergence for the distillation loss; this formulation has similarly been shown to improve stability in on-policy distillation from external teachers ( Agarwal et al., 2024 ) .
## 3 Learning without Rich Environment Feedback
We first evaluate SDPO in standard RLVR environments, where feedback is limited to scalar rewards. Instead of using the scalar reward, SDPO treats successful attempts sampled in the current batch as “feedback” for failed attempts on the same question. By comparing the student’s attempt with a correct solution, the self-teacher can identify where the student was wrong and provide dense credit assignment.
### 3.1 Experimental setting