Reinforcement Learning via Self-Distillation — part 4/6
Cleaned-text code points [47174, 62750). Bibliography omitted here; source.md includes it.
Empirically, SDPO demonstrates superior sample efficiency and wall-clock convergence compared to GRPO on reasoning tasks, even when training in standard RLVR environments without rich feedback. SDPO’s gains grow with model scale, suggesting that the capacity for self-correction scales with the model’s in-context learning capabilities. Moreover, we show that performing SDPO at test time on individual hard binary-reward tasks accelerates the discovery of solutions compared to strong baselines.
SDPO enables learning from rich feedback in a way that is arguably closer to human cognition: utilizing precise outcomes rather than just binary rewards. By allowing the model to determine retrospectively how it should have acted, we demonstrate that language models can convert diverse tokenized feedback into effective self-supervision.
##### Limitations.
Our findings show that SDPO’s performance depends on a model’s in-context learning ability, suggesting that SDPO is primarily applicable for RL-training stronger base models, while it can underperform GRPO on weaker models. Moreover, performance depends on the quality of the environment feedback. If the environment provides uninformative or misleading feedback, a model may not be able to learn from it through SDPO. Finally, SDPO adds a small computational overhead compared to GRPO for computing the log-probs of the retrospective model. While often negligible, this may be a larger overhead for smaller models with shorter generation lengths, where generation time is comparatively small.
##### Future Work.
Our work highlights several exciting directions for future research: • Long-horizon and agentic settings. RLRF is particularly appealing when trajectories are long or expose information about intermediate states. Evaluating SDPO in agentic environments is a natural next step.
• Training dynamics at scale. Beyond our evaluation on LiveCodeBench, it would be particularly interesting to scale SDPO to large multi-task RL training runs and further study its scaling properties with frontier base models.
• Beyond verifiable rewards. While we focused on verifiable code generation, many tasks provide textual feedback without a ground-truth verifier. Investigating whether SDPO’s retrospection mechanism can improve alignment in open-ended text generation or continuous-reward tasks remains an open empirical question.
• Behavioral differences in reasoning. We observed that SDPO induces qualitatively different reasoning patterns than GRPO, notably avoiding the latter’s tendency toward verbosity and superficial reasoning. Future work should systematically study how individual aspects, such as the reprompt template, influence behavior.
## Author Contributions
Jonas Hübotter conceived of the project in summer 2025 and has been working on it full-time since then, leading the team. Jonas proposed the conceptual framework of self-distillation for credit assignment with input from Lejs, implemented the algorithm with help from others, led the quantitative experiments on LCBv6, and led the writing of the paper.
Frederike Lübeck led the design of the code environment, led the design and evaluation of the TTT setting in Section 5 with input from Jonas, contributed to the project direction in discussions, and contributed significantly to the writing of the paper.
Lejs Behric noted the dense credit assignment of knowledge distillation with strong teacher models in discussions with Jonas, inspiring the idea of self-distillation. Further, Lejs led the evaluation of different teacher templates, co-led the development of a tool for qualitative analysis of runs with Marco and Daniel, helped implement parts of the algorithm, and contributed to the project direction in discussions.
Anton Baumann joined in December 2025 and led the evaluation of SDPO without rich feedback in Section 3 with input from Jonas, and contributed to the writing of the paper.
Marco Bagatella and Daniel Marta co-led the development of a tool for qualitative analysis of runs with Lejs, contributed to the training infrastructure, and contributed to the project direction in discussions.
Ido Hakimi significantly contributed to the initial codebase and experimental setup, contributed early algorithmic ideas, and contributed to the project direction in discussions.
Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause supported this project, with Idan and Carlos joining in December 2025. They made significant contributions to the project direction in discussions and gave valuable advice on our presentation. Thomas and Idan, in particular, significantly contributed to the development of core algorithmic ideas and design of experiments. Thomas further evaluated checkpoints on holdout benchmarks. Carlos suggested the qualitative analysis of reasoning traces in Figure 7 and the presentation of TTT results in Section 5 . Andreas pointed out valuable connections to existing work in RL which shaped the direction of the project.
## Acknowledgments
We would like to thank Akira Yoshiyama, Yassir Akram, Parnian Kassraie, Jonathan Thomm, Roman Vorushin, Afra Amini, Imanol Schlag, Yu Sun, and Moritz Hardt for helpful discussions. We thank Eduard Durech for helpful conversations regarding the scaling of RL fine-tuning and for his technical guidance on distributed infrastructure and long-context optimization. We are grateful to Ruixu Zhou from Tsinghua University & the Tencent Hunyuan Team for pointing out an error in the initially derived gradient estimator. Furthermore, we would like to thank Leander Diaz-Bone for supporting dataset generation.
This project was supported through the Swiss AI compute grant a156 and, in part, compute grant infra01. JH was supported by the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545. FL and MB were supported by the ETH-MPI Center for Learning Systems. TKB and IH were supported by an ETH AI Center Postdoctoral Fellowship. DM was supported by the Knut and Alice Wallenberg Foundation.
## Appendix A Implementation of SDPO
The following pseudocode in Figure 14 outlines the implementation of SDPO:
In the following, we provide further details on: • The gradient estimator used in our implementation ( Section A.1 )
• Teacher regularization ( Section A.2 )
• Approximating logit-distillation with the top- K K logits for saving GPU memory ( Section A.3 )
• Generalizing PPO-style policy gradient algorithms to logit-level advantages ( Section A.4 )
To disambiguate the notation of the self-teacher, we use q θ ( ⋅ ∣ x , f ) := π θ ( ⋅ ∣ reprompt ( x , f ) ) q_{\theta}(\cdot\mid x,f):=\pi_{\theta}(\cdot\mid\mathrm{reprompt}(x,f)) in the following. Here, reprompt denotes the reprompt template of the self-teacher.
### A.1 Gradient Estimators
In this seciton, we discuss two possible gradient estimators for the KL divergence between the current policy π θ ( y ∣ x ) \pi_{\theta}(y\mid x) and the teacher policy q θ ( y ∣ x , f ) q_{\theta}(y\mid x,f) .
##### Per-token estimator.
Deriving the gradient of the SDPO loss as defined in Equation 1 : ℒ token ( θ ) := 𝔼 y ∼ stopgrad ( π θ ( ⋅ ∣ x ) ) [ ∑ t = 1 T KL ( π θ ( ⋅ ∣ x , y < t ) ∥ stopgrad ( π θ ( ⋅ ∣ x , f , y < t ) ) ) ] \mathcal{L}_{\mathrm{token}}(\theta):=\mathbb{E}_{y\sim\mathrm{stopgrad}(\pi_{\theta}(\cdot\mid x))}\left[\sum_{t=1}^{T}\mathrm{KL}(\pi_{\theta}(\cdot\mid x,y_{<t})\|\mathrm{stopgrad}(\pi_{\theta}(\cdot\mid x,f,y_{<t})))\right] (5) leads to the following estimator (see a detailed proof in Section B.1 ), which corresponds to the sum of gradients of the KL divergence at each token: ∇ ℒ token ( θ ) = 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ ∑ t = 1 T 𝔼 y ^ t ∼ π θ ( ⋅ ∣ x , y < t ) [ ∇ θ log π θ ( y ^ t ∣ x , y < t ) ⋅ log π θ ( y ^ t ∣ x , y < t ) π θ ( y ^ t ∣ x , f , y < t ) ] ] . \boldsymbol{\nabla}\mathcal{L}_{\text{token}}(\theta)=\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}\mathbb{E}_{\hat{y}_{t}\sim\pi_{\theta}(\cdot\mid x,y_{<t})}\!\!\left[\boldsymbol{\nabla}_{\!\!\theta}\,\log\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\cdot\log\frac{\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\pi_{\theta}(\hat{y}_{t}\mid x,f,y_{<t})}\right]\right]. (6) This corresponds to the estimator presented in Proposition 2.1 . This gradient estimator effectively assumes that the sampling distribution generating y y is fixed.
##### Sequence-level estimator.
An alternative self-distillation objective minimizes the sequence-level KL divergence between student and self-teacher, i.e., ℒ seq ( θ ) := KL ( π θ ∥ q θ ) = 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ log π θ ( y ∣ x ) q θ ( y ∣ x , f ) ] = ∑ t = 1 T 𝔼 s t ∼ Π θ [ KL ( π θ ( ⋅ ∣ s t ) ∥ q θ ( ⋅ ∣ s t , f ) ) ] , \displaystyle\begin{split}\mathcal{L}_{\mathrm{seq}}(\theta):=\mathrm{KL}\left(\pi_{\theta}\|q_{\theta}\right)&=\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\log\frac{\pi_{\theta}(y\mid x)}{q_{\theta}(y\mid x,f)}\right]\\ &=\sum_{t=1}^{T}\mathbb{E}_{s_{t}\sim\Pi_{\theta}}\left[\mathrm{KL}\left(\pi_{\theta}(\cdot\mid s_{t})\|q_{\theta}(\cdot\mid s_{t},f)\right)\right],\end{split} (7) where s t = ( x , y < t ) s_{t}=(x,y_{<t}) is the prefix (“state”) at step t t and Π θ \Pi_{\theta} denotes the prefix distribution under policy π θ \pi_{\theta} . Estimating the gradient of this objective additionally takes into account how the choice of y t y_{t} influences future states y > t y_{>t} (due to the additional dependence on Π θ \Pi_{\theta} ).
Amini et al. (2025) show that the corresponding gradient estimator is given by ∇ ℒ seq ( θ ) = ∇ ℒ token ( θ ) + 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ ∑ t = 1 T KL ( π θ ( ⋅ ∣ s t ) ∥ q θ ( ⋅ ∣ s t , f ) ) ∇ θ log Π θ ( s t ) ] . \boldsymbol{\nabla}\mathcal{L}_{\text{seq}}(\theta)=\boldsymbol{\nabla}\mathcal{L}_{\text{token}}(\theta)+\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}\mathrm{KL}\left(\pi_{\theta}(\cdot\mid s_{t})\|q_{\theta}(\cdot\mid s_{t},f)\right)\boldsymbol{\nabla}_{\!\!\theta}\,\log\Pi_{\theta}(s_{t})\right]. (8)
The additional term of the sequence-level gradient captures how prefixes influence the self-distillation divergence of future tokens. We also experimented with this sequence-level gradient estimator but did not find measurable gains relative to its additional complexity.
### A.2 Regularized teacher
In contrast to standard distillation, the teacher in SDPO changes throughout training. This bootstrapping enables the teacher to improve, but it may also lead to training instability. To stabilize training, we seek to prevent the teacher q q from quickly diverging from the initial teacher q θ ref \smash{q_{\theta_{{\mathrm{ref}}}}} . We can achieve this by placing an explicit trust-region constraint on q q ( Schulman et al., 2015 ; Peng et al., 2019 ) , that is: ∑ t KL ( q ( y t ∣ x , f , y < t ) ∥ q θ ref ( y t ∣ x , f , y < t ) ) ≤ ϵ , ϵ > 0 . \sum_{t}\mathrm{KL}\left(q(y_{t}\mid x,f,y_{<t})\|q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})\right)\leq\epsilon,\quad\epsilon>0. (9) This trust-region can be implemented in two ways: 1. Explicit trust-region: We can define the teacher as the policy closest to q θ q_{\theta} while satisfying the trust-region constraint. This teacher can be expressed as q ( y t ∣ x , f , y < t ) ∝ exp ( ( 1 − α ) log q θ ref ( y t ∣ x , f , y < t ) + α log q θ ( y t ∣ x , f , y < t ) ) , q(y_{t}\mid x,f,y_{<t})\propto\exp\!\big((1-\alpha)\log q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})+\alpha\log q_{\theta}(y_{t}\mid x,f,y_{<t})\big), (10) with α ∈ ( 0 , 1 ) \alpha\in(0,1) the inverse Lagrange multiplier for the trust-region constraint. We include a full derivation in Section B.2 . We can plug this explicitly constrained teacher directly into the SDPO objective.
2. Exponential moving average (EMA): Alternatively, we can stabilize the teacher’s parameters directly; parameterizing q θ ′ q_{\theta^{\prime}} by θ ′ \theta^{\prime} and updating as θ ′ ← ( 1 − α ) θ ′ + α θ \theta^{\prime}\leftarrow(1-\alpha)\theta^{\prime}+\alpha\theta with α ∈ ( 0 , 1 ) \alpha\in(0,1) .
Note that each implementation has a different practical advantage: The EMA teacher requires additional GPU memory for θ ′ \theta^{\prime} yet does not introduce any runtime overhead. In contrast, the trust-region teacher requires an additional log-prob computation with q θ ref \smash{q_{\theta_{{\mathrm{ref}}}}} yet does not require additional GPU memory if θ ref {\theta_{{\mathrm{ref}}}} is used for explicit KL regularization.
### A.3 Approximate Logit Distillation
To save GPU memory, we perform distillation only on the top- K K tokens predicted by the student:
ℒ SDPO ( θ ) \displaystyle\mathcal{L}_{\mathrm{SDPO}}(\theta) = ∑ t = 1 T KL ( π θ ( ⋅ ∣ x , y < t ) ∥ stopgrad ( q θ ( ⋅ ∣ x , f , y < t ) ) ) \displaystyle=\sum_{t=1}^{T}\mathrm{KL}(\pi_{\theta}(\cdot\mid x,y_{<t})\|\mathrm{stopgrad}(q_{\theta}(\cdot\mid x,f,y_{<t}))) ≈ ∑ t = 1 T ∑ y ^ t ∈ top K ( π θ ) π θ ( y ^ t ∣ x , y < t ) ⋅ log π θ ( y ^ t ∣ x , y < t ) stopgrad ( q θ ( y ^ t ∣ x , f , y < t ) ) + ( 1 − ∑ y ^ t ∈ top K ( π θ ) π θ ( y ^ t ∣ x , y < t ) ) ⋅ log 1 − ∑ y ^ t ∈ top K ( π θ ) π θ ( y ^ t ∣ x , y < t ) stopgrad ( 1 − ∑ y ^ t ∈ top K ( π θ ) q θ ( y ^ t ∣ x , f , y < t ) ) ⏟ tail \displaystyle\approx\begin{multlined}\sum_{t=1}^{T}\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\cdot\log\frac{\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\mathrm{stopgrad}(q_{\theta}(\hat{y}_{t}\mid x,f,y_{<t}))}\\ +\underbrace{\Big(1-\textstyle\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\Big)\cdot\log\frac{1-\textstyle\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\mathrm{stopgrad}\Big(1-\textstyle\sum_{\hat{y}_{t}\in\mathrm{top}_{K}(\pi_{\theta})}q_{\theta}(\hat{y}_{t}\mid x,f,y_{<t})\Big)}}_{\text{tail}}\end{multlined}
Here, the top- K K is with respect to student. Without top- K K distillation, we would have to keep two copies of logits in memory: one for teacher and student each. Top- K K distillation avoids virtually any memory overhead without impacting performance significantly, since most tokens of the vocabulary are not informative at a given time.
### A.4 Off-Policy Training: Generalization to Logit-Level Losses
PPO-style clipping ( Schulman et al., 2017 ) with truncated importance sampling ( Yao et al., 2025 ) , clip-higher ( Yu et al., 2025 ) , fixed length normalization ( Liu et al., 2025b ) : ℒ token ( θ ) := − 1 ∑ i = 1 G | y i | ∑ i = 1 G ∑ t = 1 | y i | min ( w i , t TIS , ρ ) min ( w i , t A i , t , clip ( w i , t , 1 − ε low , 1 + ε high ) A i , t ) , \mathcal{L}_{\mathrm{token}}(\theta):=-{\color[rgb]{1,0.5,0}\frac{1}{\sum_{i=1}^{G}|y_{i}|}}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}{\color[rgb]{0.9492,0.3281,0.3555}\min\left(w^{\mathrm{TIS}}_{i,t},\rho\right)}\min\left(w_{i,t}A_{i,t},\text{clip}(w_{i,t},1-\varepsilon_{\text{low}},1+{\color[rgb]{0.3477,0.7344,0.168}\varepsilon_{\text{high}}})A_{i,t}\right), (13) with w i , t := π θ ( y i , t ∣ x , y i , < t ) π θ old ( y i , t ∣ x , y i , < t ) w_{i,t}:=\frac{\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{{\mathrm{old}}}}(y_{i,t}\mid x,y_{i,<t})} , w i , t TIS := π θ old ( y i , t ∣ x , y i , < t ) π θ old rollout ( y i , t ∣ x , y i , < t ) w^{\mathrm{TIS}}_{i,t}:=\frac{\pi_{\theta_{{\mathrm{old}}}}(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{{\mathrm{old}}}}^{\mathrm{rollout}}(y_{i,t}\mid x,y_{i,<t})} , and A i , t A_{i,t} denotes the per-token advantage.