Reinforcement Learning via Self-Distillation — part 5/6
Cleaned-text code points [62750, 78711). Bibliography omitted here; source.md includes it.
We extend this to a logit-level loss: ℒ logit ( θ ) := − 1 ∑ i = 1 G | y i | ∑ i = 1 G ∑ t = 1 | y i | ∑ y ^ i , t min ( π θ old ( y ^ i , t ∣ x , y i , < t ) , ρ π θ old rollout ( y ^ i , t ∣ x , y i , < t ) ) min ( w i , t ( y ^ i , t ) A i , t ( y ^ i , t ) , clip ( w i , t ( y ^ i , t ) , 1 − ε low , 1 + ε high ) A i , t ( y ^ i , t ) ) , \begin{multlined}\mathcal{L}_{\mathrm{logit}}(\theta):=-{\color[rgb]{1,0.5,0}\frac{1}{\sum_{i=1}^{G}|y_{i}|}}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}{\color[rgb]{0.168,0.3125,0.668}\sum_{\hat{y}_{i,t}}}\ {\color[rgb]{0.9492,0.3281,0.3555}\min\left(\pi_{\theta_{{\mathrm{old}}}}(\hat{y}_{i,t}\mid x,y_{i,<t}),\rho\pi_{\theta_{{\mathrm{old}}}}^{\mathrm{rollout}}(\hat{y}_{i,t}\mid x,y_{i,<t})\right)}\\ \min\left(w_{i,t}(\hat{y}_{i,t})A_{i,t}(\hat{y}_{i,t}),\text{clip}(w_{i,t}(\hat{y}_{i,t}),1-\varepsilon_{\text{low}},1+{\color[rgb]{0.3477,0.7344,0.168}\varepsilon_{\text{high}}})A_{i,t}(\hat{y}_{i,t})\right),\end{multlined} (14) where y ^ i , t \hat{y}_{i,t} sums over all possible tokens at position t t for rollout i i (or the K K most likely under π θ old \pi_{\theta_{{\mathrm{old}}}} , cf. Section A.3 ). The TIS changes since we explicitly weight each logit by its probability under π θ old \pi_{\theta_{{\mathrm{old}}}} rather than relying on a Monte Carlo estimate of the expectation over next-token predictions. Here, A i , t ( y ^ i , t ) A_{i,t}(\hat{y}_{i,t}) is a per-logit advantage.
In our experiments for SDPO, we apply the TIS term on a token-level rather than logit-level.
## Appendix B Theoretical Analysis
This section is organized as follows: • Section B.1 derives the SDPO gradient from Proposition 2.1 .
• Section B.2 derives the trust-region regularized teacher discussed in Section A.2 .
To disambiguate the notation of the self-teacher, we use q θ ( ⋅ ∣ x , f ) := π θ ( ⋅ ∣ reprompt ( x , f ) ) q_{\theta}(\cdot\mid x,f):=\pi_{\theta}(\cdot\mid\mathrm{reprompt}(x,f)) in the following. Here, reprompt denotes the reprompt template of the self-teacher.
### B.1 Proof of Proposition 2.1 .
###### Proof.
In the following, we derive the gradient of ℒ SDPO \mathcal{L}_{\mathrm{SDPO}} . ∇ θ ℒ SDPO ( θ ) \displaystyle\boldsymbol{\nabla}_{\!\!\theta}\,\mathcal{L}_{\mathrm{SDPO}}(\theta) = ∇ θ ∑ t = 1 T KL ( π θ ( ⋅ ∣ x , y < t ) ∥ stopgrad ( q θ ( ⋅ ∣ x , f , y < t ) ) ) \displaystyle=\boldsymbol{\nabla}_{\!\!\theta}\,\sum_{t=1}^{T}\mathrm{KL}(\pi_{\theta}(\cdot\mid x,y_{<t})\|\mathrm{stopgrad}(q_{\theta}(\cdot\mid x,f,y_{<t}))) = ∇ θ ∑ t = 1 T ∑ y ^ t π θ ( y ^ t ∣ x , y < t ) log ( π θ ( y ^ t ∣ x , y < t ) stopgrad ( q θ ( y ^ t ∣ x , f , y < t ) ) ) \displaystyle=\boldsymbol{\nabla}_{\!\!\theta}\,\sum_{t=1}^{T}\sum_{\hat{y}_{t}}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\log\left(\frac{\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}{\mathrm{stopgrad}(q_{\theta}(\hat{y}_{t}\mid x,f,y_{<t}))}\right) Let A t , k := log ( stopgrad ( q θ ( y ^ t ∣ x , f , y < t ) ) π θ ( y ^ t ∣ x , y < t ) ) A_{t,k}:=\log\left(\frac{\mathrm{stopgrad}(q_{\theta}(\hat{y}_{t}\mid x,f,y_{<t}))}{\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}\right) . Then, = − ∇ θ ∑ t = 1 T ∑ y ^ t π θ ( y ^ t ∣ x , y < t ) A t , k \displaystyle=-\boldsymbol{\nabla}_{\!\!\theta}\,\sum_{t=1}^{T}\sum_{\hat{y}_{t}}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})A_{t,k} = − ∑ t = 1 T ∑ y ^ t π θ ( y ^ t ∣ x , y < t ) ∇ θ A t , k + A t , k ∇ θ π θ ( y ^ t ∣ x , y < t ) . \displaystyle=-\sum_{t=1}^{T}\sum_{\hat{y}_{t}}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\boldsymbol{\nabla}_{\!\!\theta}\,A_{t,k}+A_{t,k}\boldsymbol{\nabla}_{\!\!\theta}\,\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t}). We have that ∇ θ A t , k = − ∇ θ log π θ ( y ^ t ∣ x , y < t ) \boldsymbol{\nabla}_{\!\!\theta}\,A_{t,k}=-\boldsymbol{\nabla}_{\!\!\theta}\,\log\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t}) is the negative score function. Using the score trick, π θ ( y ^ t ∣ x , y < t ) ∇ θ log π θ ( y ^ t ∣ x , y < t ) = ∇ θ π θ ( y ^ t ∣ x , y < t ) \pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\boldsymbol{\nabla}_{\!\!\theta}\,\log\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})=\boldsymbol{\nabla}_{\!\!\theta}\,\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t}) . Hence, the first term simplifies to − ∑ t = 1 T ∑ y ^ t π θ ( y ^ t ∣ x , y < t ) ∇ θ A t , k \displaystyle-\sum_{t=1}^{T}\sum_{\hat{y}_{t}}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\boldsymbol{\nabla}_{\!\!\theta}\,A_{t,k} = ∑ t = 1 T ∑ y ^ t ∇ θ π θ ( y ^ t ∣ x , y < t ) = ∑ t = 1 T ∇ θ ∑ y ^ t π θ ( y ^ t ∣ x , y < t ) ⏟ = 1 = 0 . \displaystyle=\sum_{t=1}^{T}\sum_{\hat{y}_{t}}\boldsymbol{\nabla}_{\!\!\theta}\,\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})=\sum_{t=1}^{T}\boldsymbol{\nabla}_{\!\!\theta}\,\underbrace{\sum_{\hat{y}_{t}}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})}_{=1}=0. Thus, the gradient of ℒ SDPO \mathcal{L}_{\mathrm{SDPO}} is ∇ θ ℒ SDPO \displaystyle\boldsymbol{\nabla}_{\!\!\theta}\,\mathcal{L}_{\mathrm{SDPO}} = − ∑ t = 1 T ∑ y ^ t A t , k ∇ θ π θ ( y ^ t ∣ x , y < t ) \displaystyle=-\sum_{t=1}^{T}\sum_{\hat{y}_{t}}A_{t,k}\boldsymbol{\nabla}_{\!\!\theta}\,\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t}) = − ∑ t = 1 T ∑ y ^ t π θ ( y ^ t ∣ x , y < t ) ( A t , k ∇ θ log π θ ( y ^ t ∣ x , y < t ) ) \displaystyle=-\sum_{t=1}^{T}\sum_{\hat{y}_{t}}\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\Big(A_{t,k}\boldsymbol{\nabla}_{\!\!\theta}\,\log\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\Big) = − ∑ t = 1 T 𝔼 y ^ t ∼ π θ ( ⋅ ∣ x , y < t ) [ A t , k ∇ θ log π θ ( y ^ t ∣ x , y < t ) ] . \displaystyle=-\sum_{t=1}^{T}\mathbb{E}_{\hat{y}_{t}\sim\pi_{\theta}(\cdot\mid x,y_{<t})}\left[A_{t,k}\boldsymbol{\nabla}_{\!\!\theta}\,\log\pi_{\theta}(\hat{y}_{t}\mid x,y_{<t})\right]. ∎
Notably, the above implies that the gradient of ℒ SDPO \mathcal{L}_{\mathrm{SDPO}} is equivalent to the gradient of the loss if A t , k = stopgrad ( log q θ ( y t ∣ x , f , y < t ) π θ ( y t ∣ x , y < t ) ) A_{t,k}=\mathrm{stopgrad}\left(\log\frac{q_{\theta}(y_{t}\mid x,f,y_{<t})}{\pi_{\theta}(y_{t}\mid x,y_{<t})}\right) .
### B.2 Trust-region Teacher
To stabilize training, we seek to prevent the teacher q q from diverging from the initial teacher q θ ref q_{\theta_{{\mathrm{ref}}}} . We can achieve this by placing an explicit trust-region constraint on the teacher q q ( Schulman et al., 2015 ; Peng et al., 2019 ) , that is: ∑ t KL ( q ( y t ∣ x , f , y < t ) ∥ q θ ref ( y t ∣ x , f , y < t ) ) ≤ ϵ , ϵ > 0 . \sum_{t}\mathrm{KL}\left(q(y_{t}\mid x,f,y_{<t})\|q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})\right)\leq\epsilon,\quad\epsilon>0. (15)
In the following, we derive a teacher q q which satisfies the trust-region constraint while staying close to the target q θ q_{\theta} . The following optimization problem characterizes such a q q ( Peng et al., 2019 ) : arg max q ∈ Δ ∑ t ∑ y t q ( y t ∣ x , f , y < t ) log q θ ( y t ∣ x , f , y < t ) q θ ref ( y t ∣ x , f , y < t ) s.t. ∑ t KL ( q ( y t ∣ x , f , y < t ) ∥ q θ ref ( y t ∣ x , f , y < t ) ) ≤ ϵ , \displaystyle\begin{split}\argmax_{q\in\Delta}\ &\sum_{t}\sum_{y_{t}}q(y_{t}\mid x,f,y_{<t})\log\frac{q_{\theta}(y_{t}\mid x,f,y_{<t})}{q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})}\\ \text{s.t.}\ &\sum_{t}\mathrm{KL}\left(q(y_{t}\mid x,f,y_{<t})\|q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})\right)\leq\epsilon,\end{split} (16) where Δ \Delta denotes the probability simplex. Intuitively, the solution is the q q satisfying the trust-region constraint, which is closest to q θ q_{\theta} (i.e., has minimal cross-entropy to q θ q_{\theta} ) while being farthest from q θ ref q_{\theta_{{\mathrm{ref}}}} (i.e., has maximal cross-entropy to q θ ref q_{\theta_{{\mathrm{ref}}}} ).
###### Proposition B.1 .
The solution to Equation 16 can be expressed in closed form as q ∗ ( y t ∣ x , f , y < t ) ∝ exp ( ( 1 − α ) log q θ ref ( y t ∣ x , f , y < t ) + α log q θ ( y t ∣ x , f , y < t ) ) . \displaystyle q^{*}(y_{t}\mid x,f,y_{<t})\propto\exp\!\big((1-\alpha)\log q_{\theta_{{\mathrm{ref}}}}(y_{t}\mid x,f,y_{<t})+\alpha\log q_{\theta}(y_{t}\mid x,f,y_{<t})\big). (17)
###### Proof.
To simplify notation, we omit the conditioning in the following. The Lagrangian (with λ ≥ 0 \lambda\geq 0 for the KL constraint and ν \nu for normalization) is ℒ ( q , λ , ν ) = ∑ t ∑ y t q ( y t ) log q θ ( y t ) q θ ref ( y t ) − λ ( ∑ y t q ( y t ) log q ( y t ) q θ ref ( y t ) − ϵ ) + ν ( ∑ y t q ( y t ) − 1 ) . \displaystyle\mathcal{L}(q,\lambda,\nu)=\sum_{t}\sum_{y_{t}}q({y_{t}})\log\frac{q_{\theta}({y_{t}})}{q_{\theta_{{\mathrm{ref}}}}({y_{t}})}-\lambda\Big(\sum_{y_{t}}q({y_{t}})\log\frac{q({y_{t}})}{q_{\theta_{{\mathrm{ref}}}}({y_{t}})}-\epsilon\Big)+\nu\Big(\sum_{y_{t}}q({y_{t}})-1\Big). Stationarity gives, for all y t y_{t} , 0 = ∂ ℒ ∂ q ( y t ) = log q θ ( y t ) q θ ref ( y t ) − λ ( log q ( y t ) q θ ref ( y t ) + 1 ) + ν . \displaystyle 0=\frac{\partial\mathcal{L}}{\partial q(y_{t})}=\log\frac{q_{\theta}(y_{t})}{q_{\theta_{{\mathrm{ref}}}}(y_{t})}-\lambda\Big(\log\frac{q(y_{t})}{q_{\theta_{{\mathrm{ref}}}}(y_{t})}+1\Big)+\nu. Let α := 1 / λ \alpha:=1/\lambda . Then, the solution to Equation 16 can be characterized in closed form as q ∗ ( y t ) \displaystyle q^{*}(y_{t}) ∝ q θ ref ( y t ) exp ( α log q θ ( y t ) q θ ref ( y t ) ) \displaystyle\propto q_{\theta_{{\mathrm{ref}}}}(y_{t})\exp\!\Big(\alpha\log\tfrac{q_{\theta}(y_{t})}{q_{\theta_{{\mathrm{ref}}}}(y_{t})}\Big) ∝ exp ( ( 1 − α ) log q θ ref ( y t ) + α log q θ ( y t ) ) . \displaystyle\propto\exp\!\big((1-\alpha)\log q_{\theta_{{\mathrm{ref}}}}(y_{t})+\alpha\log q_{\theta}(y_{t})\big). ∎
Chen et al. (2025c) perform a similar derivation, but use reference π θ ref \pi_{{\theta_{{\mathrm{ref}}}}} , which we observe to underperform compared to the reference q θ ref q_{\theta_{{\mathrm{ref}}}} .
## Appendix C Additional Related Work
##### Value networks and Monte Carlo advantage estimation.
Several prior approaches aim to improve credit assignment but face the same information bottleneck as GRPO. Classical RL frequently trains value networks which provide token-level advantages, but themselves are learned from scalar rewards ( Schulman et al., 2016 ; Schulman et al., 2017 ) . Furthermore, value networks incur significant computational and memory overhead and are therefore typically not used to train LLMs. Other recent work estimates token-level advantages by performing additional generations starting from various positions in the original attempt ( Kazemnejad et al., 2025 ; Zheng et al., 2025b ) . While this can learn with fewer gradient steps than GRPO it still uses only scalar rewards as signal and requires costly additional generations.
##### Dense credit assignment with a reward model.
Several recent works study dense (per-token) reward assignment given access to an external reward model, typically by exploiting the reward model’s internal structure ( Chan et al., 2024 ; Cao et al., 2025b ) . Relatedly, Li et al. (2025b) argue that a token-level reward signal is implicit in an LLM’s logits by linking next-token prediction to offline inverse reinforcement learning, effectively yielding a training-free reward model for RL fine-tuning.
##### Partial observability.
From the perspective of classical RL, many verifiable domains for LLMs are naturally partially observable : executing a proposed solution induces a latent environment state (e.g., failing tests or states of an agentic system) that is revealed only through rich feedback. This aligns with the formalism of partially observable Markov decision processes (POMDPs), where agents must act under incomplete observations of state ( Kaelbling et al., 1998 ; Sutton & Barto, 1998 ) . By contrast, RLVR and RLHF pipelines typically discard this observation channel and learn only from terminal scalar rewards or pairwise preferences.
##### Relation to test-time training.
Our setting from Section 5 can be seen as a special case of test-time training where the model itself is updated at test-time using self-distillation. Updating the model at test-time is known as test-time training ( Sun et al., 2020 ; Sun et al., 2025 ; Hardt & Sun, 2024 ; Hübotter et al., 2025a ; Hübotter et al., 2025b ; Akyürek et al., 2025 ; Behrouz et al., 2025 ; Tandon et al., 2025 ; Hübotter et al., 2026 ) . Unlike prior work, self-distillation uses the in-context learning ability of the current model to attribute credit after receiving feedback. This can be seen as simulating long-context reasoning with periodic compression of context into the model weights.
### C.1 SDPO as Maximum Entropy RL
The SDPO objective resembles the objective in maximum entropy RL ( Levine, 2018 ; Haarnoja et al., 2018 , e.g.,) with a particular choice of reward function.
##### Maximum Entropy RL
Consider optimizing arg max θ 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ ∑ t r ( y t ∣ x , y < t ) ] + λ H [ π θ ( ⋅ ∣ x ) ] , λ > 0 \argmax_{\theta}\ \mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}{}\left[\sum_{t}r(y_{t}\mid x,y_{<t})\right]+\lambda\mathrm{H}\left[\pi_{\theta}(\cdot\mid x)\right],\quad\lambda>0 (18) where π θ ( y ∣ x ) = ∏ t = 1 T π θ ( y t ∣ x , y < t ) \smash{\pi_{\theta}(y\mid x)=\prod_{t=1}^{T}\pi_{\theta}(y_{t}\mid x,y_{<t})} and H [ π θ ( ⋅ ∣ x ) ] = 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ − log π θ ( y ∣ x ) ] \smash{\mathrm{H}\left[\pi_{\theta}(\cdot\mid x)\right]=\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}{}\left[-\log\pi_{\theta}(y\mid x)\right]} is the entropy of the policy. Here, r ( y t ∣ x , y < t ) r(y_{t}\mid x,y_{<t}) is an arbitrary reward function, possibly “dense” (i.e., per-token). Equation 18 is known as maximum entropy RL. It is known that this objective is equivalent to solving a variational inference problem which discuss next.
To this end, we define a Bernoulli random variable 𝒞 \mathcal{C} which is 1 1 if the attempt y y is correct and 0 0 otherwise. We then define its distribution as p ( 𝒞 = 1 ∣ x , y ) ∝ exp ( 1 λ ∑ t r ( y t ∣ x , y < t ) ) \smash{p(\mathcal{C}=1\mid x,y)\propto\exp(\tfrac{1}{\lambda}\sum_{t}r(y_{t}\mid x,y_{<t}))} . Further assuming w.l.o.g. that the “prior” over responses is uniform, we can express the posterior conditioned on the event of correctness as π ⋆ ( y ∣ x ) := p ( y ∣ x , 𝒞 = 1 ) ∝ p ( 𝒞 = 1 ∣ x , y ) ∝ exp ( 1 λ ∑ t r ( y t ∣ x , y < t ) ) . \pi^{\star}(y\mid x):=p(y\mid x,\mathcal{C}=1)\propto p(\mathcal{C}=1\mid x,y)\propto\exp\!\left(\frac{1}{\lambda}\sum_{t}r(y_{t}\mid x,y_{<t})\right). (19) Then, Equation 18 is equivalent to minimizing the KL divergence with respect to π ⋆ \pi^{\star} : arg min θ ∑ t KL ( π θ ( y t ∣ x , y < t ) ∥ π ⋆ ( y t ∣ x , y < t ) ) . \argmin_{\theta}\ \sum_{t}\mathrm{KL}\left(\pi_{\theta}(y_{t}\mid x,y_{<t})\|\pi^{\star}(y_{t}\mid x,y_{<t})\right). (20)
##### SDPO optimizes an implicit reward defined by the teacher
Note that Equation 20 is equivalent to the SDPO objective ( Equation 1 ) with implicit reward r ( y t ∣ x , y < t ) = log q ( y t ∣ x , f , y < t ) r(y_{t}\mid x,y_{<t})=\log q(y_{t}\mid x,f,y_{<t}) and λ = 1 \lambda=1 . In this sense, SDPO can be seen as a maximum entropy RL algorithm with dense rewards constructed implicitly through the retrospective model.
This also points to a connection of SDPO to inverse RL ( Ng et al., 2000 ; Ziebart et al., 2008 ; Rafailov et al., 2023 ) , where the goal is to recover an unknown reward function. In SDPO, the student learns an implicit reward function defined by the retrospective model.
## Appendix D Additional Results & Ablations
This section is organized as follows: • Section D.1 contains results and ablations for Section 3 .
• Section D.2 contains results and ablations for Section 4 .
• Section D.3 contains results and ablations for Section 5 .
### D.1 Learning without rich environment feedback
• Table 7 reports results when optimal hyperparameters are selected for each model/task combination.
• Table 8 compares average response lengths of SDPO and GRPO.