Reinforcement Learning via Self-Distillation — part 3/6

Cleaned-text code points [31512, 47174). Bibliography omitted here; source.md includes it.

We summarize results in Table 6 where we evaluate the effect on SDPO training as well as the direct impact on the self-teacher. We find that environment output & sample solutions are complementary, each providing informative feedback. Generally, we observe that performance is not sensitive to syntactic variations of the reprompting template from Table 2 .

## 5 Solving Hard Questions via Test-Time Self-Distillation

In Sections 3 and 4 , we have demonstrated that SDPO can substantially improve over RLVR methods when performing “train-time RL” for reasoning tasks. We now turn to a test-time setting where the model is given only a single hard (binary-reward) question x x and must discover a solution as quickly as possible:

###### Definition 5.1 (Discovery time) .

The discovery time is the number of trials needed until a solution is found (i.e., the smallest k k with the k k -th attempt y k y_{k} receiving reward 1).

Based on this notion, we can define a measure of the efficacy of discovery: discovery ​ @ ​ k := ℙ ⁡ ( discovery time ≤ k ) = ℙ ⁡ ( r ⁡ ( y 1 ∣ x ) = 1 or r ⁡ ( y 2 ∣ x ) = 1 or …or r ⁡ ( y k ∣ x ) = 1 ) , \displaystyle\begin{split}\mathrm{discovery@}k:=&\ \mathbb{P}(\text{discovery time $\leq k$})\\ =&\ \mathbb{P}(\text{$r(y_{1}\mid x)=1$ or $r(y_{2}\mid x)=1$ or \ldots or $r(y_{k}\mid x)=1$}),\end{split} (4) where the probability is over any randomness in the algorithm producing y k y_{k} and the rewards. Thus, the discovery@ k k metric quantifies the probability of discovering the solution within k k steps. 8 8 8 Our proposed discovery@ k k metric is a canonical metric in the study of runtime speedup (i.e., time until termination, Dolan & Moré (2002) ). While prior work has studied discovery with continuous rewards ( Novikov et al., 2025 ; Yuksekgonul et al., 2026 , e.g.,) , discovery with language models in sparse or binary-reward settings does not allow “hill-climbing” a continuous reward and has remained less well understood.

The most naive approach to discovery in binary-reward tasks is to sample repeatedly i.i.d. from the base model, also known as best-of- k k . The canonical pass@ k k metric for best-of- k k sampling is exactly the probability of discovering at least one solution within k k independent samples from a fixed model, coinciding with discovery@ k k . The discovery@ k k metric generalizes pass@ k k to algorithms that sample attempts sequentially. A common sequential approach re-prompts the base model with additional context from previous attempts ( Madaan et al., 2023 ; Shinn et al., 2023 ) . We refer to this as multi-turn sampling. Here, the model itself does not change, only its context evolves over time.

Performing RLVR on the question x x does not improve over best-of- k k sampling from the base model, since a binary reward provides no signal until the first solution has already been found. 9 9 9 For this reason, several works consider explicitly constructing curricula of solvable questions ( Zhao et al., 2025 ; Huang et al., 2026 ; Diaz-Bone et al., 2025 ; Hübotter et al., 2025b , e.g.,) , which self-distillation avoids. Other work found that RLVR yields limited improvement on hard questions ( Yue et al., 2025 ) . An RLRF method like SDPO does not face the same limitation, as it receives rich feedback from the environment after each attempt. This rich feedback enables the model to repeatedly “correct” its mistakes as it encounters them and receives feedback, even before ever discovering a solution. In contrast to multi-turn sampling, SDPO repeatedly compresses context c = ( y k , f k ) c=(y_{k},f_{k}) by distilling π θ ( ⋅ ∣ x , c ) \pi_{\theta}(\cdot\mid x,c) into a model π θ ′ ( ⋅ ∣ x ) \pi_{\theta^{\prime}}(\cdot\mid x) as we illustrate in Figure 12 . This self-distillation enables SDPO to continually learn over long contexts, whereas the memory bottleneck of transformers inherently limits the context length of multi-turn sampling ( Vaswani et al., 2017 ) . In this section, we seek to answer the question: Can repeatedly compressing context into model weights via self-distillation accelerate discovery for hard questions?

### 5.1 Experimental setting

We consider a particularly challenging subset of questions from LCBv6 that are at Qwen3-8B’s performance ceiling and require significant test-time sampling to find any solution. Concretely, we define two groups using Qwen3-8B’s pass@ k k : Hard tasks with pass@ ​ 64 < 0.5 {\text{pass@}64<0.5} and very hard tasks with pass@ ​ 64 < 0.03 \text{pass@}64<0.03 . Among these, we retain questions for which any of best-of- k k , multi-turn, or SDPO find at least one solution within 512 512 steps across 5 5 seeds. This results in 19 hard and 9 very hard questions.

For best-of- k k sampling under the base model, we report the standard pass ​ @ ​ k \text{pass}@k estimate ( Chen et al., 2021b ) from 2944 independent rollouts. As multi-turn sampling, we sequentially reprompt the model in-context using the concatenated feedback from previous attempts. To remain within Qwen3-8B’s 40k-token context limit, we employ a first-in, first-out sliding window, discarding the earliest feedback once the maximum prompt length (32k tokens) is reached. We ablate the multi-turn reprompting strategy in Figure 19 in Appendix D and find that retaining only past feedback while forgetting earlier attempts significantly outperforms the baseline that additionally retains past attempts. We evaluate SDPO with a batch size of 16. We ablate this choice in Figure 19 in Appendix D and find that overall performance differences are marginal, yet smaller batch sizes are beneficial for improvements at low generation budgets, while larger batch sizes result in more stable updates that still learn to solve questions at later stages into the run.

### 5.2 Results

Figure 13 compares discovery ​ @ ​ k \text{discovery}@k for SDPO, multi-turn sampling, and best-of- k k sampling on very hard (left) and hard (right) questions from LCBv6. Across both difficulty levels, SDPO achieves substantially higher discovery ​ @ ​ k \text{discovery}@k rates at almost all generation budgets.

On very hard tasks, multi-turn and best-of- k k largely fail to solve questions within the available generation budget, achieving discovery@2750 of only 35.6 % 35.6\% and 41.5 % {41.5}\% , respectively, whereas SDPO discovers a solution in 53.2 % {53.2}\% of cases. SDPO not only solves more questions overall but also does so with substantially fewer attempts. Notably, to reach a 22 % 22\% discovery probability on very hard questions, SDPO requires approximately 3 × 3\times fewer generations than best-of- k k and multi-turn sampling. On hard tasks, SDPO reaches a 78 % {78}\% discovery@2750 probability while achieving a 67 % 67\% discovery probability with roughly 2.4 × 2.4\times fewer generations than best-of- k k and multi-turn sampling. Overall, multi-turn and best-of- k k sampling solve only 68.4 % {68.4}\% and 72.3 % {72.3}\% of questions, respectively. The context window length for multi-turn sampling is reached after 837 ( ± 466 \pm 466 ) steps for hard questions and after 1007 ( ± 349 \pm 349 ) steps for very hard questions, offering a possible explanation for its diminishing gains at high generation budgets.

##### Question 3 is only solved by SDPO.

SDPO solves all questions that are solved by best-of- k k and multi-turn sampling. Beyond that, SDPO uniquely discovers a solution for Q3, which is neither solvable with multi-turn sampling nor with best-of- k k sampling within 2750 attempts. In contrast, SDPO first discovers a solution for Q3 after 321 attempts, which corresponds to 20 iteration steps of self-distillation based on feedback with a batch size of 16. We include detailed per-question results in Table 10 in Appendix D .

##### The initial self-teacher does not solve hard questions.

Notably, the self-teacher’s initial accuracy is < 1 <1 % for almost all questions, and even exactly 0 0 % on 78 78 % of them ( Table 11 in Appendix D ). This shows that a single turn of in-context feedback is insufficient to solve the problem. Despite this, the self-teacher’s credit assignment is sufficiently effective for SDPO to iteratively refine the policy and eventually solve these questions.

## 6 Related Work

### 6.1 Reinforcement Learning with LLMs

Recently, large-scale RL training on diverse tasks has significantly improved the performance of LLMs on general reasoning tasks ( Guo et al., 2025 ; Kimi et al., 2025 ; Olmo et al., 2025 ; Jaech et al., 2024 ; Lambert et al., 2025 ) . This progress is primarily enabled by RLVR methods that use Monte Carlo estimates of rewards, such as STaR or GRPO ( Zelikman et al., 2022 ; Shao et al., 2024 ) , similar to the classical REINFORCE algorithm ( Williams, 1992 ) . While several traditional RLVR algorithms rely on learning separate value networks ( Schulman et al., 2017 ) , they incur substantial memory costs and retain the information bottleneck of scalar rewards.

In the RLVR setting, it is common for an (outcome) reward to be given only at the end of a sequence. To improve credit assignment, several works learn so-called process reward models (PRMs) that estimate rewards for each step in the sequence ( Lightman et al., 2023 ; Wang et al., 2024a ; Setlur et al., 2025 ) . Unlike our RLRF setting, PRMs are typically trained on scalar rewards, either on value estimates for intermediate states or on outcome rewards ( Cui et al., 2025 ) . Unlike the self-teacher in SDPO, PRMs are a distinct model from the student, introducing significant memory overhead. Our work shows that each language model is implicitly a PRM through retrospection if given rich feedback.

Conceptually, our work is related to “bootstrapping your own latent” ( Grill et al., 2020 , BYOL;) and “expert iteration” ( Anthony et al., 2017 ) where a student is bootstrapped by repeatedly imitating an improved version of itself (called the “expert”). Canonically, the expert combines the student with test-time search, such as tree search ( Anthony et al., 2017 ) or majority voting ( Zuo et al., 2025 ) . In contrast, SDPO leverages the student’s ability to learn from rich feedback provided in-context, which is related to “augmented views” in BYOL.

### 6.2 Learning from Rich Feedback and through Retrospection

Beyond scalar outcome rewards, recent works have leveraged rich execution or verbal feedback to guide generation ( Gehring et al., 2025 ; Feng et al., 2024b ; Yuksekgonul et al., 2025 ) . A primary line of research focuses on translating verbal feedback into reward functions for RL. This is often achieved by mapping feedback to discrete token-level rewards using an external frozen model ( Wang et al., 2026 ) , or by employing strong external LLMs to explicitly construct state-wise reward functions ( Goyal et al., 2019 ; Xie et al., 2024 ; Urcelay et al., 2026 ) .

Alternatively, feedback can be utilized without explicit reward modeling. Several approaches focus on in-context improvement without integrating the process into the RL optimization loop ( Chen et al., 2021a ; Madaan et al., 2023 ; Shinn et al., 2023 ; Yao et al., 2024 ; Yuksekgonul et al., 2025 ; Lee et al., 2025 ) . Others manually curate preference datasets by pairing responses before and after feedback to train with direct preference optimization ( Stephan et al., 2024 ; Lee et al., 2024 ) , though this requires additional generation and lacks the direct credit assignment of SDPO. Various recent works bootstrap thinking traces from known answers, using these answers as rich feedback ( Zhou et al., 2026 ; Hatamizadeh et al., 2026 ; Zhang et al., 2025 ) .

A central object in several recent works is a feedback-conditioned policy π θ ​ ( y ∣ x , f ) \pi_{\theta}(y\mid x,f) , which learns answers y y that lead to feedback f f ( Liu et al., 2023 ; Zhang et al., 2023 ; Luo et al., 2025 ) , typically through supervised objectives. The idea behind these approaches is to deploy a policy conditioned on desirable (i.e., positive) feedback for deployment. This approach is conceptually related to goal-conditioned RL ( Schaul et al., 2015 ; Liu et al., 2025a ) , where one can learn from negative examples through goal relabeling ( Andrychowicz et al., 2017 ) . Feedback-conditioned policies view feedback as a goal, whereas RLRF views feedback as a state that can be used to determine whether the goal x x is achieved. Unlike SDPO, these methods do not use feedback for credit assignment in negative trajectories, but rather as a data transformation for goal relabeling.

### 6.3 Distillation

Distillation is frequently employed as an alternative to supervised fine-tuning (SFT) when a strong teacher model is available. Distillation transfers capabilities by training a student to mimic the output distribution or intermediate representations of the teacher ( Hinton et al., 2015 ; Romero et al., 2015 ; Kim & Rush, 2016 ; Sanh et al., 2019 ; Xie et al., 2020 ) . While often performed on fixed off-policy datasets, to address the distribution shift between training and inference, recent works explore on-policy distillation, where the student learns from feedback on its own generations provided by an external teacher ( Agarwal et al., 2024 ; Gu et al., 2024 ; Yang et al., 2025a ; Lu & Thinking Machines Lab, 2025 ) . This mitigates the train-test mismatch, which relates closely to earlier work on online imitation learning ( Ross et al., 2011 ) .

### 6.4 Self-Distillation

The concept of self-distillation was first proposed by Snell et al. (2022) in a setting akin to supervised learning, introducing the idea of sampling from a model provided with extra context and training the same model to mimic these predictions without that context. This mechanism has proven effective for compressing behavior ( Bai et al., 2022 ; Choi et al., 2022 ; Yang et al., 2024 ; Yang et al., 2025b ) and factual information ( Eyuboglu et al., 2026 ; Kujanpää et al., 2025 ; Cao et al., 2025a ) into model weights. Beyond compressing a fixed context into model weights, recent works have used self-distillation to learn from environment feedback ( Scheurer et al., 2023 ; Dou et al., 2024 ; Zhou et al., 2025 ; Mitra & Ulukus, 2025 ; Song et al., 2026 ) . These approaches use an off-policy self-distillation objective, which we find to substantially underperform SDPO’s on-policy learning. Off-policy self-distillation trains the student on generations from the teacher, whereas SDPO trains the student to avoid mistakes in its own generations. In concurrent work, Chen et al. (2025c) apply on-policy self-distillation to grid world settings where feedback is a scalar reward, and a reflection stage in the self-teacher diagnoses possible mistakes, showing improved credit assignment compared to learning value networks for advantage estimation. Other concurrent work studies SDPO on a fixed dataset of expert demonstrations, without online environment interaction ( Shenfeld et al., 2026a ; Zhao et al., 2026 ) .

## 7 Conclusion, Limitations, and Future Work

We introduced Reinforcement Learning with Rich Feedback (RLRF), a paradigm where environments provide tokenized feedback beyond scalar rewards, and argued that this removes a key information bottleneck of RLVR. We then proposed Self-Distillation Policy Optimization (SDPO), which uses the current policy as a feedback-conditioned self-teacher and distills its corrected log-probabilities into the student. This leverages the model’s ability to learn from context for dense credit assignment. We further demonstrated that SDPO can be implemented as a minimal, drop-in modification to standard RLVR pipelines.