Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — part 3/3

Cleaned-text code points [31396, 45000). Bibliography omitted here; source.md includes it.
Our work connects to a line of research showing that LLMs can improve by generating and exploiting their own supervision signals ( Allen-Zhu and Li, 2020 ; Xu et al., 2024b ; Chen et al., 2024 ; Wang et al., 2023 ; Sun et al., 2023 ; Yuan et al., 2024 ; Yang et al., 2024 ) . Closest in spirit is context distillation ( Snell et al., 2022 ) , which uses the same underlying model as both teacher and student by providing the teacher with privileged context and then SFT the student on the teacher’s generated outputs without context. This can be viewed as off-policy , where the learning signal is a discrete token sequence. In the reasoning domain, ReST ( Gulcehre et al., 2023 ) and STaR ( Zelikman et al., 2022 ) similarly rely on iterative self-training loops—generate rationales conditioned on hints or answers, filter by rewards or ground-truth answers, and fine-tune on successful trajectories—again yielding hard distillation; Mitra and Ulukus (2025) extends this to soft distillation. In-context editing ( Qi et al., 2025 ) does on-policy sample from student and shows that context-induced knowledge can be internalized via soft distillation by minimizing divergences and demonstrates this in knowledge editing settings. OPSD differs from these approaches in that we perform on-policy, soft distillation on the student’s own rollouts for reasoning tasks: the teacher’s supervision is per-token distribution matching rather than generating a rationale for SFT. OPSD frames reasoning improvement as learning a conditional distribution induced jointly by the dataset’s ground-truth solutions and the model’s own reasoning ability. Concurrently, SDPO ( Hübotter et al., 2026 ) explored similar algorithm with environment feedbacks as privilledged information and SDFT ( Shenfeld et al., 2026 ) explored on-policy self-distillation on continual learning tasks.

On-Policy Distillation methods train a student model directly on trajectories sampled from its own policy, while a teacher model provides per-token guidance through KL-based regularization or related objectives ( Agarwal et al., 2024 ; Xu et al., 2024a ; Gu et al., 2024 ; Lu and Lab, 2025 ; Xiaomi, 2026 ; Yang et al., 2025 ) . These approaches mitigate distribution shift by optimizing directly on the student’s visitation distribution, but they typically rely on a distinct and often larger teacher model. In this work, we explore whether an LLM can teach itself by conditioning on more privileged answer information and leveraging its own reasoning capability to guide a weaker version of itself toward improved reasoning. On-policy training paradigms are also widely used in robotics and deep reinforcement learning, such as DAgger ( Ross et al., 2011 ) , where a human teacher provides corrective supervision on the states visited by the student policy.

Improving LLM Reasoning through SFT and RL. SFT and RL are two primary methods for improving LLM reasoning ability. SFT on high-quality reasoning traces has demonstrated strong performance ( Yu et al., 2023 ; LI et al., 2024 ; Paster et al., 2023 ; Team, 2025a ; Ye et al., 2025 ; Muennighoff et al., 2025 ; Zhou et al., 2023 ) . However, prior work shows that SFT can rely on memorization rather than robust generalization ( Chu et al., 2025 ) . In contrast, RL optimizes directly for outcome-based objectives can exhibit better generalization ( Huan et al., 2025 ) . More recent algorithms such as GRPO ( Guo et al., 2025 ; Shao et al., 2024 ) enable scalable RL by estimating advantages from group-level rewards without requiring an explicit critic as in PPO ( Schulman et al., 2017 ) . Building on this line of work, a growing body of research highlights the effectiveness of RLVR for reasoning tasks ( Yu et al., 2025 ; Liu et al., 2025 ; Yue et al., 2025 ; An et al., 2025 ; Zheng et al., 2025 ) .

## 6 Conclusion

We introduced On-Policy Self-Distillation (OPSD), a simple yet effective framework for post-training large language models on reasoning tasks. The intuition behind OPSD is that a sufficiently capable reasoning LLM can teach itself when it has access to privileged information about the answer to a reasoning problem, utilizing its own rationalization ability to grade its weaker self without access to the ground truth. We experimentally demonstrated that OPSD achieves better performance than off-policy distillation/SFT, and performs on par with or better than GRPO, while exhibiting significantly better sample efficiency than GRPO.

## 7 Impact Statement

This paper presents work whose goal is to advance the field of machine learning. Our method improves the efficiency of training language models for reasoning tasks, reducing computational costs compared to existing reinforcement learning approaches. We do not foresee specific negative societal consequences.

## Appendix A Limitations and Future Directions

Due to computational constraints, our experiments are limited to models up to 8B parameters. It remains an open question whether this trend continues at scales beyond 8B parameters. Several promising directions warrant further investigation. First, our current framework does not explicitly leverage correctness verification of generated answers; incorporating such signals could provide additional learning objectives beyond distribution matching. Finally, problem difficulty plays a crucial role in self-distillation: if reasoning problems exceed the model’s comprehension threshold, the teacher policy cannot provide meaningful supervision even with access to ground-truth solutions. This suggests that curriculum learning strategies—gradually increasing problem difficulty as the model improves—could enhance training effectiveness. Exploring adaptive curricula that maintain problems at the frontier of model capabilities represents an important direction for scaling OPSD to more challenging reasoning tasks.

## Appendix B Experimental Details

We provide the training and evaluation configurations for our SFT, GRPO and OPSD experiments in Tables 7 , 6 and 8 . Note that we adopt the Thinking-Mode-off student / Thinking-Mode-on teacher configuration for main OPSD experiments. For more experiment details, please refer to our released training code in https://github.com/siyan-zhao/OPSD .We didn’t conduct tuning for the clipping parameter τ \tau , optimizing this hyperparameter may yield further performance gains within the same 100-step budget for larger models.

All experiments were conducted using 8 A100 or H100 GPUs with gradient checkpointing and Flash Attention 2 for memory efficiency. We use the AdamW ( Loshchilov and Hutter, 2017 ) optimizer and bfloat16 precision for all training runs. For OPSD, unless otherwise stated, we used full-vocabulary logit distillation.

## Appendix C Token Category Definitions

We categorize tokens into style and math groups using predefined keyword lists. These keyword sets are used to analyze the per-token KL divergence stylistic tokens and mathematical knowledge tokens as in Section 4.3.1 .

##### Style Tokens.

maybe, perhaps, probably, possibly, let, okay, ok, alright, hmm, wait, because, since, so, thus, hence, therefore, but, however, although, though, yet, or, alternatively, instead, otherwise, actually, really, just, simply, basically, very, quite, pretty, rather, fairly, now, then, next, first, second, finally, try, see, check, note, recall, think, idea, strategy, approach, method, way, would, could, should, might, can, huge, large, big, small, tiny, interesting, tricky, complex, simple.

##### Math Tokens.

exponential, exponent, power, powers, base, logarithm, logarithms, log, ln, compare, comparing, comparison, less, equal, larger, smaller, greater, factor, factors, prime, divisible, equation, expression, formula, inequality, rational, irrational, real, integer, coefficient, variable, constant, sum, product, difference, quotient, fraction, denominator, numerator, root, square, cube, nth, maximum, minimum, optimize, bound.

## Appendix D Policy-Gradient Interpretation of OPSD and Comparison to STaR

Our OPSD objective in Equation 9 can be interpreted as a policy-gradient update with a dense, token-level reward signal derived from privileged information. In this section, we show: (1) OPSD can be seen as a dense-reward policy gradient, and (2) we contrast OPSD with STaR, demonstrating that STaR’s learning signal is sequence-level while OPSD is token-level .

### D.1 STaR as Sequence-Level Policy-Gradient

STaR ( Zelikman et al., 2022 ) can be viewed as an approximation to an RL-style policy gradient objective. The language model p θ p_{\theta} induces a joint distribution over rationale r r and answer y y : p θ  ( r , y ∣ x ) = p θ  ( r ∣ x )  p θ  ( y ∣ x , r ) , p_{\theta}(r,y\mid x)=p_{\theta}(r\mid x)\,p_{\theta}(y\mid x,r), where the model first samples a latent rationale r r before predicting the final answer y y . Given an indicator reward R ⁡ ( y ) = 𝟏  ( y = y ⋆ ) R(y)=\mathbf{1}(y=y^{\star}) , the expected return across the dataset 𝒮 = { ( x i , y i ⋆ ) } i = 1 N \mathcal{S}=\{(x_{i},y_{i}^{\star})\}_{i=1}^{N} is J STaR ( θ ) = ∑ i = 1 N 𝔼 ( r , y ) ∼ p θ ( ⋅ ∣ x i ) [ 𝟏 ( y = y i ⋆ ) ] . J_{\text{STaR}}(\theta)=\sum_{i=1}^{N}\mathbb{E}_{(r,y)\sim p_{\theta}(\cdot\mid x_{i})}\big[\mathbf{1}(y=y_{i}^{\star})\big]. (10) Applying the log-derivative trick yields a policy gradient: ∇ θ J STaR ( θ ) = ∑ i = 1 N 𝔼 ( r , y ) ∼ p θ ( ⋅ ∣ x i ) [ 𝟏 ( y = y i ⋆ ) ∇ θ log p θ ( r , y ∣ x i ) ] . \nabla_{\theta}J_{\text{STaR}}(\theta)=\sum_{i=1}^{N}\mathbb{E}_{(r,y)\sim p_{\theta}(\cdot\mid x_{i})}\Big[\mathbf{1}(y=y_{i}^{\star})\,\nabla_{\theta}\log p_{\theta}(r,y\mid x_{i})\Big]. (11) Note that the indicator function discards the gradient for all sampled rationales that do not lead to the correct answer y i ⋆ y_{i}^{\star} : this corresponds to the filtering step in STaR.

One limitation is that STaR’s reward is sequence-level : the binary indicator 𝟏  ( y = y ⋆ ) \mathbf{1}(y=y^{\star}) provides the same signal to all tokens in a trajectory, offering no intermediate credit assignment. When all sampled trajectories are all incorrect, the learning signal vanishes.

### D.2 OPSD as Dense-Reward Policy Gradient

The sampled-token objective in Equation 9 can also be viewed as a policy-gradient method, but with a token-level reward. Fix a training pair ( x , y ⋆ ) (x,y^{\star}) and let the student generate a trajectory y ^ ∼ p S ( ⋅ ∣ x ) \hat{y}\sim p_{S}(\cdot\mid x) . At each position n n , define the per-token reward: r n  ( x , y ^ ) ≜ log ⁡ p T  ( y ^ n ∣ x , y ⋆ , y ^ < n ) − log ⁡ p S  ( y ^ n ∣ x , y ^ < n ) . r_{n}(x,\hat{y})\triangleq\log p_{T}(\hat{y}_{n}\mid x,y^{\star},\hat{y}_{<n})-\log p_{S}(\hat{y}_{n}\mid x,\hat{y}_{<n}). This reward measures how much the privileged teacher prefers the sampled token y ^ n \hat{y}_{n} relative to the student. As stated in the main text, we treat r n r_{n} (equivalently, the advantage A n A_{n} ) as a constant with respect to θ \theta when computing gradients—that is, we stop gradients through both p T p_{T} and p S p_{S} in the reward computation. Under this treatment, the gradient of Equation 9 takes the standard policy-gradient form: ∇ θ ℒ ( θ ) = − 𝔼 ( x , y ⋆ ) ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) [ 1 | y ^ | ∑ n = 1 | y ^ | r n ( x , y ^ ) ∇ θ log p S ( y ^ n ∣ x , y ^ < n ) ] ] , \nabla_{\theta}\mathcal{L}(\theta)=-\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\left[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\left[\frac{1}{|\hat{y}|}\sum_{n=1}^{|\hat{y}|}r_{n}(x,\hat{y})\,\nabla_{\theta}\log p_{S}(\hat{y}_{n}\mid x,\hat{y}_{<n})\right]\right], which corresponds to maximizing the expected per-token reward along on-policy student rollouts: J O  P  S  D ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) [ 1 | y ^ | ∑ n = 1 | y ^ | r n ( x , y ^ ) ] ] . J_{OPSD{}}(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\left[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\left[\frac{1}{|\hat{y}|}\sum_{n=1}^{|\hat{y}|}r_{n}(x,\hat{y})\right]\right]. This reward is dense: it provides a learning signal at every token position, regardless of whether the final answer is correct.

##### Comparison.

Both STaR and OPSD can be understood as policy-gradient methods, but their reward structures differ fundamentally. STaR uses a sequence-level indicator 𝟏  ( y = y ⋆ ) \mathbf{1}(y=y^{\star}) that assigns the same signal to all tokens; when all sampled trajectories are incorrect, the learning signal vanishes entirely. In contrast, OPSD provides a token-level reward r n r_{n} at every position, enabling fine-grained credit assignment even when the final answer is wrong.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .