Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — part 1/3
Cleaned-text code points [0, 15963). Bibliography omitted here; source.md includes it.
# Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
###### Abstract
Knowledge distillation improves large language model (LLM) reasoning by compressing the knowledge of a teacher LLM to train smaller LLMs. On-policy distillation advances this approach by having the student sample its own trajectories while a teacher LLM provides dense token-level supervision, addressing the distribution mismatch between training and inference in off-policy distillation methods. However, on-policy distillation typically requires a separate, often larger, teacher LLM and does not explicitly leverage ground-truth solutions available in reasoning datasets. Inspired by the intuition that a sufficiently capable LLM can rationalize external privileged reasoning traces and teach its weaker self, we introduce On-Policy Self-Distillation (OPSD), a learning algorithm where a single LLM acts as both teacher and student with different contexts. The teacher policy conditions on privileged information (e.g., verified reasoning traces) while the student policy sees only the question; training minimizes the per-token divergence between these distributions over the student’s own rollouts. We demonstrate the efficacy of our method on multiple mathematical reasoning benchmarks, achieving superior token efficiency compared to reinforcement learning methods and better performance over off-policy distillation methods. Code repo: https://github.com/siyan-zhao/OPSD .
###### Keywords:
## 1 Introduction
Recent advances in large language models (LLMs) have demonstrated impressive capabilities in reasoning and instruction following. Achieving these capabilities during post-training typically relies on reinforcement learning methods such as Reinforcement Learning with Verifiable Rewards (RLVR) (e.g., GRPO ( Shao et al., 2024 ; Guo et al., 2025 ; Team et al., 2025 ; Rastogi et al., 2025 ; Yu et al., 2025 ) ), supervised fine-tuning (SFT) on high-quality reasoning datasets ( Guha et al., 2025 ; Team et al., 2025 ; Xiaomi, 2026 ) , or knowledge distillation, where recent work has shown that distillation from advanced teacher models can outperform RL in both performance and training efficiency ( Yang et al., 2025 ; Xiaomi, 2026 ; Lu and Lab, 2025 ) .
Despite their respective successes, each approach has inherent limitations. RLVR suffers from inefficiencies including: (1) sampling a group of responses per prompt is computationally expensive and can introduce high variance in estimating the true value function; moreover, when all samples are either correct or incorrect, the gradient signal vanishes ( Yu et al., 2025 ; Zhao et al., 2025 ) ; and (2) the reward signal is sparse and uniformly applied across all tokens in the generated output, neglecting fine-grained token-level feedback. Supervised fine-tuning suffers from exposure bias and weaker generalization ( Agarwal et al., 2024 ; Chu et al., 2025 ) . Traditional knowledge distillation provides dense token-level supervision from a teacher model but relies on off-policy data ( Hinton et al., 2015 ) . Recent advances in on-policy distillation—where a student model samples its own trajectories while a teacher policy provides dense token-level supervision—have demonstrated superior sample efficiency by combining the distributional realism of on-policy training with dense feedback ( Agarwal et al., 2024 ; Lu and Lab, 2025 ) .
While on-policy distillation has shown strong performance, it relies on a distinct teacher model to supervise the student. Given that modern LLMs already exhibit strong reasoning capabilities, we ask this research question: can a model effectively serve as its own teacher through self-distillation? Our approach is inspired by human learning: after solving a problem incorrectly, a student can examine the correct solution, rationalize its steps, and identify where their reasoning failed. Prior work has shown that for LLMs, evaluation is often easier than generation ( Sun et al., 2024 ; Naor, 1996 ) . We hypothesize that rationalization —explaining a given correct answer—is similarly easier than generation. Motivated by this, we instantiate both the teacher and student policies from a single LLM. The teacher policy is provided with privileged information y ⋆ y^{\star} , such as the ground-truth answer or a reference chain-of-thought, while the student policy conditions only on the problem x x . Concretely, the teacher policy p T ( ⋅ ∣ x , y ⋆ ) p_{T}(\cdot\mid x,y^{\star}) conditions on both the problem and the privileged answer, whereas the student policy p S ( ⋅ ∣ x ) p_{S}(\cdot\mid x) observes only the problem. We preserve the on-policy training paradigm by sampling trajectories y ^ \hat{y} exclusively from the student policy, which then receives dense, token-level supervision from the privileged teacher policy.
We therefore propose On-Policy Self-Distillation (OPSD) , a framework in which a single model plays both teacher and student roles. The student samples its own trajectories y ^ ∼ p S ( ⋅ ∣ x ) \hat{y}\sim p_{S}(\cdot\mid x) ; we then compute the per-token divergence between the student and teacher distributions and minimize it over the student’s own rollouts. This formulation (i) uses on-policy supervision (the student’s own trajectories), (ii) provides dense per-token feedback, (iii) exploits ground-truth solutions y ⋆ y^{\star} , and (iv) requires no separate teacher model. The learning process is captured by the loss ℒ OPSD \displaystyle\mathcal{L}_{\mathrm{OPSD}} ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒮 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) ∑ n = 1 | y ^ | \displaystyle(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\;\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\sum_{n=1}^{|\hat{y}|} D ( p T ( ⋅ ∣ x , y ⋆ , y ^ < n ) ∥ p S ( ⋅ ∣ x , y ^ < n ) ) . \displaystyle\quad D\!\Bigl(p_{T}\!\left(\cdot\mid x,y^{\star},\hat{y}_{<n}\right)\;\Big\|\;p_{S}\!\left(\cdot\mid x,\hat{y}_{<n}\right)\Bigr). (1)
In summary, our contributions are as follows: • We introduce On-Policy Self-Distillation (OPSD), a novel framework that enables a single model to act as both teacher and student, leveraging ground-truth answers to provide dense token-level supervision on student rollouts.
• We introduce a per-token pointwise KL clipping mechanism that stabilizes training and improves performance as we find stylistic tokens can dominate the training signal of math tokens.
• We evaluate OPSD on three competition-level mathematical reasoning tasks, demonstrating that it matches the performance of GRPO with significantly improved token efficiency and outperform supervised fine-tuning.
• We analyze the impact of different divergence objectives, the effect of student generation length, and student–teacher generation styles.
## 2 Background
### 2.1 Knowledge Distillation for Autoregressive Large Language Models
Knowledge distillation transfers knowledge from a larger teacher model to a smaller student model by training the student to mimic the teacher’s behavior ( Hinton et al., 2015 ; Kim and Rush, 2016 ; Sanh et al., 2019 ) . The core insight is that the teacher’s soft probability distribution over classes contains richer information than hard labels alone, as it reveals the teacher’s learned similarities between classes. For auto-regressive language models, given a dataset 𝒮 = { ( x , y ⋆ ) } \mathcal{S}=\{(x,y^{\star})\} where x x denotes an input and y ⋆ y^{\star} is the corresponding reference output, both teacher p T p_{T} and student p S p_{S} define token-level distributions over vocabulary 𝒱 \mathcal{V} . Traditional supervised distillation minimizes a divergence D D between teacher and student distributions averaged over a fixed dataset: ℒ Supervised Distillation ( θ ) = 𝔼 ( x , y ) ∼ 𝒮 [ D ( p T ∥ p S ) ( y | x ) ] , \mathcal{L}_{\text{Supervised Distillation}}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{S}}[D(p_{T}\|p_{S})(y|x)], (2) where D ( p T ∥ p S ) ( y | x ) = 1 | y | ∑ n = 1 | y | D ( p T ( ⋅ | y < n , x ) ∥ p S ( ⋅ | y < n , x ) ) D(p_{T}\|p_{S})(y|x)=\frac{1}{|y|}\sum_{n=1}^{|y|}D(p_{T}(\cdot|y_{<n},x)\|p_{S}(\cdot|y_{<n},x)) measures per-token discrepancy. However, this off-policy approach suffers from distribution mismatch: the student encounters different partial sequences y < n y_{<n} during auto-regressive generation at inference than those seen during training on the fixed dataset, leading to compounding errors. On-policy distillation ( Agarwal et al., 2024 ; Lu and Lab, 2025 ; Xu et al., 2024a ) addresses this by training the student on its own generated sequences y ^ ∼ p S ( ⋅ | x ) \hat{y}\sim p_{S}(\cdot|x) , obtaining dense token-level feedback from the teacher on these on-policy samples: ℒ On-Policy Distillation ( θ ) = 𝔼 x ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ | x ) [ D ( p T ∥ p S ) ( y ^ | x ) ] ] . \mathcal{L}_{\text{On-Policy Distillation}}(\theta)=\mathbb{E}_{x\sim\mathcal{S}}[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot|x)}[D(p_{T}\|p_{S})(\hat{y}|x)]]. (3) This approach connects distillation to imitation learning ( Ross et al., 2011 ) , where the student iteratively improves by learning from the teacher’s guidance on its own outputs, combining the on-policy relevance of reinforcement learning with the dense reward signal of supervised learning, thereby mitigating exposure bias while maintaining computational efficiency.
### 2.2 Reinforcement Learning with Verifiable Rewards
Reinforcement learning with verifiable rewards (RLVR) has emerged as a popular approach for post-training large language models, particularly on tasks with easily verifiable outcomes such as mathematics and coding, using algorithms like Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) and Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) .
GRPO trains by sampling a group of G G responses { o 1 , o 2 , … , o G } \{o_{1},o_{2},\ldots,o_{G}\} from the current policy π θ \pi_{\theta} for each problem x x . Each response o i o_{i} receives a binary reward r i ∈ { 0 , 1 } r_{i}\in\{0,1\} indicating correctness. The method then assigns advantages to all tokens k = 1 , … , | o i | k=1,\ldots,|o_{i}| within response o i o_{i} using a group-normalized reward: A i = r i − mean ( { r j } j = 1 G ) std ( { r j } j = 1 G ) . A_{i}=\frac{r_{i}-\text{mean}(\{r_{j}\}_{j=1}^{G})}{\text{std}(\{r_{j}\}_{j=1}^{G})}. (4) This formulation can be understood through the value function lens: mean ( { r j } j = 1 G ) \text{mean}(\{r_{j}\}_{j=1}^{G}) serves as a G G -sample Monte Carlo estimate of the value function V ( x ) V(x) , while the sparse binary reward r i r_{i} represents the (undiscounted) state-action value Q ( x , o i ) Q(x,o_{i}) . Critically, all tokens within a response share the same advantage, as the reward signal is provided only at the sequence level. The GRPO objective incorporates a clipped surrogate loss to moderate policy updates, along with a reverse KL penalty to prevent excessive deviation from a reference policy: ℒ GRPO ( θ ) = 𝔼 x ∼ 𝒮 o 1 , … , o G ∼ π θ ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | o i | ∑ n = 1 | o i | min ( ρ i n A i , clip ( ρ i n , 1 − ε , 1 + ε ) A i ) − β D KL [ π θ ( ⋅ | x ) ∥ π ref ( ⋅ | x ) ] ] \begin{split}\mathcal{L}_{\text{GRPO}}(\theta)=\mathbb{E}_{\begin{subarray}{c}x\sim\mathcal{S}\\ o_{1},\ldots,o_{G}\sim\pi_{\theta}(\cdot|x)\end{subarray}}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{n=1}^{|o_{i}|}\\ \min\left(\rho_{i}^{n}A_{i},\text{clip}\left(\rho_{i}^{n},1-\varepsilon,1+\varepsilon\right)A_{i}\right)\\ -\beta D_{\text{KL}}[\pi_{\theta}(\cdot|x)\|\pi_{\text{ref}}(\cdot|x)]\Bigg]\end{split} (5) where ρ i n = π θ ( o i n | x , o i < n ) π θ old ( o i n | x , o i < n ) \rho_{i}^{n}=\frac{\pi_{\theta}(o_{i}^{n}|x,o_{i}^{<n})}{\pi_{\theta_{\text{old}}}(o_{i}^{n}|x,o_{i}^{<n})} is the importance ratio, π θ old \pi_{\theta_{\text{old}}} is the policy before the update, and ε \varepsilon controls the clipping range.
While RLVR methods have demonstrated strong empirical performance, they face two key limitations: (1) the reward signal is sparse, providing only sequence-level feedback rather than token-level guidance on where errors occur, and (2) when all sampled responses receive identical rewards (all correct or all incorrect), the advantages become zero, preventing any policy update despite the computational cost of sampling.
## 3 Methods
### 3.1 Learning from Verifiable Reasoning Dataset
We consider a dataset of problem-solution pairs 𝒮 = { ( x i , y i ⋆ ) } i = 1 N , \mathcal{S}=\{(x_{i},y_{i}^{\star})\}_{i=1}^{N}, where each x i x_{i} denotes a problem and y i ⋆ y_{i}^{\star} is the corresponding reference solution, which may include chain-of-thought reasoning. For brevity, we omit the sample index i i and use ( x , y ⋆ ) (x,y^{\star}) to denote a generic sample from the dataset. We can exploit learning signals from this dataset from different ways: Standard supervised fine-tuning (SFT) on 𝒮 \mathcal{S} can be viewed as off-policy distillation/imitation learning using expert trajectories, but it suffers from distribution mismatch between training and inference. Reinforcement learning from verifiable rewards (RLVR), such as GRPO, addresses this by optimizing on-policy samples and assigning binary rewards by comparing generated answers against y ⋆ y^{\star} . However, RLVR is computationally expensive and the reward signal is sparse, providing same feedback across all tokens regardless of where errors occur. Alternatively, one can train a process reward model (PRM) to provide dense, token-level feedback during RL. However, acquiring labels for PRM training is prohibitively expensive and difficult to scale ( Lightman et al., 2023 ; Zhang et al., 2025 ) . On-policy distillation works ( Agarwal et al., 2024 ; Xu et al., 2024a ; Lu and Lab, 2025 ) address distribution shift by training on the student’s own samples, but require a separate, often larger, teacher model to provide supervision. We instead seek a training signal that is dense , on-policy , and does not require external teachers or reward models . This motivates our On-Policy Self-Distillation approach. We summarize the differences of these methods in Table 1 .
### 3.2 On-Policy Self-Distillation
##### Motivation: Learning by understanding solutions.
We propose a different perspective inspired by how students learn: when struggling with a problem, rather than extended trial-and-error, a student can examine the solution, understand the reasoning, and internalize the approach. Similarly, if a model has access to the correct answer or reasoning y ⋆ y^{\star} and is sufficiently capable, it can rationalize the reasoning steps and teach itself—analogous to a student reviewing a solution and retracing why it works. This intuition motivates our framework: we exploit the ground-truth solution y ⋆ y^{\star} directly as privileged information during training, enabling the model to serve as its own teacher without requiring external reward models or larger teacher models.
##### Teacher and student policies.
We instantiate two conditional distributions from the same language model p θ p_{\theta} by varying the conditioning context. The teacher policy conditions on privileged information—both the problem x x and the reference solution y ⋆ y^{\star} : p T ( ⋅ ∣ x , y ⋆ ) ≜ p θ ( ⋅ ∣ x , y ⋆ ) . p_{T}(\cdot\mid x,y^{\star})\;\triangleq\;p_{\theta}(\cdot\mid x,y^{\star}). The student policy observes only the problem statement, matching the inference-time condition: p S ( ⋅ ∣ x ) ≜ p θ ( ⋅ ∣ x ) . p_{S}(\cdot\mid x)\;\triangleq\;p_{\theta}(\cdot\mid x). Both policies share the same parameters θ \theta but differ only in their conditioning context. To encourage the teacher to naturally evaluate the student’s generation, we add a prompt asking the teacher to generate a new solution after seeing the reference solution as shown in Figure 2 . However, the teacher doesn’t generate tokens, it only does rationalization implicitly through prefilling.