# Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

###### Abstract

Knowledge distillation improves large language model (LLM) reasoning by compressing the knowledge of a teacher LLM to train smaller LLMs. On-policy distillation advances this approach by having the student sample its own trajectories while a teacher LLM provides dense token-level supervision, addressing the distribution mismatch between training and inference in off-policy distillation methods. However, on-policy distillation typically requires a separate, often larger, teacher LLM and does not explicitly leverage ground-truth solutions available in reasoning datasets. Inspired by the intuition that a sufficiently capable LLM can rationalize external privileged reasoning traces and teach its weaker self, we introduce On-Policy Self-Distillation (OPSD), a learning algorithm where a single LLM acts as both teacher and student with different contexts. The teacher policy conditions on privileged information (e.g., verified reasoning traces) while the student policy sees only the question; training minimizes the per-token divergence between these distributions over the student’s own rollouts. We demonstrate the efficacy of our method on multiple mathematical reasoning benchmarks, achieving superior token efficiency compared to reinforcement learning methods and better performance over off-policy distillation methods. Code repo: https://github.com/siyan-zhao/OPSD .

###### Keywords:

## 1 Introduction

Recent advances in large language models (LLMs) have demonstrated impressive capabilities in reasoning and instruction following. Achieving these capabilities during post-training typically relies on reinforcement learning methods such as Reinforcement Learning with Verifiable Rewards (RLVR) (e.g., GRPO ( Shao et al., 2024 ; Guo et al., 2025 ; Team et al., 2025 ; Rastogi et al., 2025 ; Yu et al., 2025 ) ), supervised fine-tuning (SFT) on high-quality reasoning datasets ( Guha et al., 2025 ; Team et al., 2025 ; Xiaomi, 2026 ) , or knowledge distillation, where recent work has shown that distillation from advanced teacher models can outperform RL in both performance and training efficiency ( Yang et al., 2025 ; Xiaomi, 2026 ; Lu and Lab, 2025 ) .

Despite their respective successes, each approach has inherent limitations. RLVR suffers from inefficiencies including: (1) sampling a group of responses per prompt is computationally expensive and can introduce high variance in estimating the true value function; moreover, when all samples are either correct or incorrect, the gradient signal vanishes ( Yu et al., 2025 ; Zhao et al., 2025 ) ; and (2) the reward signal is sparse and uniformly applied across all tokens in the generated output, neglecting fine-grained token-level feedback. Supervised fine-tuning suffers from exposure bias and weaker generalization ( Agarwal et al., 2024 ; Chu et al., 2025 ) . Traditional knowledge distillation provides dense token-level supervision from a teacher model but relies on off-policy data ( Hinton et al., 2015 ) . Recent advances in on-policy distillation—where a student model samples its own trajectories while a teacher policy provides dense token-level supervision—have demonstrated superior sample efficiency by combining the distributional realism of on-policy training with dense feedback ( Agarwal et al., 2024 ; Lu and Lab, 2025 ) .

While on-policy distillation has shown strong performance, it relies on a distinct teacher model to supervise the student. Given that modern LLMs already exhibit strong reasoning capabilities, we ask this research question: can a model effectively serve as its own teacher through self-distillation? Our approach is inspired by human learning: after solving a problem incorrectly, a student can examine the correct solution, rationalize its steps, and identify where their reasoning failed. Prior work has shown that for LLMs, evaluation is often easier than generation ( Sun et al., 2024 ; Naor, 1996 ) . We hypothesize that rationalization —explaining a given correct answer—is similarly easier than generation. Motivated by this, we instantiate both the teacher and student policies from a single LLM. The teacher policy is provided with privileged information y ⋆ y^{\star} , such as the ground-truth answer or a reference chain-of-thought, while the student policy conditions only on the problem x x . Concretely, the teacher policy p T ( ⋅ ∣ x , y ⋆ ) p_{T}(\cdot\mid x,y^{\star}) conditions on both the problem and the privileged answer, whereas the student policy p S ( ⋅ ∣ x ) p_{S}(\cdot\mid x) observes only the problem. We preserve the on-policy training paradigm by sampling trajectories y ^ \hat{y} exclusively from the student policy, which then receives dense, token-level supervision from the privileged teacher policy.

We therefore propose On-Policy Self-Distillation (OPSD) , a framework in which a single model plays both teacher and student roles. The student samples its own trajectories y ^ ∼ p S ( ⋅ ∣ x ) \hat{y}\sim p_{S}(\cdot\mid x) ; we then compute the per-token divergence between the student and teacher distributions and minimize it over the student’s own rollouts. This formulation (i) uses on-policy supervision (the student’s own trajectories), (ii) provides dense per-token feedback, (iii) exploits ground-truth solutions y ⋆ y^{\star} , and (iv) requires no separate teacher model. The learning process is captured by the loss ℒ OPSD \displaystyle\mathcal{L}_{\mathrm{OPSD}} ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒮 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) ∑ n = 1 | y ^ | \displaystyle(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\;\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\sum_{n=1}^{|\hat{y}|} D ( p T ( ⋅ ∣ x , y ⋆ , y ^ < n ) ∥ p S ( ⋅ ∣ x , y ^ < n ) ) . \displaystyle\quad D\!\Bigl(p_{T}\!\left(\cdot\mid x,y^{\star},\hat{y}_{<n}\right)\;\Big\|\;p_{S}\!\left(\cdot\mid x,\hat{y}_{<n}\right)\Bigr). (1)

In summary, our contributions are as follows: • We introduce On-Policy Self-Distillation (OPSD), a novel framework that enables a single model to act as both teacher and student, leveraging ground-truth answers to provide dense token-level supervision on student rollouts.

• We introduce a per-token pointwise KL clipping mechanism that stabilizes training and improves performance as we find stylistic tokens can dominate the training signal of math tokens.

• We evaluate OPSD on three competition-level mathematical reasoning tasks, demonstrating that it matches the performance of GRPO with significantly improved token efficiency and outperform supervised fine-tuning.

• We analyze the impact of different divergence objectives, the effect of student generation length, and student–teacher generation styles.

## 2 Background

### 2.1 Knowledge Distillation for Autoregressive Large Language Models

Knowledge distillation transfers knowledge from a larger teacher model to a smaller student model by training the student to mimic the teacher’s behavior ( Hinton et al., 2015 ; Kim and Rush, 2016 ; Sanh et al., 2019 ) . The core insight is that the teacher’s soft probability distribution over classes contains richer information than hard labels alone, as it reveals the teacher’s learned similarities between classes. For auto-regressive language models, given a dataset 𝒮 = { ( x , y ⋆ ) } \mathcal{S}=\{(x,y^{\star})\} where x x denotes an input and y ⋆ y^{\star} is the corresponding reference output, both teacher p T p_{T} and student p S p_{S} define token-level distributions over vocabulary 𝒱 \mathcal{V} . Traditional supervised distillation minimizes a divergence D D between teacher and student distributions averaged over a fixed dataset: ℒ Supervised Distillation ( θ ) = 𝔼 ( x , y ) ∼ 𝒮 [ D ( p T ∥ p S ) ( y | x ) ] , \mathcal{L}_{\text{Supervised Distillation}}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{S}}[D(p_{T}\|p_{S})(y|x)], (2) where D ( p T ∥ p S ) ( y | x ) = 1 | y | ∑ n = 1 | y | D ( p T ( ⋅ | y < n , x ) ∥ p S ( ⋅ | y < n , x ) ) D(p_{T}\|p_{S})(y|x)=\frac{1}{|y|}\sum_{n=1}^{|y|}D(p_{T}(\cdot|y_{<n},x)\|p_{S}(\cdot|y_{<n},x)) measures per-token discrepancy. However, this off-policy approach suffers from distribution mismatch: the student encounters different partial sequences y < n y_{<n} during auto-regressive generation at inference than those seen during training on the fixed dataset, leading to compounding errors. On-policy distillation ( Agarwal et al., 2024 ; Lu and Lab, 2025 ; Xu et al., 2024a ) addresses this by training the student on its own generated sequences y ^ ∼ p S ( ⋅ | x ) \hat{y}\sim p_{S}(\cdot|x) , obtaining dense token-level feedback from the teacher on these on-policy samples: ℒ On-Policy Distillation ( θ ) = 𝔼 x ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ | x ) [ D ( p T ∥ p S ) ( y ^ | x ) ] ] . \mathcal{L}_{\text{On-Policy Distillation}}(\theta)=\mathbb{E}_{x\sim\mathcal{S}}[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot|x)}[D(p_{T}\|p_{S})(\hat{y}|x)]]. (3) This approach connects distillation to imitation learning ( Ross et al., 2011 ) , where the student iteratively improves by learning from the teacher’s guidance on its own outputs, combining the on-policy relevance of reinforcement learning with the dense reward signal of supervised learning, thereby mitigating exposure bias while maintaining computational efficiency.

### 2.2 Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards (RLVR) has emerged as a popular approach for post-training large language models, particularly on tasks with easily verifiable outcomes such as mathematics and coding, using algorithms like Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) and Group Relative Policy Optimization (GRPO) ( Shao et al., 2024 ) .

GRPO trains by sampling a group of G G responses { o 1 , o 2 , … , o G } \{o_{1},o_{2},\ldots,o_{G}\} from the current policy π θ \pi_{\theta} for each problem x x . Each response o i o_{i} receives a binary reward r i ∈ { 0 , 1 } r_{i}\in\{0,1\} indicating correctness. The method then assigns advantages to all tokens k = 1 , … , | o i | k=1,\ldots,|o_{i}| within response o i o_{i} using a group-normalized reward: A i = r i − mean ​ ( { r j } j = 1 G ) std ​ ( { r j } j = 1 G ) . A_{i}=\frac{r_{i}-\text{mean}(\{r_{j}\}_{j=1}^{G})}{\text{std}(\{r_{j}\}_{j=1}^{G})}. (4) This formulation can be understood through the value function lens: mean ​ ( { r j } j = 1 G ) \text{mean}(\{r_{j}\}_{j=1}^{G}) serves as a G G -sample Monte Carlo estimate of the value function V ⁡ ( x ) V(x) , while the sparse binary reward r i r_{i} represents the (undiscounted) state-action value Q ⁡ ( x , o i ) Q(x,o_{i}) . Critically, all tokens within a response share the same advantage, as the reward signal is provided only at the sequence level. The GRPO objective incorporates a clipped surrogate loss to moderate policy updates, along with a reverse KL penalty to prevent excessive deviation from a reference policy: ℒ GRPO ( θ ) = 𝔼 x ∼ 𝒮 o 1 , … , o G ∼ π θ ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | o i | ∑ n = 1 | o i | min ⁡ ( ρ i n ​ A i , clip ​ ( ρ i n , 1 − ε , 1 + ε ) ​ A i ) − β D KL [ π θ ( ⋅ | x ) ∥ π ref ( ⋅ | x ) ] ] \begin{split}\mathcal{L}_{\text{GRPO}}(\theta)=\mathbb{E}_{\begin{subarray}{c}x\sim\mathcal{S}\\ o_{1},\ldots,o_{G}\sim\pi_{\theta}(\cdot|x)\end{subarray}}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{n=1}^{|o_{i}|}\\ \min\left(\rho_{i}^{n}A_{i},\text{clip}\left(\rho_{i}^{n},1-\varepsilon,1+\varepsilon\right)A_{i}\right)\\ -\beta D_{\text{KL}}[\pi_{\theta}(\cdot|x)\|\pi_{\text{ref}}(\cdot|x)]\Bigg]\end{split} (5) where ρ i n = π θ ​ ( o i n | x , o i < n ) π θ old ​ ( o i n | x , o i < n ) \rho_{i}^{n}=\frac{\pi_{\theta}(o_{i}^{n}|x,o_{i}^{<n})}{\pi_{\theta_{\text{old}}}(o_{i}^{n}|x,o_{i}^{<n})} is the importance ratio, π θ old \pi_{\theta_{\text{old}}} is the policy before the update, and ε \varepsilon controls the clipping range.

While RLVR methods have demonstrated strong empirical performance, they face two key limitations: (1) the reward signal is sparse, providing only sequence-level feedback rather than token-level guidance on where errors occur, and (2) when all sampled responses receive identical rewards (all correct or all incorrect), the advantages become zero, preventing any policy update despite the computational cost of sampling.

## 3 Methods

### 3.1 Learning from Verifiable Reasoning Dataset

We consider a dataset of problem-solution pairs 𝒮 = { ( x i , y i ⋆ ) } i = 1 N , \mathcal{S}=\{(x_{i},y_{i}^{\star})\}_{i=1}^{N}, where each x i x_{i} denotes a problem and y i ⋆ y_{i}^{\star} is the corresponding reference solution, which may include chain-of-thought reasoning. For brevity, we omit the sample index i i and use ( x , y ⋆ ) (x,y^{\star}) to denote a generic sample from the dataset. We can exploit learning signals from this dataset from different ways: Standard supervised fine-tuning (SFT) on 𝒮 \mathcal{S} can be viewed as off-policy distillation/imitation learning using expert trajectories, but it suffers from distribution mismatch between training and inference. Reinforcement learning from verifiable rewards (RLVR), such as GRPO, addresses this by optimizing on-policy samples and assigning binary rewards by comparing generated answers against y ⋆ y^{\star} . However, RLVR is computationally expensive and the reward signal is sparse, providing same feedback across all tokens regardless of where errors occur. Alternatively, one can train a process reward model (PRM) to provide dense, token-level feedback during RL. However, acquiring labels for PRM training is prohibitively expensive and difficult to scale ( Lightman et al., 2023 ; Zhang et al., 2025 ) . On-policy distillation works ( Agarwal et al., 2024 ; Xu et al., 2024a ; Lu and Lab, 2025 ) address distribution shift by training on the student’s own samples, but require a separate, often larger, teacher model to provide supervision. We instead seek a training signal that is dense , on-policy , and does not require external teachers or reward models . This motivates our On-Policy Self-Distillation approach. We summarize the differences of these methods in Table 1 .

### 3.2 On-Policy Self-Distillation

##### Motivation: Learning by understanding solutions.

We propose a different perspective inspired by how students learn: when struggling with a problem, rather than extended trial-and-error, a student can examine the solution, understand the reasoning, and internalize the approach. Similarly, if a model has access to the correct answer or reasoning y ⋆ y^{\star} and is sufficiently capable, it can rationalize the reasoning steps and teach itself—analogous to a student reviewing a solution and retracing why it works. This intuition motivates our framework: we exploit the ground-truth solution y ⋆ y^{\star} directly as privileged information during training, enabling the model to serve as its own teacher without requiring external reward models or larger teacher models.

##### Teacher and student policies.

We instantiate two conditional distributions from the same language model p θ p_{\theta} by varying the conditioning context. The teacher policy conditions on privileged information—both the problem x x and the reference solution y ⋆ y^{\star} : p T ( ⋅ ∣ x , y ⋆ ) ≜ p θ ( ⋅ ∣ x , y ⋆ ) . p_{T}(\cdot\mid x,y^{\star})\;\triangleq\;p_{\theta}(\cdot\mid x,y^{\star}). The student policy observes only the problem statement, matching the inference-time condition: p S ( ⋅ ∣ x ) ≜ p θ ( ⋅ ∣ x ) . p_{S}(\cdot\mid x)\;\triangleq\;p_{\theta}(\cdot\mid x). Both policies share the same parameters θ \theta but differ only in their conditioning context. To encourage the teacher to naturally evaluate the student’s generation, we add a prompt asking the teacher to generate a new solution after seeing the reference solution as shown in Figure 2 . However, the teacher doesn’t generate tokens, it only does rationalization implicitly through prefilling.

##### On-policy sampling from the student.

Given a problem x x , the student generates an on-policy response y ^ = ( y ^ 1 , … , y ^ | y ^ | ) ∼ p S ( ⋅ ∣ x ) . \hat{y}=(\hat{y}_{1},\ldots,\hat{y}_{|\hat{y}|})\sim p_{S}(\cdot\mid x). Both policies then evaluate this student-generated trajectory. At each position n n , they induce next-token distributions over y n ∈ 𝒱 y_{n}\in\mathcal{V} conditioned on the same student prefix: p S ​ ( y n ∣ x , y ^ < n ) , p T ​ ( y n ∣ x , y ⋆ , y ^ < n ) , p_{S}\!\left(y_{n}\mid x,\hat{y}_{<n}\right),\qquad p_{T}\!\left(y_{n}\mid x,y^{\star},\hat{y}_{<n}\right), where y ^ < n ≜ ( y ^ 1 , … , y ^ n − 1 ) \hat{y}_{<n}\triangleq(\hat{y}_{1},\ldots,\hat{y}_{n-1}) .

##### Training objective: Full-vocabulary logit distillation.

We instantiate a full-vocabulary divergence objective that matches the teacher and student next-token distributions at each position. Given a student-generated sequence y ^ \hat{y} , define the trajectory-averaged, token-wise divergence D ( p T ∥ p S ) ( y ^ ∣ x ) ≜ 1 | y ^ | ∑ n = 1 | y ^ | D ( p T ( ⋅ ∣ x , y ⋆ , y ^ < n ) ∥ p S ( ⋅ ∣ x , y ^ < n ) ) , \begin{split}D\bigl(p_{T}\,\|\,p_{S}\bigr)(\hat{y}\mid x)&\triangleq\frac{1}{|\hat{y}|}\sum_{n=1}^{|\hat{y}|}D\biggl(p_{T}\!\left(\cdot\mid x,y^{\star},\hat{y}_{<n}\right)\\ &\qquad\big\|\;p_{S}\!\left(\cdot\mid x,\hat{y}_{<n}\right)\biggr),\end{split} (6) where p S ( ⋅ ∣ x , y ^ < n ) p_{S}(\cdot\mid x,\hat{y}_{<n}) and p T ( ⋅ ∣ x , y ⋆ , y ^ < n ) p_{T}(\cdot\mid x,y^{\star},\hat{y}_{<n}) denote distributions over the next token y n ∈ 𝒱 y_{n}\in\mathcal{V} . Here, D D can be any distribution divergence measure such as the generalized Jensen-Shannon divergence JSD β \operatorname{JSD}_{\beta} , defined for a weight β ∈ [ 0 , 1 ] \beta\in[0,1] as: JSD β ( p T ∥ p S ) = β D K ​ L ( p T ∥ m ) + ( 1 − β ) D K ​ L ( p S ∥ m ) \operatorname{JSD}_{\beta}(p_{T}\|p_{S})=\beta D_{KL}(p_{T}\|m)+(1-\beta)D_{KL}(p_{S}\|m) (7) where m = β ​ p T + ( 1 − β ) ​ p S m=\beta p_{T}+(1-\beta)p_{S} is the interpolated mixture distribution. This full-vocabulary formulation provides dense, token-level feedback: the teacher, informed by y ⋆ y^{\star} , exposes the student to the entire distribution over plausible next tokens and guides it toward reasoning paths that lead to the correct answer.

We minimize the expected divergence between teacher and student over on-policy student samples: ℒ ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) [ D ( p T ∥ p S ) ( y ^ ∣ x ) ] ] . \mathcal{L}(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\left[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\left[D\bigl(p_{T}\,\|\,p_{S}\bigr)(\hat{y}\mid x)\right]\right]. (8) Gradients are backpropagated only through the student policy p S p_{S} , while the teacher p T p_{T} acts as a fixed full-distribution target conditioned on privileged information ( x , y ⋆ ) (x,y^{\star}) .

##### Per-Token Pointwise Divergence Clipping.

In our experiments, we observe that token-level divergence is highly skewed across vocabulary entries: a small subset of stylistic tokens exhibits much higher divergence than mathematically meaningful tokens (see Table 5 ). This imbalance causes the training signal to be dominated by stylistic patterns. To address this, we apply pointwise clipping to the vocabulary-level divergence contributions. Let D f ( p T ∥ p S ) D_{f}(p_{T}\|p_{S}) denote an f f -divergence. At each token position n n and vocabulary entry v v , define: ℓ n , v ( f ) = p T ( v ∣ ⋅ ) f ( p S ( v ∣ ⋅ ) p T ( v ∣ ⋅ ) ) . \ell_{n,v}^{(f)}=p_{T}(v\mid\cdot)\;f\!\left(\frac{p_{S}(v\mid\cdot)}{p_{T}(v\mid\cdot)}\right). We compute the clipped divergence: D clip ( f ) ( p T ∥ p S ) = 1 | y ^ | ∑ n = 1 | y ^ | ∑ v ∈ 𝒱 min ( ℓ n , v ( f ) , τ ) . D_{\mathrm{clip}}^{(f)}(p_{T}\|p_{S})=\frac{1}{|\hat{y}|}\sum_{n=1}^{|\hat{y}|}\sum_{v\in\mathcal{V}}\min(\ell_{n,v}^{(f)},\tau).

##### Alternative objective: Sampled-token distillation through policy gradient.

Following recent on-policy distillation methods ( Lu and Lab, 2025 ) , we form a sampled-token reward signal (a reverse-KL signal on sampled actions) and optimize with policy gradient. For each position n n in a sampled sequence y ^ \hat{y} , define the advantage term A n ​ ( x , y ^ ) = log ⁡ p T ​ ( y ^ n ∣ x , y ⋆ , y ^ < n ) − log ⁡ p S ​ ( y ^ n ∣ x , y ^ < n ) , A_{n}(x,\hat{y})=\log p_{T}\!\left(\hat{y}_{n}\mid x,y^{\star},\hat{y}_{<n}\right)-\log p_{S}\!\left(\hat{y}_{n}\mid x,\hat{y}_{<n}\right), and optimize the policy-gradient-style objective ℒ ⁡ ( θ ) = − 𝔼 ( x , y ⋆ ) ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) [ 1 | y ^ | ∑ n = 1 | y ^ | A n ( x , y ^ ) × log p S ( y ^ n ∣ x , y ^ < n ) ] ] . \begin{split}\mathcal{L}(\theta)&=-\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\biggl[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\biggl[\frac{1}{|\hat{y}|}\sum_{n=1}^{|\hat{y}|}A_{n}(x,\hat{y})\\ &\qquad\times\log p_{S}\!\left(\hat{y}_{n}\mid x,\hat{y}_{<n}\right)\biggr]\biggr].\end{split} (9) A n ​ ( x , y ^ ) A_{n}(x,\hat{y}) is treated as a constant with respect to θ \theta (i.e., gradients do not flow through the advantage), so that gradients take the usual policy-gradient form A n ​ ∇ θ ​ log ⁡ p S A_{n}\nabla_{\theta}\log p_{S} . Compared to the full-vocabulary divergence objective, this on-policy shaping objective operates only on sampled tokens, using the teacher’s log-probabilities to provide dense, trajectory-level shaping signals without explicitly matching the full distribution at each step.

##### OPSD as dense-reward policy gradient and comparison to STaR.

The objective in Equation 9 can be seen as policy gradient with dense, token-level rewards. In Appendix Appendix D , we formalize this and contrast with STaR ( Zelikman et al., 2022 ) , a closely related method that also uses the same model to generate reasoning traces, then performs rejection sampling followed by SFT on correct traces. This procedure can be viewed as policy gradient with a sequence-level binary reward that assigns identical credit to all tokens and vanishes when samples are incorrect. In contrast, OPSD provides feedback at every token position regardless of final-answer correctness.

## 4 Experiments

We conduct comprehensive experiments to answer the following research questions: (1) How does OPSD compare to SFT and GRPO in reasoning performance and sample efficiency? (§ 4.2 )

(2) How does per-token pointwise KL clipping in OPSD help stabilizing training? (§ 4.3.3 )

(3) What is the effect of generation style, generation length on performance? (§ 4.3.4 )

(4) Does full-vocabulary logit distillation provide benefits over sampled-token policy gradient? (§ 4.3.5 )

### 4.1 Experimental Setup

Models and datasets. We experiment with the Qwen3 ( Team, 2025b ) model family at three scales: Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, using the instruct-tuned versions. For training data, we use the mathematical reasoning subset of OpenThoughts ( Guha et al., 2025 ) , sampling up to 30K problem-solution pairs with chain-of-thought reasoning. We evaluate on competition-level mathematics benchmarks including AIME 2024, AIME 2025, HMMT 2025.

Baselines. We compare against two methods trained on the same dataset: (1) SFT , standard supervised fine-tuning on expert trajectories, which can be seen as off-policy distillation from a more powerful LLM that generated the reasoning traces; (2) GRPO ( Shao et al., 2024 ) , group relative policy optimization with binary outcome rewards verified against ground-truth answers. The max generation length is set to 16k.

Implementation details. We fix the teacher policy to be the initial policy, rather than the currently updating learning policy, as we find this helps stabilize training and implicitly acts as regularization to prevent excessive deviation from the initial policy. We use full-vocabulary logit distillation in our experiments. All experiments are conducted on A100 or H100 GPUs with LoRA ( Hu et al., 2022 ) . More experimental details are in Appendix B .

### 4.2 Main Results

Table 2 reports results on competition-level mathematical reasoning benchmarks. OPSD consistently outperforms SFT and improves over the base model across all scales, matching or exceeding GRPO in every setting. Notably, OPSD achieves these gains using only a single rollout per problem and converges within 100 steps, with each problem requiring only 1024 sampled tokens, whereas GRPO requires 8 rollouts of 16k tokens each and may exhibit performance degradation in later steps due to entropy collapse—with most of reward standard deviations within a group being zero under this OpenThoughts dataset, yielding no learning signal and wasting sampling budget. We also observe consistent performance degradation under SFT across tasks and model scales when trained on the same dataset, which we attribute to the concise reasoning style of the ground truth solutions which has reduced reasoning lengths at test time. We attribute OPSD’s token efficiency to dense token-level supervision from the teacher distribution, and we hypothesize that earlier tokens may contribute more to effective distillation as they could represent more critical branching points in the reasoning process.

As shown in Figure 3 , OPSD achieves higher token learning efficiency within 100 steps of training as compared to GRPO. Within 100 steps, GRPO’s performance stagnates with less learning signal when the outcome reward within as sampling group remains the same, leading to zero gradient. These results suggest that OPSD may extract learning signal from the same reasoning datasets more efficiently than both GRPO and SFT, while substantially reducing training time.

### 4.3 Ablation Studies & Discussions

In this section, we conduct extensive ablations to study key design choices in OPSD, including (1) the divergence objective, (2) the generation styles of the student and teacher (e.g., thinking-mode on/off), (3) the effect of per-token KL clipping, (4) the impact of student generation length, and (5) comparison between full-vocabulary logit distillation with sampled-token distillation.

#### 4.3.1 Effect of Divergence Objective

A key design choice in OPSD is the divergence used for per-token distribution matching between the privileged teacher and the student. We compare forward KL, reverse KL, and JSD on AIME25 with Qwen3-1.7B in Table 3 . All objectives are evaluated under the same pointwise clipping scheme for stability. Forward KL consistently yields the strongest gains, improving performance from 36.7 to 43.9 at step 50 and remaining above the baseline at step 100. In contrast, reverse KL and JSD provide limited or negative improvements. We therefore adopt forward KL in all remaining experiments.

#### 4.3.2 Effect of Generation Styles and per-token KL Clipping

Another key design choice in OPSD is the generation style of the student and teacher models, as it determines both which tokens the student learns from and the style of supervision provided by the teacher. Qwen3 models support two generation modes: Thinking Mode on (TM-on), in which the model produces self-reflective chain-of-thought tokens, and Thinking Mode off (TM-off), in which it generates responses directly. To determine which combination yields the most effective learning signal, we analyze the forward KL divergence KL ( p T ∥ p S ) \mathrm{KL}(p_{T}\|p_{S}) across all four student/teacher mode pairings, categorizing tokens into three groups: math (numerals, operators, and mathematical keywords), style (reasoning connectives), and other . Table 5 reports the mean per-token KL within each category.

Across all model sizes, the TM-off student paired with a TM-on teacher yields the largest KL on math tokens, indicating stronger supervision on mathematically relevant tokens. The reported KL values correspond to the expected divergence over the vocabulary at each position; as shown in Table 5 , this expectation is highly skewed, with stylistic tokens contributing disproportionately large values. This motivates our use of pointwise clipping to control such heavy-tailed contributions. Empirically, this configuration achieves the best downstream performance. We therefore adopt the TM-off student / TM-on teacher configuration.

#### 4.3.3 Effect of Per-Token Pointwise Clipping

As shown in Table 5 , stylistic tokens can exhibit higher KL divergence than math-related tokens, causing them to dominate the training signal. We mitigate this issue using per-token pointwise clipping. As shown in Figure 4 for Qwen3-1.7B, clipping stabilizes training and prevents performance degradation, which is particularly important given that OPSD converges rapidly within a hundred steps of training.

#### 4.3.4 Effect of Generation Length

Since our objective operates at the token level (Eq. 6 ), the number of generated tokens per sample directly determines the amount of supervision signal available to the student. Longer sequences expose the student to more teacher feedback, but they also increase computational cost and may introduce noisy or uninformative continuations. To study this trade-off, we conduct an ablation on Qwen3-1.7B by varying the generation length of on-policy sampled student responses among 1024 and 4096 tokens and use full-vocabulary logit distillation. As shown in Figure 5 , increasing the generation length does not lead to consistent improvements across either task. We attribute this to early tokens being more critical for learning: as the student generation grows longer, later tokens become increasingly predictable to the teacher when conditioned on a sufficiently long student prefix so less penalties are applied to later tokens. This phenomenon is also noted in ( Lu and Lab, 2025 ) .

#### 4.3.5 Learning Objective Comparison: Full Vocabulary Logits Distillation vs. Sampled-Token Distillation

Our objective in Eq. 6 is defined as a per-token discrepancy between the teacher and student distributions . In practice, OPSD can instantiate this objective in two ways. (1) Full-vocabulary logit distillation (as in GKD ( Agarwal et al., 2024 ) ): for each token position, we compute D ( p T ∥ p S ) D(p_{T}\,\|\,p_{S}) over the entire vocabulary via a full softmax, yielding a proper token-level f f -divergence between the two policies. (2) Sampled-token advantage policy-gradient objective (as in the on-policy distillation method of Lu and Lab (2025) ): we evaluate teacher and student log-probabilities only at the token actually sampled by the student, y ^ n \hat{y}_{n} , and use the reverse-KL term as a scalar advantage inside a policy-gradient-style loss. Thus, the first variant directly matches full token distributions, whereas the second optimizes an on-policy RL objective shaped by the teacher’s log-probabilities rather than a full-distribution divergence. We compare these variants on Qwen3-4B using a 2048-token generation budget during distillation. Table 4 summarizes the results. The full-vocabulary divergence objective provides a consistent gain over the sampled-token objective. This suggests that exposing the student to the full teacher distribution offers richer supervision than relying solely on per-token on-policy shaping. However, the full-vocabulary computation incurs higher peak memory usage due to storing vocabulary-sized logits at every position, indicating a trade-off between performance and efficiency.

## 5 Related Work

##### LLM Self-Training.

Our work connects to a line of research showing that LLMs can improve by generating and exploiting their own supervision signals ( Allen-Zhu and Li, 2020 ; Xu et al., 2024b ; Chen et al., 2024 ; Wang et al., 2023 ; Sun et al., 2023 ; Yuan et al., 2024 ; Yang et al., 2024 ) . Closest in spirit is context distillation ( Snell et al., 2022 ) , which uses the same underlying model as both teacher and student by providing the teacher with privileged context and then SFT the student on the teacher’s generated outputs without context. This can be viewed as off-policy , where the learning signal is a discrete token sequence. In the reasoning domain, ReST ( Gulcehre et al., 2023 ) and STaR ( Zelikman et al., 2022 ) similarly rely on iterative self-training loops—generate rationales conditioned on hints or answers, filter by rewards or ground-truth answers, and fine-tune on successful trajectories—again yielding hard distillation; Mitra and Ulukus (2025) extends this to soft distillation. In-context editing ( Qi et al., 2025 ) does on-policy sample from student and shows that context-induced knowledge can be internalized via soft distillation by minimizing divergences and demonstrates this in knowledge editing settings. OPSD differs from these approaches in that we perform on-policy, soft distillation on the student’s own rollouts for reasoning tasks: the teacher’s supervision is per-token distribution matching rather than generating a rationale for SFT. OPSD frames reasoning improvement as learning a conditional distribution induced jointly by the dataset’s ground-truth solutions and the model’s own reasoning ability. Concurrently, SDPO ( Hübotter et al., 2026 ) explored similar algorithm with environment feedbacks as privilledged information and SDFT ( Shenfeld et al., 2026 ) explored on-policy self-distillation on continual learning tasks.

On-Policy Distillation methods train a student model directly on trajectories sampled from its own policy, while a teacher model provides per-token guidance through KL-based regularization or related objectives ( Agarwal et al., 2024 ; Xu et al., 2024a ; Gu et al., 2024 ; Lu and Lab, 2025 ; Xiaomi, 2026 ; Yang et al., 2025 ) . These approaches mitigate distribution shift by optimizing directly on the student’s visitation distribution, but they typically rely on a distinct and often larger teacher model. In this work, we explore whether an LLM can teach itself by conditioning on more privileged answer information and leveraging its own reasoning capability to guide a weaker version of itself toward improved reasoning. On-policy training paradigms are also widely used in robotics and deep reinforcement learning, such as DAgger ( Ross et al., 2011 ) , where a human teacher provides corrective supervision on the states visited by the student policy.

Improving LLM Reasoning through SFT and RL. SFT and RL are two primary methods for improving LLM reasoning ability. SFT on high-quality reasoning traces has demonstrated strong performance ( Yu et al., 2023 ; LI et al., 2024 ; Paster et al., 2023 ; Team, 2025a ; Ye et al., 2025 ; Muennighoff et al., 2025 ; Zhou et al., 2023 ) . However, prior work shows that SFT can rely on memorization rather than robust generalization ( Chu et al., 2025 ) . In contrast, RL optimizes directly for outcome-based objectives can exhibit better generalization ( Huan et al., 2025 ) . More recent algorithms such as GRPO ( Guo et al., 2025 ; Shao et al., 2024 ) enable scalable RL by estimating advantages from group-level rewards without requiring an explicit critic as in PPO ( Schulman et al., 2017 ) . Building on this line of work, a growing body of research highlights the effectiveness of RLVR for reasoning tasks ( Yu et al., 2025 ; Liu et al., 2025 ; Yue et al., 2025 ; An et al., 2025 ; Zheng et al., 2025 ) .

## 6 Conclusion

We introduced On-Policy Self-Distillation (OPSD), a simple yet effective framework for post-training large language models on reasoning tasks. The intuition behind OPSD is that a sufficiently capable reasoning LLM can teach itself when it has access to privileged information about the answer to a reasoning problem, utilizing its own rationalization ability to grade its weaker self without access to the ground truth. We experimentally demonstrated that OPSD achieves better performance than off-policy distillation/SFT, and performs on par with or better than GRPO, while exhibiting significantly better sample efficiency than GRPO.

## 7 Impact Statement

This paper presents work whose goal is to advance the field of machine learning. Our method improves the efficiency of training language models for reasoning tasks, reducing computational costs compared to existing reinforcement learning approaches. We do not foresee specific negative societal consequences.

## References

Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. R. Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes . In The twelfth international conference on learning representations , Cited by: §1 , §2.1 , §3.1 , §4.3.5 , Table 4 , §5 .

Allen-Zhu and Li (2020) Z. Allen-Zhu and Y. Li Towards understanding ensemble, knowledge distillation and self-distillation in deep learning . In The Eleventh International Conference on Learning Representations , Cited by: §5 .

An et al. (2025) C. An, Z. Xie, X. Li, L. Li, J. Zhang, S. Gong, M. Zhong, J. Xu, X. Qiu, M. Wang, and L. Kong POLARIS: a post-training recipe for scaling reinforcement learning on advanced reasoning models . External Links: Link Cited by: §5 .

Chen et al. (2024) Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu Self-play fine-tuning converts weak language models to strong language models . In International Conference on Machine Learning , pp. 6621–6642 . Cited by: §5 .

Chu et al. (2025) T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma Sft memorizes, rl generalizes: a comparative study of foundation model post-training . arXiv preprint arXiv:2501.17161 . Cited by: §1 , §5 .

Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models . In ICLR , Cited by: §5 .

Guha et al. (2025) E. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. Sprague, A. Suvarna, B. Feuer, L. Chen, Z. Khan, E. Frankel, S. Grover, C. Choi, N. Muennighoff, S. Su, W. Zhao, J. Yang, S. Pimpalgaonkar, K. Sharma, C. C. Ji, Y. Deng, S. Pratt, V. Ramanujan, J. Saad-Falcon, J. Li, A. Dave, A. Albalak, K. Arora, B. Wulfe, C. Hegde, G. Durrett, S. Oh, M. Bansal, S. Gabriel, A. Grover, K. Chang, V. Shankar, A. Gokaslan, M. A. Merrill, T. Hashimoto, Y. Choi, J. Jitsev, R. Heckel, M. Sathiamoorthy, A. G. Dimakis, and L. Schmidt OpenThoughts: data recipes for reasoning models . External Links: 2506.04178 , Link Cited by: §1 , §4.1 .

Gulcehre et al. (2023) C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling . arXiv preprint arXiv:2308.08998 . Cited by: §5 .

Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948 . Cited by: §1 , §5 .

Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network . External Links: 1503.02531 , Link Cited by: §1 , §2.1 .

Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models . In International Conference on Learning Representations , External Links: Link Cited by: §4.1 .

Huan et al. (2025) M. Huan, Y. Li, T. Zheng, X. Xu, S. Kim, M. Du, R. Poovendran, G. Neubig, and X. Yue Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning . arXiv preprint arXiv:2507.00432 . Cited by: §5 .

Hübotter et al. (2026) J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation . arXiv preprint arXiv:2601.20802 . Cited by: §5 .

Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation . In Proceedings of the 2016 conference on empirical methods in natural language processing , pp. 1317–1327 . Cited by: §2.1 .

LI et al. (2024) J. LI, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. C. Huang, K. Rasul, L. Yu, A. Jiang, Z. Shen, Z. Qin, B. Dong, L. Zhou, Y. Fleureau, G. Lample, and S. Polu NuminaMath . Numina . Note: https://github.com/project-numina/aimo-progress-prize/blob/main/report/numina_dataset.pdf Cited by: §5 .

Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step . In The Twelfth International Conference on Learning Representations , Cited by: §3.1 .

Liu et al. (2025) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective . arXiv preprint arXiv:2503.20783 . Cited by: §5 .

Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization . arXiv preprint arXiv:1711.05101 . Cited by: Appendix B .

Lu and Lab (2025) K. Lu and T. M. Lab On-policy distillation . Thinking Machines Lab: Connectionism . Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §1 , §1 , §2.1 , §3.1 , §3.2 , §4.3.4 , §4.3.5 , Table 4 , §5 .

Mitra and Ulukus (2025) P. Mitra and S. Ulukus Semantic soft bootstrapping: long context reasoning in llms without reinforcement learning . arXiv preprint arXiv:2512.05105 . Cited by: §5 .

Muennighoff et al. (2025) N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto S1: simple test-time scaling . arXiv preprint arXiv:2501.19393 . Cited by: §5 .

Naor (1996) M. Naor Evaluation may be easier than generation . In Proceedings of the twenty-eighth annual ACM symposium on Theory of computing , pp. 74–83 . Cited by: §1 .

Paster et al. (2023) K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba OpenWebMath: an open dataset of high-quality mathematical web text . External Links: 2310.06786 Cited by: §5 .

Qi et al. (2025) S. Qi, B. Yang, K. Jiang, X. Wang, J. Li, Y. Zhong, Y. Yang, and Z. Zheng In-context editing: learning knowledge from self-induced distributions . In The Thirteenth International Conference on Learning Representations , Cited by: §5 .

Rastogi et al. (2025) A. Rastogi, A. Q. Jiang, A. Lo, G. Berrada, G. Lample, J. Rute, J. Barmentlo, K. Yadav, K. Khandelwal, K. R. Chandu, et al. Magistral . arXiv preprint arXiv:2506.10910 . Cited by: §1 .

Ross et al. (2011) S. Ross, G. Gordon, and D. Bagnell A reduction of imitation learning and structured prediction to no-regret online learning . In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pp. 627–635 . Cited by: §2.1 , §5 .

Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, and T. Wolf DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter . arXiv preprint arXiv:1910.01108 . Cited by: §2.1 .

Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms . arXiv preprint arXiv:1707.06347 . Cited by: §2.2 , §5 .

Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models . arXiv preprint arXiv:2402.03300 . Cited by: §1 , §2.2 , §4.1 , §5 .

Shenfeld et al. (2026) I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal Self-distillation enables continual learning . External Links: 2601.19897 , Link Cited by: §5 .

Snell et al. (2022) C. Snell, D. Klein, and R. Zhong Learning by distilling context . arXiv preprint arXiv:2209.15189 . Cited by: §5 .

Sun et al. (2023) Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, and C. Gan Principle-driven self-alignment of language models from scratch with minimal human supervision . In Thirty-seventh Conference on Neural Information Processing Systems , External Links: Link Cited by: §5 .

Sun et al. (2024) Z. Sun, L. Yu, Y. Shen, W. Liu, Y. Yang, S. Welleck, and C. Gan Easy-to-hard generalization: scalable alignment beyond human supervision . Advances in Neural Information Processing Systems 37 , pp. 51118–51168 . Cited by: §1 .

Team et al. (2025) K. Team, Y. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen, Y. Chen, et al. Kimi k2: open agentic intelligence . arXiv preprint arXiv:2507.20534 . Cited by: §1 .

Team (2025a) O. Team Open Thoughts . Note: https://open-thoughts.ai Cited by: §5 .

Team (2025b) Q. Team Qwen3 technical report . External Links: 2505.09388 , Link Cited by: §4.1 .

Wang et al. (2023) Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi Self-instruct: aligning language models with self-generated instructions . In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers) , pp. 13484–13508 . Cited by: §5 .

Xiaomi (2026) L. Xiaomi MiMo-v2-flash technical report . External Links: 2601.02780 , Link Cited by: §1 , §5 .

Xu et al. (2024a) W. Xu, R. Han, Z. Wang, L. Le, D. Madeka, L. Li, W. Y. Wang, R. Agarwal, C. Lee, and T. Pfister Speculative knowledge distillation: bridging the teacher-student gap through interleaved sampling . In The Thirteenth International Conference on Learning Representations , Cited by: §2.1 , §3.1 , §5 .

Xu et al. (2024b) X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou A survey on knowledge distillation of large language models . CoRR . Cited by: §5 .

Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report . arXiv preprint arXiv:2505.09388 . Cited by: §1 , §5 .

Yang et al. (2024) Z. Yang, T. Pang, H. Feng, H. Wang, W. Chen, M. Zhu, and Q. Liu Self-distillation bridges distribution gap in language model fine-tuning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1028–1043 . Cited by: §5 .

Ye et al. (2025) Y. Ye, Z. Huang, Y. Xiao, E. Chern, S. Xia, and P. Liu LIMO: less is more for reasoning . External Links: 2502.03387 , Link Cited by: §5 .

Yu et al. (2023) L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu MetaMath: bootstrap your own mathematical questions for large language models . arXiv preprint arXiv:2309.12284 . Cited by: §5 .

Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale, 2025 . URL https://arxiv. org/abs/2503.14476 . Cited by: §1 , §1 , §5 .

Yuan et al. (2024) W. Yuan, R. Y. Pang, K. Cho, X. Li, S. Sukhbaatar, J. Xu, and J. E. Weston Self-rewarding language models . In International Conference on Machine Learning , pp. 57905–57923 . Cited by: §5 .

Yue et al. (2025) Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du, et al. Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks . arXiv preprint arXiv:2504.05118 . Cited by: §5 .

Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman Star: bootstrapping reasoning with reasoning . Advances in Neural Information Processing Systems 35 , pp. 15476–15488 . Cited by: §D.1 , §3.2 , §5 .

Zhang et al. (2025) Z. Zhang, C. Zheng, Y. Wu, B. Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and J. Lin The lessons of developing process reward models in mathematical reasoning . arXiv preprint arXiv:2501.07301 . Cited by: §3.1 .

Zhao et al. (2025) S. Zhao, M. Liu, J. Huang, M. Liu, C. Wang, B. Liu, Y. Tian, G. Pang, S. Bell, A. Grover, et al. Inpainting-guided policy optimization for diffusion large language models . arXiv preprint arXiv:2509.10396 . Cited by: §1 .

Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. Group sequence policy optimization . arXiv preprint arXiv:2507.18071 . Cited by: §5 .

Zhou et al. (2023) C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. LIMA: less is more for alignment . In Proceedings of the 37th International Conference on Neural Information Processing Systems , pp. 55006–55021 . Cited by: §5 .

## Appendix A Limitations and Future Directions

Due to computational constraints, our experiments are limited to models up to 8B parameters. It remains an open question whether this trend continues at scales beyond 8B parameters. Several promising directions warrant further investigation. First, our current framework does not explicitly leverage correctness verification of generated answers; incorporating such signals could provide additional learning objectives beyond distribution matching. Finally, problem difficulty plays a crucial role in self-distillation: if reasoning problems exceed the model’s comprehension threshold, the teacher policy cannot provide meaningful supervision even with access to ground-truth solutions. This suggests that curriculum learning strategies—gradually increasing problem difficulty as the model improves—could enhance training effectiveness. Exploring adaptive curricula that maintain problems at the frontier of model capabilities represents an important direction for scaling OPSD to more challenging reasoning tasks.

## Appendix B Experimental Details

We provide the training and evaluation configurations for our SFT, GRPO and OPSD experiments in Tables 7 , 6 and 8 . Note that we adopt the Thinking-Mode-off student / Thinking-Mode-on teacher configuration for main OPSD experiments. For more experiment details, please refer to our released training code in https://github.com/siyan-zhao/OPSD .We didn’t conduct tuning for the clipping parameter τ \tau , optimizing this hyperparameter may yield further performance gains within the same 100-step budget for larger models.

All experiments were conducted using 8 A100 or H100 GPUs with gradient checkpointing and Flash Attention 2 for memory efficiency. We use the AdamW ( Loshchilov and Hutter, 2017 ) optimizer and bfloat16 precision for all training runs. For OPSD, unless otherwise stated, we used full-vocabulary logit distillation.

## Appendix C Token Category Definitions

We categorize tokens into style and math groups using predefined keyword lists. These keyword sets are used to analyze the per-token KL divergence stylistic tokens and mathematical knowledge tokens as in Section 4.3.1 .

##### Style Tokens.

maybe, perhaps, probably, possibly, let, okay, ok, alright, hmm, wait, because, since, so, thus, hence, therefore, but, however, although, though, yet, or, alternatively, instead, otherwise, actually, really, just, simply, basically, very, quite, pretty, rather, fairly, now, then, next, first, second, finally, try, see, check, note, recall, think, idea, strategy, approach, method, way, would, could, should, might, can, huge, large, big, small, tiny, interesting, tricky, complex, simple.

##### Math Tokens.

exponential, exponent, power, powers, base, logarithm, logarithms, log, ln, compare, comparing, comparison, less, equal, larger, smaller, greater, factor, factors, prime, divisible, equation, expression, formula, inequality, rational, irrational, real, integer, coefficient, variable, constant, sum, product, difference, quotient, fraction, denominator, numerator, root, square, cube, nth, maximum, minimum, optimize, bound.

## Appendix D Policy-Gradient Interpretation of OPSD and Comparison to STaR

Our OPSD objective in Equation 9 can be interpreted as a policy-gradient update with a dense, token-level reward signal derived from privileged information. In this section, we show: (1) OPSD can be seen as a dense-reward policy gradient, and (2) we contrast OPSD with STaR, demonstrating that STaR’s learning signal is sequence-level while OPSD is token-level .

### D.1 STaR as Sequence-Level Policy-Gradient

STaR ( Zelikman et al., 2022 ) can be viewed as an approximation to an RL-style policy gradient objective. The language model p θ p_{\theta} induces a joint distribution over rationale r r and answer y y : p θ ​ ( r , y ∣ x ) = p θ ​ ( r ∣ x ) ​ p θ ​ ( y ∣ x , r ) , p_{\theta}(r,y\mid x)=p_{\theta}(r\mid x)\,p_{\theta}(y\mid x,r), where the model first samples a latent rationale r r before predicting the final answer y y . Given an indicator reward R ⁡ ( y ) = 𝟏 ​ ( y = y ⋆ ) R(y)=\mathbf{1}(y=y^{\star}) , the expected return across the dataset 𝒮 = { ( x i , y i ⋆ ) } i = 1 N \mathcal{S}=\{(x_{i},y_{i}^{\star})\}_{i=1}^{N} is J STaR ( θ ) = ∑ i = 1 N 𝔼 ( r , y ) ∼ p θ ( ⋅ ∣ x i ) [ 𝟏 ( y = y i ⋆ ) ] . J_{\text{STaR}}(\theta)=\sum_{i=1}^{N}\mathbb{E}_{(r,y)\sim p_{\theta}(\cdot\mid x_{i})}\big[\mathbf{1}(y=y_{i}^{\star})\big]. (10) Applying the log-derivative trick yields a policy gradient: ∇ θ J STaR ( θ ) = ∑ i = 1 N 𝔼 ( r , y ) ∼ p θ ( ⋅ ∣ x i ) [ 𝟏 ( y = y i ⋆ ) ∇ θ log p θ ( r , y ∣ x i ) ] . \nabla_{\theta}J_{\text{STaR}}(\theta)=\sum_{i=1}^{N}\mathbb{E}_{(r,y)\sim p_{\theta}(\cdot\mid x_{i})}\Big[\mathbf{1}(y=y_{i}^{\star})\,\nabla_{\theta}\log p_{\theta}(r,y\mid x_{i})\Big]. (11) Note that the indicator function discards the gradient for all sampled rationales that do not lead to the correct answer y i ⋆ y_{i}^{\star} : this corresponds to the filtering step in STaR.

One limitation is that STaR’s reward is sequence-level : the binary indicator 𝟏 ​ ( y = y ⋆ ) \mathbf{1}(y=y^{\star}) provides the same signal to all tokens in a trajectory, offering no intermediate credit assignment. When all sampled trajectories are all incorrect, the learning signal vanishes.

### D.2 OPSD as Dense-Reward Policy Gradient

The sampled-token objective in Equation 9 can also be viewed as a policy-gradient method, but with a token-level reward. Fix a training pair ( x , y ⋆ ) (x,y^{\star}) and let the student generate a trajectory y ^ ∼ p S ( ⋅ ∣ x ) \hat{y}\sim p_{S}(\cdot\mid x) . At each position n n , define the per-token reward: r n ​ ( x , y ^ ) ≜ log ⁡ p T ​ ( y ^ n ∣ x , y ⋆ , y ^ < n ) − log ⁡ p S ​ ( y ^ n ∣ x , y ^ < n ) . r_{n}(x,\hat{y})\triangleq\log p_{T}(\hat{y}_{n}\mid x,y^{\star},\hat{y}_{<n})-\log p_{S}(\hat{y}_{n}\mid x,\hat{y}_{<n}). This reward measures how much the privileged teacher prefers the sampled token y ^ n \hat{y}_{n} relative to the student. As stated in the main text, we treat r n r_{n} (equivalently, the advantage A n A_{n} ) as a constant with respect to θ \theta when computing gradients—that is, we stop gradients through both p T p_{T} and p S p_{S} in the reward computation. Under this treatment, the gradient of Equation 9 takes the standard policy-gradient form: ∇ θ ℒ ( θ ) = − 𝔼 ( x , y ⋆ ) ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) [ 1 | y ^ | ∑ n = 1 | y ^ | r n ( x , y ^ ) ∇ θ log p S ( y ^ n ∣ x , y ^ < n ) ] ] , \nabla_{\theta}\mathcal{L}(\theta)=-\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\left[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\left[\frac{1}{|\hat{y}|}\sum_{n=1}^{|\hat{y}|}r_{n}(x,\hat{y})\,\nabla_{\theta}\log p_{S}(\hat{y}_{n}\mid x,\hat{y}_{<n})\right]\right], which corresponds to maximizing the expected per-token reward along on-policy student rollouts: J O ​ P ​ S ​ D ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒮 [ 𝔼 y ^ ∼ p S ( ⋅ ∣ x ) [ 1 | y ^ | ∑ n = 1 | y ^ | r n ( x , y ^ ) ] ] . J_{OPSD{}}(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{S}}\left[\mathbb{E}_{\hat{y}\sim p_{S}(\cdot\mid x)}\left[\frac{1}{|\hat{y}|}\sum_{n=1}^{|\hat{y}|}r_{n}(x,\hat{y})\right]\right]. This reward is dense: it provides a learning signal at every token position, regardless of whether the final answer is correct.

##### Comparison.

Both STaR and OPSD can be understood as policy-gradient methods, but their reward structures differ fundamentally. STaR uses a sequence-level indicator 𝟏 ​ ( y = y ⋆ ) \mathbf{1}(y=y^{\star}) that assigns the same signal to all tokens; when all sampled trajectories are incorrect, the learning signal vanishes entirely. In contrast, OPSD provides a token-level reward r n r_{n} at every position, enabling fine-grained credit assignment even when the final answer is wrong.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .