Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — part 1/5

Cleaned-text code points [0, 15467). Bibliography omitted here; source.md includes it.

\setheadertext Rethinking On-Policy Distillation \pretitlefigure

# Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

###### Abstract

On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student’s perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%–99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD’s apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

## 1 Introduction

On-policy distillation (OPD) has rapidly emerged as a core technique for large language model (LLM) post-training. Recent industry efforts, including Qwen3 [ Yang et al., 2025 ] , MiMo [ Xiao et al., 2026 ] and GLM-5 [ Zeng et al., 2026 ] , all adopt OPD in their post-training pipelines and report substantial gains, establishing it as a competitive complement to conventional supervised fine-tuning (SFT) and outcome-reward reinforcement learning (RL). Thinking Machines Lab [ Lu and Lab, 2025 ] replicates the Qwen3 OPD recipe at a fraction of the RL compute cost, independently confirming that on-policy, dense supervision is a practically efficient alternative.

Unlike off-policy distillation, which trains the student on fixed teacher-generated sequences and suffers from exposure bias [ Bengio et al., 2015 ] , OPD has the student generate its own rollouts and leverages the teacher’s per-token log-probabilities as a dense reward signal to refine behavior on states the student actually visits. Recently, this has been extended to self-distillation settings where a single model serves as its own teacher given privileged information, demonstrating that the framework can drive continual self-improvement [ Hübotter et al., 2026 , Zhao et al., 2026b , Shenfeld et al., 2026 ] .

However, despite these successes, OPD remains poorly understood and fragile in practice. We observe a striking failure mode: a stronger teacher can completely fail to improve a student, even when a weaker teacher succeeds from lower initial alignment. Yet few studies have investigated why the teacher’s token-level signal steers the student distribution in the desired direction, or the conditions under which it fails.

⇒ \boldsymbol{\Rightarrow} Why does OPD work at token level? ⇒ \boldsymbol{\Rightarrow} How to rescue failing OPD?

We present a systematic study of OPD training dynamics, progressing from empirical conditions through token-level mechanism to practical recipe.

Phenomenology (§ 3 ). We investigate the empirical patterns that distinguish effective from ineffective OPD, and identify two governing factors. (i) Thinking-pattern consistency : the student and teacher should share consistent thinking patterns (e.g. higher overlap ratio in their top- k k token distributions). Even when the teacher achieves higher benchmark scores, mismatched thinking patterns produce low initial overlap that training cannot fully recover. (ii) Higher scores ≠ \neq new knowledge : even with consistent thinking patterns and higher benchmark scores, the teacher should offer knowledge that the student has not already acquired. When both models are trained on the same data and recipe, they converge to similar distributions at their respective scales, leaving the teacher with little transferable signal. Only when the teacher carries knowledge beyond what the student has already seen can OPD yield substantial gains. We validate both conditions through reverse distillation experiments, which further reveal that OPD fundamentally learns thinking patterns rather than merely benefiting from pattern consistency, and that training dynamics can be entirely decoupled from benchmark scores.

Mechanism (§ 4 ). We then investigate the token-level mechanism based on these conditions. Across all settings studied, effective OPD exhibits a consistent signature where the student and teacher distributions become progressively more similar on student-visited states. The high-probability tokens increasingly coincide (overlap ratio rising from 72% to 91%), the entropy gap between the two distributions narrows, and the shared top- k k tokens concentrate 97–99% of the combined probability mass. By contrast, failing runs show stagnant overlap and persistent entropy mismatch from the outset. We further show that restricting supervision to overlap tokens alone matches full top- k k performance, confirming that the overlap set is the principal locus of OPD’s gradient signal.

Recipe (§ 5 ). Guided by the phenomenological analysis, we propose two complementary strategies that recover OPD in otherwise failing configurations: (i) off-policy cold start , a warmup SFT phase on teacher-generated rollouts before OPD that bridges the thinking-pattern gap by raising initial overlap ratio; (ii) teacher-aligned prompt selection , which uses prompts drawn from the teacher’s post-training data to sharpen alignment on high-probability tokens, though at the cost of substantially lower student entropy that necessitates mixing with out-of-distribution prompts. In both cases, the recovered runs exhibit the same dynamic signature as naturally successful ones as shown in § 4 : steadily rising overlap ratio, improving token-level advantage, and a narrowing entropy gap.

Finally, we examine the cost of OPD’s dense supervision (§ 6 ). We show that reward quality degrades systematically with trajectory depth and that instability originates at later tokens before propagating backward through the trajectory. Surprisingly, even failing teachers provide reward that is globally correlated with rollout correctness, suggesting that the failure is not one of signal quality but of local optimization geometry. A larger teacher may induce a reward landscape that is locally flat around the student’s policy, making token-level gradients ineffective despite an informative global signal. These findings reveal a fundamental tension between supervision density and supervision reliability, and point to the limitations of current OPD for long-horizon reasoning and agentic settings.

## 2 Preliminaries

### 2.1 Notation

Let x = ( x 1 , … , x n ) x=(x_{1},\ldots,x_{n}) denote an input prompt and y = ( y 1 , … , y m ) y=(y_{1},\ldots,y_{m}) a response. We write y < t ≜ ( y 1 , … , y t − 1 ) y_{<t}\triangleq(y_{1},\ldots,y_{t-1}) for the prefix up to step t t . We consider two LLMs: a student π θ \pi_{\theta} and a teacher π T \pi_{T} , each defining a next-token distribution π ( ⋅ ∣ x , y < t ) \pi(\cdot\mid x,y_{<t}) over a vocabulary 𝒱 \mathcal{V} . We write y ∼ π θ ( ⋅ ∣ x ) y\sim\pi_{\theta}(\cdot\mid x) for a response sampled autoregressively from the student. 𝒟 = { ( x ( i ) , y ( i ) ) } i = 1 N \mathcal{D}=\{(x^{(i)},y^{(i)})\}_{i=1}^{N} denotes a fixed dataset of prompt–response pairs with teacher-generated outputs, and 𝒟 x ≜ { x ( i ) } i = 1 N \mathcal{D}_{x}\triangleq\{x^{(i)}\}_{i=1}^{N} the corresponding prompt set. Knowledge distillation (KD) transfers knowledge from π T \pi_{T} to π θ \pi_{\theta} by minimizing the divergence between the two distributions. A standard choice is the Kullback-Leibler (KL) divergence, defined for two distributions P P and Q Q over 𝒱 \mathcal{V} as D KL ( P ∥ Q ) = ∑ v ∈ 𝒱 P ( v ) log P ⁡ ( v ) Q ⁡ ( v ) D_{\mathrm{KL}}(P\|Q)=\sum_{v\in\mathcal{V}}P(v)\log\frac{P(v)}{Q(v)} .

### 2.2 On-Policy Distillation

On-Policy Distillation (OPD) computes supervision on trajectories sampled from the current student π θ \pi_{\theta} . Given a prompt x ∼ 𝒟 x x\sim\mathcal{D}_{x} , the student samples a response y ^ = ( y ^ 1 , … , y ^ T ) ∼ π θ ( ⋅ ∣ x ) \hat{y}=(\hat{y}_{1},\ldots,\hat{y}_{T})\sim\pi_{\theta}(\cdot\mid x) , where T ≜ | y ^ | T\triangleq|\hat{y}| denotes the rollout length. Both models are then evaluated on the student-generated prefixes y ^ < t \hat{y}_{<t} , yielding two next-token distributions at each step t t : p t ​ ( v ) ≜ π θ ​ ( v ∣ x , y ^ < t ) p_{t}(v)\triangleq\pi_{\theta}(v\mid x,\hat{y}_{<t}) and q t ​ ( v ) ≜ π T ​ ( v ∣ x , y ^ < t ) q_{t}(v)\triangleq\pi_{T}(v\mid x,\hat{y}_{<t}) for v ∈ 𝒱 v\in\mathcal{V} .

A standard formulation minimizes the sequence-level reverse KL over student-generated trajectories: ℒ OPD ( θ ) = 𝔼 x ∼ 𝒟 x [ D KL ( π θ ( ⋅ ∣ x ) ∥ π T ( ⋅ ∣ x ) ) ] . \mathcal{L}_{\mathrm{OPD}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{x}}\Bigl[D_{\mathrm{KL}}\bigl(\pi_{\theta}(\cdot\mid x)\;\|\;\pi_{T}(\cdot\mid x)\bigr)\Bigr]. (1)

Using the autoregressive factorization, this sequence-level objective admits the exact token-level decomposition: ℒ OPD ( θ ) = 𝔼 x ∼ 𝒟 x , y ^ ∼ π θ ( ⋅ ∣ x ) [ ∑ t = 1 T D KL ( p t ∥ q t ) ] . \mathcal{L}_{\mathrm{OPD}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{x},\;\hat{y}\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}D_{\mathrm{KL}}(p_{t}\|q_{t})\right]. (2) In practice, different implementations vary in how this exact per-token reverse KL is computed: full-vocabulary OPD optimizes Eq. ( 2 ) directly, sampled-token OPD uses an unbiased Monte Carlo estimator of each token-level KL term, and top- k k OPD replaces the full-vocabulary KL with a subset-based approximation. We now describe these three common supervision granularities.

#### Sampled-Token OPD.

The most lightweight variant evaluates only the token sampled by the student, and is also the most common implementation in prior on-policy distillation work [ Lu and Lab, 2025 , Xiao et al., 2026 , Yang et al., 2026b ] . Given y ^ t ∼ p t \hat{y}_{t}\sim p_{t} , the per-token loss is ℓ t sample ≜ log ⁡ p t ​ ( y ^ t ) − log ⁡ q t ​ ( y ^ t ) \ell_{t}^{\mathrm{sample}}\triangleq\log p_{t}(\hat{y}_{t})-\log q_{t}(\hat{y}_{t}) , aggregated as: ℒ OPD sample ( θ ) = 𝔼 x ∼ 𝒟 x , y ^ ∼ π θ ( ⋅ ∣ x ) [ ∑ t = 1 T ℓ t sample ] . \mathcal{L}_{\mathrm{OPD}}^{\mathrm{sample}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{x},\;\hat{y}\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}\ell_{t}^{\mathrm{sample}}\right]. (3) Since 𝔼 y ^ t ∼ p t [ ℓ t sample ] = D KL ( p t ∥ q t ) \mathbb{E}_{\hat{y}_{t}\sim p_{t}}[\ell_{t}^{\mathrm{sample}}]=D_{\mathrm{KL}}(p_{t}\|q_{t}) , each ℓ t sample \ell_{t}^{\mathrm{sample}} is an unbiased single-sample estimator of the token-level reverse KL.

#### Full-Vocabulary OPD.

At the other extreme, one computes the divergence over the entire vocabulary at each prefix: ℒ OPD full ( θ ) = 𝔼 x ∼ 𝒟 x , y ^ ∼ π θ ( ⋅ ∣ x ) [ ∑ t = 1 T D KL ( p t ∥ q t ) ] . \mathcal{L}_{\mathrm{OPD}}^{\mathrm{full}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{x},\;\hat{y}\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}D_{\mathrm{KL}}(p_{t}\|q_{t})\right]. (4) This yields denser gradients compared to sampled-token OPD, at the cost of O ⁡ ( B ​ T ​ M ) O(BTM) memory for batch size B B , sequence length T T and vocabulary size M = | 𝒱 | M=|\mathcal{V}| .

#### Top- k k OPD.

Top- k k OPD provides an intermediate design between sampled-token and full-vocabulary OPD by restricting the divergence computation to a subset S t ⊆ 𝒱 S_{t}\subseteq\mathcal{V} . Here we focus on the student top- k k variant, which selects the k k tokens assigned the highest probability under the student, namely S t = TopK ⁡ ( p t , k ) S_{t}=\operatorname{TopK}(p_{t},k) . Define the renormalized student and teacher distributions on S t S_{t} as: p ¯ t ( S t ) ​ ( v ) = p t ( v ) 1 [ v ∈ S t ] ∑ u ∈ S t p t ​ ( u ) , q ¯ t ( S t ) ​ ( v ) = q t ( v ) 1 [ v ∈ S t ] ∑ u ∈ S t q t ​ ( u ) . \bar{p}_{t}^{(S_{t})}(v)=\frac{p_{t}(v)\,\mathbf{1}[v\in S_{t}]}{\sum_{u\in S_{t}}p_{t}(u)},\qquad\bar{q}_{t}^{(S_{t})}(v)=\frac{q_{t}(v)\,\mathbf{1}[v\in S_{t}]}{\sum_{u\in S_{t}}q_{t}(u)}. Distillation is then performed by minimizing the subset KL divergence D KL ( p ¯ t ( S t ) ∥ q ¯ t ( S t ) ) D_{\mathrm{KL}}\!\bigl(\bar{p}_{t}^{(S_{t})}\,\|\,\bar{q}_{t}^{(S_{t})}\bigr) , yielding the trajectory-level objective: ℒ OPD top ​ - ​ k ( θ ) = 𝔼 x ∼ 𝒟 x , y ^ ∼ π θ ( ⋅ ∣ x ) [ ∑ t = 1 T D KL ( p ¯ t ( S t ) ∥ q ¯ t ( S t ) ) ] . \mathcal{L}_{\mathrm{OPD}}^{\mathrm{top\text{-}k}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{x},\;\hat{y}\sim\pi_{\theta}(\cdot\mid x)}\left[\sum_{t=1}^{T}D_{\mathrm{KL}}\!\bigl(\bar{p}_{t}^{(S_{t})}\,\|\,\bar{q}_{t}^{(S_{t})}\bigr)\right]. (5)

This formulation discards mass outside S t S_{t} and therefore remains an approximation to the full-vocabulary reverse KL, but it substantially reduces teacher-query cost while preserving multi-token supervision on the student’s high-probability region.

### 2.3 Dynamic Metrics

We define the student’s and teacher’s top- k k sets at step t t as S t ( p ) = TopK ⁡ ( p t , k ) S_{t}^{(p)}=\operatorname{TopK}(p_{t},k) and S t ( q ) = TopK ⁡ ( q t , k ) S_{t}^{(q)}=\operatorname{TopK}(q_{t},k) , respectively. The following metrics are monitored throughout OPD training in later experiments.

#### Overlap Ratio.

This metric quantifies the alignment between the student’s and teacher’s candidate spaces. It is defined as the average proportion of tokens that appear simultaneously in both the student’s and the teacher’s top- k k sets: ℳ overlap ≜ 𝔼 t ​ [ | S t ( p ) ∩ S t ( q ) | k ] . \mathcal{M}_{\text{overlap}}\triangleq\mathbb{E}_{t}\left[\frac{|S_{t}^{(p)}\cap S_{t}^{(q)}|}{k}\right]. (6) A low overlap ratio indicates that the student’s probability mass is concentrated on a disjoint set of tokens from the teacher, suggesting significant policy divergence or “mode mismatch”. Conversely, a ratio nearing 1.0 1.0 implies the student has successfully located the teacher’s support region.

#### Overlap-Token Advantage.

To measure distributional agreement within the overlap tokens, we define A t ​ ( v ) ≜ p ¯ t ​ ( v ) ​ ( log ⁡ q ¯ t ​ ( v ) − log ⁡ p ¯ t ​ ( v ) ) A_{t}(v)\triangleq\bar{p}_{t}(v)(\log\bar{q}_{t}(v)-\log\bar{p}_{t}(v)) where p ¯ t , q ¯ t \bar{p}_{t},\bar{q}_{t} are the renormalized student and teacher distributions over S t ( p ) ∩ S t ( q ) S_{t}^{(p)}\cap S_{t}^{(q)} . The metric averages this quantity: ℳ adv ≜ 𝔼 t ​ [ 1 | S t ( p ) ∩ S t ( q ) | ​ ∑ v ∈ S t ( p ) ∩ S t ( q ) A t ​ ( v ) ] . \mathcal{M}_{\text{adv}}\triangleq\mathbb{E}_{t}\!\left[\frac{1}{|S_{t}^{(p)}\cap S_{t}^{(q)}|}\sum_{v\in S_{t}^{(p)}\cap S_{t}^{(q)}}A_{t}(v)\right]. (7) A value close to zero indicates high-quality alignment where the student places mass on teacher-preferred tokens with appropriate confidence. Conversely, a large negative value indicates that within the intersection, the student is overconfident compared to the teacher (high p t p_{t} but lower q t q_{t} ).

#### Entropy and Entropy Gap.