Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — part 2/5

Cleaned-text code points [15467, 31036). Bibliography omitted here; source.md includes it.

To monitor the distributional properties of the policies, we track the entropy of both the student H ⁡ ( p t ) H(p_{t}) and the teacher H ⁡ ( q t ) H(q_{t}) on the student’s rollouts, and define the entropy gap as: Δ ​ H t = | H ⁡ ( q t ) − H ⁡ ( p t ) | . \Delta H_{t}=\left|H(q_{t})-H(p_{t})\right|. (8) Δ ​ H t \Delta H_{t} is a state-specific indicator of mode alignment. A large gap suggests a substantial mismatch between the student and teacher in confidence and diversity over the same visited states, while convergence toward zero indicates that the student has matched the teacher’s uncertainty profile along its generated trajectories.

## 3 Phenomenology of On-Policy Distillation

Before investigating the token-level mechanism of OPD, we first ask a broader question: what conditions govern the effectiveness of OPD? A natural assumption is that a stronger teacher should always yield better distillation outcomes, yet we observe configurations where this fails. We compare OPD runs under controlled settings and identify two conditions that jointly govern the outcome.

### 3.1 Thinking-Pattern Consistency

We first study whether OPD requires compatible thinking patterns between the student and the teacher. A stronger teacher does not guarantee better distillation: a large mismatch in reasoning pattern can weaken the distillation signal regardless of the teacher’s benchmark advantage.

#### Setup.

We use Qwen3-1.7B-Base [ Yang et al., 2025 ] as the student and compare two teachers: Qwen3-4B (Non-thinking) [ Yang et al., 2025 ] and Qwen3-4B-Base-GRPO, where the latter is obtained by applying zero-RL to Qwen3-4B-Base [ Yang et al., 2025 ] using GRPO [ Shao et al., 2024 ] (detailed training settings are provided in Appendix A.1 ). Since the student is also a base model, we expect its thinking pattern to be closer to that of the GRPO-trained teacher. We conduct two OPD experiments using the DAPO-Math-17K dataset [ Yu et al., 2025 ] , differing only in the choice of teacher model. Unless otherwise specified, all experiments use the default hyperparameters described in Appendix A.2 and are evaluated on AIME 2024 [ Li et al., 2024 ] , AIME 2025 [ Balunović et al., 2025 ] and AMC 2023 [ Li et al., 2024 ] . Following standard practice, we sample 16 solutions per problem with temperature 0.7 and top- p p 0.95, using a maximum validation response length of 31,744 tokens. We report average accuracy over 16 samples (avg@16) as the primary evaluation metric.

#### Results.

As shown in Figure 2 , distillation from Qwen3-4B-Base-GRPO consistently outperforms distillation from Qwen3-4B (Non-thinking), although the two teachers have broadly comparable performance as shown in Figure 3 . Despite underperforming on benchmarks, the GRPO teacher exhibits a higher initial overlap ratio, suggesting that its thinking pattern aligns more closely with the student. Although the two overlap curves converge later in training, the performance gap persists, suggesting that early-stage thinking-pattern mismatch causes a loss of distillation benefit that cannot be recovered later. We report the validation accuracy for each benchmark individually in Appendix A.3 , where the same overall trend holds across all datasets.

### 3.2 New Knowledge, Not Just Scale

Thinking-pattern consistency alone does not explain all of our observations. Even when the teacher scores higher and shares a consistent thinking pattern with the student, OPD can still fail.

#### Setup.

We construct two controlled comparisons across model families. In the DeepSeek family, we use DeepSeek-R1-Distill-Qwen-1.5B (R1-Distill-1.5B) [ Guo et al., 2025 ] as the student and compare two teachers: DeepSeek-R1-Distill-Qwen-7B (R1-Distill-7B) [ Guo et al., 2025 ] and Skywork-OR1-Math-7B [ He et al., 2025b ] , where the latter is obtained by applying RL post-training to R1-Distill-7B. In the Qwen family, we use Qwen3-1.7B (Non-thinking) [ Yang et al., 2025 ] as the student and compare two teachers: Qwen3-4B (Non-thinking) and Qwen3-4B-Non-Thinking-RL-Math [ Yang et al., 2026b ] , where the latter is obtained by applying RL to Qwen3-4B (Non-thinking) on a 57K subset of DeepMath [ He et al., 2025c ] . In both settings, the key contrast lies between a teacher from the same training pipeline and one that has acquired additional capabilities through further RL. All runs use the same dataset and training recipe as before.

#### Results.

As shown in Figure 4 , both families exhibit a consistent pattern. Same-pipeline teachers yield limited improvement, while the post-trained teachers produce substantially stronger gains across all benchmarks. Importantly, the post-trained teachers not only achieve higher absolute performance but also recover a much larger fraction of the teacher-student gap, measured by the gap recovery rate ( Acc after OPD − Acc before OPD ) / ( Acc teacher − Acc before OPD ) (\mathrm{Acc}_{\text{after OPD}}-\mathrm{Acc}_{\text{before OPD}})/(\mathrm{Acc}_{\text{teacher}}-\mathrm{Acc}_{\text{before OPD}}) . This indicates that the additional capabilities acquired by these teachers are more transferable through OPD. Since the post-trained teachers are derived from the same base checkpoints, their thinking patterns remain broadly aligned, which is also observed by the overlap ratio dynamic. The improvement therefore stems from new capabilities of the teacher acquired through RL.

### 3.3 Validation via Reverse Distillation

We design a reverse-distillation experiment as the comparison that simultaneously validates both conditions and reveals deeper insights into the nature of OPD.

#### Setup.

JustRL-DeepSeek-1.5B (JustRL-1.5B) [ He et al., 2025a ] is obtained by RL from R1-Distill-1.5B. We now reverse this direction, using JustRL-1.5B as the student and distilling from R1-Distill-1.5B (its own pre-RL checkpoint). We also use R1-Distill-7B as a teacher for the comparison. Note that R1-Distill-7B achieves slightly higher benchmark scores than JustRL-1.5B, while R1-Distill-1.5B is substantially weaker.

#### Results.

Figure 5 reveals two striking phenomena. First, distilling JustRL-1.5B toward R1-Distill-1.5B, its own pre-RL checkpoint, causes the student to regress almost exactly to its pre-RL performance, removing all gains acquired through RL. Second, when we replace the teacher with R1-Distill-7B, a substantially larger and even slightly stronger model from the same family, the training trajectory is nearly indistinguishable: despite outscoring JustRL-1.5B on benchmarks, R1-Distill-7B drives the student to the same regressed level as the weaker 1.5B teacher. Since OPD minimizes reverse KL divergence over student-generated trajectories, this convergence implies that the two teachers induce nearly identical local target distributions on student-visited states, despite their difference in scale.

These results yield several conclusions: • Thinking pattern matters, and OPD fundamentally learns thinking patterns. Distilling from R1-Distill-1.5B into JustRL-1.5B causes JustRL-1.5B to regress to its pre-RL performance. This suggests that OPD actively acquires the teacher’s thinking patterns and overwrites the student’s own. This is precisely why consistency in thinking patterns is important: if the gap is too large, the student may fail to learn effectively.

• Benchmark performance does not predict OPD outcome. R1-Distill-7B scores higher than JustRL-1.5B, yet the distillation produces no improvement and instead causes regression. This shows that OPD’s training dynamics can be completely independent of the teacher’s benchmark performance, and may even move in the opposite direction.

• Higher scores do not imply new knowledge for OPD. R1-Distill-7B and R1-Distill-1.5B are within the same model family and differ only in scale. The indistinguishable effects of the two models on the student already confirm that: (i) a higher score (R1-Distill-7B) may merely reflect a different degree of fit to the same data, rather than genuinely novel capabilities. For OPD to produce gains, the teacher should possess knowledge beyond what the student has already seen during training; and (ii) despite the difference in scale, R1-Distill-7B and 1.5B exhibit the same thinking patterns.

The reverse distillation experiments and the forward comparisons in Sections 3.1 and 3.2 consolidate the two conditions. Thinking-pattern consistency is associated with higher initial overlap and stronger OPD outcomes, while new knowledge (such as from further post-training) enables larger transferable gains even when overlap is already high.

## 4 Mechanism of On-Policy Distillation

Section 3 identified two conditions, thinking-pattern consistency and new knowledge beyond the same model family, that govern OPD effectiveness. We now investigate the token-level mechanism through which these conditions manifest during training. By comparing successful and failing OPD runs, we show that effective distillation is driven by progressive alignment on high-probability tokens.

### 4.1 Progressive Alignment of High-Probability Tokens

We compare the dynamics of a single student distilled from two different teachers under the same settings, one yielding clear improvement and the other yielding none. We find that successful OPD is essentially driven by learning the high-probability tokens shared between the student and teacher.

#### Setup.

We choose R1-Distill-1.5B as the student and compare two teachers: JustRL-1.5B and R1-Distill-7B. The two teachers exhibit comparable math performance, with the latter being slightly stronger. We use the same DAPO-Math-17K dataset and training settings as before, and monitor three dynamic metrics during OPD.

#### Results.

Figure 6 shows sharply different outcomes. Distillation from JustRL-1.5B yields consistent gains, with the final student recovering more than 80 % 80\% of the performance gap to the teacher, whereas distillation from R1-Distill-7B fails to yield any improvement despite the teacher being stronger overall. The training dynamics ( Figure 6 , bottom) reveal the underlying divergence. In the successful run, the overlap ratio rises steadily, the overlap-token advantage improves toward zero, and the entropy gap narrows, indicating that the student progressively locates the teacher’s high-probability region, calibrates its mass within that region, and matches the teacher’s local confidence. In the failing run, all three metrics stagnate.

Two observations deserve emphasis. First, the overlap tokens carry 97 % 97\% - 99 % 99\% of the total probability mass for both models throughout training (see Appendix B.1 ), so the rising overlap reflects alignment on the probabilistically dominant tokens, not merely a set-level coincidence. Second, the improvement in overlap-token advantage suggests that OPD’s primary optimization signal lies in reweighting probability within the overlap region rather than in tokens outside it.

We also report auxiliary optimization metrics (policy loss, gradient norm, and extreme-advantage token probability differences) in Appendix B.2 , which show consistent secondary patterns: the successful run exhibits decreasing loss and sustained gradient magnitude, while the failing run shows weak gradients and persistent probability discrepancies. We further verify that these findings generalize across different model pairs in Appendix B.3 , using R1-Distill-7B as the student with two different teachers under the same settings.

### 4.2 Optimizing Shared Tokens Alone Suffices

The above analysis shows that high-probability token alignment correlates with OPD success. We further investigate whether this correlation is causal: whether the overlap region is not only where alignment emerges, but also the region that drives optimization. We design a targeted ablation that decomposes the top- k k support into its overlap and non-overlap parts, training on each in isolation.

Setup. Using the successful OPD setting from Section 4.1 (JustRL-1.5B → \to R1-Distill-1.5B), we compare three variants that differ only in which tokens the distillation loss covers: (i) Student Top- k k , which optimizes on the full student top- k k support S t ( p ) S_{t}^{(p)} ; (ii) Overlap Top- k k , which restricts optimization to the intersection of the student and teacher top- k k sets S t ( p ) ∩ S t ( q ) S_{t}^{(p)}\cap S_{t}^{(q)} ; and (iii) Non-Overlap Top- k k , which restricts optimization to their symmetric difference S t ( p ) ​ △ ​ S t ( q ) S_{t}^{(p)}\triangle S_{t}^{(q)} . We set default k k to 16 16 .

#### Results.

As shown in Figure 7 , optimizing only the overlap region is sufficient to recover nearly the full benefit of standard Student Top- k k OPD on all three benchmarks, while Non-Overlap Top- k k remains consistently weaker. This suggests that the main gains of OPD come from gradients on the shared high-probability region, rather than non-overlap tokens. This also explains why Student Top- k k and Overlap Top- k k behave so similarly. The extra tokens in the student-only support carry very little probability mass. Consistently, the overlap-token advantage curves of Student Top- k k and Overlap Top- k k are nearly indistinguishable, whereas Non-Overlap Top- k k has much smaller magnitude, indicating a much weaker effective gradient on the overlap tokens.

#### Overlap optimization is self-reinforcing.

Both Student Top- k k and Overlap Top- k k raise the overlap ratio steadily from about 72 % 72\% to above 91 % 91\% , while Non-Overlap Top- k k first decreases and then only partially recovers ( Figure 7 , bottom-left). This reveals a self-reinforcing dynamic: once a token enters the shared high-probability region and is favored by the teacher, reverse-KL updates concentrate more mass on it, gradually pushing competing non-overlap tokens out of the student’s top- k k set. The overlap region thus grows not despite but because of the optimization, creating a virtuous cycle that sustains alignment throughout training.

Overall, these results support a unified mechanism for OPD: its primary effect is to progressively refine the student’s distribution over teacher-supported high-probability tokens at student-visited states. This alignment is both the signature of successful OPD and its operative locus, where optimizing only the overlap tokens suffices, and non-overlap tokens contribute little. When the conditions identified in Section 3 are met, this self-reinforcing dynamic drives steady improvement; when they are not, overlap stagnates and training fails to progress.

## 5 Practical Recipe

In Section 3 , we identified two conditions for successful OPD. While possessing new knowledge is an intrinsic property of the teacher, the thinking-pattern gap between the teacher and the student can be narrowed through training design. In this section, we present two complementary strategies that recover OPD in otherwise failing configurations by improving the overlap dynamics.

### 5.1 Off-Policy Distillation from Teacher Rollouts as Cold Start

When the student and teacher have substantially different thinking patterns, pure OPD can be ineffective because the teacher’s token-level supervision is difficult for the student to exploit from its initial policy. To mitigate this mismatch, we consider a two-stage framework: we first perform off-policy distillation by supervised fine-tuning (SFT) the student on teacher-generated rollouts to bring it closer to the teacher’s thinking pattern, and then continue training with standard OPD.

#### Setup.