Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — part 4/5

Cleaned-text code points [45934, 61893). Bibliography omitted here; source.md includes it.

MiniLLM [ Gu et al., 2023 ] first formalized on-policy distillation (OPD) for LLMs under a reverse KL objective optimized via policy gradient, arguing that reverse KL’s mode-seeking behavior prevents the student from spreading probability mass over regions the teacher considers unlikely. GKD [ Agarwal et al., 2024 ] introduced a unified framework interpolating between on-policy and off-policy data across multiple divergences, demonstrating consistent gains over other KD baselines. Yang et al. [2026b] later formalized OPD theoretically as a special case of dense KL-constrained RL, showing that the teacher’s per-token log-ratio constitutes an implicit reward and that scaling this reward beyond its standard weight can push the student past the teacher’s performance boundary. OPD has since been adopted in industry post-training pipelines [ Yang et al., 2025 , Lu and Lab, 2025 , Zeng et al., 2026 , Xiao et al., 2026 , Ko et al., 2026 , Jin et al., 2026 , Jang et al., 2026 , Fu et al., 2026 , Yang et al., 2026b ] , and extended to scalable self-distillation [ Hübotter et al., 2026 , Zhao et al., 2026b , He et al., 2026 , Shenfeld et al., 2026 , Ye et al., 2026b , Sang et al., 2026 , Kim et al., 2026 , Ye et al., 2026a , Yang et al., 2026a , Li et al., 2026 , Zhao et al., 2026a , Ding, 2026 ] , where a single model acts as its own teacher by conditioning on privileged information such as ground-truth solutions or execution feedback. Despite this growing body of work, existing studies focus on demonstrating OPD’s promise, such as dense rewards and mitigated exposure bias, across varied objectives, tasks, and teacher-student pairs, without systematically analyzing when or why OPD fails.

#### Capacity Gap and Distillability.

A recurring observation in knowledge distillation is that large teacher-student capacity gaps can degrade or even reverse the benefit of distillation. Cho and Hariharan [2019] demonstrate that distillation can hurt student performance when the teacher is substantially more capable, and Mirzadeh et al. [2020] propose an intermediate-sized teacher assistant to bridge the gap. Busbridge et al. [2025] provide a quantitative treatment via distillation scaling laws, showing that student loss follows a power law as a function of teacher quality, student size, and data volume, identifying a U-shaped capacity regime where teacher over-capability degrades distillation efficiency. For LLM reasoning, Li et al. [2025] document a “learnability gap” showing that training small models on long chain-of-thought traces from strong reasoning teachers consistently underperforms simpler approaches, suggesting that the reasoning complexity of teacher outputs must be matched to student capacity. These findings call for caution regarding the universality of distillation. However, the existing analyses have largely centered on off-policy knowledge distillation. In particular, the issues of capacity gap and distillability in OPD remain underexplored.

## 8 Conclusion and Future Work

This work provides a systematic analysis of OPD, decomposing its success into two governing conditions: thinking-pattern consistency and the presence of genuinely new knowledge beyond what the student has seen during training. When these conditions are unmet, off-policy cold start and teacher-aligned prompt selection provide effective remedies. We also reveal a practical ceiling imposed by reward degradation over long trajectories.

Future Work Building on our findings, we identify several directions for future research: • Beyond Mathematical Reasoning: All experiments in this work are conducted on mathematical benchmarks. Whether the same conditions and token-level mechanisms govern OPD in other domains such as code and open-ended settings remains an important open question.

• Impact of Pre-Training: The “new knowledge” condition implicitly depends on differences in pre-training corpora, but isolating this factor is challenging. Current studies mainly rely on cross-family distillation (e.g., Qwen → LLaMA), which confounds data divergence with tokenizer mismatch and architectural differences, while controlled pre-training ablations remain prohibitively expensive. As a result, measuring the effect of pre-training data on OPD remains an open problem.

• Self-Distillation Dynamics: Recent work increasingly adopts self-distillation, where a single model serves as its own teacher given privileged information. Extending these insights to the self-distillation regime, where thinking-pattern consistency is guaranteed but knowledge novelty arises from privileged access rather than a separate teacher, is a natural next step.

• Long-Horizon and Agentic Settings: The trajectory-length ceiling revealed in Section 6 motivates hybrid approaches that combine dense token-level supervision on short segments with sparse outcome-level rewards for longer horizons, as well as curriculum strategies that progressively extend the supervised horizon during training.

## Appendix A Details for Section 3

### A.1 GRPO Training Details

#### Base Model.

We initialize GRPO training from Qwen3-4B-Base.

#### Training Dataset.

We use the processed DAPO-Math-17K dataset for GRPO training. Specifically, each question is augmented with the following instruction:

#### Training and Evaluation Settings.

We train the teacher model using GRPO. During training, we sample n = 8 n=8 responses for each prompt. The maximum prompt length and maximum response length are set to 1,024 and 7,168 tokens, respectively. Training is conducted for one epoch on 8 A800 80G GPUs with a learning rate of 1 × 10 − 6 1\times 10^{-6} . We set both the student sampling temperature and the teacher temperature to 1.0, use a repetition penalty of 1.0, disable KL regularization, and adopt token-mean loss aggregation. The main hyperparameters are summarized in Table 1 .

### A.2 Experimental Setup

Unless otherwise noted, all experiments use the default OPD hyperparameters listed in Table 2 .

### A.3 Benchmark-wise breakdown of thinking-pattern compatibility

To further unpack the averaged result in Figure 2 , Figure 17 presents a benchmark-wise breakdown. The advantage of distillation from Qwen3-4B-Base-GRPO is broadly consistent across datasets rather than being driven by a single benchmark. The gap is more pronounced on AMC 2023 and AIME 2024, and smaller but still generally present on AIME 2025. This per-benchmark view supports the interpretation that better early-stage thinking-pattern compatibility leads to better downstream distillation performance, and the loss from an early mismatch is not fully recovered later in training.

## Appendix B Details for Section 4

### B.1 Additional Analysis of Token Overlap Mass

To quantify how much probability mass each model assigns to the overlap top- k k region, we define: ℳ overlap-mass ( p ) = 𝔼 t [ ∑ v ∈ S t ( p ) ∩ S t ( q ) p t ( v ) ] , \mathcal{M}_{\text{overlap-mass}}^{(p)}=\mathbb{E}_{t}\left[\sum_{v\in S_{t}^{(p)}\cap S_{t}^{(q)}}p_{t}(v)\right], (9) and ℳ overlap-mass ( q ) = 𝔼 t [ ∑ v ∈ S t ( p ) ∩ S t ( q ) q t ( v ) ] , \mathcal{M}_{\text{overlap-mass}}^{(q)}=\mathbb{E}_{t}\left[\sum_{v\in S_{t}^{(p)}\cap S_{t}^{(q)}}q_{t}(v)\right], (10) which measure the fraction of total probability mass that the student and teacher, respectively, assign to the shared tokens in their top- k k sets. In our experiments, the overlap tokens carry 97 % 97\% – 99 % 99\% of the total probability mass for both models throughout training, as shown in Figure 18 .

### B.2 Auxiliary Optimization Dynamics

To complement the analysis in Section 4.1 , we report several additional optimization diagnostics for the same contrastive setting. Throughout this appendix, we fix the student to R1-Distill-1.5B and compare two teachers under the same Student Top- k k OPD training recipe: JustRL-1.5B, which yields a successful run, and R1-Distill-7B, which yields a failing run under otherwise matched conditions. These diagnostics are not intended as primary evidence; rather, they provide a complementary view of how the optimization signal differs between successful and failing OPD.

#### Diagnostics.

We monitor three additional quantities. The first is the batch-averaged OPD training loss, denoted as PG Loss in Figure 19 . The second is the gradient norm, which measures the overall magnitude of the update signal reaching the student. The third is the probability difference p t ( v ) − q t ( v ) p_{t}(v)-q_{t}(v) on the token with the largest absolute advantage, which tracks whether the student can reduce the most pronounced local disagreement with the teacher on the tokens that carry the strongest optimization signal. Together, these metrics help distinguish between successful and failing OPD: in the former, the student receives a usable signal and progressively reduces mismatch, whereas in the latter, the signal is too weak or too poorly aligned to drive substantial improvement.

#### Results.

The trends in Figure 19 are consistent with the main conclusion of Section 4.1 . First, the successful run with JustRL-1.5B shows a pronounced reduction in training loss over the course of optimization. Starting from a much larger initial mismatch, the loss decreases steadily for most of training before flattening at a low value. By contrast, the failing run with R1-Distill-7B begins with a much smaller loss and changes only modestly thereafter. This pattern suggests that the smaller loss in the failing run does not indicate better optimization. Rather, it reflects a weak teacher-induced training signal from the outset, which remains too small to drive substantial policy improvement.

Second, the gradient norm shows an even clearer separation between the two runs. In the successful run, the gradient norm is initially large and remains substantial through a long portion of training, indicating that the student continues to receive a meaningful corrective signal. In the failing run, the gradient norm is consistently much smaller, with only limited variation over time. Thus, even though optimization proceeds under the same algorithm and training budget, the student trained against R1-Distill-7B experiences a much weaker update signal. This observation is consistent with the finding that failure is associated with poor alignment on high-probability tokens: when the student does not meaningfully enter the teacher-supported region, the resulting gradients remain weak.

Third, the right panel shows that the successful run steadily reduces the probability discrepancy on the token with the largest absolute advantage, whereas the failing run maintains a noticeably larger gap throughout training. In other words, when OPD succeeds, the student progressively corrects the local mistakes that matter most under the teacher-induced advantage signal. When OPD fails, these high-advantage discrepancies persist rather than being resolved. This is again consistent with the interpretation that the decisive signal in OPD lies on a small set of high-probability, high-advantage tokens, and failure occurs when the student cannot effectively exploit that signal.

Taken together, these auxiliary dynamics reinforce the interpretation developed in Section 4.1 . Successful OPD is characterized not only by increasing overlap on high-probability tokens, but also by a training regime in which the student receives gradients of sufficient magnitude to reduce the most important local distributional mismatches. In contrast, failing OPD is accompanied by weak gradients, limited loss reduction, and persistent disagreement on the tokens with the strongest advantage signal. While these diagnostics are supportive rather than central, they provide an optimization-level view that is fully consistent with that the useful learning signal of OPD is concentrated on high-probability tokens at student-visited states, and training degrades when that signal is too weak or too misaligned to drive effective updates.

### B.3 Cross-Model Validation of High-Probability-Token Alignment

We further test whether the phenomenon in Section 4.1 generalizes to another model pair. Here we fix the student model to R1-Distill-7B and choose Skywork-OR1-Math-7B and DeepSeek-R1-Distill-Qwen-14B (R1-Distill-14B) as teachers, using the same training and evaluation setup as in Section 4.1 .

#### Results.

Figure 20 shows the same pattern as Figure 6 . With Skywork-OR1-Math-7B as the teacher, distillation improves student performance and is accompanied by steadily increasing overlap ratio, overlap-token advantage approaching zero, and a small entropy gap. In contrast, with R1-Distill-14B as the teacher, training shows little improvement and the alignment metrics remain poor or unstable. This provides additional evidence that successful OPD consistently coincides with the emergence of high-probability-token alignment at student-visited states.

## Appendix C Details for Section 5

### C.1 Cold-Start Distillation Details

#### Offline teacher rollout.

To construct the cold-start SFT data, we sample 200K math prompts from the math subset of OpenThoughts3-1.2M [ Guha et al., 2025 ] and use Qwen3-4B (Non-thinking) to generate one offline response for each prompt. For each prompt, we use the following template:

We decode with temperature 0.7 0.7 , top- p = 0.95 p=0.95 , top- k = − 1 k=-1 , and a maximum generation length of 12,288 tokens. After generation, we filter out incomplete responses (e.g., truncated outputs that do not finish properly) and degenerate repetitive responses. The remaining prompt-response pairs are used as the supervised distillation corpus for training the student.

#### Student SFT.

Starting from Qwen3-1.7B-Base, we perform full-parameter SFT on the filtered 200K teacher-generated samples using the LLaMA-Factory framework [ Zheng et al., 2024 ] , yielding Qwen3-1.7B-SFT. We summarize the detailed hyperparameters in Table 3 .

### C.2 Additional Analysis of Overlap Mass

To better understand why the base-initialized student can occasionally exhibit a comparable or even slightly better Overlap-Token Advantage while still underperforming overall, we further examine the probability mass covered by the overlap set from both the student and teacher sides. As shown in Figure 21 , the SFT-initialized student maintains both student overlap mass and teacher overlap mass at consistently high levels throughout training. This indicates that the overlap tokens cover most of the high-probability regions of both the student and teacher distributions, suggesting a strong and stable alignment from the beginning of OPD. In contrast, the base-initialized student exhibits substantially lower and more unstable overlap mass, especially in the early stage of training.

This analysis helps explain why Overlap-Token Advantage alone can sometimes be misleading. Since it is averaged only over overlap tokens, it can appear relatively favorable even when the overlap set itself misses substantial high-probability teacher tokens. Overlap mass complements this view by revealing whether the shared support actually covers the most important parts of the two distributions. From this perspective, the SFT cold start leads to a substantially better and more stable match between student and teacher.

### C.3 Deduplication Details for the DeepMath Subset

For the cross-size setting, we construct a DeepMath subset deduplicated against DAPO-Math-17K in order to compare prompts aligned with the teacher’s RL post-training data against prompts that are only in-domain.

Our deduplication is performed in two stages: exact-match deduplication and semantic deduplication.

#### Question extraction.

For both DAPO-Math-17K and DeepMath, we extract the question content and remove the instruction suffix in the prompt, so that deduplication is performed based on the question text alone.

#### Stage 1: Exact-match deduplication.