Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — part 3/5

Cleaned-text code points [31036, 45934). Bibliography omitted here; source.md includes it.

We study this setting using Qwen3-1.7B-Base as the student and Qwen3-4B (Non-thinking) as the teacher. We use the math-domain subset of OpenThoughts3-1.2M [ Guha et al., 2025 ] as the prompt source for SFT. The teacher generates 200K responses on a subset of this dataset, and we use these teacher rollouts to perform SFT on the student as a cold start, yielding Qwen3-1.7B-SFT. We then continue training with OPD from this SFT initialization, using the remaining prompts from OpenThoughts after deduplicating against the SFT prompt subset (approximately 30K prompts). As a control, we compare against a pure-OPD baseline that starts directly from Qwen3-1.7B-Base and uses the same teacher and OPD prompt set, but performs no cold-start distillation before OPD. Detailed offline rollout and SFT configurations are provided in Appendix C.1 .

#### Results.

As shown in Figure 8 , the two-stage approach substantially outperforms pure OPD. Starting from Qwen3-1.7B-SFT yields consistently better validation performance than starting directly from Qwen3-1.7B-Base. Moreover, the performance gap persists throughout training, indicating that the off-policy cold start improves not only early optimization, but also the final performance ceiling of subsequent OPD.

The overlap dynamics support the same conclusion. The SFT-initialized student begins with a much higher overlap ratio and maintains a smooth, stable trajectory, whereas the base-initialized student starts lower and exhibits pronounced instability before gradually recovering. The entropy gap is also substantially smaller for the SFT-initialized student, indicating a closer match to the teacher’s confidence profile from the outset. These observations confirm that off-policy distillation reduces the initial pattern mismatch, making the teacher’s token-level supervision immediately exploitable once OPD begins. A more detailed analysis of the overlap mass dynamics is provided in Appendix C.2 .

### 5.2 Leveraging Teacher Post-Training Prompts

The previous section narrows the thinking-pattern gap by moving the student closer to the teacher through SFT. Another way to improve alignment is from the data side. Since the teacher’s policy is shaped by the prompts seen during post-training, we find that using teacher-aligned prompts during OPD yields more effective supervision.

#### Setup.

We conduct experiments at two granularities: whether matching the prompt template matters, and whether matching the prompt content matters.

• Prompt template: The teacher is JustRL-1.5B and the student is R1-Distill-1.5B. The prompt set is DAPO-Math-17K, with only the prompt template differing. The original template is the standard DAPO format used in all previous experiments unless otherwise specified, while the teacher-aligned template matches the format used during JustRL post-training:

Thus, the two runs contain the same math problems but differ in how the task is presented to the model. This design isolates the effect of prompt-template alignment with the teacher while keeping the underlying problem content unchanged.

• Prompt content: The teacher is Qwen3-4B-Base-GRPO introduced in Section 3.1 and the student is Qwen3-1.7B-Base. We compare two prompt sets of matched size: DAPO-Math-17K (aligned with the teacher’s RL training datasets) and a subset of DeepMath, deduplicated against DAPO-Math-17K (see Appendix C.3 ). This design tests whether OPD benefits from using prompts that are identical to the teacher’s post-training data, rather than prompts that are merely in-domain.

#### Results.

The prompt template setting in Figure 9 shows that simply switching to the teacher-aligned template improves validation performance on all three benchmarks. The overlap dynamics support this result: the teacher-aligned template run begins with a higher overlap ratio and converges to a higher level, which indicates that even a minor change in prompt template can materially affect OPD by making the student’s generated states more compatible with the teacher. The benchmark-wise breakdown in Appendix C.4 shows the same trend.

The prompt content setting in Figure 10 shows a similar downstream advantage but with a subtlety: teacher-aligned prompts produce a lower overlap ratio throughout training. However, the cumulative student probability mass on the overlap tokens is substantially higher, indicating that the student concentrates its mass on fewer but more strongly shared tokens. The effective alignment on high-probability tokens is therefore stronger, even though the overlap set is smaller.

At the same time, we observe that using teacher-aligned prompts leads to substantially lower student entropy during training. This suggests that performing OPD only on prompts seen during teacher post-training may not always be ideal, as it can overly reduce policy entropy. In practice, a more robust strategy may be to mix teacher-aligned prompts with prompts outside the teacher’s post-training data in order to preserve policy entropy and maintain the student’s capacity for exploration.

Overall, these results suggest that OPD benefits not only from an appropriate teacher, but also from a well-matched prompt set. Prompts closer to the teacher’s post-training data can improve downstream performance and sharpen alignment on the most important shared tokens, but they should be used with care to avoid overly suppressing student entropy.

## 6 Discussion

The appeal of OPD lies in its dense supervision, where every token receives a reward signal from the teacher, in contrast to the sparse outcome-level reward used in RL. However, this increased supervision density comes at a cost. The above sections all implicitly depend on the teacher’s token-level reward being reliable in student-visited states, yet we have seen that this assumption can break down. In this section, we investigate the reward signal itself and examine its properties and limitations.

### 6.1 Reward Quality Degrades with Trajectory Depth

We first investigate how the teacher’s reward quality varies with response length.

#### Response length exhibits a sweet spot.

The supervision at position t t depends on the teacher’s conditional π T ​ ( y t ∣ x , y < t ) \pi_{T}(y_{t}\mid x,y_{<t}) under a student-generated prefix y < t y_{<t} , which may drift from trajectories the teacher would naturally produce. We train R1-Distill-1.5B against JustRL-1.5B across six maximum response lengths for 200 steps. As shown in Figure 11 (a), very short responses (0.5K and 1K) provide too few supervised tokens for sample-efficient learning, while moderate lengths (3K and 7K) yield the strongest results. Beyond this range (10K and 15K), performance plateaus or declines. The training dynamics in Figure 12 confirm that moderate lengths produce smooth overlap growth, whereas 10K and 15K exhibit late-stage collapse, with the overlap ratio dropping sharply, accompanied by spikes in student entropy and gradient norm.

#### Instability originates at later tokens.

Where does this collapse begin? In the 15K setting, analyzing student entropy as a function of output position across training steps reveals a clear back-to-front pattern: as shown in Figure 13 , high entropy first appears at the end of the response and progressively propagates toward earlier tokens as training proceeds. Teacher entropy exhibits a similar suffix-to-prefix trend (see Appendix D.1 ), consistent with the teacher encountering increasingly unfamiliar prefixes at later positions and producing noisier reward that in turn destabilizes the student.

#### Teacher continuation degrades with prefix depth.

We further probe this by testing whether the teacher can still improve upon the student’s continuation when starting from a student-generated prefix. We sample 2K prompts from DAPO-Math-17K, generate full student rollouts, and select those exceeding 16K tokens. We then truncate each rollout at multiple positions and let the teacher continue from the resulting prefix. Figure 11 (b) shows that the teacher’s accuracy advantage decreases monotonically, from +0.37 at a 1K prefix to just +0.02 at a 16K prefix.

Together, these results reveal a fundamental tradeoff in OPD’s token-level supervision. Dense reward is effective on moderately long reasoning traces, but its reliability degrades with depth as the student prefix drifts further from the states familiar to the teacher. This suggests that OPD may not extend cleanly to longer-horizon settings such as extended chain-of-thought or agentic multi-turn interaction.

### 6.2 Globally Informative Reward Does Not Guarantee Local Exploitability

The previous subsection shows that reward quality degrades with trajectory depth. A natural follow-up question is whether the reward signal is fundamentally uninformative in failing OPD configurations or whether the source of failure lies elsewhere.

#### Setup.

We revisit the controlled comparison from Section 4.1 , with R1-Distill-1.5B as the student and two teachers: JustRL-1.5B (successful OPD) and R1-Distill-7B (failed OPD). For each student rollout y y , we compute the sequence mean reward r ¯ ​ ( y ) = 1 T ​ ∑ t = 1 T [ log ⁡ π T ​ ( y t ∣ x , y < t ) − log ⁡ π θ ​ ( y t ∣ x , y < t ) ] \bar{r}(y)=\frac{1}{T}\sum_{t=1}^{T}\left[\log\pi_{T}(y_{t}\mid x,y_{<t})-\log\pi_{\theta}(y_{t}\mid x,y_{<t})\right] under sampled-token OPD, and compare the distribution of r ¯ ​ ( y ) \bar{r}(y) between correct and incorrect rollouts.

#### Global reward structure is preserved in both settings.

Figure 14 shows that, for both teachers, correct rollouts consistently receive higher sequence mean reward than incorrect ones, with comparable AUROC values (0.73 for JustRL-1.5B, 0.75 for R1-Distill-7B). The failing 7B teacher does not produce a weaker global signal, which is equally correlated with rollout correctness.

#### A hypothesis on local optimization geometry.

If the reward is globally informative in both cases, why does OPD fail with the 7B teacher? The training dynamics from Section 4.1 offer a clue. As shown in Figure 6 , when R1-Distill-7B serves as the teacher, the overlap-token advantage becomes larger in magnitude than in the JustRL setting during the later stages of training, yet the gradient norm remains persistently smaller (see Appendix B.2 ). One possible explanation is that the 7B teacher’s per-token advantages, while individually large, are anisotropic across positions within each sequence. When these heterogeneous signals are aggregated into a gradient update, they partially cancel, yielding small effective gradients despite large per-token rewards. By contrast, JustRL-1.5B, which shares a compatible thinking pattern with the student, may concentrate its advantage on a more coherent subset of tokens. The resulting gradient, though composed of smaller per-token signals, points in a consistent direction that reverse KL can amplify through its mode-seeking behavior.

We have not directly verified this anisotropy hypothesis, and doing so would require analyzing the directional structure of per-token gradients, which we leave to future work. Nonetheless, the co-occurrence of high per-token advantage and low gradient norm is suggestive and points to an important distinction that a globally informative reward does not guarantee a locally exploitable one. Understanding the geometry of OPD’s reward landscape, and developing objectives that can exploit anisotropic reward structures, remains an open question.

### 6.3 Sampled-Token Reward Is Already Sufficient

A natural question about OPD’s reward is how many tokens per position are needed to compute a useful gradient. Top- k k OPD aggregates the reward over the k k highest-probability tokens at each position, and one might expect that larger support always leads to better or more stable learning. We investigate this by varying k k and comparing against the simpler sampled-token OPD, which uses only a single token drawn from the student distribution at each position.

#### Setup.

We use R1-Distill-1.5B as the student and JustRL-1.5B as the teacher, and compare Top- k k OPD with k ∈ { 1 , 4 , 16 , 64 } k\in\{1,4,16,64\} against sampled-token OPD, keeping all other hyperparameters fixed.

#### Results.

Figure 15 shows that sampled-token OPD achieves performance comparable to that of the Top- k k settings averaged on three benchmarks. The only clearly worse configuration is Top- 1 1 , which consistently underperforms. Enlarging k k beyond 4 brings negligible additional gain while leading to greater computational overhead. Figure 16 shows the training dynamics and reveals where the differences arise. Top- 1 1 exhibits unstable overlap growth, accompanied by sharp spikes in entropy and gradient norm. Top- 4 4 is substantially more stable but still shows a late-stage dip. Top- 16 16 and Top- 64 64 remain smooth throughout.

Overall, these results suggest that the support size may not be a critical design choice for OPD, as long as the degenerate Top- 1 1 setting is avoided. The reason sampled-token OPD works well despite using only one token per position is that it draws a different token at each step proportionally to the student’s own distribution, providing unbiased coverage of the high-probability region across training. Top- 1 1 , by contrast, always selects the argmax token, thereby concentrating the reward on a single mode. Small policy changes can flip which token occupies rank 1, creating an unstable reward signal that does not average out over training. The failure of Top- 1 1 is therefore not about using too few tokens, but about using a biased, mode-concentrated selection rule.

## 7 Related Work

#### Knowledge Distillation.

Knowledge distillation (KD) [ Hinton et al., 2015 ] transfers knowledge from a large model to a smaller one by training a student network on the soft output distributions of a teacher. For autoregressive sequence models, Kim and Rush [2016] extended this to sequence-level distillation by training students on teacher-generated outputs, establishing the dominant off-policy distillation baseline [ Sanh et al., 2019 , Jiao et al., 2020 , Wang et al., 2020 ] . In parallel, supervised fine-tuning (SFT) has been directly applied to improve performance on a variety of downstream tasks [ Chung et al., 2024 , Sanh et al., 2021 , Wei et al., 2021 ] . A fundamental limitation shared by all off-policy approaches is the train-inference distribution mismatch. The student is optimized on teacher-generated or reference sequences, but must generate from its own distribution at inference, which is an instance of the exposure bias [ Bengio et al., 2015 ] that accumulates errors over long generations. This mismatch motivates shifting distillation to the student’s own on-policy distribution, which is the central idea behind on-policy distillation.

#### On-Policy Distillation.