Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? — part 4/4

Cleaned-text code points [47444, 59358). Bibliography omitted here; source.md includes it.

We additionally measure the per-token KL divergence between the student π θ ( ⋅ ∣ x , y < t ) \pi_{\theta}(\cdot\mid x,y_{<t}) and the self-teacher π θ ( ⋅ ∣ x , c , y < t ) \pi_{\theta}(\cdot\mid x,c,y_{<t}) for each surface-form variant of the epistemic tokens in 𝒯 \mathcal{T} . As shown in Table 12 , every variant exceeds the corpus-wide mean KL of 0.076 0.076 , ranging from 1.9 × 1.9\times ( ‘Wait’ ) to 6.8 × 6.8\times ( ‘ Hmm’ ). Notably, sentence-initial variants (e.g., ‘ Hmm’ at 6.8 × 6.8\times , ‘ wait’ at 3.8 × 3.8\times ) tend to exhibit larger divergence than mid-sentence forms, suggesting that the self-teacher most strongly disagrees at positions where the student initiates epistemic deliberation. This provides a quantitative explanation for the suppression of epistemic verbalization under SDPO.

## Appendix G Analysis of Changes in Model Internal Uncertainty under Self-Distillation

To examine whether self-distillation also reduces the model’s internal uncertainty, we further analyze token-level entropy as a measure of internal uncertainty. Building on the analysis in Section 3 , we revisit the comparison between unguided and solution-guided generation to assess whether access to richer information is associated with lower predictive uncertainty.

As shown in Table 13 , solution-guided generation reduces average token-level entropy by approximately 43%, indicating substantially lower predictive uncertainty when richer information is available. Moreover, under unguided generation, epistemic tokens have an average entropy of 1.34, approximately 3.5 × 3.5\times higher than the overall average, indicating that epistemic verbalization tends to occur at positions of high predictive uncertainty.

We next examine whether this lower-entropy generation behavior is transferred to the student through self-distillation.

Table 14 and Figure 17 show a consistent pattern in which GRPO increases token-level entropy for both models, whereas SDPO reduces it, and this distinction persists throughout training. These results indicate that the suppression of uncertainty expressions through self-distillation is accompanied by a genuine reduction in the model’s internal uncertainty.

## Appendix H Can Uncertainty-Encouraging Prompts Mitigate the Suppression?

We further investigate whether prompting the teacher to produce more verbose reasoning with explicit expressions of uncertainty can mitigate the performance degradation caused by the suppression of uncertainty expressions. To this end, we design a modified regeneration prompt that encourages detailed, step-by-step reasoning with uncertainty, and repeat the reasoning behavior analysis (Section 3 ) and on-policy self-distillation experiments (Section 5 ) using this prompt.

Table 15 shows that the uncertainty-encouraging prompt partially recovers the suppressed reasoning behavior: compared to the vanilla solution-guided prompt, it produces longer responses (3,098 vs. 1,873 tokens) and more epistemic expressions (31.4 vs. 8.8). However, both metrics remain far below those of unguided generation, indicating that the presence of a reference solution fundamentally constrains the model’s reasoning verbosity regardless of the prompt instruction.

Figure 18 shows that the uncertainty-encouraging prompt leads to slightly higher training scores and longer responses compared to the vanilla setting, and this improvement carries over to AIME24 evaluation. Nevertheless, both variants still suffer from substantial performance degradation, suggesting that prompting alone is insufficient to resolve the fundamental issue of uncertainty suppression in solution-guided generation.

## Appendix I Comparison with OPSD from Zhao et al. (2026)

Recently, Zhao et al. (2026) proposed OPSD and demonstrated performance gains in mathematical reasoning through on-policy self-distillation, particularly on the Qwen3 series. Unlike our setup, where both the student and the teacher either enable or disable thinking mode, OPSD adopts a hybrid configuration in which the student operates with thinking mode disabled while the teacher has it enabled.

As our experiments also confirm, enabling thinking mode produces substantially longer responses with a greater number of epistemic tokens, making this hybrid setup function more akin to conventional teacher–student distillation, despite using the same underlying model. We note that this configuration is inherently limited to model families such as Qwen3 that support toggling thinking mode on and off.

For training efficiency, OPSD trains only on a response prefix (1,024 tokens by default), rather than the full student response. Unlike OPSD, SDPO and our setup use full fine-tuning with verl ( Sheng et al., 2024 ) , whereas OPSD adopts LoRA fine-tuning with trl ( von Werra et al., 2020 ) and a smaller batch size of 32 (vs. 256 in our setup), resulting in higher training efficiency but smaller parameter updates per step.

Under this hybrid setup with prefix learning in Qwen3-1.7B, as shown in Figure 19a ( orange line ), we observe an intriguing training dynamic. In the early stages, the thinking-enabled teacher encourages the student to generate longer responses while simultaneously improving performance, highlighting the effectiveness of hybrid distillation during the initial phase of training. However, as training progresses, performance steadily deteriorates despite this early improvement. In contrast, under our homogeneous setup, where both the student and teacher operate with thinking mode enabled ( pink line ), performance declines consistently throughout training. Response length and epistemic token count likewise decrease steadily, consistent with our analysis.

While this hybrid distillation setup for the Qwen3 series presents an interesting research direction with its own unique training dynamics, such as why performance initially improves before declining and whether this stems from changes in reasoning behavior or chat template mismatch, a thorough investigation is beyond the scope of this work and is left for future exploration.

## Appendix J More On-Policy Self-Distillation Results

### J.1 Qwen3-8B (Thinking Mode: OFF)

As an extension of Figure 5 in Section 5.3 , we additionally compare AMC23 evaluation scores and the change in epistemic token usage on AIME24. As shown in Figure 20 , GRPO significantly increases response length and yields substantial performance gains on both benchmarks. In contrast, SDPO exhibits divergent trends: on AMC23, acc@16 increases from 0.67 to 0.73 while reducing response length by roughly half, whereas on AIME24, acc@16 slightly decreases from 0.25 to 0.23, with pass@16 dropping more substantially. Notably on AMC23, SDPO achieves approximately a 6-point improvement with shorter responses, whereas GRPO obtains a much larger improvement of around 36 points at the cost of considerably longer responses. Achieving large performance gains while maintaining a reasonable response length remains an open challenge.

### J.2 Qwen3-1.7B (Thinking Mode: ON)

To investigate whether our findings generalize to smaller models, we conducted the same on-policy self-distillation experiment on Qwen3-1.7B. As shown in Figure 21 , our finding that self-distillation reduces uncertainty and makes outputs more concise, while simultaneously degrading mathematical reasoning performance, remains consistent across model sizes. Notably, the performance degradation is more severe for the 1.7B model (45.9%) compared to Qwen3-8B with Thinking Mode ON (12.1%) and DeepSeek-R1-Distill-Qwen-7B (40.7%). This suggests that the externalization of uncertainty may be particularly important for robust reasoning in smaller models, which possess more limited inherent capabilities.

### J.3 Olmo-3-7B-Instruct

In addition to DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B, we further evaluate on-policy self-distillation on OLMo-3-7B-Instruct from a different model family. As shown in Figure 22 , consistent with our previous analyses, SDPO also degrades reasoning performance on this model, with OOD evaluation scores falling below those of the base model. This confirms that our findings are not model-dependent but reflect robust characteristics of reasoning behavior across diverse model families.

### J.4 Pass@16 Score

In addition to the acc@16 scores for DeepSeek-Distill-7B and Qwen3-8B (Thinking Mode: ON) presented in Figures 3b , 3c , 4b , and 4c , we report pass@16 scores in Figure 23 . The gap between GRPO and SDPO in pass@16 is larger for DeepSeek-Distill-7B than for Qwen3-8B, and more pronounced on the harder benchmark (AIME24) compared to AMC23.

## Appendix K Qualitative Analysis

### K.1 Unguided vs. Solution-Guided Generation

We analyze examples of 𝒟 ug \mathcal{D}_{\text{ug}} (Unguided responses) and 𝒟 sg \mathcal{D}_{\text{sg}} (Solution-guided responses) introduced in Section 4 . Below, we present the two responses generated for the same problem. As illustrated, the unguided response exhibits substantially more uncertainty and exploratory reasoning, frequently revisiting assumptions, considering alternative approaches, and self-correcting along the way. In contrast, the solution-guided response proceeds in a more concise and direct manner, arriving at each intermediate result without detours.

Problem. How many ways can 8 8 mutually non-attacking rooks be placed on a 9 × 9 9\times 9 chessboard so that all 8 8 rooks are on squares of the same color? (Two rooks are said to be attacking each other if they are placed in the same row or column of the board.) Please reason step by step, and put your final answer within \boxed{\phantom{0}} .

### K.2 GRPO vs. SDPO Generation

#### Problem (AIME24).

Find the number of ways to place a digit in each cell of a 2 × \times 3 grid so that the sum of the two numbers formed by reading left to right is 999, and the sum of the three numbers formed by reading top to bottom is 99. The grid below is an example of such an arrangement because 8 + 991 = 999 8+991=999 and 9 + 9 + 81 = 99 9+9+81=99 : 0 0 8 9 9 1 \begin{array}[]{|c|c|c|}\hline\cr 0&0&8\\ \hline\cr 9&9&1\\ \hline\cr\end{array} The correct answer is 45 \boxed{45} .

GRPO explores with repeated reflections, deriving the extra constraint a + b + c = 8 a+b+c=8 beyond the pairwise sums, and correctly counts ( 10 2 ) = 45 \binom{10}{2}=45 solutions. SDPO proceeds directly and stops after deriving a + d = b + e = c + f = 9 a+d=b+e=c+f=9 , multiplying 10 × 10 × 10 = 1000 10\times 10\times 10=1000 without checking the row-sum equation, thereby missing the additional constraint a + b + c = 8 a+b+c=8 .

## Appendix L Disclosure of LLM Usage

We used LLMs for conducting the LLM-as-a-Judge experiments described in Appendix E.5 , refining plot designs, and improving the clarity of our writing. LLMs were not involved in research ideation, and their role in writing was limited to polishing author-written text.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .