Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? — part 1/4
Cleaned-text code points [0, 15647). Bibliography omitted here; source.md includes it.
# Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
###### Abstract
Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of epistemic verbalization —the model’s expression of uncertainty during reasoning. Through controlled experiments varying conditioning context richness and task coverage , we show that conditioning the teacher on rich information suppresses uncertainty expression, enabling rapid in-domain optimization with limited task coverage but harming OOD performance, where unseen problems benefit from expressing uncertainty and adjusting accordingly. Across Qwen3-1.7B/8B, DeepSeek-Distill-Qwen-7B, and Olmo3-7B-Instruct, we observe performance drops of up to 40%. Our findings highlight that exposing appropriate levels of uncertainty is crucial for robust reasoning and underscore the importance of optimizing reasoning behavior beyond merely reinforcing correct answer traces.
## 1 Introduction
Recently, self-distillation ( Snell et al., 2022 ) has attracted increasing attention in the post-training of large language models (LLMs). In this paradigm, two instances of the same model are employed: one conditioned on privileged information (e.g., ground-truth solutions) serves as a teacher, providing dense reward signals for responses generated by another instance that does not have access to that information. Several studies have demonstrated that combining this framework with post-training methods such as Reinforcement Learning from Verifiable Rewards (RLVR) leads to highly efficient performance gains ( Zhu et al., 2026 ; Hübotter et al., 2026 ; Zhao et al., 2026 ; Penaloza et al., 2026 ; Liu et al., 2026 ; Ye et al., 2026 ) . These methods have shown particularly strong improvements in domains such as agentic environments and scientific reasoning, especially under in-domain evaluation settings. Interestingly, a consistent trend observed across these works is that performance improves as response length decreases, suggesting that self-distillation promotes more concise and effective reasoning.
However, when we apply the same self-distillation approach to mathematical reasoning tasks, we observe a markedly different phenomenon. Figure 1 compares the effects of a representative self-distillation algorithm, SDPO, in the Chemistry domain (a) and the Math domain (b). As shown in the figure, in the Chemistry domain, self-distillation substantially reduces response length compared to GRPO while rapidly improving performance. In contrast, in the Math domain, although response length consistently decreases as training progresses, performance drops significantly, contrary to prior findings.
This raises a question: ”Why does performance sometimes degrade despite the model being trained to move toward the correct answer?”
Our analysis reveals a consistent pattern: the more informative the context provided to the teacher, the more concise and confident the resulting reasoning becomes, with substantially fewer expressions of uncertainty and, particularly in math reasoning, degraded performance. We trace this effect to the suppression of epistemic verbalization ( Kim et al., 2026 ) , whereby models explicitly verbalize and incorporate uncertainty during reasoning. Strong reasoning models such as DeepSeek-R1 ( Guo et al., 2025 ) frequently express uncertainty using tokens like “Wait” or “Hmm”. Although these expressions may not directly advance the reasoning, removing them discards important signals that a reasoning path may be flawed, leading to significant performance drops ( Kim et al., 2026 ) .
To systematically understand when and why self-distillation suppresses epistemic verbalization and thus affects performance, we conduct a comprehensive empirical study and identify two key factors: information richness and task coverage . The more privileged the information the teacher receives, such as the correct solution, the more its reasoning traces suppress expressions of uncertainty, encouraging the student to imitate a confident reasoning style that presupposes information unavailable at inference time. When task coverage is limited, this compression enables rapid in-domain optimization. However, as coverage increases, removing epistemic verbalization can interfere with optimization across diverse tasks, degrading performance on more challenging or unseen problems.
More broadly, our results show that even when the training objective faithfully guides the model toward correct reasoning traces, the resulting reasoning style can quietly shift in ways that hurt generalization. The suppression of epistemic verbalization is not penalized by standard objectives, yet negatively impacts out-of-distribution (OOD) performance. This suggests that post-training objectives need to account not only for answer correctness, but also for eliciting and preserving uncertainty-aware reasoning behaviors. We believe these findings offer a useful step toward a deeper understanding of reasoning in self-distillation and post-training more broadly.
## 2 Preliminaries
#### Self-Distillation
Let x ∈ 𝒳 x\in\mathcal{X} denote an input and y = ( y 1 , … , y T ) y=(y_{1},\dots,y_{T}) a sequence generated by a language model π θ \pi_{\theta} . The model defines an autoregressive distribution π θ ( y | x ) = ∏ t = 1 T π θ ( y t ∣ x , y < t ) . \pi_{\theta}(y|x)=\prod_{t=1}^{T}\pi_{\theta}(y_{t}\mid x,y_{<t}). In self-distillation, the same model acts as both a student and a teacher under different conditioning contexts. The student first generates a sequence y ∼ π θ ( ⋅ ∣ x ) y\sim\pi_{\theta}(\cdot\mid x) . The teacher policy is obtained by conditioning the model on a richer context c c that provides additional information about the input (e.g., solutions, environment feedback, or other auxiliary signals). Training minimizes the divergence between the student and teacher next-token distributions: ℒ Self-Distillation ( θ ) = ∑ t KL ( π θ ( ⋅ ∣ x , y < t ) ∥ stopgrad ( π θ ( ⋅ ∣ x , c , y < t ) ) ) . \mathcal{L}_{\mathrm{\text{Self-Distillation}}}(\theta)=\sum_{t}\mathrm{KL}\!\left(\pi_{\theta}(\cdot\mid x,y_{<t})\;\|\;\mathrm{stopgrad}\big(\pi_{\theta}(\cdot\mid x,c,y_{<t})\big)\right). (1) This objective encourages the student to match the teacher’s predictions under the richer context, enabling the model to improve by distilling privileged information available at training time without requiring an external teacher.
#### Key Characteristics of Math Reasoning
In LLMs, math reasoning can be viewed as self-Bayesian reasoning , where each step is generated conditioned only on the prompt x x and previously generated tokens y = ( y 1 , … , y T ) y=(y_{1},\dots,y_{T}) , allowing the model to iteratively update its belief over intermediate hypotheses ( Kim et al., 2026 ) . Such reasoning spans diverse tasks such as arithmetic, algebra, geometry, word problems, and logical pattern recognition, so evaluation benchmarks are frequently OOD relative to training data due to compositional and reasoning-depth shifts.
Within this process, verbalized uncertainty toward correct response y ∗ y^{*} , referred to as epistemic verbalization ( Kim et al., 2026 ) , can serve as an informative signal rather than stylistic redundancy. As shown in Figure 2(2a) , reasoning without such signals may prematurely commit to incorrect hypotheses with little room for recovery, whereas epistemic verbalization preserves alternative hypotheses and supports gradual uncertainty reduction.
## 3 LLM Reasoning Behavior Under Richer Information
In self-distillation, the teacher has access to a richer context c c , which provides strong hints and enables it to generate reasoning traces with minimal expressed uncertainty (Figure 2(2b) ). While this yields more concise responses, it could eliminate not only unnecessary verbosity but also valuable epistemic signals, potentially hindering the student’s ability to perform uncertainty-aware reasoning. This raises two questions we investigate: (i) how does the richness of c c relate to the suppression of uncertainty expression and to reasoning performance? and (ii) if epistemic verbalization is beneficial, why does suppressing it improve performance in domains such as Chemistry while degrading it in Math?
To study these questions, we first need a way to quantify how informative the conditioning context is. Let y ∗ y^{*} denote a random variable representing an ideal correct response to x x . We formalize the information that c c provides about y ∗ y^{*} as the conditional mutual information I ( y ∗ ; c ∣ x ) = H ( y ∗ ∣ x ) − H ( y ∗ ∣ x , c ) , I(y^{*};\,c\mid x)\;=\;H(y^{*}\mid x)\;-\;H(y^{*}\mid x,\,c), (2) which captures the reduction in uncertainty about y ∗ y^{*} once the additional context c c is given.
Building on this measure, we construct a controlled comparison to examine how the model’s reasoning behavior changes as the conditioning information is varied. We use the DAPO-Math-17k dataset ( Yu et al., 2026 ) with DeepSeek-R1-Distill-Qwen-7B ( Guo et al., 2025 ) as the base model, selecting 100 problems on which the base model achieves accuracy between 0.125 and 0.5 over 8 rollouts. Let s s denote the full solution (including chain-of-thought in <think> tags), s ∖ think s_{\setminus\text{think}} the solution with <think> content removed, and y r ∼ π θ ( ⋅ ∣ x , s ) y_{r}\sim\pi_{\theta}(\cdot\mid x,s) a response previously generated under full solution guidance. We compare the model’s responses across four generation settings:
• (1) Unguided generation ( c = ∅ c=\emptyset ): I ( y ∗ ; c ∣ x ) = 0 I(y^{*};\,c\mid x)=0 by definition.
• (2) Solution-guided generation ( c = s c=s ): provides maximal guidance and yields the largest I ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) .
• (3) Solution-guided generation without think contents ( c = s ∖ think c=s_{\setminus\text{think}} ): since s ∖ think s_{\setminus\text{think}} is a strict informational subset of s s , we have I ( y ∗ ; s ∖ think ∣ x ) ≤ I ( y ∗ ; s ∣ x ) I(y^{*};\,s_{\setminus\text{think}}\mid x)\leq I(y^{*};\,s\mid x) .
• (4) Regeneration-conditioned generation ( c = y r c=y_{r} , where y r ∼ π θ ( ⋅ ∣ x , s ) y_{r}\sim\pi_{\theta}(\cdot\mid x,s) ): since y r y_{r} is derived from s s , the data processing inequality gives I ( y ∗ ; y r ∣ x ) ≤ I ( y ∗ ; s ∣ x ) I(y^{*};\,y_{r}\mid x)\leq I(y^{*};\,s\mid x) .
These settings induce the following ordering over the conditional mutual information: I ( y ∗ ; ∅ ∣ x ) = 0 ⏟ (1) < I ( y ∗ ; s ∖ think ∣ x ) ⏟ (3) ≤ I ( y ∗ ; y r ∣ x ) ⏟ (4) ≤ I ( y ∗ ; s ∣ x ) ⏟ (2) . \underbrace{I(y^{*};\,\emptyset\mid x)=0}_{\text{(1)}}\;<\;\underbrace{I(y^{*};\,s_{\setminus\text{think}}\mid x)}_{\text{(3)}}\;\leq\;\underbrace{I(y^{*};\,y_{r}\mid x)}_{\text{(4)}}\;\leq\;\underbrace{I(y^{*};\,s\mid x)}_{\text{(2)}}. (3)
#### Prompts
The prompts used for unguided and solution-guided settings are as follows. For regeneration, we used the same prompts as in Hübotter et al. (2026) . A comparative analysis with other regeneration prompts is provided in Appendix H .
#### Epistemic Tokens
Following Kim et al. (2026) , we define a set of 10 epistemic markers 𝒯 = { wait , hmm , perhaps , maybe , actually , alternatively , seems , might , likely , check } \mathcal{T}=\{\textit{wait},\,\textit{hmm},\,\textit{perhaps},\,\textit{maybe},\,\textit{actually},\,\textit{alternatively},\,\textit{seems},\,\textit{might},\,\textit{likely},\,\textit{check}\} as practical indicators of regions where the externalization of uncertainty is likely to occur. These tokens are intuitively recognizable as uncertainty expressions, with their selection further supported by a cross-family analysis of 9,600 reasoning traces (Appendix B ). We measure the epistemic token count of a response y y as E ( y ) = ∑ t ∈ 𝒯 count ( t , y ) E(y)=\sum_{t\in\mathcal{T}}\mathrm{count}(t,\,y) .
#### Results
We analyze how different forms of conditioning information toward the correct answer affect the model’s reasoning behavior by comparing the average response length 𝔼 [ L ( y ) ] \mathbb{E}[L(y)] , score, and the epistemic token count 𝔼 [ E ( y ) ] \mathbb{E}[E(y)] across the four settings. As shown in Table 1 , both quantities decrease monotonically as I ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) increases: 𝔼 [ L ( y ) ] | ( 1 ) > 𝔼 [ L ( y ) ] | ( 3 ) > 𝔼 [ L ( y ) ] | ( 4 ) > 𝔼 [ L ( y ) ] | ( 2 ) , \mathbb{E}\bigl[L(y)\bigr]\Big|_{(1)}\;>\;\mathbb{E}\bigl[L(y)\bigr]\Big|_{(3)}\;>\;\mathbb{E}\bigl[L(y)\bigr]\Big|_{(4)}\;>\;\mathbb{E}\bigl[L(y)\bigr]\Big|_{(2)}, (4) and analogously for 𝔼 [ E ( y ) ] \mathbb{E}[E(y)] , confirming that richer conditioning information leads to more concise and confident reasoning.
Specifically, unguided generation ( c = ∅ c=\emptyset ) produces substantially longer responses with the highest epistemic token counts. When the full solution s s is provided in (2), the model follows the given reasoning trace with high confidence, and its concise output can be viewed as a compressed representation of the reasoning in s s . In (3), removing the <think> portion retains only s ∖ think s_{\setminus\text{think}} (640 out of 13,054 response tokens), and both 𝔼 [ L ( y ) ] \mathbb{E}[L(y)] and 𝔼 [ E ( y ) ] \mathbb{E}[E(y)] increase again toward the unguided level, reflecting the substantial information loss. Setting (4), conditioning on the regenerated response y r y_{r} , yields intermediate values (lower than (3) but higher than (2)), indicating that y r y_{r} preserves much of the informative structure of the full solution. Detailed per-token breakdowns are reported in Appendix E.1 .
## 4 Supervised Finetuning with Self-Distillation
A natural follow-up question is whether the suppression of epistemic verbalization under high I ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) is merely stylistic or has a tangible impact on reasoning capability. To test this, we conduct off-policy self-distillation (SFT) using DeepSeek-R1-Distill-Qwen-7B ( Guo et al., 2025 ) on two datasets, each containing 800 correct responses:
• 𝒟 ug \mathcal{D}_{\text{ug}} : unguided responses ( c = ∅ c=\emptyset ), with high 𝔼 [ E ( y ) ] \mathbb{E}[E(y)] and 𝔼 [ L ( y ) ] ≈ 12 k \mathbb{E}[L(y)]\approx 12\text{k} tokens.
• 𝒟 sg \mathcal{D}_{\text{sg}} : solution-guided responses ( c = s c=s ), with low 𝔼 [ E ( y ) ] \mathbb{E}[E(y)] and 𝔼 [ L ( y ) ] ≈ 2 k \mathbb{E}[L(y)]\approx 2\text{k} tokens.
Both datasets consist entirely of correct trajectories. The key difference lies in the epistemic density of the training signal. We evaluate the resulting checkpoints across multiple math benchmarks (examples from each dataset are presented in Appendix K.1 ).
As shown in Table 2 , training on 𝒟 sg \mathcal{D}_{\text{sg}} leads to substantial degradation across all benchmarks, despite the dataset consisting of correct answers, whereas training on 𝒟 ug \mathcal{D}_{\text{ug}} produces no significant performance change. This asymmetry arises because solution-guided responses are concise precisely due to the external context s s . Using them as SFT targets forces the model to imitate a reasoning style that presupposes information unavailable at inference time, effectively suppressing the epistemic tokens that support autonomous exploration and error correction.
## 5 On-Policy Self-Distillation