# Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

###### Abstract

Self-distillation has emerged as an effective post-training paradigm for LLMs, often improving performance while shortening reasoning traces. However, in mathematical reasoning, we find that it can reduce response length while degrading performance. We trace this degradation to the suppression of epistemic verbalization —the model’s expression of uncertainty during reasoning. Through controlled experiments varying conditioning context richness and task coverage , we show that conditioning the teacher on rich information suppresses uncertainty expression, enabling rapid in-domain optimization with limited task coverage but harming OOD performance, where unseen problems benefit from expressing uncertainty and adjusting accordingly. Across Qwen3-1.7B/8B, DeepSeek-Distill-Qwen-7B, and Olmo3-7B-Instruct, we observe performance drops of up to 40%. Our findings highlight that exposing appropriate levels of uncertainty is crucial for robust reasoning and underscore the importance of optimizing reasoning behavior beyond merely reinforcing correct answer traces.

## 1 Introduction

Recently, self-distillation ( Snell et al., 2022 ) has attracted increasing attention in the post-training of large language models (LLMs). In this paradigm, two instances of the same model are employed: one conditioned on privileged information (e.g., ground-truth solutions) serves as a teacher, providing dense reward signals for responses generated by another instance that does not have access to that information. Several studies have demonstrated that combining this framework with post-training methods such as Reinforcement Learning from Verifiable Rewards (RLVR) leads to highly efficient performance gains ( Zhu et al., 2026 ; Hübotter et al., 2026 ; Zhao et al., 2026 ; Penaloza et al., 2026 ; Liu et al., 2026 ; Ye et al., 2026 ) . These methods have shown particularly strong improvements in domains such as agentic environments and scientific reasoning, especially under in-domain evaluation settings. Interestingly, a consistent trend observed across these works is that performance improves as response length decreases, suggesting that self-distillation promotes more concise and effective reasoning.

However, when we apply the same self-distillation approach to mathematical reasoning tasks, we observe a markedly different phenomenon. Figure 1 compares the effects of a representative self-distillation algorithm, SDPO, in the Chemistry domain (a) and the Math domain (b). As shown in the figure, in the Chemistry domain, self-distillation substantially reduces response length compared to GRPO while rapidly improving performance. In contrast, in the Math domain, although response length consistently decreases as training progresses, performance drops significantly, contrary to prior findings.

This raises a question: ”Why does performance sometimes degrade despite the model being trained to move toward the correct answer?”

Our analysis reveals a consistent pattern: the more informative the context provided to the teacher, the more concise and confident the resulting reasoning becomes, with substantially fewer expressions of uncertainty and, particularly in math reasoning, degraded performance. We trace this effect to the suppression of epistemic verbalization ( Kim et al., 2026 ) , whereby models explicitly verbalize and incorporate uncertainty during reasoning. Strong reasoning models such as DeepSeek-R1 ( Guo et al., 2025 ) frequently express uncertainty using tokens like “Wait” or “Hmm”. Although these expressions may not directly advance the reasoning, removing them discards important signals that a reasoning path may be flawed, leading to significant performance drops ( Kim et al., 2026 ) .

To systematically understand when and why self-distillation suppresses epistemic verbalization and thus affects performance, we conduct a comprehensive empirical study and identify two key factors: information richness and task coverage . The more privileged the information the teacher receives, such as the correct solution, the more its reasoning traces suppress expressions of uncertainty, encouraging the student to imitate a confident reasoning style that presupposes information unavailable at inference time. When task coverage is limited, this compression enables rapid in-domain optimization. However, as coverage increases, removing epistemic verbalization can interfere with optimization across diverse tasks, degrading performance on more challenging or unseen problems.

More broadly, our results show that even when the training objective faithfully guides the model toward correct reasoning traces, the resulting reasoning style can quietly shift in ways that hurt generalization. The suppression of epistemic verbalization is not penalized by standard objectives, yet negatively impacts out-of-distribution (OOD) performance. This suggests that post-training objectives need to account not only for answer correctness, but also for eliciting and preserving uncertainty-aware reasoning behaviors. We believe these findings offer a useful step toward a deeper understanding of reasoning in self-distillation and post-training more broadly.

## 2 Preliminaries

#### Self-Distillation

Let x ∈ 𝒳 x\in\mathcal{X} denote an input and y = ( y 1 , … , y T ) y=(y_{1},\dots,y_{T}) a sequence generated by a language model π θ \pi_{\theta} . The model defines an autoregressive distribution π θ ​ ( y | x ) = ∏ t = 1 T π θ ​ ( y t ∣ x , y < t ) . \pi_{\theta}(y|x)=\prod_{t=1}^{T}\pi_{\theta}(y_{t}\mid x,y_{<t}). In self-distillation, the same model acts as both a student and a teacher under different conditioning contexts. The student first generates a sequence y ∼ π θ ( ⋅ ∣ x ) y\sim\pi_{\theta}(\cdot\mid x) . The teacher policy is obtained by conditioning the model on a richer context c c that provides additional information about the input (e.g., solutions, environment feedback, or other auxiliary signals). Training minimizes the divergence between the student and teacher next-token distributions: ℒ Self-Distillation ( θ ) = ∑ t KL ( π θ ( ⋅ ∣ x , y < t ) ∥ stopgrad ( π θ ( ⋅ ∣ x , c , y < t ) ) ) . \mathcal{L}_{\mathrm{\text{Self-Distillation}}}(\theta)=\sum_{t}\mathrm{KL}\!\left(\pi_{\theta}(\cdot\mid x,y_{<t})\;\|\;\mathrm{stopgrad}\big(\pi_{\theta}(\cdot\mid x,c,y_{<t})\big)\right). (1) This objective encourages the student to match the teacher’s predictions under the richer context, enabling the model to improve by distilling privileged information available at training time without requiring an external teacher.

#### Key Characteristics of Math Reasoning

In LLMs, math reasoning can be viewed as self-Bayesian reasoning , where each step is generated conditioned only on the prompt x x and previously generated tokens y = ( y 1 , … , y T ) y=(y_{1},\dots,y_{T}) , allowing the model to iteratively update its belief over intermediate hypotheses ( Kim et al., 2026 ) . Such reasoning spans diverse tasks such as arithmetic, algebra, geometry, word problems, and logical pattern recognition, so evaluation benchmarks are frequently OOD relative to training data due to compositional and reasoning-depth shifts.

Within this process, verbalized uncertainty toward correct response y ∗ y^{*} , referred to as epistemic verbalization ( Kim et al., 2026 ) , can serve as an informative signal rather than stylistic redundancy. As shown in Figure 2(2a) , reasoning without such signals may prematurely commit to incorrect hypotheses with little room for recovery, whereas epistemic verbalization preserves alternative hypotheses and supports gradual uncertainty reduction.

## 3 LLM Reasoning Behavior Under Richer Information

In self-distillation, the teacher has access to a richer context c c , which provides strong hints and enables it to generate reasoning traces with minimal expressed uncertainty (Figure 2(2b) ). While this yields more concise responses, it could eliminate not only unnecessary verbosity but also valuable epistemic signals, potentially hindering the student’s ability to perform uncertainty-aware reasoning. This raises two questions we investigate: (i) how does the richness of c c relate to the suppression of uncertainty expression and to reasoning performance? and (ii) if epistemic verbalization is beneficial, why does suppressing it improve performance in domains such as Chemistry while degrading it in Math?

To study these questions, we first need a way to quantify how informative the conditioning context is. Let y ∗ y^{*} denote a random variable representing an ideal correct response to x x . We formalize the information that c c provides about y ∗ y^{*} as the conditional mutual information I ⁡ ( y ∗ ; c ∣ x ) = H ⁡ ( y ∗ ∣ x ) − H ⁡ ( y ∗ ∣ x , c ) , I(y^{*};\,c\mid x)\;=\;H(y^{*}\mid x)\;-\;H(y^{*}\mid x,\,c), (2) which captures the reduction in uncertainty about y ∗ y^{*} once the additional context c c is given.

Building on this measure, we construct a controlled comparison to examine how the model’s reasoning behavior changes as the conditioning information is varied. We use the DAPO-Math-17k dataset ( Yu et al., 2026 ) with DeepSeek-R1-Distill-Qwen-7B ( Guo et al., 2025 ) as the base model, selecting 100 problems on which the base model achieves accuracy between 0.125 and 0.5 over 8 rollouts. Let s s denote the full solution (including chain-of-thought in <think> tags), s ∖ think s_{\setminus\text{think}} the solution with <think> content removed, and y r ∼ π θ ( ⋅ ∣ x , s ) y_{r}\sim\pi_{\theta}(\cdot\mid x,s) a response previously generated under full solution guidance. We compare the model’s responses across four generation settings:

• (1) Unguided generation ( c = ∅ c=\emptyset ): I ⁡ ( y ∗ ; c ∣ x ) = 0 I(y^{*};\,c\mid x)=0 by definition.

• (2) Solution-guided generation ( c = s c=s ): provides maximal guidance and yields the largest I ⁡ ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) .

• (3) Solution-guided generation without think contents ( c = s ∖ think c=s_{\setminus\text{think}} ): since s ∖ think s_{\setminus\text{think}} is a strict informational subset of s s , we have I ⁡ ( y ∗ ; s ∖ think ∣ x ) ≤ I ⁡ ( y ∗ ; s ∣ x ) I(y^{*};\,s_{\setminus\text{think}}\mid x)\leq I(y^{*};\,s\mid x) .

• (4) Regeneration-conditioned generation ( c = y r c=y_{r} , where y r ∼ π θ ( ⋅ ∣ x , s ) y_{r}\sim\pi_{\theta}(\cdot\mid x,s) ): since y r y_{r} is derived from s s , the data processing inequality gives I ⁡ ( y ∗ ; y r ∣ x ) ≤ I ⁡ ( y ∗ ; s ∣ x ) I(y^{*};\,y_{r}\mid x)\leq I(y^{*};\,s\mid x) .

These settings induce the following ordering over the conditional mutual information: I ⁡ ( y ∗ ; ∅ ∣ x ) = 0 ⏟ (1) < I ⁡ ( y ∗ ; s ∖ think ∣ x ) ⏟ (3) ≤ I ⁡ ( y ∗ ; y r ∣ x ) ⏟ (4) ≤ I ⁡ ( y ∗ ; s ∣ x ) ⏟ (2) . \underbrace{I(y^{*};\,\emptyset\mid x)=0}_{\text{(1)}}\;<\;\underbrace{I(y^{*};\,s_{\setminus\text{think}}\mid x)}_{\text{(3)}}\;\leq\;\underbrace{I(y^{*};\,y_{r}\mid x)}_{\text{(4)}}\;\leq\;\underbrace{I(y^{*};\,s\mid x)}_{\text{(2)}}. (3)

#### Prompts

The prompts used for unguided and solution-guided settings are as follows. For regeneration, we used the same prompts as in Hübotter et al. (2026) . A comparative analysis with other regeneration prompts is provided in Appendix H .

#### Epistemic Tokens

Following Kim et al. (2026) , we define a set of 10 epistemic markers 𝒯 = { wait , hmm , perhaps , maybe , actually , alternatively , seems , might , likely , check } \mathcal{T}=\{\textit{wait},\,\textit{hmm},\,\textit{perhaps},\,\textit{maybe},\,\textit{actually},\,\textit{alternatively},\,\textit{seems},\,\textit{might},\,\textit{likely},\,\textit{check}\} as practical indicators of regions where the externalization of uncertainty is likely to occur. These tokens are intuitively recognizable as uncertainty expressions, with their selection further supported by a cross-family analysis of 9,600 reasoning traces (Appendix B ). We measure the epistemic token count of a response y y as E ⁡ ( y ) = ∑ t ∈ 𝒯 count ⁡ ( t , y ) E(y)=\sum_{t\in\mathcal{T}}\mathrm{count}(t,\,y) .

#### Results

We analyze how different forms of conditioning information toward the correct answer affect the model’s reasoning behavior by comparing the average response length 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] , score, and the epistemic token count 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] across the four settings. As shown in Table 1 , both quantities decrease monotonically as I ⁡ ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) increases: 𝔼 ⁡ [ L ⁡ ( y ) ] | ( 1 ) > 𝔼 ⁡ [ L ⁡ ( y ) ] | ( 3 ) > 𝔼 ⁡ [ L ⁡ ( y ) ] | ( 4 ) > 𝔼 ⁡ [ L ⁡ ( y ) ] | ( 2 ) , \mathbb{E}\bigl[L(y)\bigr]\Big|_{(1)}\;>\;\mathbb{E}\bigl[L(y)\bigr]\Big|_{(3)}\;>\;\mathbb{E}\bigl[L(y)\bigr]\Big|_{(4)}\;>\;\mathbb{E}\bigl[L(y)\bigr]\Big|_{(2)}, (4) and analogously for 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] , confirming that richer conditioning information leads to more concise and confident reasoning.

Specifically, unguided generation ( c = ∅ c=\emptyset ) produces substantially longer responses with the highest epistemic token counts. When the full solution s s is provided in (2), the model follows the given reasoning trace with high confidence, and its concise output can be viewed as a compressed representation of the reasoning in s s . In (3), removing the <think> portion retains only s ∖ think s_{\setminus\text{think}} (640 out of 13,054 response tokens), and both 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] and 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] increase again toward the unguided level, reflecting the substantial information loss. Setting (4), conditioning on the regenerated response y r y_{r} , yields intermediate values (lower than (3) but higher than (2)), indicating that y r y_{r} preserves much of the informative structure of the full solution. Detailed per-token breakdowns are reported in Appendix E.1 .

## 4 Supervised Finetuning with Self-Distillation

A natural follow-up question is whether the suppression of epistemic verbalization under high I ⁡ ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) is merely stylistic or has a tangible impact on reasoning capability. To test this, we conduct off-policy self-distillation (SFT) using DeepSeek-R1-Distill-Qwen-7B ( Guo et al., 2025 ) on two datasets, each containing 800 correct responses:

• 𝒟 ug \mathcal{D}_{\text{ug}} : unguided responses ( c = ∅ c=\emptyset ), with high 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] and 𝔼 ⁡ [ L ⁡ ( y ) ] ≈ 12 ​ k \mathbb{E}[L(y)]\approx 12\text{k} tokens.

• 𝒟 sg \mathcal{D}_{\text{sg}} : solution-guided responses ( c = s c=s ), with low 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] and 𝔼 ⁡ [ L ⁡ ( y ) ] ≈ 2 ​ k \mathbb{E}[L(y)]\approx 2\text{k} tokens.

Both datasets consist entirely of correct trajectories. The key difference lies in the epistemic density of the training signal. We evaluate the resulting checkpoints across multiple math benchmarks (examples from each dataset are presented in Appendix K.1 ).

As shown in Table 2 , training on 𝒟 sg \mathcal{D}_{\text{sg}} leads to substantial degradation across all benchmarks, despite the dataset consisting of correct answers, whereas training on 𝒟 ug \mathcal{D}_{\text{ug}} produces no significant performance change. This asymmetry arises because solution-guided responses are concise precisely due to the external context s s . Using them as SFT targets forces the model to imitate a reasoning style that presupposes information unavailable at inference time, effectively suppressing the epistemic tokens that support autonomous exploration and error correction.

## 5 On-Policy Self-Distillation

We now turn to on-policy self-distillation ( Hübotter et al., 2026 ; Zhao et al., 2026 ; Penaloza et al., 2026 ) , where the model learns from dense reward signals provided by a self-teacher with access to the correct solution, based on its own responses. We compare GRPO with Reinforcement Learning via Self-Distillation (SDPO) ( Hübotter et al., 2026 ) on the DAPO-Math-17k dataset ( Yu et al., 2026 ) , using Qwen3-8B ( Yang et al., 2025 ) and DeepSeek-R1-Distill-Qwen-7B ( Guo et al., 2025 ) . Additional results for Qwen3-1.7B and Olmo-3-7B-Instruct ( Olmo et al., 2025 ) are in Appendix J.3 . For each model, we track training score, response length, and OOD performance on AIME24 and AMC23. We fix the teacher to the initial policy, which performs better than a moving target (see Section 5.5 for a comparison).

The behavior of on-policy self-distillation depends on two factors: (i) the extent of epistemic verbalization already present in the base model, and (ii) the richness of the conditioning context c c . To disentangle these, we compare GRPO and SDPO under two settings: c = s c=s (full solution) and c = s ∖ think c=s_{\setminus\text{think}} (solution without <think> content).

### 5.1 DeepSeek-R1-Distill-Qwen-7B

DeepSeek-R1-Distill-Qwen-7B is a representative high-reasoning model that generates rich epistemic verbalizations within <think> tags, averaging about 180 epistemic tokens per response, as shown in Section 3 , leading to lengthy yet high-performing reasoning outputs.

#### Training Performance

As shown in Figure 3a , GRPO training slightly increases 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] with a modest improvement in score. In contrast, SDPO with c = s c=s causes a sharp initial drop in both 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] and score. Performance gradually recovers but remains below GRPO throughout training. When the conditioning is reduced to c = s ∖ think c=s_{\setminus\text{think}} , the drop in 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] is attenuated and the score trajectory approaches that of GRPO, consistent with the relationship between I ⁡ ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) and epistemic suppression discussed in Section 3 .

#### OOD Evaluation - AIME24, AMC23

Consistent with the training trends, GRPO yields modest gains on both OOD benchmarks (AIME24: 54.7 → \to 56.0; AMC23: 89.3 → \to 91.1, Figures 3b and 3c ) with a slight increase in 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] . SDPO with c = s c=s degrades performance substantially ( ∼ 40 % {\sim}40\% on AIME24, ∼ 15 % {\sim}15\% on AMC23). Reducing the conditioning to c = s ∖ think c=s_{\setminus\text{think}} mitigates the drop, though performance still remains below the base model.

#### Reasoning Pattern

Figure 3d illustrates the epistemic token counts of the trained models. GRPO increases 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] , whereas SDPO suppresses it more aggressively, consistent with the observed correlation between epistemic suppression and performance degradation throughout our analysis. Additional analyses on total versus epistemic token changes, as well as the number of uncertainty expressions per reasoning trace, evaluated using an LLM-as-Judge, are provided in Appendices E.4 and E.5 . Furthermore, a comparison of the KL divergence of epistemic tokens with respect to the mean is presented in Appendix F . Example responses generated by GRPO and SDPO are included in Appendix K.2 .

### 5.2 Qwen3-8B (Thinking Mode: ON)

Qwen3-8B is a hybrid reasoning model that can operate in either a thinking mode for deliberate reasoning or a non-thinking mode for direct responses. With thinking mode enabled, Qwen3-8B produces longer responses than DeepSeek-R1-Distill-Qwen-7B, with a higher number of epistemic tokens (see Appendix E.2 for a comparison of epistemic tokens across models).

#### Training Performance

As shown in Figure 4a , 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] decreases under both GRPO and SDPO, with SDPO exhibiting a larger reduction and a correspondingly larger performance drop. Notably, 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] first drops sharply, then increases slightly. Since the teacher policy is fixed as the reference policy, shortening the response by ∼ 900 {\sim}900 tokens reduces the informativeness of c c , i.e., decreases I ⁡ ( y ∗ , c ∣ x ) I(y^{*},c\mid x) . As the context becomes less informative, the model compensates by increasing epistemic verbalization, causing the length to partially recover.

#### OOD Evaluation - AIME24, AMC23

The gap becomes more pronounced on OOD benchmarks (Figures 4b and 4c ): GRPO maintains largely stable performance with gradually decreasing 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] , whereas SDPO falls below the base model, particularly with c = s c=s . Notably, although GRPO and SDPO with c = s ∖ think c=s_{\setminus\text{think}} achieve comparable training performance, their OOD results diverge, especially on the more challenging AIME24, where SDPO with c = s ∖ think c=s_{\setminus\text{think}} shows progressive performance degradation as training proceeds.

#### Reasoning Pattern

Figure 4d reveals that both methods reduce 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] relative to the base model, with SDPO more aggressively so. This suggests that Qwen3-8B originally generates more epistemic verbalization than necessary. While both methods mitigate this redundancy, overly aggressive suppression risks removing epistemic signals that carry useful reasoning information.

### 5.3 Qwen3-8B (Thinking Mode: OFF)

When Qwen3-8B is used without thinking mode, the <think> tag is absent, so we consider only c = s c=s . The model initially produces much shorter responses and exhibits significantly lower performance. GRPO rapidly increases 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] by promoting epistemic verbalization (as shown in Appendix J.1 ), quickly achieving a high training score. In contrast, SDPO reduces 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] and improves much more slowly. Even when the training score slightly increases, as shown in Figure 5b , performance on AIME24 slightly declines ( 0.25 → 0.23 0.25\to 0.23 ), further illustrating the cost of epistemic suppression under self-distillation.

### 5.4 Causal Study: Epistemic Token Suppression during GRPO

To further examine whether the suppression of epistemic verbalization is a causal contributor to the observed performance degradation, we explicitly penalized epistemic tokens during GRPO training and tested whether this intervention induces a similar degradation pattern.

Following the epistemic marker set 𝒯 \mathcal{T} defined in Section 3 , we override the standard GRPO advantage at every position t t whose generated token realizes a marker in 𝒯 \mathcal{T} , setting A ~ t = min ⁡ ( A t , − κ ) \tilde{A}_{t}=\min(A_{t},-\kappa) if y t ∈ 𝒯 y_{t}\in\mathcal{T} and A ~ t = A t \tilde{A}_{t}=A_{t} otherwise, with suppression strength κ = 0.5 \kappa=0.5 . Since A ~ t ≤ − κ \tilde{A}_{t}\leq-\kappa holds even for correct rollouts, the policy gradient consistently reduces 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] throughout training.

As shown in Figure 6 , explicitly penalizing epistemic tokens during GRPO produces shorter responses and lower performance, qualitatively reproducing the degradation pattern observed with SDPO. The degradation is less pronounced than under SDPO, possibly because the predefined epistemic token set does not capture all linguistic realizations of uncertainty or epistemic reasoning, as discussed further in Appendix E.5 . Nevertheless, this result provides causal evidence that suppressing uncertainty expression contributes to the performance decline under SDPO.

### 5.5 Ablation Study: Fixed vs. Moving Target Teacher

In naive on-policy self-distillation, the teacher and student share a continuously updated policy, making the teacher a moving target that can introduce training instability ( Zhao et al., 2026 ; Ye et al., 2026 ) . To mitigate this, SDPO uses an EMA-smoothed teacher (EMA rate: 0.05). However, we find that setting the EMA rate to 0.0 (i.e., fixing the teacher to the initial policy) yields better performance, and thus Section 5 follows this setting.

Figure 7a shows additional comparison results when the teacher is updated during training. As shown, even slow updates (e.g., rate 0.05) lead to a sharper reduction in response length, resulting in larger performance degradation. This can be interpreted as a feedback loop in self-distillation. The model is trained to produce increasingly confident outputs, and when this is used as the teacher, it generates even more confident responses, amplifying the effect over iterations. Further ablations on learning rate, top- k k logits, and the effect of limiting the maximum response length to 4,096 tokens are provided in Appendix D .

## 6 Relationship Between Task Coverage, Epistemic Verbalization and Generalization Ability

Across the off-policy and on-policy settings analyzed above, self-distillation consistently produces more confident responses with reduced 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] . This aligns with the findings of Hübotter et al. (2026) , which reports that SDPO learns to reason concisely: on Science Q&A (Chemistry, Physics, Biology, and Materials Science) ( Feng et al., 2024 ) , tool use ( Tang et al., 2023 ) , and LiveCodeBench v6 ( Jain et al., 2025 ) , SDPO achieves higher accuracy than GRPO while producing substantially shorter outputs with fewer epistemic markers.

In other words, in these domains, self-distillation suppresses epistemic verbalization and improves performance simultaneously. The key question is why the same mechanism leads to performance degradation in our math-focused setup . We hypothesize that the answer lies in differences in task coverage between the training and evaluation distributions.

### 6.1 Comparison of Task Coverage

To test this hypothesis, we compare the dataset characteristics of the settings where SDPO outperformed GRPO against our experimental setup. As shown in Table 3 , the Chemistry dataset, despite its large size, draws from only six main problem types that differ primarily in surface details rather than underlying structure. LiveCodeBench v6 contains diverse problems but only 131 in total, leading to repeated exposure during training with identical train/eval splits. In contrast, DAPO-Math-17k exposes the model to 14,000 distinct problems (78% of the 25,600 samples drawn over 100 steps, due to repeated sampling), spanning a broad, non-overlapping range of problem types, and evaluation is performed on unseen problem types.

### 6.2 Relationship Between Task Coverage and Learning Performance

To further investigate the interplay between task coverage and generalization, we vary the number of training questions | 𝒟 | ∈ { 1 , 8 , 64 , 128 , 512 } |\mathcal{D}|\in\{1,8,64,128,512\} from DAPO-Math-17k and train with both GRPO and SDPO. All experiments use Qwen3-8B (Thinking Mode OFF).

#### Training Logs

As shown in Figure 8 , GRPO and SDPO exhibit distinct training dynamics as | 𝒟 | |\mathcal{D}| varies. When | 𝒟 | ≤ 128 |\mathcal{D}|\leq 128 , SDPO quickly achieves high scores while reducing 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] by up to 8 × 8\times , indicating higher training efficiency on a small task set. However, at | 𝒟 | = 512 |\mathcal{D}|=512 , further reductions in 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] begin to hurt the training score relative to GRPO, whose 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] gradually increases with | 𝒟 | |\mathcal{D}| .

This difference can be interpreted through task coverage . As | 𝒟 | |\mathcal{D}| grows, the model must accommodate a broader range of reasoning patterns. GRPO addresses this by increasing 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] , allowing the model to express greater uncertainty and adapt its reasoning accordingly. SDPO instead encourages confident, concise responses— effective when task coverage is small but limiting when the problem set becomes larger and more diverse.

#### OOD Evaluation - AIME24, MATH500

The distinction between GRPO and SDPO becomes more pronounced on OOD benchmarks (Figure 9 ). Under GRPO, performance scales consistently with | 𝒟 | |\mathcal{D}| : | 𝒟 | = 1 |\mathcal{D}|=1 converges quickly but soon stops improving, while larger | 𝒟 | |\mathcal{D}| yields progressively higher final scores accompanied by increasing 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] . Under SDPO, the pattern reverses: smaller | 𝒟 | |\mathcal{D}| leads to more severe OOD degradation. Even at the largest | 𝒟 | |\mathcal{D}| (DAPO setting), SDPO still underperforms the base model. Changes in GRPO and SDPO reasoning patterns across task coverage are provided in Appendix E.3 .

## 7 Conclusion

In this work, we provide an empirical analysis of on-policy self-distillation, motivated by an information-theoretic view of conditioning context richness. Our experiments suggest that the effectiveness of self-distillation is closely tied to how information is provided to the model and how the model expresses uncertainty during reasoning. We observe that self-distillation tends to produce answers with higher confidence and shorter reasoning traces. While this effect enables more compact reasoning and can quickly improve in-domain performance when task coverage is limited, it becomes less effective when task coverage is broad and may even harm OOD performance.

Importantly, the fact that performance can degrade even when a mathematically sound objective function is designed to elicit correct CoT reasoning suggests that the choice of optimization objective alone may not be sufficient for preserving robust reasoning, and that we need to pay closer attention to how training reshapes the model’s reasoning behavior, beyond answer correctness. We hope that our analysis contributes to a deeper understanding of self-distillation and other LLM post-training methods, and provides insights for developing more robust training strategies.

## Acknowledgments

We thank Li Dong, Tianzhu Ye, and Sojeong Rhee for their valuable discussions. This work was supported by Microsoft Research and partly by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(RS-2025-00557589).

## Ethics Statement

This work aims to analyze the causes of reasoning performance degradation during self-distillation of LLMs. All experiments were conducted using publicly available models (Qwen3, DeepSeek-R1-Distill-Qwen, OLMo) and datasets (DAPO-Math-17k, AIME, AMC, MATH500). As this study does not involve any new data collection or human subjects research, concerns related to privacy or human subjects ethics do not apply.

A key finding of this work is that self-distillation can suppress epistemic verbalization, thereby diminishing a model’s ability to express uncertainty during reasoning. This observation carries important implications for the design of LLM post-training methods. The ability to appropriately express uncertainty is directly related to model reliability and safety. Training procedures that indiscriminately suppress such expressions may increase the risk of models producing incorrect answers with unwarranted confidence. We hope that our analysis contributes to the development of more robust and trustworthy reasoning models.

## Reproducibility Statement

We release our implementation and experiment scripts at GitHub (beanie00/self-distillation-analysis) , along with our training logs on Weights & Biases , and trained checkpoints for the main model settings on Hugging Face (beanie00/self-distillation-analysis) .

The repository provides instructions and scripts for data preparation, GRPO/SDPO training, task-coverage experiments, and evaluation. Detailed prompts, training hyperparameters, and evaluation configurations are provided in Appendix C .

## References

Biewald (2020) L. Biewald Experiment tracking with weights and biases . Note: Software available from wandb.com External Links: Link Cited by: Figure 1 .

Deng et al. (2026) J. Deng, S. Liang, J. Li, H. Li, and Y. Xie ConPress: learning efficient reasoning from multi-question contextual pressure . In Forty-third International Conference on Machine Learning , External Links: Link Cited by: Appendix A .

Feng et al. (2024) K. Feng, X. Shen, W. Wang, X. Zhuang, Y. Tang, Q. Zhang, and K. Ding Sciknoweval: evaluating multi-level scientific knowledge of large language models . arXiv preprint arXiv:2406.09098 . Cited by: §6 .

Gao et al. (2024) L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou The language model evaluation harness . Zenodo . External Links: Document , Link Cited by: Appendix C .

Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948 . Cited by: Appendix B , §1 , §3 , §4 , §5 .

Hübotter et al. (2026) J. Hübotter, F. Lübeck, L. D. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation . In Forty-third International Conference on Machine Learning , External Links: Link Cited by: Appendix A , Appendix H , Figure 1 , §1 , §3 , §3 , §5 , §6 .

Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code . In The Thirteenth International Conference on Learning Representations , External Links: Link Cited by: §6 .

Kim et al. (2026) J. Kim, X. Luo, M. Kim, S. Lee, D. Li, and Y. Yang Understanding reasoning in llms through strategic information allocation under uncertainty . arXiv preprint arXiv:2603.15500 . Cited by: Appendix A , §1 , §2 , §2 , §3 .

Liu et al. (2026) Z. Liu, J. Kim, X. Luo, D. Li, and Y. Yang Exploratory memory-augmented LLM agent via hybrid on- and off-policy optimization . In The Fourteenth International Conference on Learning Representations , External Links: Link Cited by: Appendix A , §1 .

Luo et al. (2026) Q. Luo, S. Ren, X. Chen, R. Liu, J. Fang, N. Tan, and S. Huang Compress the easy, explore the hard: difficulty-aware entropy regularization for efficient llm reasoning . arXiv preprint arXiv:2602.22642 . Cited by: Appendix A .

Olmo et al. (2025) T. Olmo, A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, et al. Olmo 3 . arXiv preprint arXiv:2512.13961 . Cited by: Appendix B , §5 .

Penaloza et al. (2026) E. Penaloza, D. Vattikonda, N. Gontier, A. Lacoste, L. Charlin, and M. Caccia Privileged information distillation for language models . In Forty-third International Conference on Machine Learning , External Links: Link Cited by: Appendix A , §1 , §5 .

Sang et al. (2026) H. Sang, Y. Xu, Z. Zhou, R. He, Z. Wang, and J. Sun On-policy self-distillation for reasoning compression . arXiv preprint arXiv:2603.05433 . Cited by: Appendix A .

Sheng et al. (2024) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient rlhf framework . arXiv preprint arXiv: 2409.19256 . Cited by: Appendix I .

Shrivastava et al. (2026) V. Shrivastava, A. H. Awadallah, V. Balachandran, S. Garg, H. Behl, and D. Papailiopoulos Sample more to think less: group filtered policy optimization for concise reasoning . In The Fourteenth International Conference on Learning Representations , External Links: Link Cited by: Appendix A .

Singh et al. (2025) A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. Openai gpt-5 system card . arXiv preprint arXiv:2601.03267 . Cited by: §E.5 .

Snell et al. (2022) C. Snell, D. Klein, and R. Zhong Learning by distilling context . arXiv preprint arXiv:2209.15189 . Cited by: Appendix A , §1 .

Tang et al. (2023) Q. Tang, Z. Deng, H. Lin, X. Han, Q. Liang, B. Cao, and L. Sun Toolalpaca: generalized tool learning for language models with 3000 simulated cases . arXiv preprint arXiv:2306.05301 . Cited by: §6 .

von Werra et al. (2020) L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec TRL: Transformers Reinforcement Learning . External Links: Link Cited by: Appendix I .

Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report . arXiv preprint arXiv:2505.09388 . Cited by: Appendix B , §5 .

Yang et al. (2026) Z. Yang, Z. Guo, Y. Huang, Y. Wang, W. Shi, Y. Wang, X. Liang, and J. Tang Accordion-thinking: self-regulated step summaries for efficient and readable llm reasoning . arXiv preprint arXiv:2602.03249 . Cited by: Appendix A .

Ye et al. (2026) T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei On-policy context distillation for language models . arXiv preprint arXiv:2602.12275 . Cited by: Appendix A , §1 , §5.5 .

Yu et al. (2026) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale . Advances in Neural Information Processing Systems 38 , pp. 113222–113244 . Cited by: §3 , §5 .

Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models . In Forty-third International Conference on Machine Learning , External Links: Link Cited by: Appendix A , Appendix I , Appendix I , §1 , §5.5 , §5 .

Zhu et al. (2026) M. Zhu, Y. Liu, Z. Fu, Q. Wang, and Y. Zhang In-token rationality optimization: towards accurate and concise llm reasoning via self-feedback . In Proceedings of the AAAI Conference on Artificial Intelligence , pp. 35195–35203 . Cited by: §1 .

## Appendix A Related Works

#### Self-Distillation for LLM Reasoning.

Self-distillation, where a single model acts as both teacher and student under asymmetric conditioning ( Snell et al., 2022 ) , has recently been widely adopted for LLM post-training. SDPO ( Hübotter et al., 2026 ) conditions the self-teacher on its previously generated correct trajectories or rich environment feedback to provide dense credit assignment, while OPSD ( Zhao et al., 2026 ) leverages ground-truth solutions as privileged information, achieving strong efficiency gains in both cases. Our analysis builds upon the SDPO codebase, and a detailed comparison with OPSD is provided in Appendix I . Liu et al. (2026) ; Penaloza et al. (2026) ; Ye et al. (2026) further utilize this paradigm in settings that incorporate additional signals, such as privileged information, self-generated hints derived from environmental interactions, or contextual guidance, with the goal of internalizing this external knowledge into the model’s parameters.

While these works demonstrate the effectiveness of self-distillation, they do not investigate when or why self-distillation may degrade reasoning performance, especially in scenarios where a model must solve problems entirely on its own, without interaction with the external environment, which is the central question of our work.

#### Uncertainty Expression in LLM Reasoning.

Kim et al. (2026) introduce an information-theoretic framework that identifies epistemic verbalization, the explicit externalization of uncertainty, as informationally necessary for robust reasoning. Our work answers a question this framework cannot readily explain: Under what conditions is this behavior encouraged or suppressed? And if epistemic verbalization is necessary, why does suppressing it lead to better performance in some domains, such as Chemistry, while degrading performance in others? We trace this to two factors: conditioning context richness (Section 3 ) and task coverage (Section 6 ), supported by a systematic analysis of recent on-policy self-distillation methods across various models with varying pretrained capabilities.

#### Reasoning Compression.

Several recent methods aim to shorten reasoning traces while preserving accuracy. GFPO ( Shrivastava et al., 2026 ) filters training rollouts by length and token efficiency within enlarged sample groups, reducing GRPO’s length inflation by 46–85% on STEM and coding benchmarks. OPSDC ( Sang et al., 2026 ) uses a conciseness instruction as privileged context, achieving 57–59% token reduction on MATH-500. ConPress ( Deng et al., 2026 ) exploits multi-question contextual pressure to elicit self-compressed traces, yielding 30–60% reductions via SFT. Accordion-Thinking ( Yang et al., 2026 ) trains models to produce periodic step summaries that can be discarded, achieving 3 × \times throughput with a gap-vanishing phenomenon during RL training. CEEH ( Luo et al., 2026 ) proposes difficulty-aware entropy regularization to prevent entropy collapse under length constraints.

Our analysis complements these efforts by showing that compression which suppresses uncertainty expression, even when producing correct answers, can harm OOD generalization, and that this effect is modulated by task coverage. Therefore, the key challenge is to filter out non-informative content while retaining epistemic expressions that enable iterative belief refinement, rather than blindly compressing the reasoning process.

## Appendix B Basis for Selecting Epistemic Tokens

To provide an empirical basis for our selection of epistemic tokens, we analyzed uncertainty expressions across a diverse set of reasoning models. We collected 9,600 reasoning traces from eight models spanning four model families, including the Qwen3 family ( Yang et al., 2025 ) (Qwen3-1.7B, Qwen3-8B, Qwen3-14B, and Qwen3-32B), the QwQ family ( Yang et al., 2025 ) (QwQ-32B), the DeepSeek-R1-Distill-Qwen family ( Guo et al., 2025 ) (7B and 32B), and the OLMo family ( Olmo et al., 2025 ) (OLMo-3-7B-Instruct).

The traces were generated on 600 math problems drawn from AIME 2024/2025, AMC 2023, and MATH500, with two samples per problem for each model. Each reasoning trace was then provided to GPT-5 with the following prompt:

This procedure yielded over 99,000 ( span , trigger_token ) (\textit{span},\textit{trigger\_token}) pairs. We then ranked trigger tokens by frequency within each model family (Qwen3, QwQ, DeepSeek, and OLMo).

As shown in Table 4 , the ten markers maybe , wait , check , hmm , seems , perhaps , might , actually , likely , and alternatively , together with their natural variants, consistently appeared among the top-ranked uncertainty expressions, supporting their inclusion across model families. While several tokens, such as let me try/verify , I think , and make sure , were excluded, our selected markers cover 75–93% of the occurrences of the phrases in Table 4 .

Moreover, to account for uncertainty expressions not captured by our token set, we conducted an independent LLM-as-a-Judge annotation experiment (Appendix E.5 ) that captures the full range of uncertainty expressions. The resulting distribution closely mirrors that obtained with our token-based approach.

Finally, although the linguistic patterns used to express uncertainty vary slightly across models, Qwen3, QwQ, and DeepSeek exhibit highly similar distributions. OLMo also shares largely the same set of uncertainty tokens, but differs in the relative frequency with which individual tokens are used. The selected markers largely overlap with expressions that human readers would naturally recognize as signals of uncertainty, plausibly because these models are ultimately trained on human-generated text.

## Appendix C Experimental Details

#### Training

For GRPO and SDPO training, we built upon the SDPO implementation ( https://github.com/lasgroup/SDPO ) and additionally incorporated the DAPO-Math-17k dataset. The original DAPO-Math-17k dataset uses the following prompt format:

Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\n{question}\nRemember to put your answer on its own line after "Answer:".

We replaced this with a simpler format:

{question}\nPlease reason step by step, and put your final answer within \boxed{}.

as we observed that this format consistently yielded higher evaluation performance. For reward verification, we used the scoring function from the verl framework, which extracts the answer from the \boxed{} expression and verifies correctness via exact match followed by mathematical equivalence checking using math-verify, adapted from EleutherAI’s lm-evaluation-harness ( Gao et al., 2024 ) .

The GRPO and SDPO training hyperparameters are listed in Tables 5 , 6 , and 7 . For the experiments on the relationship between task coverage and learning performance in Figure 9 , we reduced the question batch size to 64 due to the use of smaller training questions.

#### Evaluation

We evaluated all models using the hyperparameters listed in Table 8 , following the recommended settings from each model’s official documentation. 1 1 1 https://huggingface.co/Qwen/Qwen3-8B , 2 2 2 https://huggingface.co/allenai/OLMo-3-7B-Instruct

#### Chat Templates for Different Model Series

Table 9 summarizes the chat template formats used by several open-weight model families, each using distinct special tokens to delineate user and assistant turns. The placeholder {prompt} denotes the user message.

## Appendix D More Ablation Study

### D.1 Effect of Maximum Response Length

In our main experiments, as detailed in Table 5 , we set the maximum training response length to 20,480 tokens to avoid prematurely truncating the long reasoning trajectories. Under a budget-constrained setting, we additionally examine how the training behavior of GRPO and SDPO changes when the maximum response length is reduced to 4,096 tokens.

As shown in Figure 10 , under the 4,096-token constraint, SDPO achieves relatively high training scores early in training, whereas GRPO improves more gradually and eventually reaches comparable training performance. More importantly, GRPO consistently achieves higher performance on AIME24 and AMC23, while SDPO plateaus at lower performance. Under a strict response budget, SDPO’s tendency to produce shorter reasoning traces allows more responses to be completed within the available token budget, which likely contributes to its rapid initial gains. However, this early advantage does not translate into continued improvement in reasoning performance.

### D.2 Top- k k Distillation and Learning Rate

To examine the effect of various training hyperparameters on self-distillation behavior, we conduct additional experiments by varying the top- k k distillation parameter and the learning rate. As shown in Figure 11a , increasing top- k k from 100 to 256 yields no significant difference in training dynamics or final performance. Meanwhile, as illustrated in Figure 11b , reducing the learning rate from 1 ​ e − 5 1\mathrm{e}{-5} to 1 ​ e − 6 1\mathrm{e}{-6} slows the rate of degradation, but the model ultimately converges to the same reasoning behavior.

## Appendix E Additional Analysis of Epistemic Verbalization

### E.1 Per-Token Analysis of Epistemic Verbalization

In Table 1 of Section 3 , we compared the average number of ten epistemic tokens per response. Figure 12 further extends this analysis by examining how the average per-response count of each individual token changes under varying levels of conditioning information. When examining the per-token counts, all tokens exhibit a consistent trend: 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 1 ) > 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 3 ) > 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 4 ) > 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 2 ) , \mathbb{E}\bigl[E(y)\bigr]\Big|_{(1)}>\mathbb{E}\bigl[E(y)\bigr]\Big|_{(3)}>\mathbb{E}\bigl[E(y)\bigr]\Big|_{(4)}>\mathbb{E}\bigl[E(y)\bigr]\Big|_{(2)}, (5) where tokens such as wait , maybe , and perhaps are particularly prominent.

### E.2 Comparison of Epistemic Token Usage Across Models

Following the analysis of DeepSeek-R1-Distill-Qwen-7B (DeepSeek-Distill-7B) in Section 3 , we further compare epistemic token usage across three settings: DeepSeek-Distill-7B, Qwen3-8B (Thinking Mode: ON), and Qwen3-8B (Thinking Mode: OFF).

As shown in Figure 13 , both DeepSeek-Distill-7B and Qwen3-8B (Thinking Mode: ON) produce substantially more epistemic tokens than Qwen3-8B (Thinking Mode: OFF). While the two thinking-enabled models share a similar tendency to express uncertainty, they differ in their preferred epistemic tokens. For instance, DeepSeek-Distill-7B frequently uses wait and employs perhaps and maybe at comparable rates, whereas Qwen3-8B uses perhaps relatively less and favors maybe . Qwen3-8B also uses alternatively and check far more than DeepSeek-Distill-7B, and overall embeds a greater amount of uncertainty within its reasoning.

Extending the discussion in Section 3 , we also observe that Qwen3-8B generates far fewer epistemic tokens under solution-guided generation than under unguided generation. Across all three settings, Qwen3-8B (Thinking Mode: ON) produces the most epistemic tokens, followed by DeepSeek-Distill-7B, and then Qwen3-8B (Thinking Mode: OFF).

### E.3 Analysis of GRPO and SDPO Reasoning Patterns by Task Coverage

To provide a more in-depth analysis of the results in Figure 9 (Section 6.2 ), we compare the change in epistemic token counts relative to the base model on AIME24 across six training configurations: GRPO and SDPO, each with | D | ∈ { 1 , 64 , 512 } |D|\in\{1,64,512\} . Figure 14 shows that GRPO tends to increase epistemic token usage relative to the base model as | D | |D| grows, whereas SDPO reduces epistemic token usage, with smaller | D | |D| leading to a greater reduction.

### E.4 Comparative Analysis of Epistemic and Overall Word Usage

In addition to analyzing epistemic tokens, we examine how the overall word distribution in model responses shifts across the base model, GRPO-trained model, and SDPO-trained model. We first extract English words from each response using regular expressions and construct a reference vocabulary 𝒱 \mathcal{V} from all unique words appearing in the base model responses. For each word w ∈ 𝒱 w\in\mathcal{V} and model m ∈ { base , GRPO , SDPO } m\in\{\text{base},\text{GRPO},\text{SDPO}\} , we compute the average per-response frequency: μ m ​ ( w ) = 1 N ​ ∑ i = 1 N f ⁡ ( w , r i ( m ) ) , \mu_{m}(w)=\frac{1}{N}\sum_{i=1}^{N}f(w,r_{i}^{(m)}), (6) where f ⁡ ( w , r ) f(w,r) denotes the number of occurrences of word w w in response r r . We then define the frequency shift Δ m ​ ( w ) = μ m ​ ( w ) − μ base ​ ( w ) \Delta_{m}(w)=\mu_{m}(w)-\mu_{\text{base}}(w) to capture how much each word’s usage changes relative to the base model.

Using these frequency shifts, we compare the behavior of the predefined set of epistemic tokens T T (as defined in Section 3 ) against the rest of the vocabulary. Specifically, we compare { Δ m ​ ( w ) ∣ w ∈ T } \{\Delta_{m}(w)\mid w\in T\} against { Δ m ​ ( w ) ∣ w ∈ 𝒱 ∖ T } \{\Delta_{m}(w)\mid w\in\mathcal{V}\setminus T\} to determine whether the changes in epistemic token usage are disproportionately large relative to the overall shift in word distribution. This allows us to distinguish whether the training procedure specifically amplifies epistemic expressions or merely induces a uniform change across the vocabulary.

As shown in Figure 15 , the disparity between the two panels confirms that the observed epistemic shifts are not byproducts of general vocabulary drift. Across all model-method pairs, the average per-word shift over the full vocabulary remains near zero ( | Δ | < 1 |\Delta|<1 ), whereas epistemic tokens exhibit shifts up to 30 × 30\times – 40 × 40\times larger in magnitude (e.g., − 11.9 -11.9 and − 12.2 -12.2 for SDPO). This indicates that both training objectives selectively target epistemic expressions rather than uniformly perturbing the output distribution.

### E.5 Uncertainty Annotation via LLM-as-Judge

We employed epistemic markers as indicators of uncertainty expressions, as they serve as practical and interpretable cues for regions where uncertainty externalization is likely to occur. To more comprehensively capture diverse forms of uncertainty expression, we additionally leveraged GPT-5.4 ( Singh et al., 2025 ) as an automated annotator. Specifically, GPT-5.4 was prompted to identify and count uncertainty expressions in each model-generated response. The prompt used for this annotation is shown below.

Table 10 presents representative examples of uncertainty expressions identified by the annotator. The majority of identified expressions coincide with the epistemic tokens examined in the previous analysis (e.g., maybe , perhaps , wait ), while others manifest as multi-token phrases such as “I’m not entirely sure” that cannot be captured by single-token frequency alone. This confirms that epistemic markers serve as a reliable proxy for uncertainty externalization, while the LLM-as-Judge approach provides complementary coverage over more diverse expression forms.

Table 11 reports the average number of such expressions per response for each method. When evaluated through LLM-as-Judge annotation, SDPO again substantially reduces uncertainty expressions compared to both the baseline and GRPO, consistent with the token-level analysis. Figure 16 complements this with a box plot comparison across methods: notably, GRPO exhibits considerably greater variance than the baseline, suggesting that RLVR can amplify inconsistency in hedging behavior. In contrast, SDPO maintains consistently low counts with a narrow spread, confirming that on-policy self-distillation suppresses unnecessary hedging and self-doubt during reasoning.

## Appendix F Student–Teacher KL Divergence on Epistemic Tokens

We additionally measure the per-token KL divergence between the student π θ ( ⋅ ∣ x , y < t ) \pi_{\theta}(\cdot\mid x,y_{<t}) and the self-teacher π θ ( ⋅ ∣ x , c , y < t ) \pi_{\theta}(\cdot\mid x,c,y_{<t}) for each surface-form variant of the epistemic tokens in 𝒯 \mathcal{T} . As shown in Table 12 , every variant exceeds the corpus-wide mean KL of 0.076 0.076 , ranging from 1.9 × 1.9\times ( ‘Wait’ ) to 6.8 × 6.8\times ( ‘ Hmm’ ). Notably, sentence-initial variants (e.g., ‘ Hmm’ at 6.8 × 6.8\times , ‘ wait’ at 3.8 × 3.8\times ) tend to exhibit larger divergence than mid-sentence forms, suggesting that the self-teacher most strongly disagrees at positions where the student initiates epistemic deliberation. This provides a quantitative explanation for the suppression of epistemic verbalization under SDPO.

## Appendix G Analysis of Changes in Model Internal Uncertainty under Self-Distillation

To examine whether self-distillation also reduces the model’s internal uncertainty, we further analyze token-level entropy as a measure of internal uncertainty. Building on the analysis in Section 3 , we revisit the comparison between unguided and solution-guided generation to assess whether access to richer information is associated with lower predictive uncertainty.

As shown in Table 13 , solution-guided generation reduces average token-level entropy by approximately 43%, indicating substantially lower predictive uncertainty when richer information is available. Moreover, under unguided generation, epistemic tokens have an average entropy of 1.34, approximately 3.5 × 3.5\times higher than the overall average, indicating that epistemic verbalization tends to occur at positions of high predictive uncertainty.

We next examine whether this lower-entropy generation behavior is transferred to the student through self-distillation.

Table 14 and Figure 17 show a consistent pattern in which GRPO increases token-level entropy for both models, whereas SDPO reduces it, and this distinction persists throughout training. These results indicate that the suppression of uncertainty expressions through self-distillation is accompanied by a genuine reduction in the model’s internal uncertainty.

## Appendix H Can Uncertainty-Encouraging Prompts Mitigate the Suppression?

We further investigate whether prompting the teacher to produce more verbose reasoning with explicit expressions of uncertainty can mitigate the performance degradation caused by the suppression of uncertainty expressions. To this end, we design a modified regeneration prompt that encourages detailed, step-by-step reasoning with uncertainty, and repeat the reasoning behavior analysis (Section 3 ) and on-policy self-distillation experiments (Section 5 ) using this prompt.

Table 15 shows that the uncertainty-encouraging prompt partially recovers the suppressed reasoning behavior: compared to the vanilla solution-guided prompt, it produces longer responses (3,098 vs. 1,873 tokens) and more epistemic expressions (31.4 vs. 8.8). However, both metrics remain far below those of unguided generation, indicating that the presence of a reference solution fundamentally constrains the model’s reasoning verbosity regardless of the prompt instruction.

Figure 18 shows that the uncertainty-encouraging prompt leads to slightly higher training scores and longer responses compared to the vanilla setting, and this improvement carries over to AIME24 evaluation. Nevertheless, both variants still suffer from substantial performance degradation, suggesting that prompting alone is insufficient to resolve the fundamental issue of uncertainty suppression in solution-guided generation.

## Appendix I Comparison with OPSD from Zhao et al. (2026)

Recently, Zhao et al. (2026) proposed OPSD and demonstrated performance gains in mathematical reasoning through on-policy self-distillation, particularly on the Qwen3 series. Unlike our setup, where both the student and the teacher either enable or disable thinking mode, OPSD adopts a hybrid configuration in which the student operates with thinking mode disabled while the teacher has it enabled.

As our experiments also confirm, enabling thinking mode produces substantially longer responses with a greater number of epistemic tokens, making this hybrid setup function more akin to conventional teacher–student distillation, despite using the same underlying model. We note that this configuration is inherently limited to model families such as Qwen3 that support toggling thinking mode on and off.

For training efficiency, OPSD trains only on a response prefix (1,024 tokens by default), rather than the full student response. Unlike OPSD, SDPO and our setup use full fine-tuning with verl ( Sheng et al., 2024 ) , whereas OPSD adopts LoRA fine-tuning with trl ( von Werra et al., 2020 ) and a smaller batch size of 32 (vs. 256 in our setup), resulting in higher training efficiency but smaller parameter updates per step.

Under this hybrid setup with prefix learning in Qwen3-1.7B, as shown in Figure 19a ( orange line ), we observe an intriguing training dynamic. In the early stages, the thinking-enabled teacher encourages the student to generate longer responses while simultaneously improving performance, highlighting the effectiveness of hybrid distillation during the initial phase of training. However, as training progresses, performance steadily deteriorates despite this early improvement. In contrast, under our homogeneous setup, where both the student and teacher operate with thinking mode enabled ( pink line ), performance declines consistently throughout training. Response length and epistemic token count likewise decrease steadily, consistent with our analysis.

While this hybrid distillation setup for the Qwen3 series presents an interesting research direction with its own unique training dynamics, such as why performance initially improves before declining and whether this stems from changes in reasoning behavior or chat template mismatch, a thorough investigation is beyond the scope of this work and is left for future exploration.

## Appendix J More On-Policy Self-Distillation Results

### J.1 Qwen3-8B (Thinking Mode: OFF)

As an extension of Figure 5 in Section 5.3 , we additionally compare AMC23 evaluation scores and the change in epistemic token usage on AIME24. As shown in Figure 20 , GRPO significantly increases response length and yields substantial performance gains on both benchmarks. In contrast, SDPO exhibits divergent trends: on AMC23, acc@16 increases from 0.67 to 0.73 while reducing response length by roughly half, whereas on AIME24, acc@16 slightly decreases from 0.25 to 0.23, with pass@16 dropping more substantially. Notably on AMC23, SDPO achieves approximately a 6-point improvement with shorter responses, whereas GRPO obtains a much larger improvement of around 36 points at the cost of considerably longer responses. Achieving large performance gains while maintaining a reasonable response length remains an open challenge.

### J.2 Qwen3-1.7B (Thinking Mode: ON)

To investigate whether our findings generalize to smaller models, we conducted the same on-policy self-distillation experiment on Qwen3-1.7B. As shown in Figure 21 , our finding that self-distillation reduces uncertainty and makes outputs more concise, while simultaneously degrading mathematical reasoning performance, remains consistent across model sizes. Notably, the performance degradation is more severe for the 1.7B model (45.9%) compared to Qwen3-8B with Thinking Mode ON (12.1%) and DeepSeek-R1-Distill-Qwen-7B (40.7%). This suggests that the externalization of uncertainty may be particularly important for robust reasoning in smaller models, which possess more limited inherent capabilities.

### J.3 Olmo-3-7B-Instruct

In addition to DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B, we further evaluate on-policy self-distillation on OLMo-3-7B-Instruct from a different model family. As shown in Figure 22 , consistent with our previous analyses, SDPO also degrades reasoning performance on this model, with OOD evaluation scores falling below those of the base model. This confirms that our findings are not model-dependent but reflect robust characteristics of reasoning behavior across diverse model families.

### J.4 Pass@16 Score

In addition to the acc@16 scores for DeepSeek-Distill-7B and Qwen3-8B (Thinking Mode: ON) presented in Figures 3b , 3c , 4b , and 4c , we report pass@16 scores in Figure 23 . The gap between GRPO and SDPO in pass@16 is larger for DeepSeek-Distill-7B than for Qwen3-8B, and more pronounced on the harder benchmark (AIME24) compared to AMC23.

## Appendix K Qualitative Analysis

### K.1 Unguided vs. Solution-Guided Generation

We analyze examples of 𝒟 ug \mathcal{D}_{\text{ug}} (Unguided responses) and 𝒟 sg \mathcal{D}_{\text{sg}} (Solution-guided responses) introduced in Section 4 . Below, we present the two responses generated for the same problem. As illustrated, the unguided response exhibits substantially more uncertainty and exploratory reasoning, frequently revisiting assumptions, considering alternative approaches, and self-correcting along the way. In contrast, the solution-guided response proceeds in a more concise and direct manner, arriving at each intermediate result without detours.

Problem. How many ways can 8 8 mutually non-attacking rooks be placed on a 9 × 9 9\times 9 chessboard so that all 8 8 rooks are on squares of the same color? (Two rooks are said to be attacking each other if they are placed in the same row or column of the board.) Please reason step by step, and put your final answer within \boxed{\phantom{0}} .

### K.2 GRPO vs. SDPO Generation

#### Problem (AIME24).

Find the number of ways to place a digit in each cell of a 2 × \times 3 grid so that the sum of the two numbers formed by reading left to right is 999, and the sum of the three numbers formed by reading top to bottom is 99. The grid below is an example of such an arrangement because 8 + 991 = 999 8+991=999 and 9 + 9 + 81 = 99 9+9+81=99 : 0 0 8 9 9 1 \begin{array}[]{|c|c|c|}\hline\cr 0&0&8\\ \hline\cr 9&9&1\\ \hline\cr\end{array} The correct answer is 45 \boxed{45} .

GRPO explores with repeated reflections, deriving the extra constraint a + b + c = 8 a+b+c=8 beyond the pairwise sums, and correctly counts ( 10 2 ) = 45 \binom{10}{2}=45 solutions. SDPO proceeds directly and stops after deriving a + d = b + e = c + f = 9 a+d=b+e=c+f=9 , multiplying 10 × 10 × 10 = 1000 10\times 10\times 10=1000 without checking the row-sum equation, thereby missing the additional constraint a + b + c = 8 a+b+c=8 .

## Appendix L Disclosure of LLM Usage

We used LLMs for conducting the LLM-as-a-Judge experiments described in Appendix E.5 , refining plot designs, and improving the clarity of our writing. LLMs were not involved in research ideation, and their role in writing was limited to polishing author-written text.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .