Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? — part 3/4

Cleaned-text code points [31527, 47444). Bibliography omitted here; source.md includes it.

Self-distillation, where a single model acts as both teacher and student under asymmetric conditioning ( Snell et al., 2022 ) , has recently been widely adopted for LLM post-training. SDPO ( Hübotter et al., 2026 ) conditions the self-teacher on its previously generated correct trajectories or rich environment feedback to provide dense credit assignment, while OPSD ( Zhao et al., 2026 ) leverages ground-truth solutions as privileged information, achieving strong efficiency gains in both cases. Our analysis builds upon the SDPO codebase, and a detailed comparison with OPSD is provided in Appendix I . Liu et al. (2026) ; Penaloza et al. (2026) ; Ye et al. (2026) further utilize this paradigm in settings that incorporate additional signals, such as privileged information, self-generated hints derived from environmental interactions, or contextual guidance, with the goal of internalizing this external knowledge into the model’s parameters.

While these works demonstrate the effectiveness of self-distillation, they do not investigate when or why self-distillation may degrade reasoning performance, especially in scenarios where a model must solve problems entirely on its own, without interaction with the external environment, which is the central question of our work.

#### Uncertainty Expression in LLM Reasoning.

Kim et al. (2026) introduce an information-theoretic framework that identifies epistemic verbalization, the explicit externalization of uncertainty, as informationally necessary for robust reasoning. Our work answers a question this framework cannot readily explain: Under what conditions is this behavior encouraged or suppressed? And if epistemic verbalization is necessary, why does suppressing it lead to better performance in some domains, such as Chemistry, while degrading performance in others? We trace this to two factors: conditioning context richness (Section 3 ) and task coverage (Section 6 ), supported by a systematic analysis of recent on-policy self-distillation methods across various models with varying pretrained capabilities.

#### Reasoning Compression.

Several recent methods aim to shorten reasoning traces while preserving accuracy. GFPO ( Shrivastava et al., 2026 ) filters training rollouts by length and token efficiency within enlarged sample groups, reducing GRPO’s length inflation by 46–85% on STEM and coding benchmarks. OPSDC ( Sang et al., 2026 ) uses a conciseness instruction as privileged context, achieving 57–59% token reduction on MATH-500. ConPress ( Deng et al., 2026 ) exploits multi-question contextual pressure to elicit self-compressed traces, yielding 30–60% reductions via SFT. Accordion-Thinking ( Yang et al., 2026 ) trains models to produce periodic step summaries that can be discarded, achieving 3 × \times throughput with a gap-vanishing phenomenon during RL training. CEEH ( Luo et al., 2026 ) proposes difficulty-aware entropy regularization to prevent entropy collapse under length constraints.

Our analysis complements these efforts by showing that compression which suppresses uncertainty expression, even when producing correct answers, can harm OOD generalization, and that this effect is modulated by task coverage. Therefore, the key challenge is to filter out non-informative content while retaining epistemic expressions that enable iterative belief refinement, rather than blindly compressing the reasoning process.

## Appendix B Basis for Selecting Epistemic Tokens

To provide an empirical basis for our selection of epistemic tokens, we analyzed uncertainty expressions across a diverse set of reasoning models. We collected 9,600 reasoning traces from eight models spanning four model families, including the Qwen3 family ( Yang et al., 2025 ) (Qwen3-1.7B, Qwen3-8B, Qwen3-14B, and Qwen3-32B), the QwQ family ( Yang et al., 2025 ) (QwQ-32B), the DeepSeek-R1-Distill-Qwen family ( Guo et al., 2025 ) (7B and 32B), and the OLMo family ( Olmo et al., 2025 ) (OLMo-3-7B-Instruct).

The traces were generated on 600 math problems drawn from AIME 2024/2025, AMC 2023, and MATH500, with two samples per problem for each model. Each reasoning trace was then provided to GPT-5 with the following prompt:

This procedure yielded over 99,000 ( span , trigger_token ) (\textit{span},\textit{trigger\_token}) pairs. We then ranked trigger tokens by frequency within each model family (Qwen3, QwQ, DeepSeek, and OLMo).

As shown in Table 4 , the ten markers maybe , wait , check , hmm , seems , perhaps , might , actually , likely , and alternatively , together with their natural variants, consistently appeared among the top-ranked uncertainty expressions, supporting their inclusion across model families. While several tokens, such as let me try/verify , I think , and make sure , were excluded, our selected markers cover 75–93% of the occurrences of the phrases in Table 4 .

Moreover, to account for uncertainty expressions not captured by our token set, we conducted an independent LLM-as-a-Judge annotation experiment (Appendix E.5 ) that captures the full range of uncertainty expressions. The resulting distribution closely mirrors that obtained with our token-based approach.

Finally, although the linguistic patterns used to express uncertainty vary slightly across models, Qwen3, QwQ, and DeepSeek exhibit highly similar distributions. OLMo also shares largely the same set of uncertainty tokens, but differs in the relative frequency with which individual tokens are used. The selected markers largely overlap with expressions that human readers would naturally recognize as signals of uncertainty, plausibly because these models are ultimately trained on human-generated text.

## Appendix C Experimental Details

#### Training

For GRPO and SDPO training, we built upon the SDPO implementation ( https://github.com/lasgroup/SDPO ) and additionally incorporated the DAPO-Math-17k dataset. The original DAPO-Math-17k dataset uses the following prompt format:

Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.\n\n{question}\nRemember to put your answer on its own line after "Answer:".

We replaced this with a simpler format:

{question}\nPlease reason step by step, and put your final answer within \boxed{}.

as we observed that this format consistently yielded higher evaluation performance. For reward verification, we used the scoring function from the verl framework, which extracts the answer from the \boxed{} expression and verifies correctness via exact match followed by mathematical equivalence checking using math-verify, adapted from EleutherAI’s lm-evaluation-harness ( Gao et al., 2024 ) .

The GRPO and SDPO training hyperparameters are listed in Tables 5 , 6 , and 7 . For the experiments on the relationship between task coverage and learning performance in Figure 9 , we reduced the question batch size to 64 due to the use of smaller training questions.

#### Evaluation

We evaluated all models using the hyperparameters listed in Table 8 , following the recommended settings from each model’s official documentation. 1 1 1 https://huggingface.co/Qwen/Qwen3-8B , 2 2 2 https://huggingface.co/allenai/OLMo-3-7B-Instruct

#### Chat Templates for Different Model Series

Table 9 summarizes the chat template formats used by several open-weight model families, each using distinct special tokens to delineate user and assistant turns. The placeholder {prompt} denotes the user message.

## Appendix D More Ablation Study

### D.1 Effect of Maximum Response Length

In our main experiments, as detailed in Table 5 , we set the maximum training response length to 20,480 tokens to avoid prematurely truncating the long reasoning trajectories. Under a budget-constrained setting, we additionally examine how the training behavior of GRPO and SDPO changes when the maximum response length is reduced to 4,096 tokens.

As shown in Figure 10 , under the 4,096-token constraint, SDPO achieves relatively high training scores early in training, whereas GRPO improves more gradually and eventually reaches comparable training performance. More importantly, GRPO consistently achieves higher performance on AIME24 and AMC23, while SDPO plateaus at lower performance. Under a strict response budget, SDPO’s tendency to produce shorter reasoning traces allows more responses to be completed within the available token budget, which likely contributes to its rapid initial gains. However, this early advantage does not translate into continued improvement in reasoning performance.

### D.2 Top- k k Distillation and Learning Rate

To examine the effect of various training hyperparameters on self-distillation behavior, we conduct additional experiments by varying the top- k k distillation parameter and the learning rate. As shown in Figure 11a , increasing top- k k from 100 to 256 yields no significant difference in training dynamics or final performance. Meanwhile, as illustrated in Figure 11b , reducing the learning rate from 1 ​ e − 5 1\mathrm{e}{-5} to 1 ​ e − 6 1\mathrm{e}{-6} slows the rate of degradation, but the model ultimately converges to the same reasoning behavior.

## Appendix E Additional Analysis of Epistemic Verbalization

### E.1 Per-Token Analysis of Epistemic Verbalization

In Table 1 of Section 3 , we compared the average number of ten epistemic tokens per response. Figure 12 further extends this analysis by examining how the average per-response count of each individual token changes under varying levels of conditioning information. When examining the per-token counts, all tokens exhibit a consistent trend: 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 1 ) > 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 3 ) > 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 4 ) > 𝔼 ⁡ [ E ⁡ ( y ) ] | ( 2 ) , \mathbb{E}\bigl[E(y)\bigr]\Big|_{(1)}>\mathbb{E}\bigl[E(y)\bigr]\Big|_{(3)}>\mathbb{E}\bigl[E(y)\bigr]\Big|_{(4)}>\mathbb{E}\bigl[E(y)\bigr]\Big|_{(2)}, (5) where tokens such as wait , maybe , and perhaps are particularly prominent.

### E.2 Comparison of Epistemic Token Usage Across Models

Following the analysis of DeepSeek-R1-Distill-Qwen-7B (DeepSeek-Distill-7B) in Section 3 , we further compare epistemic token usage across three settings: DeepSeek-Distill-7B, Qwen3-8B (Thinking Mode: ON), and Qwen3-8B (Thinking Mode: OFF).

As shown in Figure 13 , both DeepSeek-Distill-7B and Qwen3-8B (Thinking Mode: ON) produce substantially more epistemic tokens than Qwen3-8B (Thinking Mode: OFF). While the two thinking-enabled models share a similar tendency to express uncertainty, they differ in their preferred epistemic tokens. For instance, DeepSeek-Distill-7B frequently uses wait and employs perhaps and maybe at comparable rates, whereas Qwen3-8B uses perhaps relatively less and favors maybe . Qwen3-8B also uses alternatively and check far more than DeepSeek-Distill-7B, and overall embeds a greater amount of uncertainty within its reasoning.

Extending the discussion in Section 3 , we also observe that Qwen3-8B generates far fewer epistemic tokens under solution-guided generation than under unguided generation. Across all three settings, Qwen3-8B (Thinking Mode: ON) produces the most epistemic tokens, followed by DeepSeek-Distill-7B, and then Qwen3-8B (Thinking Mode: OFF).

### E.3 Analysis of GRPO and SDPO Reasoning Patterns by Task Coverage

To provide a more in-depth analysis of the results in Figure 9 (Section 6.2 ), we compare the change in epistemic token counts relative to the base model on AIME24 across six training configurations: GRPO and SDPO, each with | D | ∈ { 1 , 64 , 512 } |D|\in\{1,64,512\} . Figure 14 shows that GRPO tends to increase epistemic token usage relative to the base model as | D | |D| grows, whereas SDPO reduces epistemic token usage, with smaller | D | |D| leading to a greater reduction.

### E.4 Comparative Analysis of Epistemic and Overall Word Usage

In addition to analyzing epistemic tokens, we examine how the overall word distribution in model responses shifts across the base model, GRPO-trained model, and SDPO-trained model. We first extract English words from each response using regular expressions and construct a reference vocabulary 𝒱 \mathcal{V} from all unique words appearing in the base model responses. For each word w ∈ 𝒱 w\in\mathcal{V} and model m ∈ { base , GRPO , SDPO } m\in\{\text{base},\text{GRPO},\text{SDPO}\} , we compute the average per-response frequency: μ m ​ ( w ) = 1 N ​ ∑ i = 1 N f ⁡ ( w , r i ( m ) ) , \mu_{m}(w)=\frac{1}{N}\sum_{i=1}^{N}f(w,r_{i}^{(m)}), (6) where f ⁡ ( w , r ) f(w,r) denotes the number of occurrences of word w w in response r r . We then define the frequency shift Δ m ​ ( w ) = μ m ​ ( w ) − μ base ​ ( w ) \Delta_{m}(w)=\mu_{m}(w)-\mu_{\text{base}}(w) to capture how much each word’s usage changes relative to the base model.

Using these frequency shifts, we compare the behavior of the predefined set of epistemic tokens T T (as defined in Section 3 ) against the rest of the vocabulary. Specifically, we compare { Δ m ​ ( w ) ∣ w ∈ T } \{\Delta_{m}(w)\mid w\in T\} against { Δ m ​ ( w ) ∣ w ∈ 𝒱 ∖ T } \{\Delta_{m}(w)\mid w\in\mathcal{V}\setminus T\} to determine whether the changes in epistemic token usage are disproportionately large relative to the overall shift in word distribution. This allows us to distinguish whether the training procedure specifically amplifies epistemic expressions or merely induces a uniform change across the vocabulary.

As shown in Figure 15 , the disparity between the two panels confirms that the observed epistemic shifts are not byproducts of general vocabulary drift. Across all model-method pairs, the average per-word shift over the full vocabulary remains near zero ( | Δ | < 1 |\Delta|<1 ), whereas epistemic tokens exhibit shifts up to 30 × 30\times – 40 × 40\times larger in magnitude (e.g., − 11.9 -11.9 and − 12.2 -12.2 for SDPO). This indicates that both training objectives selectively target epistemic expressions rather than uniformly perturbing the output distribution.

### E.5 Uncertainty Annotation via LLM-as-Judge

We employed epistemic markers as indicators of uncertainty expressions, as they serve as practical and interpretable cues for regions where uncertainty externalization is likely to occur. To more comprehensively capture diverse forms of uncertainty expression, we additionally leveraged GPT-5.4 ( Singh et al., 2025 ) as an automated annotator. Specifically, GPT-5.4 was prompted to identify and count uncertainty expressions in each model-generated response. The prompt used for this annotation is shown below.

Table 10 presents representative examples of uncertainty expressions identified by the annotator. The majority of identified expressions coincide with the epistemic tokens examined in the previous analysis (e.g., maybe , perhaps , wait ), while others manifest as multi-token phrases such as “I’m not entirely sure” that cannot be captured by single-token frequency alone. This confirms that epistemic markers serve as a reliable proxy for uncertainty externalization, while the LLM-as-Judge approach provides complementary coverage over more diverse expression forms.

Table 11 reports the average number of such expressions per response for each method. When evaluated through LLM-as-Judge annotation, SDPO again substantially reduces uncertainty expressions compared to both the baseline and GRPO, consistent with the token-level analysis. Figure 16 complements this with a box plot comparison across methods: notably, GRPO exhibits considerably greater variance than the baseline, suggesting that RLVR can amplify inconsistency in hedging behavior. In contrast, SDPO maintains consistently low counts with a narrow spread, confirming that on-policy self-distillation suppresses unnecessary hedging and self-doubt during reasoning.

## Appendix F Student–Teacher KL Divergence on Epistemic Tokens