Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? — part 2/4

Cleaned-text code points [15647, 31527). Bibliography omitted here; source.md includes it.

We now turn to on-policy self-distillation ( Hübotter et al., 2026 ; Zhao et al., 2026 ; Penaloza et al., 2026 ) , where the model learns from dense reward signals provided by a self-teacher with access to the correct solution, based on its own responses. We compare GRPO with Reinforcement Learning via Self-Distillation (SDPO) ( Hübotter et al., 2026 ) on the DAPO-Math-17k dataset ( Yu et al., 2026 ) , using Qwen3-8B ( Yang et al., 2025 ) and DeepSeek-R1-Distill-Qwen-7B ( Guo et al., 2025 ) . Additional results for Qwen3-1.7B and Olmo-3-7B-Instruct ( Olmo et al., 2025 ) are in Appendix J.3 . For each model, we track training score, response length, and OOD performance on AIME24 and AMC23. We fix the teacher to the initial policy, which performs better than a moving target (see Section 5.5 for a comparison).

The behavior of on-policy self-distillation depends on two factors: (i) the extent of epistemic verbalization already present in the base model, and (ii) the richness of the conditioning context c c . To disentangle these, we compare GRPO and SDPO under two settings: c = s c=s (full solution) and c = s ∖ think c=s_{\setminus\text{think}} (solution without <think> content).

### 5.1 DeepSeek-R1-Distill-Qwen-7B

DeepSeek-R1-Distill-Qwen-7B is a representative high-reasoning model that generates rich epistemic verbalizations within <think> tags, averaging about 180 epistemic tokens per response, as shown in Section 3 , leading to lengthy yet high-performing reasoning outputs.

#### Training Performance

As shown in Figure 3a , GRPO training slightly increases 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] with a modest improvement in score. In contrast, SDPO with c = s c=s causes a sharp initial drop in both 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] and score. Performance gradually recovers but remains below GRPO throughout training. When the conditioning is reduced to c = s ∖ think c=s_{\setminus\text{think}} , the drop in 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] is attenuated and the score trajectory approaches that of GRPO, consistent with the relationship between I ⁡ ( y ∗ ; c ∣ x ) I(y^{*};\,c\mid x) and epistemic suppression discussed in Section 3 .

#### OOD Evaluation - AIME24, AMC23

Consistent with the training trends, GRPO yields modest gains on both OOD benchmarks (AIME24: 54.7 → \to 56.0; AMC23: 89.3 → \to 91.1, Figures 3b and 3c ) with a slight increase in 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] . SDPO with c = s c=s degrades performance substantially ( ∼ 40 % {\sim}40\% on AIME24, ∼ 15 % {\sim}15\% on AMC23). Reducing the conditioning to c = s ∖ think c=s_{\setminus\text{think}} mitigates the drop, though performance still remains below the base model.

#### Reasoning Pattern

Figure 3d illustrates the epistemic token counts of the trained models. GRPO increases 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] , whereas SDPO suppresses it more aggressively, consistent with the observed correlation between epistemic suppression and performance degradation throughout our analysis. Additional analyses on total versus epistemic token changes, as well as the number of uncertainty expressions per reasoning trace, evaluated using an LLM-as-Judge, are provided in Appendices E.4 and E.5 . Furthermore, a comparison of the KL divergence of epistemic tokens with respect to the mean is presented in Appendix F . Example responses generated by GRPO and SDPO are included in Appendix K.2 .

### 5.2 Qwen3-8B (Thinking Mode: ON)

Qwen3-8B is a hybrid reasoning model that can operate in either a thinking mode for deliberate reasoning or a non-thinking mode for direct responses. With thinking mode enabled, Qwen3-8B produces longer responses than DeepSeek-R1-Distill-Qwen-7B, with a higher number of epistemic tokens (see Appendix E.2 for a comparison of epistemic tokens across models).

#### Training Performance

As shown in Figure 4a , 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] decreases under both GRPO and SDPO, with SDPO exhibiting a larger reduction and a correspondingly larger performance drop. Notably, 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] first drops sharply, then increases slightly. Since the teacher policy is fixed as the reference policy, shortening the response by ∼ 900 {\sim}900 tokens reduces the informativeness of c c , i.e., decreases I ⁡ ( y ∗ , c ∣ x ) I(y^{*},c\mid x) . As the context becomes less informative, the model compensates by increasing epistemic verbalization, causing the length to partially recover.

#### OOD Evaluation - AIME24, AMC23

The gap becomes more pronounced on OOD benchmarks (Figures 4b and 4c ): GRPO maintains largely stable performance with gradually decreasing 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] , whereas SDPO falls below the base model, particularly with c = s c=s . Notably, although GRPO and SDPO with c = s ∖ think c=s_{\setminus\text{think}} achieve comparable training performance, their OOD results diverge, especially on the more challenging AIME24, where SDPO with c = s ∖ think c=s_{\setminus\text{think}} shows progressive performance degradation as training proceeds.

#### Reasoning Pattern

Figure 4d reveals that both methods reduce 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] relative to the base model, with SDPO more aggressively so. This suggests that Qwen3-8B originally generates more epistemic verbalization than necessary. While both methods mitigate this redundancy, overly aggressive suppression risks removing epistemic signals that carry useful reasoning information.

### 5.3 Qwen3-8B (Thinking Mode: OFF)

When Qwen3-8B is used without thinking mode, the <think> tag is absent, so we consider only c = s c=s . The model initially produces much shorter responses and exhibits significantly lower performance. GRPO rapidly increases 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] by promoting epistemic verbalization (as shown in Appendix J.1 ), quickly achieving a high training score. In contrast, SDPO reduces 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] and improves much more slowly. Even when the training score slightly increases, as shown in Figure 5b , performance on AIME24 slightly declines ( 0.25 → 0.23 0.25\to 0.23 ), further illustrating the cost of epistemic suppression under self-distillation.

### 5.4 Causal Study: Epistemic Token Suppression during GRPO

To further examine whether the suppression of epistemic verbalization is a causal contributor to the observed performance degradation, we explicitly penalized epistemic tokens during GRPO training and tested whether this intervention induces a similar degradation pattern.

Following the epistemic marker set 𝒯 \mathcal{T} defined in Section 3 , we override the standard GRPO advantage at every position t t whose generated token realizes a marker in 𝒯 \mathcal{T} , setting A ~ t = min ⁡ ( A t , − κ ) \tilde{A}_{t}=\min(A_{t},-\kappa) if y t ∈ 𝒯 y_{t}\in\mathcal{T} and A ~ t = A t \tilde{A}_{t}=A_{t} otherwise, with suppression strength κ = 0.5 \kappa=0.5 . Since A ~ t ≤ − κ \tilde{A}_{t}\leq-\kappa holds even for correct rollouts, the policy gradient consistently reduces 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] throughout training.

As shown in Figure 6 , explicitly penalizing epistemic tokens during GRPO produces shorter responses and lower performance, qualitatively reproducing the degradation pattern observed with SDPO. The degradation is less pronounced than under SDPO, possibly because the predefined epistemic token set does not capture all linguistic realizations of uncertainty or epistemic reasoning, as discussed further in Appendix E.5 . Nevertheless, this result provides causal evidence that suppressing uncertainty expression contributes to the performance decline under SDPO.

### 5.5 Ablation Study: Fixed vs. Moving Target Teacher

In naive on-policy self-distillation, the teacher and student share a continuously updated policy, making the teacher a moving target that can introduce training instability ( Zhao et al., 2026 ; Ye et al., 2026 ) . To mitigate this, SDPO uses an EMA-smoothed teacher (EMA rate: 0.05). However, we find that setting the EMA rate to 0.0 (i.e., fixing the teacher to the initial policy) yields better performance, and thus Section 5 follows this setting.

Figure 7a shows additional comparison results when the teacher is updated during training. As shown, even slow updates (e.g., rate 0.05) lead to a sharper reduction in response length, resulting in larger performance degradation. This can be interpreted as a feedback loop in self-distillation. The model is trained to produce increasingly confident outputs, and when this is used as the teacher, it generates even more confident responses, amplifying the effect over iterations. Further ablations on learning rate, top- k k logits, and the effect of limiting the maximum response length to 4,096 tokens are provided in Appendix D .

## 6 Relationship Between Task Coverage, Epistemic Verbalization and Generalization Ability

Across the off-policy and on-policy settings analyzed above, self-distillation consistently produces more confident responses with reduced 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] . This aligns with the findings of Hübotter et al. (2026) , which reports that SDPO learns to reason concisely: on Science Q&A (Chemistry, Physics, Biology, and Materials Science) ( Feng et al., 2024 ) , tool use ( Tang et al., 2023 ) , and LiveCodeBench v6 ( Jain et al., 2025 ) , SDPO achieves higher accuracy than GRPO while producing substantially shorter outputs with fewer epistemic markers.

In other words, in these domains, self-distillation suppresses epistemic verbalization and improves performance simultaneously. The key question is why the same mechanism leads to performance degradation in our math-focused setup . We hypothesize that the answer lies in differences in task coverage between the training and evaluation distributions.

### 6.1 Comparison of Task Coverage

To test this hypothesis, we compare the dataset characteristics of the settings where SDPO outperformed GRPO against our experimental setup. As shown in Table 3 , the Chemistry dataset, despite its large size, draws from only six main problem types that differ primarily in surface details rather than underlying structure. LiveCodeBench v6 contains diverse problems but only 131 in total, leading to repeated exposure during training with identical train/eval splits. In contrast, DAPO-Math-17k exposes the model to 14,000 distinct problems (78% of the 25,600 samples drawn over 100 steps, due to repeated sampling), spanning a broad, non-overlapping range of problem types, and evaluation is performed on unseen problem types.

### 6.2 Relationship Between Task Coverage and Learning Performance

To further investigate the interplay between task coverage and generalization, we vary the number of training questions | 𝒟 | ∈ { 1 , 8 , 64 , 128 , 512 } |\mathcal{D}|\in\{1,8,64,128,512\} from DAPO-Math-17k and train with both GRPO and SDPO. All experiments use Qwen3-8B (Thinking Mode OFF).

#### Training Logs

As shown in Figure 8 , GRPO and SDPO exhibit distinct training dynamics as | 𝒟 | |\mathcal{D}| varies. When | 𝒟 | ≤ 128 |\mathcal{D}|\leq 128 , SDPO quickly achieves high scores while reducing 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] by up to 8 × 8\times , indicating higher training efficiency on a small task set. However, at | 𝒟 | = 512 |\mathcal{D}|=512 , further reductions in 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] begin to hurt the training score relative to GRPO, whose 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] gradually increases with | 𝒟 | |\mathcal{D}| .

This difference can be interpreted through task coverage . As | 𝒟 | |\mathcal{D}| grows, the model must accommodate a broader range of reasoning patterns. GRPO addresses this by increasing 𝔼 ⁡ [ E ⁡ ( y ) ] \mathbb{E}[E(y)] , allowing the model to express greater uncertainty and adapt its reasoning accordingly. SDPO instead encourages confident, concise responses— effective when task coverage is small but limiting when the problem set becomes larger and more diverse.

#### OOD Evaluation - AIME24, MATH500

The distinction between GRPO and SDPO becomes more pronounced on OOD benchmarks (Figure 9 ). Under GRPO, performance scales consistently with | 𝒟 | |\mathcal{D}| : | 𝒟 | = 1 |\mathcal{D}|=1 converges quickly but soon stops improving, while larger | 𝒟 | |\mathcal{D}| yields progressively higher final scores accompanied by increasing 𝔼 ⁡ [ L ⁡ ( y ) ] \mathbb{E}[L(y)] . Under SDPO, the pattern reverses: smaller | 𝒟 | |\mathcal{D}| leads to more severe OOD degradation. Even at the largest | 𝒟 | |\mathcal{D}| (DAPO setting), SDPO still underperforms the base model. Changes in GRPO and SDPO reasoning patterns across task coverage are provided in Appendix E.3 .

## 7 Conclusion

In this work, we provide an empirical analysis of on-policy self-distillation, motivated by an information-theoretic view of conditioning context richness. Our experiments suggest that the effectiveness of self-distillation is closely tied to how information is provided to the model and how the model expresses uncertainty during reasoning. We observe that self-distillation tends to produce answers with higher confidence and shorter reasoning traces. While this effect enables more compact reasoning and can quickly improve in-domain performance when task coverage is limited, it becomes less effective when task coverage is broad and may even harm OOD performance.

Importantly, the fact that performance can degrade even when a mathematically sound objective function is designed to elicit correct CoT reasoning suggests that the choice of optimization objective alone may not be sufficient for preserving robust reasoning, and that we need to pay closer attention to how training reshapes the model’s reasoning behavior, beyond answer correctness. We hope that our analysis contributes to a deeper understanding of self-distillation and other LLM post-training methods, and provides insights for developing more robust training strategies.

## Acknowledgments

We thank Li Dong, Tianzhu Ye, and Sojeong Rhee for their valuable discussions. This work was supported by Microsoft Research and partly by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(RS-2025-00557589).

## Ethics Statement

This work aims to analyze the causes of reasoning performance degradation during self-distillation of LLMs. All experiments were conducted using publicly available models (Qwen3, DeepSeek-R1-Distill-Qwen, OLMo) and datasets (DAPO-Math-17k, AIME, AMC, MATH500). As this study does not involve any new data collection or human subjects research, concerns related to privacy or human subjects ethics do not apply.

A key finding of this work is that self-distillation can suppress epistemic verbalization, thereby diminishing a model’s ability to express uncertainty during reasoning. This observation carries important implications for the design of LLM post-training methods. The ability to appropriately express uncertainty is directly related to model reliability and safety. Training procedures that indiscriminately suppress such expressions may increase the risk of models producing incorrect answers with unwarranted confidence. We hope that our analysis contributes to the development of more robust and trustworthy reasoning models.

## Reproducibility Statement

We release our implementation and experiment scripts at GitHub (beanie00/self-distillation-analysis) , along with our training logs on Weights & Biases , and trained checkpoints for the main model settings on Hugging Face (beanie00/self-distillation-analysis) .

The repository provides instructions and scripts for data preparation, GRPO/SDPO training, task-coverage experiments, and evaluation. Detailed prompts, training hyperparameters, and evaluation configurations are provided in Appendix C .

## Appendix A Related Works

#### Self-Distillation for LLM Reasoning.