Self-Distillation Enables Continual Learning — part 4/4

Cleaned-text code points [46658, 54154). Bibliography omitted here; source.md includes it.

We analyze which components of the teacher context are essential for the effectiveness of our method in the Knowledge Acquisition setting. Recent self-distillation approaches for knowledge injection perform offline distillation using only the raw corpus as context ( Eyuboglu et al., 2025 ; Kujanpää et al., 2025 ) . In contrast, our approach differs along two dimensions: (i) the teacher is conditioned not only on the source text but also on a worked answer, and (ii) distillation is performed on-policy.

In this subsection, we isolate the effect of the teacher context while holding the on-policy training procedure fixed. Specifically, we compare three variants: conditioning the teacher on only the article text , only the answer , and the full text-plus-answer context. A direct comparison to offline distillation methods is deferred to Section 4.6 .

#### Results.

The results are shown in Figure 7 . Conditioning the teacher on the full text-plus-answer context yields the strongest performance, achieving 89% strict accuracy. Using only the article text substantially underperforms, consistent with prior findings that text-only distillation provides a weak and noisy supervisory signal. Conditioning on text alone improves performance over answers-only context but still falls short of the full context. These results suggest that answer-conditioned context plays a critical role by providing stronger guidance for the student policy.

### A.3 Choice of teacher model

We ablate the choice of teacher policy used for distillation. While our framework does not require an external teacher, the stability of training depends critically on how the teacher is instantiated.

Using the frozen base model as the teacher yields stable training but consistently underperforms, as the teacher fails to reflect improvements acquired during learning. At the other extreme, using the student model itself as the teacher leads to severe instabilities. In this setting, small stochastic fluctuations in token-level probability updates can be rapidly amplified through the on-policy feedback loop, causing training to diverge. We find that maintaining an exponential moving average (EMA) of the student parameters provides an effective compromise. As shown in Figure 8 , the EMA teacher tracks the student’s progress while smoothing high-variance updates, resulting in both stable training and superior final performance.

## Appendix B Training and Evaluation details

### B.1 Training Details

All experiments were conducted using the Hugging Face TRL library. Each experiment was conducted on a single NVIDIA H200 GPU. We perform full fine-tuning of the entire model’s parameters. For each method, we performed a hyperparameter sweep over learning rates, batch sizes, and training epochs. We report test results for the model checkpoint that achieved the best validation performance on the target task.

Tables 3 and 4 present the full hyperparameter search spaces and final selected values for the Skill Learning and Knowledge Acquisition settings, respectively. Across all tasks, we found that SDFT benefits from training for multiple epochs—typically 2 epochs for Skill Learning tasks and 4 epochs for Knowledge Acquisition. In contrast, SFT tends to overfit rapidly and showed no performance gains beyond a single epoch in most cases.

For SDFT, The teacher context was constructed using the prompt template shown in Section 3 . We employed the analytic per-token KL gradient estimator (see Appendix A.1 ) with a single on-policy rollout per training example.

### B.2 Evaluation Details

Sampling Strategy. For accuracy metrics, we used greedy decoding (temperature = 0). For pass@k experiments, we used temperature = 1.0 with nucleus sampling (top-p = 0.95).

Statistical Reporting. Unless mentioned otherwise, all experiments were run over 3 random seeds. We report mean performance and 95% confidence intervals across seeds.

Prior Capabilities Evaluation. We assessed performance on general capabilities using the suite of benchmarks described in Section 4.1 : HellaSwag ( Zellers et al., 2019 ) , TruthfulQA ( Lin et al., 2021 ) , MMLU ( Hendrycks et al., 2020 ) , IFEval ( Zhou et al., 2023 ) , Winogrande ( Sakaguchi et al., 2021 ) , and HumanEval ( Chen et al., 2021 ) . All benchmark evaluations were conducted using the Language Model Evaluation Harness ( Gao et al., 2024 ) .

### B.3 Dataset Details

#### Science Q&A.

We used the Chemistry L-3 subset from SciKnowEval ( Feng et al., 2024 ) , splitting the data into approximately 75% train, 5% validation, and 20% test. To construct expert demonstrations, we queried GPT-4o, sampling up to 8 responses per prompt and retaining a single response that matched the correct final answer. This procedure yielded valid demonstrations for 100% of training examples. Since this is a multiple-choice dataset, accuracy was computed by exact match between the model’s final answer choice and the ground truth.

#### Tool Use.

We used the ToolAlpaca dataset ( Tang et al., 2023 ) , following the original train-test split provided by the authors. Expert demonstrations were included in the original dataset. Accuracy was evaluated using regex matching against the ground-truth API call, accounting for variations in argument ordering.

#### Medical.

We built upon the HuatuoGPT-o1 dataset ( Chen et al., 2024 ) , which provides both an SFT training set and a collection of problems with only final answer (used in the original paper for RL training). For training, we used only the English-language questions, yielding approximately 20,000 examples. For evaluation, we randomly sampled 1,000 verifiable questions from the verifiable problem set. Since these are open-ended clinical reasoning questions, we used GPT-5-mini as an automated evaluator with the following prompt:

#### Knowledge Acquisition.

We constructed a corpus of Wikipedia articles describing natural disasters that occurred in 2025 (after the model’s knowledge cutoff), including:

2025 Myanmar earthquake, 2025 Kamchatka earthquake, 2025 Uttarakhand flash flood, Typhoon Kalmaegi, Tropical Storm Wipha, Cyclonic Ditwah, Hurricane Melissa, Kentwood Carson Tornado, July 2025 Central Texas floods.

Following Mecklenburg et al. (2024) , we used GPT-5 to generate question-answer pairs from these articles, using the following prompt:

We also manually verify that the same question wasn’t generated more than once. For evaluation, we used GPT-5-mini as an automated evaluator with the following prompt:

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .