Self-Distillation Enables Continual Learning — part 2/4

Cleaned-text code points [15314, 30962). Bibliography omitted here; source.md includes it.

In a standard IRL setting, π k + 1 ∗ \pi_{k+1}^{*} is unknown. However, our key idea is that the model’s own in-context learning capabilities provide a robust approximation of this optimal policy. We introduce our In-Context Assumption - given a demonstration c c , the model conditioned on c c approximates the optimal next policy. π k + 1 ∗ ​ ( y | x ) ≈ π ⁡ ( y | x , c ) \pi_{k+1}^{*}(y|x)\approx\pi(y|x,c) (4) This substitution posits that the behavioral shift induced by observing a demonstration reflects the expert’s true intent. Substituting this into Eq. (6), we derive an intrinsic reward function: r ⁡ ( y , x , c ) = log ⁡ π ⁡ ( y | x , c ) − log ⁡ π k ​ ( y | x ) r(y,x,c)=\log\pi(y|x,c)-\log\pi_{k}(y|x) (5) We drop β \beta and C C since linear transformations of reward do not affect the optimal policy ( Sutton et al., 1998 ) . While this defines a trajectory-level reward, our model has an autoregressive structure. Therefore, we decompose the reward into token-level rewards r t r_{t} via token-level probabilities. We define the instantaneous reward as the immediate log-probability change: r t ​ ( y t | y < t , x , c ) = log ⁡ π ⁡ ( y t | y < t , x , c ) π k ​ ( y t | y < t , x ) , r_{t}(y_{t}\,|\,y_{<t},x,c)=\log\frac{\pi(y_{t}\,|\,y_{<t},x,c)}{\pi_{k}(y_{t}\,|\,y_{<t},x)}, and indeed for all y y , we have ∑ t r t ​ ( y t | y < t , x , c ) = r ⁡ ( y , x , c ) \sum_{t}r_{t}(y_{t}\,|\,y_{<t},x,c)=r(y,x,c) . Finally, we demonstrate that optimizing the policy with respect to this reward is equivalent to the reverse-KL distillation used in our method. The policy gradient under the current policy π k \pi_{k} is: ∇ θ J ​ ( π k ) = 𝔼 y ∼ π k ​ [ r ⁡ ( y , x , c ) ​ ∇ θ ​ log ⁡ π k ​ ( y | x ) ] \nabla_{\theta}J(\pi_{k})=\mathbb{E}_{y\sim\pi_{k}}\left[r(y,x,c)\nabla_{\theta}\log\pi_{k}(y|x)\right] Substituting our derived reward from Equation 5 : ∇ θ J ​ ( π k ) = 𝔼 y ∼ π k ​ [ log ⁡ π ⁡ ( y | x , c ) π k ​ ( y | x ) ​ ∇ θ ​ log ​ π k ​ ( y | x ) ] \nabla_{\theta}J(\pi_{k})=\mathbb{E}_{y\sim\pi_{k}}\left[\log\frac{\pi(y|x,c)}{\pi_{k}(y|x)}\nabla_{\theta}\log\pi_{k}(y|x)\right] (6)

We observe that this is equivalent in expectation to the gradient of the reverse KL divergence D KL ( π k ( ⋅ | x ) ∥ π ( ⋅ | x , c ) ) D_{\mathrm{KL}}(\pi_{k}(\cdot|x)\|\pi(\cdot|x,c)) in Equation 2 . Thus, our method can be viewed as an on-policy RL algorithm that maximizes rewards inferred by comparing the student’s current behavior to its own “wiser,” demonstration-aware counterpart.

### 3.2 Validating the ICL Assumption

The core hypothesis of SDFT can be seen as the assumption in Equation 4 , which states that a model conditioned on an expert demonstration behaves like the (unknown) optimal policy for that task π k + 1 ∗ ​ ( y | x ) ≈ π ⁡ ( y | x , c ) \pi_{k+1}^{*}(y|x)\approx\pi(y|x,c) and therefore it can be a good teacher. The quality of this approximation depends on 2 conditions: 1. Optimality : The teacher’s expected reward must match that of the unknown optimal policy: 𝔼 y ∼ π ⁡ ( y | x , c ) ​ [ r ⁡ ( y , x ) ] ≈ 𝔼 y ∼ π k + 1 ∗ ​ [ r ⁡ ( y , x ) ] \mathbb{E}_{y\sim\pi(y|x,c)}[r(y,x)]\approx\mathbb{E}_{y\sim\pi_{k+1}^{*}}[r(y,x)] In other words, samples drawn from the demonstration-conditioned policy should achieve near-maximal reward on the task.

2. Minimal Deviation : Due to the trust-region regularization in Equation 3 , the optimal policy π k + 1 ∗ ​ ( y | x ) \pi_{k+1}^{*}(y|x) is the one closest to the current model among all the ones that maximize reward. Thus, we require: D KL ( π ( ⋅ | x , c ) | | π k ( ⋅ | x ) ) ≈ D KL ( π k + 1 ∗ ( ⋅ | x ) | | π k ( ⋅ | x ) ) D_{\mathrm{KL}}\left(\pi(\cdot|x,c)||\pi_{k}(\cdot|x)\right)\approx D_{\mathrm{KL}}\left(\pi_{k+1}^{*}(\cdot|x)||\pi_{k}(\cdot|x)\right) That is, among all policies that achieve optimal reward, the teacher should be close, in the KL sense, to the π k \pi_{k} .

The second requirement, remaining close to the current policy, is crucial for practical viability. If the demonstration-conditioned teacher simply mimicked the example verbatim, it would deviate substantially from the base model, losing the benefits of on-policy learning. What makes the teacher valuable is that it produces new, task-appropriate behavior while remaining anchored to the base model. Moreover, prior work shows that distributions close to the pretrained distribution suffer significantly less catastrophic forgetting and better preserve general capabilities ( Shenfeld et al., 2025 ; Chen et al., 2025 ) .

Empirical Validation. While we cannot verify these conditions theoretically, we evaluate each empirically. We use the Qwen-2.5-7B-Instruct model ( Hui et al., 2024 ) as the base policy and the ToolAlpaca dataset ( Tang et al., 2023 ) . In this benchmark, the model receives a tool-API specification and a user request, and must identify the correct tool call. Without demonstrations, the base model solves only 42% of examples. When provided with the appropriate demonstration c c for each prompt x x , the teacher achieves a 100% success rate. To further test reward proximity, we manually inspected 50 teacher reasoning traces. In all cases, not only were the final tool calls correct, but the intermediate chain-of-thought was valid and semantically grounded. This suggests that the teacher is reconstructing a correct reasoning process rather than merely copying the expert output. These observations provide evidence for the first requirement, that the demonstration-conditioned model behaves as an optimal policy.

To verify the second requirement, we measure the KL divergence to the base policy D KL ( π ∥ π 0 ) D_{\mathrm{KL}}(\pi\|\pi_{0}) as a proxy for the distance to the policy during training π k \pi_{k} . We compare this divergence for both the SFT model trained on demonstrations and the demonstration-conditioned teacher. As shown in Figure 2 (right panel), the SFT model deviates substantially from the base model (1.26 nats), whereas the teacher remains significantly closer (0.68 nats)—nearly half the divergence. This validates that the teacher produces high-quality outputs while maintaining proximity to the base policy, precisely the balance required by the trust-region formulation.

## 4 Experiments

### 4.1 Experimental Setting

We evaluate our method in two settings that reflect common forms of post-training adaptation: Skill Learning and Knowledge Acquisition . These correspond to improving performance on a new task, and integrating novel factual information into a pretrained model.

In Skill Learning , we study whether a pretrained LLM with broad capabilities can acquire a new, narrowly defined skill without degrading its existing abilities. We choose to experiment with tasks the models had not been explicitly fine-tuned on (unlike Math or Coding) to show the benefits of continual learning. Therefore, we test our method on three domains:

• Science Q&A : Undergraduate-level scientific reasoning, using the Chemistry L-3 subset of SciKnowEval ( Feng et al., 2024 ) .

• Tool Use : Mapping a tool-API specification and user request to the correct tool call, using ToolAlpaca ( Tang et al., 2023 ) .

• Medical : Clinical reasoning questions, with training data from stage 1 of the HuatuoGPT-o1 pipeline and evaluation from stage 2 ( Chen et al., 2024 ) .

In Knowledge Acquisition , the objective is different: the model must integrate genuinely new factual content not present in its pretraining data. We construct a corpus of Wikipedia articles describing natural disasters that occurred in 2025 (after the training knowledge cutoff), totaling approximately 200K tokens. Following Mecklenburg et al. (2024) , we generate question–answer pairs about these articles, yielding an SFT dataset roughly 5× larger than the source corpus. These questions probe factual content such as “which regions were affected by the 2025 Myanmar earthquake?”.

This setting tests whether the model can absorb newly injected knowledge rather than merely improving skills it already has.

#### Evaluation.

For each task, we evaluate along two primary axes: • In-Distribution Accuracy: Accuracy on held-out test data for the newly introduced task. For Knowledge Acquisition, we use two variants: (1) All details correct (Strict Accuracy). (2) The answer contains correct information and no incorrect statements (Lenient Accuracy).

• Previous Capabilities: Performance on a suite of established benchmarks that probe general reasoning and world knowledge: HellaSwag ( Zellers et al., 2019 ) , TruthfulQA ( Lin et al., 2021 ) , MMLU ( Hendrycks et al., 2020 ) , IFEval ( Zhou et al., 2023 ) , Winogrande ( Sakaguchi et al., 2021 ) , and HumanEval ( Chen et al., 2021 ) . We report the average performance across these datasets as a measure of catastrophic forgetting.

For the Knowledge Acquisition setting, we include a third metric: • Out-of-Distribution Accuracy: “Indirect” questions whose answers depend on the injected knowledge but do not directly reference it (e.g., “Which countries required international humanitarian aid in 2025?”). This measures whether the new information has been properly integrated into the model’s internal memory rather than memorized in a narrow form.

#### Baselines.

In the Skill Learning setting, we compare our method to standard SFT and to DFT ( Wu et al., 2025b ) , which uses importance sampling to treat the offline dataset as on-policy samples. We also include the recently proposed ” Re-invocation ” method ( Lu and Lab, 2025 ) , which performs additional on-policy distillation from the base policy on general-purpose prompts after SFT to restore prior capabilities.

In the Knowledge Acquisition setting, we compare our method to CPT (Continual Pre-Training), which trains directly on the text corpus using next token prediction loss and SFT , which trains on the question-answer pairs. In addition, we also compare with pure ICL methods. Because the full corpus exceeds the model’s context window, we evaluate RAG with an oracle retriever that always provides the correct article for each question.

Unless otherwise noted, all experiments were performed on the Qwen2.5-7B-Instruct model. For each baseline, we perform a hyperparameter sweep and report results for the model achieving the highest validation performance on the target task. Full datasets, hyperparameters, and training protocols are provided in Appendix B .

### 4.2 On-policy learning leads to better generalization

Prior work has shown that on policy learning achieves better in-distribution performance than SFT ( Ross et al., 2011 ) , as well as superior out-of-distribution generalization ( Chu et al., 2025 ) . We investigate whether these advantages also arise in our on-policy distillation framework. For that, we measure performance on test set on all our training tasks, as well as OOD generalization in the Knowledge Acquisition setting.

#### Results.

Results for Skill Learning , as shown in Figure 4 , indicate that our method achieves higher new-task accuracy than SFT, which represents better in-distribution generalization. We attribute these gains to the fact that off-policy learning trains only on expert-induced trajectories; errors at test-time can push the policy into unseen states, causing compounding errors. On-policy imitation learning avoids this mismatch by training on the state distribution induced by the learned policy itself ( Ross et al., 2011 ) .

The results for Knowledge Acquisition appear in Table 1 . Since the new knowledge was not included in the base model’s training, it cannot answer any of the questions correctly. Consistent with earlier observations ( Mecklenburg et al., 2024 ) , continual pretraining performs poorly. SFT on questions improves performance substantially but still lags behind our SDFT. On strict accuracy, it reaches 80% while our on-policy method achieves 89% and nearly closes the gap to the oracle RAG model. The advantage becomes even clearer on out-of-distribution questions, where our method achieves close to perfect accuracy, while SFT’s performance remains low. This disparity underscores a key limitation of SFT: it teaches the model to reproduce specific answers but does not reliably incorporate the underlying facts into the model’s broader knowledge base.

Finally, with on-policy RL there is a concern for superficial improvements through entropy reduction rather than acquisition of new behaviors ( Yue et al., 2025 ; Wu et al., 2025a ) . To ensure our gains are not merely due to distributional sharpening, we evaluate pass@ k k for k k up to 128 in the Skill Learning Setting. As shown in Figure 5 (right), the performance gains over both the base model and SFT persist uniformly across all k k . This indicates that the improvements reflect genuine skill acquisition rather than entropy collapse.

### 4.3 Learning without forgetting

A central claim of SDFT is that, due to its on-policy nature, it can acquire new skills while mitigating catastrophic forgetting. To test this, we perform the following experiments: 1. Single Task Learning. A convenient case study for continual learning is fine-tuning a model on a single task. Using the Skill Learning setting, we compare the broad capabilities of our models before and after training on each task.

2. Multi-Task Continual Learning. We investigate a more complex continual learning experiment in which a single model is trained sequentially on each task. The goal here is to measure catastrophic forgetting over longer training and to see whether the model retains the capabilities it learned at each stage of training.

.

#### Results.

The results for single-task training, presented in Figure 4 , show that our method is the only approach to improve performance on the new task without significant degradation in prior capabilities. In contrast, standard SFT produces substantial catastrophic forgetting across all evaluated benchmarks. Augmenting SFT with the “re-invoke” procedure partially restores lost abilities but does not recover the base model’s full capabilities. DFT, which performs approximate on-policy updates, exhibits reduced forgetting relative to SFT but still results in noticeable degradation. For the breakdown of the score over prior tasks, see Table 5 .

We now turn to the more challenging setting of long-horizon continual learning, where a single model is trained sequentially on all three skills. Figure 3 shows that SDFT enables stable accumulation of skills over time. As training progresses, the model improves on each newly introduced task while maintaining performance on previously learned ones. In contrast, SFT exhibits severe interference—performance on earlier skills rapidly degrades once training shifts to a new task, resulting in oscillatory behavior rather than cumulative learning. These results demonstrate that SDFT supports true continual learning, allowing a single model to incrementally acquire multiple skills without catastrophic forgetting.

### 4.4 Effect of model size

Our method relies fundamentally on the model’s in-context learning ability. The teacher signal comes from the model conditioned on demonstrations, and the quality of that signal depends on how well the model can interpret and extrapolate from those examples. This suggests that larger models, whose in-context learning abilities are known to improve with scale ( Brown et al., 2020 ) , should yield stronger teacher policies and therefore better SDFT updates. To test this hypothesis, we conduct a scaling experiment using several sizes from the Qwen 2.5 family ( Hui et al., 2024 ) , evaluating each on the Science Q&A task.