Self-Distillation Enables Continual Learning — part 1/4
Cleaned-text code points [0, 15314). Bibliography omitted here; source.md includes it.
# Self-Distillation Enables Continual Learning
###### Abstract
Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce forgetting, it requires explicit reward functions that are often unavailable. Learning from expert demonstrations, the primary alternative, is dominated by supervised fine-tuning (SFT), which is inherently off-policy. We introduce Self-Distillation Fine-Tuning (SDFT) , a simple method that enables on-policy learning directly from demonstrations. SDFT leverages in-context learning by using a demonstration-conditioned model as its own teacher, generating on-policy training signals that preserve prior capabilities while acquiring new skills. Across skill learning and knowledge acquisition tasks, SDFT consistently outperforms SFT, achieving higher new-task accuracy while substantially reducing catastrophic forgetting. In sequential learning experiments, SDFT enables a single model to accumulate multiple skills over time without performance regression, establishing on-policy distillation as a practical path to continual learning from demonstrations. Code and Datasets are available at http://idanshenfeld.com/SDFT .
## 1 Introduction
Foundation models have achieved remarkable success in recent years, powering AI applications across language, vision, robotics, and beyond. However, despite their impressive capabilities, today’s AI systems remain static after deployment. While they can adapt their behavior at inference time through mechanisms such as retrieval or prompting, they do not update their parameters to acquire new skills, internalize new knowledge, or improve from experience. To enable the next generation of foundation models, we must solve the problem of continual learning: enabling AI systems to keep learning and improving over time, similar to how humans accumulate knowledge and refine skills throughout their lives ( Hassabis et al., 2017 ; De Lange et al., 2021 ) .
A growing body of recent work has highlighted the importance of on-policy learning for continual learning. When models learn from data generated by their current policy, they exhibit substantially reduced catastrophic forgetting compared to off-policy alternatives ( Shenfeld et al., 2025 ; Chen et al., 2025 ) . To date, most successful on-policy approaches have been developed in the context of reinforcement learning (RL), where feedback is provided through an explicit reward function. However, in many real-world settings such rewards are unavailable or difficult to specify. Instead, learning typically proceeds from datasets of expert demonstrations. The dominant paradigm in this regime is supervised fine-tuning (SFT), which trains the model to imitate expert actions under a fixed, offline data distribution. While simple and scalable, SFT is inherently off-policy, and prior work has shown that sequential SFT can lead to poor generalization and severe catastrophic forgetting when models are adapted to new tasks or domains ( Kirkpatrick et al., 2017 ; Li and Hoiem, 2017 ) . This tension raises a fundamental challenge for continual learning: how can we obtain the benefits of on-policy learning when only demonstrations are available?
The challenges of off-policy learning can, in principle, be overcome by first learning a reward function from demonstrations (i.e., Inverse Reinforcement Learning or IRL), and then performing on-policy RL ( Ng et al., 2000 ; Abbeel and Ng, 2004 ) . While IRL is conceptually elegant, effectively recovering rewards typically requires strong priors over the reward structure, which has limited its practical adoption to settings where such assumptions are justified, such as RLHF ( Peng et al., 2018 ; Stiennon et al., 2020 ) .
Rather than inferring an explicit reward function, we propose Self-Distillation Fine-Tuning (SDFT), an on-policy distillation ( Ross et al., 2011 ; Agarwal et al., 2024 ) framework for learning directly from demonstrations. SDFT relies on the observation that large pretrained models exhibit strong in-context learning—the ability to adapt their behavior when conditioned on examples, without parameter updates ( Brown et al., 2020 ) . We exploit this property by using the same model in two roles: a teacher, conditioned on both the task input and an expert demonstration, and a student, conditioned only on the task input. Training distills the teacher’s predictions into the student on trajectories generated by the student itself, yielding on-policy updates that incorporate information from demonstrations without explicit reward inference or offline imitation.
We evaluate SDFT in two continual learning settings: skill learning , where demonstrations are used to improve performance on a task, and knowledge acquisition , where new information must be incorporated into the model. Across both settings, SDFT provides stable on-policy updates that enable learning while substantially reducing catastrophic forgetting compared to supervised learning. Consistent with prior work on on-policy learning ( Ross et al., 2011 ; Chu et al., 2025 ) , SDFT also improves generalization both in-distribution and out-of-distribution, making it beneficial even in settings where retaining prior capabilities is not the primary objective. In a sequential learning experiment involving three distinct skills, SDFT enables a single model to acquire each skill in turn while preserving performance on previously learned skills as well as on unrelated, pre-existing capabilities — demonstrating that continual learning from demonstrations is possible.
## 2 Related Work
#### Off-policy versus On-policy Learning.
A long line of work highlights the advantages of on-policy learning, i.e., training on trajectories induced by the model itself, over off-policy learning. The seminal result of Ross et al. (2011) shows that off-policy imitation learning suffers from compounding errors at inference time, as the learned policy drifts away from the states covered in the demonstrations, errors accumulate rapidly, a failure mode that on-policy algorithms avoid by continually training under their own state distribution. More recent empirical studies reinforce this distinction. Models fine-tuned with on-policy RL have been shown to generalize better beyond the training distribution ( Agarwal et al., 2024 ; Han et al., 2025 ; Chu et al., 2025 ; Li et al., 2025 ) and transfer more effectively to related tasks ( Huan et al., 2025 ) than models trained purely off-policy. In continual learning settings, on-policy updates also reduce catastrophic forgetting when adapting to new tasks ( Shenfeld et al., 2025 ; Lai et al., 2025 ) . These findings collectively motivate our goal - to enable on-policy learning from demonstrations, thereby retaining the benefits of on-policy RL while avoiding the need for explicit reward engineering.
#### Inverse Reinforcement Learning.
IRL ( Ng et al., 2000 ) provides a classical solution to the problem faced in many RL settings: the agent must learn a policy when no explicit reward function is available, only demonstrations. Rather than cloning the expert’s actions, IRL seeks to infer the underlying reward for which those demonstrations would be optimal. This perspective avoids the issues of off-policy imitation learning, since the inferred reward can support on-policy updates ( Xu et al., 2020 ) . While this idea has deep theoretical appeal, traditional IRL methods are known not to scale well ( Lazzati et al., 2024 ; Arora and Doshi, 2021 ) .
A common thread across all successful IRL formulations is that they rely on strong structural assumptions to make the reward identifiable. Maximum-entropy IRL assumes that experts follow a soft-optimal Boltzmann policy ( Ziebart et al., 2008 ; Wulfmeier et al., 2015 ) ; adversarial IRL methods ( Ho and Ermon, 2016 ) assume that expert and learner trajectories can be distinguished by a classifier; and preference-based IRL methods, such as RLHF ( Ziegler et al., 2019 ; Ouyang et al., 2022 ) , assume access to pairs of positive–negative demonstrations. These priors are essential—without them, IRL is either ill-posed or too expensive to be practical. In our approach, rather than imposing an explicit learning reward function, we leverage the model’s in-context learning to extract an on-policy learning signal.
#### Context Distillation.
Our method also relates to the growing line of work on context distillation, in which a model conditioned on additional information acts as a teacher for a version of itself without that information ( Bai et al., 2022 ; Snell et al., 2022 ) . Prior approaches typically rely on offline distillation from static contexts, such as few-shot examples or behavioral guidelines, and supervise the student on trajectories drawn from the teacher’s distribution. Our algorithm differs in two important ways. First, the distillation is on-policy: the student is trained under its own induced trajectory distribution, allowing the teacher to correct errors as they arise ( Ross et al., 2011 ; Agarwal et al., 2024 ) . Second, the context provided to the teacher is not a fixed prompt prefix but a specific demonstration chosen for each query . This dynamic, instance-wise conditioning enables the teacher to express fine-grained task intent, rather than a single global behavioral prior. Together, these differences allow context distillation to function not merely as a form of prompt compression but as an IRL-like mechanism that extracts and transfers the underlying reasoning induced by demonstrations.
#### Self-Distillation.
Concurrently and independently of our work, two other papers have suggested the Self-Distillation algorithm Zhao et al. (2026) ; Hübotter et al. (2026) . We see these works as complementary to our own, demonstrating the potential in this algorithm for training LLMs.
## 3 Self-Distillation Fine-Tuning
Our approach builds on the framework of student-teacher distillation, where a student model is trained to match the behavior of a teacher model by minimizing the divergence between their output distributions. Traditionally, distillation uses separate models, typically a larger, more capable teacher and a smaller student ( Hinton et al., 2015 ) . Our key innovation is that we can use the same model as both teacher and student by exploiting its in-context learning abilities. Specifically, given a foundation model with policy π \pi , we construct the teacher by conditioning it on expert demonstrations: π ( ⋅ | x , c ) \pi(\cdot|x,c) , where x x is the task prompt and c c is a demonstration. The student is simply the base model without this conditioning π θ ( ⋅ | x ) \pi_{\theta}(\cdot|x) .
To construct the teacher for a given prompt x x , we condition the model on both the prompt and a demonstration using the following simple prompt:
<Question> This is an example for a response to the question: <Demonstration> Now answer with a response of your own, including the thinking process:
We find that this prompt is sufficient to prevent the policy from outputting c c verbatim and instead elicits a response that reflects the model’s understanding of the intent behind the demonstration, leveraging its in-context learning capabilities. See subsection 3.2 for further analysis of the conditioned policy’s outputs.
As mentioned before, we hypothesize that on-policy learning is necessary for continual learning; therefore, we train the student using on-policy distillation from the teacher * * * Even without the continual learning constraints, learning from a teacher with privileged information requires on-policy learning ( Swamy et al., 2022 ) . . For every prompt x x , our algorithm, SDFT, samples responses from the student policy y ∼ π θ ( ⋅ | x ) y\sim\pi_{\theta}(\cdot|x) and minimizes the reverse Kullback-Leibler (KL) divergence between the student and the teacher distributions: ℒ ( θ ) = D K L ( π θ ( ⋅ | x ) ∥ π ( ⋅ | x , c ) ) = 𝔼 y ∼ π θ ( y | x ) [ log π θ ( y | x ) π ( y | x , c ) ] \mathcal{L}(\theta)=D_{KL}\left(\pi_{\theta}(\cdot|x)\parallel\pi(\cdot|x,c)\right)=\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[\log\frac{\pi_{\theta}(y|x)}{\pi(y|x,c)}\right] (1) Leveraging the autoregressive nature of the model, we decompose this objective into a token-level loss (see Tang and Munos (2025) for a derivation) and take the gradient with respect to the student parameters θ \theta while treating the teacher distribution as fixed. This results in the following gradient estimator: ∇ θ ℒ ( θ ) = 𝔼 y ∼ π θ [ ∑ t ∑ y t ∈ 𝒱 π θ ( y t | y < t , x ) log π θ ( y t | y < t , x ) π ( y t | y < t , x , c ) ∇ θ log π θ ( y t | y < t , x ) ] \nabla_{\theta}\mathcal{L}(\theta)=\mathbb{E}_{y\sim\pi_{\theta}}\left[\sum_{t}\sum_{y_{t}\in\mathcal{V}}\pi_{\theta}(y_{t}|y_{<t},x)\log\frac{\pi_{\theta}(y_{t}|y_{<t},x)}{\pi(y_{t}|y_{<t},x,c)}\nabla_{\theta}\log\pi_{\theta}(y_{t}|y_{<t},x)\right] (2) where 𝒱 \mathcal{V} is the token vocabulary.
Practical Implementation. There are a few critical components to the practical implementation of SDFT. The first is the parameterization of the teacher model used to compute the likelihood ratios. Appendix A.3 includes an ablation regarding this design choice. Unless mentioned otherwise, we use an exponential moving average (EMA) of the student parameters for the teacher. Another one is the logit-level loss taken between the teacher and the students Agarwal et al. (2024) . Although the theory points to Reverse KL as a suitable loss, we found in practice that Forward KL yields the best performance. A full detailed description of our algorithm can be found in Algorithm 1 in the appendix.
### 3.1 Self-Distillation as Inverse RL
Although we present our algorithm from a student-teacher distillation perspective, it can also be interpreted in the IRL framework, where it maximizes an implicit reward function. In the following section, we formally show that our self-distillation objective is mathematically equivalent to maximizing an implicit reward function defined by the expert demonstrations and the model’s in-context learning capabilities.
We begin with the standard formulation of trust-region-regularized reinforcement learning Schulman et al. (2015) , where the policy update in step k + 1 k+1 is constrained to stay close to the current policy π k \pi_{k} : π k + 1 = max π 𝔼 y ∼ π [ r ( y , x ) ] − β D KL ( π ( ⋅ | x ) ∥ π k ( ⋅ | x ) ) \begin{split}\pi_{k+1}=\max_{\pi}\mathbb{E}_{y\sim\pi}[r(y,x)]-\beta D_{\mathrm{KL}}(\pi(\cdot|x)\|\pi_{k}(\cdot|x))\end{split} (3)
For this objective, the optimal policy π k + 1 ∗ \pi_{k+1}^{*} takes the known closed-form expression of a tilted distribution ( Korbak et al., 2022 ; Rafailov et al., 2023 ) : π k + 1 ∗ ( y | x ) ∝ π k ( y | x ) exp ( 1 β r ( y , x ) ) \pi_{k+1}^{*}(y|x)\propto\pi_{k}(y|x)\,\exp(\frac{1}{\beta}\,r(y,x)) Rearranging this equation allows us to express the underlying reward as a function of the divergence between the optimal and previous policies: r ( y , x ) = β [ log π k + 1 ∗ ( y | x ) − log π k ( y | x ) ] + C r(y,x)=\beta\left[\log\pi_{k+1}^{*}(y|x)-\log\pi_{k}(y|x)\right]+C