To Retain or to Adapt? Generalizing Continual Learning — part 1/15
Cleaned-text code points [0, 15987). Bibliography omitted here; source.md includes it.
# To Retain or to Adapt? Generalizing Continual Learning
###### Abstract
The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting. This objective rests on a pervasive, often unstated assumption: that a lifelong learner should approximate the Joint-Task Learning (JTL) solution and retain all previously acquired knowledge. We challenge this retention-centered premise, arguing that in non-stationary environments prioritizing retention can impede real-time adaptation. Shifting the focus to the Average Lifelong Error (ALE), we formalize CL as an online optimization problem governed by the interaction between environmental and learning dynamics. We introduce Transfer Efficiency as a quantitative measure of the tension between Instability , the bias inherited from conflicting past experience, and Transient Error , the optimization cost of learning new tasks from scratch. Under mild convergence conditions, holding across linear and neural network models, this decomposition yields a Critical Task Duration : a closed-form threshold beyond which historical knowledge transitions from a warm-start advantage to an optimization liability whenever retention induces a positive stationary bias. We validate these theoretical predictions on continual image classification and reinforcement learning benchmarks. Finally, by connecting continual learning to the online learning framework of predictable sequences, we show that JTL is only one instance of a broader family of objectives, and we propose a new general class of continual learning algorithms, which we call Predictive Continual Learning . Predictive CL algorithms optimize expected future performance under an explicit, dynamically updated model of future tasks. As a proof of concept, we analyze a Window algorithm that interpolates between JTL and Independent-Task Learning (ITL), outperforming both under controlled distributional drift.
### 1 Introduction
Real-world environments are inherently nonstationary, evolving over time and requiring systems that can adapt continually and do so efficiently. Continual learning (CL) ( Ring, 1994 ; Thrun and Mitchell, 1995 ; Parisi et al., 2019 ; Hadsell et al., 2020 , e.g.) —also known as lifelong learning—addresses this challenge by developing algorithms that can acquire, retain, and update knowledge over extended sequences of experience. Unlike conventional machine learning, where the data distribution is fixed, continual learning operates under distributional drift, where performance must be maintained relative to a moving target.
Progress in continual learning has historically been guided and measured by a diverse list of metrics, which reflect the many desiderata of an adaptive agent, namely: forward transfer , minimal forgetting , preservation of plasticity , and computational efficiency ( Schwarz et al., 2018 ; Hadsell et al., 2020 ; Mundt et al., 2023 ) . This distinction of separate metrics has inevitably led to the development of methods focused only on one or a subset of those, at the expense of the other desiderata ( Wołczyk et al., 2021 ; Dohare et al., 2024 ) . Recently, Kumar et al. (2025) formally proved that maximizing real-time performance naturally leads to a trade-off between these requirements—one that is driven dynamically by the environment rather than fixed a priori. Their proposed Average Lifelong Error (ALE) objective expresses continual learning as the minimization of the online expected error in a dynamic environment, subject to resource constraints. Importantly, this perspective frames forgetting as a functional necessity to adapt within finite computations. As Kumar et al. (2025) shows, a learner with limited compute must selectively retain only the information predictive of the future , and systematically discard the rest to maintain adaptive capacity.
In the real world, it is difficult to know in advance which piece of past information will be useful in the future. Consequently, any lifelong learning algorithm must rely on assumptions or an explicit model about the future. We argue that algorithms designed to preserve performance on all previously encountered tasks, including most continual learning methods aimed at minimizing catastrophic forgetting, implicitly encode such a model. Specifically, the Joint-Task Learning (JTL) objective—the average loss over all tasks observed up to the current time— corresponds to a particular assumption about future structure: the future tasks are drawn from the same distribution as past tasks. Equivalently, JTL is the special case of ALE objective under an ergodic environment, where there statistical properties of the task stream remain stationary over time.
While this assumption can be reasonable in many scenarios, e.g. when the agent gradually explores a larger but fundamentally stable world, it breaks down when the environment changes irreversibly and past data become progressively less predictive. In such settings, optimizing the JTL objective can place excessive weight on outdated information, hindering adaptation and yielding suboptimal lifelong performance.
To make this perspective concrete, we contrast the JTL formulation with the Independent-Task Learning (ITL) objective, which optimizes only on the current task and retain no information from past tasks. This comparison highlights that neither objective is universally optimal. In rapidly changing environments, the ITL objective can achieve lower lifelong error by avoiding the implicit bias toward past distributions. Conversely, when past experience remains predictive of the future, the JTL objective benefits from this bias to accelerate learning and improve generalization. By analyzing these two extremes in both controlled quadratic models and deep neural networks, we characterize the conditions under which memory becomes a liability for learning. This analysis introduces three key concepts: Transfer Efficiency , which quantifies this trade-off; Instability , which measures the cost of bias inherited from previous tasks; and Transient Error , which captures the cost of learning a new task from scratch.
Our framework connects several adjacent literatures—online learning, multi-task learning, and learning to learn— revealing a shared theoretical foundation. It offers analytical tools to challenge and extend existing theories. In particular, the connection to the online learning literature allows us to interpret the JTL and ITL objectives as part of a family of algorithms based on the concept of predictable sequences ( Rakhlin and Sridharan, 2013 ) . In light of this connection, we propose a generalization of continual learning algorithms beyond the restrictive assumption of ergodicity. Rather than optimizing for perfect retention or complete adaptation, the objective becomes minimizing the expected future error under a model of future tasks, where the predictive model itself may be learned and updated online. We refer to algorithms that optimize this generalized objective as Predictive Continual Learning (Predictive CL) . This formulation transforms the space between JTL and ITL from a gap between two extremes into a principled design space, enabling continual learning algorithms to adapt their retention strategy to the structure of the environment rather than to a fixed, a priori objective.
###### Contributions.
We summarize the main contributions of this paper as follows: 1. We recast continual learning as an online optimization problem centered on Average Lifelong Error (ALE), showing that the widely adopted Joint-Task Learning (JTL) objective is not a universally optimal target in non-stationary environments, but rather corresponds to a particular assumption about future task structure.
2. We formalize CL as the interaction between environment dynamics and learning dynamics, revealing that the utility of memory depends jointly on task distributions, optimization dynamics, and model properties.
3. We introduce three analytical quantities: Transfer Efficiency , Instability , and Transient Error that decompose the benefit of transferring past knowledge. This decomposition yields a Critical Task Duration under which retained knowledge transitions from accelerating learning to hindering adaptation.
4. We derive closed-form expressions for these quantities in quadratic objectives, showing that the environment Growth Rate determines the asymptotic benefits of long term memory. We further demonstrate that the same mechanisms emerge empirically in deep linear models, continual image classification, and reinforcement learning benchmarks.
5. We connect continual learning to the online learning literature, unifying JTL and ITL within a broader family of objectives. Building on this perspective, we propose Predictive CL , a generalized framework in which algorithms optimize expected future performance under an explicit predictive model of future tasks. As a proof of concept, we analyze a Window algorithm that interpolates between ITL and JTL and outperforms both under controlled distributional drift.
### 2 Background & Setup
#### 2.1 A General Objective for Continual Learning
We begin by recalling the continual learning objective introduced by Kumar et al. (2025) , and instantiate it in the supervised learning setting. 1 1 1 We adopt a slightly different notation from the original formulation to better align with the continual learning literature. We review the main concepts here and provide an in-depth exposition in the supplementary material ( Section A.1 ).
###### General setting.
During its lifetime, an agent accumulates a history H T = { ( a 0 , o 1 ) , … , ( a T − 1 , o T ) } H_{T}=\{(a_{0},o_{1}),\dots,(a_{T-1},o_{T})\} of action-observation pairs in A × O A\times O . At each time step n n the agent samples an action a n ∼ π n ( ⋅ ∣ H n ) a_{n}\sim\pi_{n}(\cdot\mid H_{n}) from a policy π n \pi_{n} conditioned on the history. The environment then produces a new observation o n + 1 ∼ 𝒫 ( ⋅ ∣ H n , a n ) o_{n+1}\sim\mathcal{P}(\cdot\mid H_{n},a_{n}) , and the learner uses the new experience ( a n , o n + 1 ) (a_{n},o_{n+1}) together with the pre-existing history H n H_{n} to produce a new policy estimate π n + 1 \pi_{n+1} .
In this work we model the learning step as a stochastic process guided by an algorithm 𝒜 \mathcal{A} , which updates the policy according to π n ∼ 𝒜 ( ⋅ | H n ) \pi_{n}\sim\mathcal{A}(\cdot|H_{n}) . Therefore, at each time step there are two interacting stochastic processes (depicted in Figure 2 ): the learning process , by which π n \pi_{n} and then a n a_{n} are sampled, and the environment process , by which o n + 1 o_{n+1} is sampled. Together, these coupled processes determine the agent’s lifetime trajectory.
The supervised learning problem is a special case of this general formulation in which the environment process is independent of the learning process —though the converse does not hold, since the learner’s policy still depends on past observations. Concretely, the new observation o n + 1 o_{n+1} is independent of the action a n a_{n} , so o n + 1 ∼ 𝒫 ( ⋅ ∣ H n ) o_{n+1}\sim\mathcal{P}(\cdot\mid H_{n}) .
###### Continual learning objective.
Let ℓ ( π , o ) \ell(\pi,o) be a non-negative measure of error for a learner policy π \pi when given an observation o o 2 2 2 Note that here we’re bypassing the action sampling, which is incorporated in the definition of ℓ \ell and denote by 𝒫 n \mathcal{P}_{n} the conditional distribution 𝒫 ( ⋅ | H n , a n ) \mathcal{P}(\cdot|H_{n},a_{n}) . Lower values of ℓ \ell correspond to better performance, a convention we keep throughout the paper.
Kumar et al. (2025) define the general continual learning objective as the minimization of the real-time error : inf 𝒜 1 T ∑ n = 1 T 𝔼 π n ∼ 𝒜 ( ⋅ ∣ H n ) , o n + 1 ∼ 𝒫 n [ ℓ ( π n , o n + 1 ) ] ⏟ ALE ( 𝒜 ) s.t. computational constraints \displaystyle\inf_{\mathcal{A}}\,\,\underbrace{\frac{1}{T}\sum_{n=1}^{T}\mathbb{E}_{\pi_{n}\sim\mathcal{A}(\cdot\mid H_{n}),\,o_{n+1}\sim\mathcal{P}_{n}}\,\left[\ell(\pi_{n},o_{n+1})\right]}_{\mathrm{ALE}(\mathcal{A})}\qquad\text{s.t. computational constraints} (1) The computational constraints are intentionally left generic in the original formulation (Eq. 2 of Kumar et al. (2025) ), as the precise formulation depends on the specific form of 𝒜 \mathcal{A} .
Two features of this formulation are central to what follows. First, the learner is evaluated on the next-step observation o n + 1 o_{n+1} : in stationary settings this corresponds to the conventional notion of test-error, but under non-stationarity the data distribution may change in successive time steps ( 𝒫 n ≠ 𝒫 n + 1 \mathcal{P}_{n}\neq\mathcal{P}_{n+1} ), causing previously acquired knowledge to lose predictive value. Second, the computational constraint on 𝒜 \mathcal{A} induces the essential trade-off at the heart of CL. With unlimited computational resources, an agent could retain all past information while remaining adaptive. Under realistic resource constraints, it may be better to forget past data once they cease to predict the future. As we show in the following sections, these trade-offs emerge even in simple quadratic learning problems.
#### 2.2 Analysis Setup and Formalism
The general formulation above accommodates arbitrary forms of non-stationarity. Following the CL literature, we introduce a task-based abstraction, where the data-generating process (described by 𝒫 n \mathcal{P}_{n} ) is locally stationary .
###### Definition 1 (Task-based data generating process.) .
A given history H T H_{T} is a task-based sequence if the sampling distribution 𝒫 n \mathcal{P}_{n} is locally stationary. The timeline is partitioned into time intervals called tasks S k = [ a k , b k ] S_{k}=[a_{k},b_{k}] such that at each time step n ∈ S k n\in S_{k} an observation o n o_{n} is sampled from a task-specific distribution 𝒫 k \mathcal{P}^{k} .
Unless otherwise stated, we assume the history is a task-based sequence with known task switch points . We denote by z ( n ) z(n) the task identity at time n n and by N k = | S k | N_{k}=|S_{k}| the duration of task k k .
With a slight abuse of notation we use ℓ ( θ , o ) \ell(\theta,o) to indicate ℓ ( π θ , o ) \ell(\pi_{\theta},o) , where θ \theta are the parameters of the policy π \pi . We denote the expected loss for task k k over the model parameters θ \theta as ℒ k ( θ ) = 𝔼 o ∼ 𝒫 k [ ℓ ( θ , o ) ] \mathcal{L}^{k}(\theta)=\mathbb{E}_{o\sim\mathcal{P}^{k}}[\ell(\theta,o)] We refer to the corresponding oracle minimizer θ ⋆ k ∈ arg min θ ℒ k ( θ ) \theta^{k}_{\star}\in\arg\min_{\theta}\mathcal{L}^{k}(\theta) as a task solution vector . Within this formalism, we define the Average Task Error as the real-time error accumulated during a single task: ALE k ( 𝒜 ) = 1 N k ∑ n = a k b k 𝔼 θ ∼ 𝒜 ( ⋅ | H n ) [ ℒ k ( θ ) ] \mathrm{ALE}^{k}(\mathcal{A})=\frac{1}{N_{k}}\sum_{n=a_{k}}^{b_{k}}\mathbb{E}_{\theta\sim\mathcal{A}(\cdot|H_{n})}[\mathcal{L}^{k}(\theta)] It follows that the total lifelong error is a weighted sum of these task errors ALE ( 𝒜 ) = ∑ k = 1 z ( T ) w ¯ k ⋅ ALE k ( 𝒜 ) \mathrm{ALE}(\mathcal{A})=\sum_{k=1}^{z(T)}\bar{w}_{k}\cdot\mathrm{ALE}^{k}(\mathcal{A}) , where the weight of each task w ¯ k = N k / T \bar{w}_{k}=N_{k}/T is simply its relative presence over the agent’s lifetime.
Literature note. Defining c k = ℒ k ( θ ⋆ k ) c^{k}=\mathcal{L}^{k}(\theta^{k}_{\star}) the optimal value of the loss for task k k for the chosen model class, we get the average regret of the optimization algorithm: ALE k ( 𝒜 ) − c k \mathrm{ALE}^{k}(\mathcal{A})-c^{k} . The notion of regret is common in online learning ( Cesa-Bianchi and Lugosi, 2006 ; Shalev-Shwartz, 2012 ; Zinkevich, 2003 ; Alquier et al., 2017 ) , and measures the cumulative loss of a learner along its trajectory with respect to a given comparator .
#### 2.3 The algorithm as a Markov Process