To Retain or to Adapt? Generalizing Continual Learning — part 4/15

Cleaned-text code points [47250, 62800). Bibliography omitted here; source.md includes it.

Figure 8 shows that the three TE fronts differ markedly even though the function class is the same; per-width fronts are in Figures 22 and 23 . Examining the constituents ( Figure 24 ) reveals two observations consistent with Theorem 8 : Instability is several orders of magnitude higher under NTP , yielding substantially lower JTL performance; and the JTL Transient is several orders of magnitude below the ITL Transient and Instability at every width . Two implications: first, the framework continues to hold: Transfer Efficiency is governed by − ℐ -\mathcal{I} plus a vanishing transient gap, but the value of ℐ \mathcal{I} is determined by implementation-level choices, not by the task sequence alone. Second, this highlights the limit of any purely statistical account of transfer (cf. the early Learning-to-Learn literature): the design of adaptive algorithms must jointly consider the stochastic processes defined by the data and by the optimizer.

### 5 Generalizing CL beyond the JTL/ITL dichotomy

#### 5.1 From implicit to explicit future models

We have shown, empirically and theoretically, that neither the JTL nor the ITL algorithm is universally optimal: each minimizes the Average Lifelong Error ( Equation 1 ) only under specific environmental regimes. This raises the natural question: is there a single optimal objective for continual learning, one that provably minimizes the ALE?

Foundational results in online convex optimization ( Besbes et al., 2015 ) established that, when the environment is unpredictable (i.e. adversarial), no first-order stochastic method can outperform online gradient descent ( Zinkevich, 2003 ) . However, Rakhlin and Sridharan (2013) tell us that much more can be hoped for when the future is somewhat predictable . If 𝒫 n + 1 \mathcal{P}_{n+1} were known, we could choose 𝒟 n \mathcal{D}_{n} to match it and minimize the next-step loss directly; 𝒫 n + 1 \mathcal{P}_{n+1} is not known in general, but one can always approximate the future based on some assumptions.

This lets us reinterpret the sampling choice (the choice of 𝒟 n \mathcal{D}_{n} ), and more generally the choice of objective, as an implicit assumption about the future. For JTL, the assumption is that the environment is ergodic and the future will resemble the past — the classical i.i.d. premise underlying empirical risk minimization . For ITL, the assumption is the opposite: that no structure can be reliably extrapolated, in which case online gradient descent on the current task is minimax-optimal ( Zinkevich, 2003 ; Besbes et al., 2015 ) .

Algorithms that maximize average accuracy or minimize forgetting can be viewed as implicit (or, for replay-based methods, explicit) approximations of the JTL objective. Thus the proposed re-reading opens a new way to analyze and to build CL algorithms. More importantly, the same framing suggests a constructive path: rather than fix an objective implicitly, one can make the future model an explicit design choice. We call the resulting family Predictive CL algorithms .

###### Definition 9 (Predictive CL Algorithm) .

Let ℋ < k \mathcal{H}_{<k} denote the observable history up to task k k , including past data, environments, and learning trajectories. A prediction mapping Φ \Phi maps this history to a predicted target distribution for the current task, 𝒫 ~ k = Φ ⁡ ( ℋ < k ) \tilde{\mathcal{P}}^{k}=\Phi(\mathcal{H}_{<k}) . A Predictive CL algorithm optimizes its parameters by minimizing the expected loss under the predicted distribution 𝒫 ~ k \tilde{\mathcal{P}}^{k} : inf 𝒜 1 N k ∑ n = a k b k 𝔼 θ n ∼ 𝒜 ( ⋅ ∣ H n ) 𝔼 o ∼ 𝒫 ~ k [ ℓ ( θ n , o ) ] s.t. computational constraints \displaystyle\inf_{\mathcal{A}}\,\,\frac{1}{N_{k}}\sum_{n=a_{k}}^{b_{k}}\mathbb{E}_{\theta_{n}\sim\mathcal{A}(\cdot\mid H_{n})\,}\mathbb{E}_{o\sim\tilde{\mathcal{P}}^{k}}\,\left[\ell(\theta_{n},o)\right]\qquad\text{s.t. computational constraints} (6)

Definition 9 generalizes existing CL objectives by making the future model explicit. JTL is recovered by choosing Φ ⁡ ( ℋ < k ) = ∑ j ≤ k w j k ​ 𝒫 j \Phi(\mathcal{H}_{<k})=\sum_{j\leq k}w_{j}^{k}\,\mathcal{P}^{j} (the empirical past is the prediction); ITL is recovered by Φ ⁡ ( ℋ < k ) = 𝒫 k \Phi(\mathcal{H}_{<k})=\mathcal{P}^{k} (the current task is the prediction). The Window algorithm introduced below sits between these limits. Crucially, Φ \Phi need not be fixed: it can itself be a learned model of the future.

#### 5.2 Proof of Concept: the Window Predictive CL Algorithm

Consider again the quadratic losses case of Section 4 . We now instantiate, as an example, a Predictive CL algorithm which we call the Window (W) algorithm. The Window algorithm predicts the future using a finite recent history: it is defined by a hyper-parameter W W , which we call window size . For each task k k the SGD sampling distribution 𝒟 W k \mathcal{D}^{k}_{W} is a mix of the past W W tasks 𝒟 W k = ∑ j = k − W + 1 k w j k ​ 𝒫 j . \mathcal{D}^{k}_{W}=\sum_{j=k-W+1}^{k}w_{j}^{k}\,\mathcal{P}^{j}. Like the JTL agent, the Window agent parameters are not reset at the beginning of a new task. As W W grows the window agent converges to the JTL agent - which can be seen as an infinite window agent-, whereas for W = 1 W=1 it coincides with an ITL agent with warm starts .

In contrast to the JTL agent, whose bias is determined by the global growth of the task solutions relative to the origin, the Window agent’s bias depends on the local variations of the environment over the horizon W W . In order to capture this bias asymptotically we need to introduce a new measure, akin to the Growth Rate, which we call the Mean Squared Displacement .

###### Definition 10 (Mean Squared Displacement (MSD)) .

The Mean Squared Displacement m : ℕ × ℕ → ℝ + m:\mathbb{N}\times\mathbb{N}\to\mathbb{R}^{+} characterizes the expected squared Euclidean distance between two task solutions separated by a lag W W at time k k : 𝔼 ⁡ [ ‖ θ ⋆ k − θ ⋆ k − W ‖ 2 ] ∈ Θ ⁡ ( m ⁡ ( k , W ) ) . \mathbb{E}[\|\theta_{\star}^{k}-\theta_{\star}^{k-W}\|^{2}]\in\Theta(m(k,W)). (7)

Based on the MSD measure, we can compute the asymptotic Transfer Efficiency of the Window Algorithm compared to ITL training (as by Section 4.4 ), and we can solve for the window size W W which maximises the Transfer Efficiency.

###### Lemma 11 (Asymptotic Optimality of Window Size) .

Let m ⁡ ( k , W ) m(k,W) be the Mean Squared Displacement of tasks within a window W W . The optimal window size W ∗ W^{*} maximizing transfer depends on the environment’s non-stationarity regime. In particular consider three examples: 1. Bounded Regime ( ψ ⁡ ( k ) = 1 \psi(k)=1 ): If m ⁡ ( k , W ) ∈ Θ ⁡ ( 1 ) m(k,W)\in\Theta(1) , transfer increases monotonically with W W . The asymptotically optimal strategy is the JTL agent ( W ∗ → ∞ W^{*}\to\infty ).

2. Linear Drift Regime ( ψ ⁡ ( k ) = k 2 \psi(k)=k^{2} ): If m ⁡ ( k , W ) ∈ Θ ⁡ ( W 2 ) m(k,W)\in\Theta(W^{2}) , transfer decreases monotonically with W W . The asymptotically optimal strategy is the ITL agent ( W ∗ = 1 W^{*}=1 ).

3. Diffusive Regime ( ψ ⁡ ( k ) = k \psi(k)=k ): If m ⁡ ( k , W ) ∈ Θ ⁡ ( W ) m(k,W)\in\Theta(W) , transfer is strictly convex in W W . The asymptotically optimal strategy is a Window agent with finite memory ( 1 < W ∗ < ∞ 1<W^{*}<\infty ).

This result formally establishes that the best objective for real-time performance depends on the environment and the model ( m ⁡ ( k , W ) m(k,W) is model-dependent). The bounded and linear-drift cases recover the JTL and ITL extremes respectively; the most interesting case is the diffusive regime, where the optimal strategy is neither and depends on further environmental parameters.

For the Continuous Drift Model of Section 4.4 , we derive a non-asymptotic closed form for W ⋆ W^{\star} ( Equation 121 ; full derivation in Section A.7 ). Crucially, W ⋆ W^{\star} depends on the learning rate η \eta —not just on environment parameters—confirming that the optimal objective for fast adaptation is jointly determined by the environment and the learning algorithm.

We run simulations of the Window agent in the Continuous Drift Environment (details in Section B.1 ). In Figure 9 we compare the loss of the ITL, JTL and Window agents with optimal window size: the Window agent consistently outperforms both baselines across all tested task durations. The optimal window shrinks as N N grows, recovering ITL-with-warm-starts for large N N and JTL for sufficiently small N N . The full transfer-efficiency landscape across window sizes and drift magnitudes ( Figure 15 in the appendix) confirms these theoretical predictions.

###### Beyond the Window: richer future models.

The Window algorithm is a deliberately simple instance of Definition 9 : it encodes the prior that recent tasks are the most informative about the next one, summarized by a single scalar W W . The same framework accommodates far richer prediction mappings Φ \Phi —for instance, a generative model that produces samples resembling future tasks, or a learned sufficient summary of the history tailored to next-task prediction. In both cases the design question becomes which statistics of the past predict the future , and the bias-variance trade-off our framework formalizes provides the criterion under which competing Φ \Phi can be compared. We leave the exploration of such mappings to future work.

### 6 Interdisciplinary Literature Review

One of the aims of this work is to draw connections between the continual learning problem and the adjacent literatures that have developed conceptual structures and theoretical results bearing on its core questions. Our contribution is two-fold: providing CL with new analytical machinery, and offering existing fields fresh problems where their results can be tested and applied.

###### Online Learning and Stochastic Optimization.

Our analysis sits at the intersection of Online Convex Optimization (OCO) and Stochastic Optimization ( Cesa-Bianchi and Lugosi, 2006 ; Hazan, 2016 ; Orabona, 2019 ) . The Average Lifelong Error metric is a finite-horizon analogue of the OCO static regret ( Zinkevich, 2003 ) , and across multiple tasks the average ALE corresponds to a notion of dynamic regret ( Herbster and Warmuth, 1998 ; Besbes et al., 2015 ) . Besbes et al. (2015) establish a minimax dynamic regret bound of O ⁡ ( T 2 / 3 ​ V T 1 / 3 ) O(T^{2/3}V_{T}^{1/3}) that grows with the comparator’s path variation V T V_{T} , formalizing how non-stationarity inflates regret with time. The role played there by the path variation V T V_{T} is played in our task-indexed setting by the Growth Rate ψ ⁡ ( k ) \psi(k) : both quantify how non-stationarity accumulates as the environment expands. Our Critical Task Duration N max ∞ N_{\max}^{\infty} then converts this descriptor into a horizon on the task-duration axis: it characterizes the point beyond which the environment’s Growth Rate ψ ⁡ ( k ) \psi(k) makes accumulating past observations actively harmful relative to learning each task afresh.

Our transient analysis of the JTL and ITL agents recovers and extends classical Lyapunov-type results on constant step-size SGD ( Murata, 1999 ; Bach and Moulines, 2013 ; Mandt et al., 2017 ) ; the Transient Potential Law (Theorem 25 ) makes the geometric role of the relative-curvature matrix ℛ k \mathcal{R}^{k} in those bounds explicit.

A central conceptual import from this literature is the framework of predictable sequences ( Rakhlin and Sridharan, 2013 ) : an online learner can improve over the worst-case minimax dynamic regret of Zinkevich (2003) ; Besbes et al. (2015) when given a useful hypothesis about future losses. Our generalization of CL beyond JTL ( Section 5 ) is precisely this idea translated to non-stationary task sequences. From this perspective, JTL (and similarly most of the continual learning algorithms it embodies) is not a new algorithm but a particular instantiation of predictable-sequence learning under an ergodicity hypothesis.

###### Multi-Task Learning (MTL).

The comparison of JTL and ITL aligns with a foundational question of Multi-Task Learning: does jointly training over multiple tasks benefit performance relative to learning each independently? JTL, as we use it, is best understood as an online, cumulative version of MTL: the task pool grows over time, and joint training is performed online over the accumulated history rather than offline on a fixed pool. The classical MTL literature operates in a different regime, typically a fixed task pool sampled i.i.d. and evaluated post-hoc; the underlying tension between sharing and interference is, however, identical.

Statistical foundations show that shared representations reduce sample complexity when the task family is self-contained ( Baxter, 2000 ; Maurer et al., 2016 ; Tripuraneni et al., 2020 ; Du et al., 2021 ) , but abstract away the optimization process; a separate line considers optimization-time interference and gradient conflict ( Standley et al., 2020 ; Sener and Koltun, 2018 ; Yu et al., 2020a ; Liu et al., 2023 ) . Neither line jointly tracks environment and optimization processes as they co-evolve, nor formalizes the resulting trade-off through an ALE-like metric.

###### Learning To Learn and Online Meta-Learning.

Our framework has its closest ties to Learning-to-Learn and Online Meta-Learning ( Kuzborskij and Orabona, 2017 ; Denevi et al., 2018 ) . The Online-Within-Online setting ( Alquier et al., 2017 ; Denevi et al., 2019 ; Khodak et al., 2019 ; Nazari and Khorram, 2021 ) considers regret accumulated across a sequence of tasks adapted via inner-loop optimization — formally analogous to our ALE. Denevi et al. (2019) ; Khodak et al. (2019) prove that the regret-optimal running prior is the centroid of past task minimizers; under our task-alignment assumption ( Assumption 22 ), this centroid coincides exactly with the JTL minimizer θ ⋆ J ​ T ​ L , k \theta_{\star}^{JTL,k} . The two algorithm families are therefore not merely similar in spirit but provably equivalent in their asymptotic target. The substantive difference lies in what they consume: meta-learning algorithms typically use past gradients or update statistics, whereas JTL uses the source data directly via replay, with consequences for variance and robustness beyond convex regimes.

A direct corollary of the MTL/MAML equivalence in over-parametrized networks ( Wang et al., 2021 ) is that our Instability bound for JTL also bounds the asymptotic interference cost of MAML; the transient trajectories of the two families, which are the focus of much of our analysis, remain distinct. Our Transfer Efficiency also reconciles the warm-start puzzle of Ash and Adams (2020) : warm-starting helps when Instability is small relative to the ITL Transient, and hurts when this inequality reverses.

###### Continual Learning.

We finally situate our contribution within continual learning. The defining aim of modern CL is to mitigate catastrophic forgetting ( Parisi et al., 2019 ; Hadsell et al., 2020 ; De Lange et al., 2022 ) , pursued via mechanisms as diverse as regularization, replay, parameter isolation, expert mixtures, and prompt-based adaptation. Although these methods differ substantially in implementation, they implicitly share a common target: maximize average accuracy across all tasks seen so far. In our terminology, each is an approximation of the JTL objective. The variety of CL methods reflects different practical strategies for approximating this common objective, not different objectives in themselves.