To Retain or to Adapt? Generalizing Continual Learning β€” part 2/15

Cleaned-text code points [15987, 31622). Bibliography omitted here; source.md includes it.

We treat the algorithm as a stochastic process. This is the most generic framing of an iterative learner: the rest of the theory in this paper instantiates it on SGD, but the definitions below cover any first-order stochastic-approximation method. More formally, we define π’œ \mathcal{A} to be a discrete-time stochastic process { ΞΈ n } n β‰₯ 0 \{\theta_{n}\}_{n\geq 0} taking values in the parameter space, and we denote by Q n Q_{n} the resulting parameter distribution at time n n . The evolution of the parameters is governed by a stochastic update rule U U . In this work we consider algorithms where, at step n n , the update rule depends on the current parameters ΞΈ n \theta_{n} and a data batch ΞΎ n \xi_{n} drawn from an algorithm-specific sampling distribution π’Ÿ n \mathcal{D}_{n} : ΞΈ n + 1 = U ⁑ ( ΞΈ n , ΞΎ n ) , where ​ ΞΎ n ∼ π’Ÿ n . \theta_{n+1}=U(\theta_{n},\xi_{n}),\quad\text{where }\xi_{n}\sim\mathcal{D}_{n}. This formula covers SGD as our running example and, by augmenting ΞΈ n \theta_{n} to include auxiliary variables, momentum-based methods, Adam, and more broadly the entire family of stochastic approximation algorithms based on first-order Markov processes .

Crucially, the sampling distribution π’Ÿ n \mathcal{D}_{n} is distinct from the environmental distribution 𝒫 n \mathcal{P}_{n} . While 𝒫 n \mathcal{P}_{n} dictates the incoming data stream, π’Ÿ n \mathcal{D}_{n} dictates the data used for optimization; for replay-based algorithmsβ€”such as the JTL algorithm presented hereβ€” π’Ÿ n \mathcal{D}_{n} may include data from previous tasks. Moreover, we assume the sampling strategy is consistent within task boundaries, such that π’Ÿ n = π’Ÿ k \mathcal{D}_{n}=\mathcal{D}^{k} for all n ∈ S k n\in S_{k} .

Importantly, the computational cost of such algorithms depends on the complexity of the update function U U . For example, for SGD the complexity is proportional to the batch size used at each step, since that determines the number of forward and backward passes through the model.

Throughout this paper we reason about Q n Q_{n} (the parameter distribution at step n n ) and its stationary limit Q ∞ Q_{\infty} ; in practice, expectations under Q n Q_{n} are estimated from the empirical ensemble of training trajectories induced by multiple random seeds, and the deep-network experiments treat each trained model as a sample from Q n Q_{n} at the corresponding step.

#### 2.4 JTL and ITL processes

Within these boundaries, many different learning algorithms can be implemented through choices of U U or π’Ÿ n \mathcal{D}_{n} . Here we introduce two specific examples, that will be the paper’s object of study.

###### The Joint-Task Learning algorithm ( π’œ JTL \mathcal{A}_{\text{JTL}} ).

The JTL agent aims to optimize the average loss over all seen tasks. This is equivalent to experience replay with an infinitely large bufferβ€”often used in continual learning as an upper bound on performance. To maintain time-homogeneous dynamics within a task, we model the experience replay as a fixed mixture approximation. Specifically, the sampling distribution within task k k , π’Ÿ J ​ T ​ L k \mathcal{D}^{k}_{JTL} , is a weighted mixture of the current and all preceding task distributions: π’Ÿ J ​ T ​ L k = βˆ‘ j = 1 k w j k β‹… 𝒫 j , where ​ w j k = N j b k . \mathcal{D}^{k}_{JTL}=\sum_{j=1}^{k}w_{j}^{k}\cdot\mathcal{P}^{j},\qquad\text{where }w_{j}^{k}=\frac{N_{j}}{b_{k}}\,.

###### The Independent-Task Learning algorithm ( π’œ ITL \mathcal{A}_{\text{ITL}} ).

The ITL agent optimizes solely based on the current data stream. Consequently, the sampling distribution is identical to the current task distribution, π’Ÿ I ​ T ​ L k = 𝒫 k . \mathcal{D}^{k}_{ITL}=\mathcal{P}^{k}\,. At the boundary between tasks, the ITL agent resets the parameters to a fixed initialization distribution, effectively discarding prior knowledge to avoid negative transfer and loss of plasticity ( Dohare et al., 2024 ) . A related variant warm-starts each task from the previous parameters instead of resetting; the differences are minor for our analysis, and we discuss them in Section A.2.2 .

###### Equivalent computational costs.

By construction, ITL and JTL share the same optimization compute per iteration: the cost scales linearly with the examples processed in a gradient update, compute ​ ( π’œ ) ∈ Θ ​ ( B ) \text{compute}(\mathcal{A})\in\Theta(B) for batch size B B . 3 3 3 Hardware optimizations might violate this linear relationship; for the present discussion the linear scaling suffices. This is the cost appearing in Equation 1 . JTL may require additional memory or data-access machinery to implement replay, but those are outside the optimization budget studied here. Our experiments match this budget by design, using the same number of parameter updates and the same per-update batch budget for both agents.

We study JTL and ITL not as competing heuristics but as the two limiting cases of a single design axis: both are instances of the broader family of Predictive CL algorithms we formalize in Section 5 ( Definition 9 ), each corresponding to a particular assumption about how the future relates to the past.

### 3 Transfer Efficiency

We wish to quantify when a JTL model is preferable to an ITL one. To this end, we define the concept of Transfer Efficiency , which is central to our analysis.

###### Definition 2 (Transfer Efficiency) .

The Transfer Efficiency for task k k is the performance gain of a Joint-Task Learning (JTL) agent over an Independent-Task Learning (ITL) baseline: TE k := ALE k ​ ( π’œ ITL ) βˆ’ ALE k ​ ( π’œ JTL ) . \mathrm{TE}^{k}:=\mathrm{ALE}^{k}(\mathcal{A}_{\text{ITL}})-\mathrm{ALE}^{k}(\mathcal{A}_{\text{JTL}}).

A positive value TE k > 0 \mathrm{TE}^{k}>0 indicates positive transfer , where the JTL agent’s past experience improves its performance on task k k relative to the ITL agent; conversely, TE k < 0 \mathrm{TE}^{k}<0 indicates negative transfer . Because ALE k ​ ( π’œ ) \mathrm{ALE}^{k}(\mathcal{A}) represents the regret of π’œ \mathcal{A} on task k k (up to a constant), the Transfer Efficiency can equivalently be read as the regret reduction obtained by JTL relative to ITL.

To understand the mechanisms of transfer, we split the error into two components: a stationary component , the error at equilibrium, and a transient component , the error accumulated while the algorithm is actively moving toward that equilibrium. 4 4 4 We use equilibrium , stationary distribution , and convergence point interchangeably throughout the paper. These terms originate in different communities (statistical physics, Markov chains, and optimization respectively), but for the iterate sequence { θ n } \{\theta_{n}\} they all refer to the same object: the limiting distribution Q ∞ Q_{\infty} to which Q n Q_{n} converges as the algorithm runs on a fixed task. We avoid critical point , which in optimization denotes a zero-gradient parameter value rather than a distribution over parameters. Equilibrium is only well-defined if the optimization process converges , which we henceforth assume.

We denote by Q ∞ π’œ Q_{\infty}^{\mathcal{A}} the stationary parameter distribution to which Q n Q_{n} converges under algorithm π’œ \mathcal{A} , given a sufficiently long task duration. It is dictated jointly by the task data and the agent’s initialization at the task boundary. For the JTL and ITL algorithms we write Q ∞ J ​ T ​ L Q_{\infty}^{JTL} and Q ∞ I ​ T ​ L Q_{\infty}^{ITL} respectively. We also use β„’ k ​ ( Q ) := 𝔼 ΞΈ ∼ Q ​ [ β„’ k ​ ( ΞΈ ) ] \mathcal{L}^{k}(Q):=\mathbb{E}_{\theta\sim Q}[\mathcal{L}^{k}(\theta)] to lift the expected loss to a distribution over parameters.

###### Definition 3 (Instability) .

The Instability of a task k k ( ℐ k \mathcal{I}^{k} ) is the performance gap between the JTL and ITL agent measured at convergence ℐ k := β„’ k ​ ( Q ∞ J ​ T ​ L ) βˆ’ β„’ k ​ ( Q ∞ I ​ T ​ L ) . \mathcal{I}^{k}:=\mathcal{L}^{k}(Q_{\infty}^{JTL})-\mathcal{L}^{k}(Q_{\infty}^{ITL}).

###### Definition 4 (Transient Error) .

The Transient Error of a task k k and algorithm π’œ \mathcal{A} ( Ξ΄ k ​ ( π’œ ) \delta^{k}(\mathcal{A}) ) is the average deviation from the stationary risk over the task duration S k S_{k} : Ξ΄ k ​ ( π’œ ) = 1 N k ​ βˆ‘ n ∈ S k ( β„’ k ​ ( Q n π’œ ) βˆ’ β„’ k ​ ( Q ∞ π’œ ) ) . \delta^{k}(\mathcal{A})=\frac{1}{N_{k}}\sum_{n\in S_{k}}\left(\mathcal{L}^{k}(Q^{\mathcal{A}}_{n})-\mathcal{L}^{k}(Q^{\mathcal{A}}_{\infty})\right).

Combining these definitions yields a handy decomposition of Transfer Efficiency, separating the β€œquality of the final solution” from the β€œspeed of learning” (visualized in Figure 20 , appendix).

###### Lemma 5 (TE Decomposition) .

For any task k k , assuming the algorithms converge, the Transfer Efficiency decomposes exactly as: TE k = βˆ’ ℐ k ⏟ Stationary Gap + ( Ξ΄ k ​ ( π’œ ITL ) βˆ’ Ξ΄ k ​ ( π’œ JTL ) ) ⏟ Learning Speed Gap . \mathrm{TE}^{k}=\underbrace{-\mathcal{I}^{k}}_{\text{Stationary Gap}}+\underbrace{(\delta^{k}(\mathcal{A}_{\text{ITL}})-\delta^{k}(\mathcal{A}_{\text{JTL}}))}_{\text{Learning Speed Gap}}. (2)

Literature note. The Transient Error has been extensively studied in the stochastic optimization literature ( Murata, 1999 ) , whereas the Instability has been the object of study of the multi-task learning and transfer-learning literature ( Baxter, 2000 ; Caruana, 1997 ) . These two fields have mostly developed independently, except for work in online Meta-Learning ( Denevi et al., 2018 ; Nazari and Khorram, 2021 ) where both online performance and between-task transfer are considered, and where joint-task and independent-task baselines are routinely compared. To place our definitions within this rich literature, we provide a map in Figure 3 . We discuss these connections and their context in depth in Section 6 .

#### 3.1 The Critical Task Duration

The decomposition in Lemma 5 yields a first general result, holding under only minimal assumptions: when the stationary JTL solution is worse than the stationary ITL solution, there is a threshold task duration beyond which the initial advantage of transfer is lost.

###### Lemma 6 (The Critical Task Duration) .

Assume a sequence of K K tasks of equal duration N N . Let Ξ” k ​ ( π’œ ) := N β‹… Ξ΄ k ​ ( π’œ ) \Delta^{k}(\mathcal{A}):=N\cdot\delta^{k}(\mathcal{A}) denote the cumulative Transient Error of algorithm π’œ \mathcal{A} on task k k , let Ξ” Β― ​ ( π’œ ) = 1 K ​ βˆ‘ k = 1 K Ξ” k ​ ( π’œ ) \bar{\Delta}(\mathcal{A})=\frac{1}{K}\sum_{k=1}^{K}\Delta^{k}(\mathcal{A}) be its average across tasks, and let ℐ Β― \bar{\mathcal{I}} be the average Instability. If the cumulative Transient Error is sub-linear in N N , i.e. Ξ” k ​ ( π’œ ) ∈ o ​ ( N ) \Delta^{k}(\mathcal{A})\in o(N) , and ℐ Β― > 0 \bar{\mathcal{I}}>0 , then there exists a critical task horizon N max ∞ N^{\infty}_{\max} beyond which transfer becomes detrimental ( TE < 0 \mathrm{TE}<0 ). The breakeven horizon is: N max ∞ = Ξ” Β― ​ ( π’œ ITL ) βˆ’ Ξ” Β― ​ ( π’œ JTL ) ℐ Β― . N^{\infty}_{\max}=\frac{\bar{\Delta}(\mathcal{A}_{\text{ITL}})-\bar{\Delta}(\mathcal{A}_{\text{JTL}})}{\bar{\mathcal{I}}}. (3)

In other words, if the task duration exceeds N max ∞ N^{\infty}_{\max} , the ITL agent has enough time to overcome its poor initialization, eventually outperforming the JTL agent. The Critical Task Duration is inversely proportional to the average Instability in the agent’s lifetime: the more β€˜unpredictable’ the world, the narrower the window for transfer. When ℐ Β― ≀ 0 \bar{\mathcal{I}}\leq 0 , the stationary JTL solution is not worse on average, and the decomposition predicts no finite horizon at which retention must become detrimental. The sub-linearity condition Ξ” k ​ ( π’œ ) ∈ o ​ ( N ) \Delta^{k}(\mathcal{A})\in o(N) requires only that the learning algorithm converges toward its stationary risk β€” a mild assumption holding across standard optimization regimes, from Convex Optimization to Neural Networks (see Lemma 16 ).

#### 3.2 Empirical Validation of the Framework

We use three benchmarks, chosen to span a range of Instabilities: CLEAR ( Lin et al., 2021 ) , ten image classes evolving smoothly over 2004–2014; MD5 , a sequence we construct from five unrelated public classification datasets with sharp semantic transitions (introduced in this paper); MT10 ( Yu et al., 2020b ) , a 10-task suite of Meta-World robotic-manipulation environments previously used to highlight continual-learning interference ( WoΕ‚czyk et al., 2021 ) . We use the same task duration N N across tasks and tune hyperparameters for the lowest ALE per algorithm; full setup details are in Appendix B .

Throughout the paper, classification performance is reported as accuracy in [ 0 , 1 ] [0,1] and reinforcement-learning performance as success rate in [ 0 , 1 ] [0,1] . We summarize each agent by its Average Lifelong Accuracy ( ALA ) or Average Lifelong Success Rate ( ALS )β€”monotone (higher-is-better) re-readings of the ALEβ€”and we sign Transfer Efficiency so that TE > 0 \mathrm{TE}>0 always means JTL outperforms ITL.

Running ITL and JTL with matched compute on each, the resulting learning curves ( Figure 4 ) and average lifelong performance ( Table 1 ) already tell the qualitative story: where Instability 5 5 5 In the experiments, expectations under Q n Q_{n} and Q ∞ Q_{\infty} are estimated empirically: we run multiple training seeds for each algorithm and treat the resulting ensemble of trained models as samples from the parameter distribution at step n n . The expected loss β„’ k ​ ( Q n ) \mathcal{L}^{k}(Q_{n}) is then approximated by the average test loss across seeds, and the Instability ℐ k \mathcal{I}^{k} and Transient Error Ξ΄ k \delta^{k} in the decomposition are estimated from this empirical ensemble after the algorithms have converged on each task. is large (MD5, MT10), joint training confers no advantage and is at times actively worse; where Instability is small (CLEAR), it confers a large, compounding advantage. In short, the sign of Transfer Efficiency tracks the sign of βˆ’ ℐ -\mathcal{I} , exactly as Equation 2 predicts.

Beyond this qualitative pattern, the decomposition in Equation 2 and the Critical Task Duration of Lemma 6 make two sharper, quantitative predictions about real continual-learning systems:

β€’ (P1) Instability shifts Transfer Efficiency by an amount independent of task duration; Transfer Efficiency then decays with task duration at a 1 / N 1/N rate. Direct prediction of Equation 2 : ℐ k \mathcal{I}^{k} enters as an additive constant in N N , while the transient gap is averaged over N N steps.

β€’ (P2) For sequences with ℐ Β― > 0 \bar{\mathcal{I}}>0 , increasing N N drives Transfer Efficiency from positive to negative; and once N N is large enough for both agents to converge, TE \mathrm{TE} is dominated by ℐ \mathcal{I} . Direct prediction of Lemma 6 : the transient gap shrinks with N N , leaving the stationary Instability as the residual.

Next, we test these predictions directly, using controlled benchmarks that let us vary Instability and task duration in isolation.

###### (P1) Controlled Instability: Permuted- and Shuffled-CIFAR.

We manipulate Instability by perturbing the input and target distribution in CIFAR-10, creating two separate benchmarks: Permuted-CIFAR (PC-16/32) shuffles input pixels within a centred patch (preserving targets, transforming only the input distribution), and Shuffled-CIFAR (SC-10/30/50) randomly re-assigns labels to a fraction of training samples (transforming only the target distribution). We expect higher Instability in Shuffled- than in Permuted-CIFAR, as label shuffling forces the same input to map to inconsistent labels across tasks. This is confirmed by our empirical measurements, collected in Table 11 .