To Retain or to Adapt? Generalizing Continual Learning — part 6/15

Cleaned-text code points [77920, 93765). Bibliography omitted here; source.md includes it.

We view the learning algorithm 𝒜 \mathcal{A} as a discrete-time stochastic process { θ n } n ≥ 0 \{\theta_{n}\}_{n\geq 0} taking values in the parameter space Θ ⊆ ℝ d \Theta\subseteq\mathbb{R}^{d} . Parameter evolution is governed by a stochastic update rule U U . At step n n , the update depends on the current parameters θ n \theta_{n} and a data batch ξ n \xi_{n} drawn from an algorithm-specific sampling distribution 𝒟 n \mathcal{D}_{n} : θ n + 1 = U ⁡ ( θ n , ξ n ) , where ​ ξ n ∼ 𝒟 n . \theta_{n+1}=U(\theta_{n},\xi_{n}),\quad\text{where }\xi_{n}\sim\mathcal{D}_{n}. (14) Crucially, the optimization distribution 𝒟 n \mathcal{D}_{n} is distinct from the environmental distribution 𝒫 n \mathcal{P}_{n} . While 𝒫 n \mathcal{P}_{n} dictates the incoming data stream, 𝒟 n \mathcal{D}_{n} determines the data used for the update. For replay-based algorithms (such as our JTL agent), 𝒟 n \mathcal{D}_{n} may incorporate data from previous tasks.

Assuming the sampling strategy is consistent within task boundaries ( 𝒟 n = 𝒟 k \mathcal{D}_{n}=\mathcal{D}^{k} for all n ∈ S k n\in S_{k} ), the process induces a time-homogeneous Markov Transition Operator 𝒯 k \mathcal{T}^{k} . For any bounded measurable test function ϕ : Θ → ℝ \phi:\Theta\to\mathbb{R} , the operator is defined as: ( 𝒯 k ​ ϕ ) ​ ( θ ) := 𝔼 ξ ∼ 𝒟 k ​ [ ϕ ⁡ ( U ⁡ ( θ , ξ ) ) ] . (\mathcal{T}^{k}\phi)(\theta):=\mathbb{E}_{\xi\sim\mathcal{D}^{k}}[\phi(U(\theta,\xi))]. (15) The sequence of parameter distributions { Q n } n ∈ S k \{Q_{n}\}_{n\in S_{k}} evolves according to this linear operator, Q n + 1 = Q n ​ 𝒯 k Q_{n+1}=Q_{n}\mathcal{T}^{k} (following the Markov chain convention where measures act as row vectors).

###### Convergence.

A central aspect of our analysis is characterizing the long-term behavior of the learning algorithm. An algorithm is said to converge if its transition operator 𝒯 k \mathcal{T}^{k} admits a stationary distribution Q ∞ Q_{\infty} ; if run indefinitely, the process will converge to this distribution. This property allows us to rigorously decouple the effect of starting conditions (the transient phase) from asymptotic performance (the stationary phase).

###### Stochastic Gradient Descent Instantiations.

We analyze two canonical instances of SGD with a constant step-size η \eta . Both employ the standard gradient step θ n + 1 = θ n − η ∇ ℓ ( θ n , ξ n ) \theta_{n+1}=\theta_{n}-\eta\nabla\ell(\theta_{n},\xi_{n}) , differing only in their sampling distributions 𝒟 k \mathcal{D}^{k} and task-boundary initialization strategies.

###### ITL SGD ( 𝒜 ITL \mathcal{A}_{\text{ITL}} ).

The ITL agent optimizes solely on the current data stream. Its sampling distribution matches the current task distribution ( 𝒟 ITL k = 𝒫 k \mathcal{D}^{k}_{\text{ITL}}=\mathcal{P}^{k} ). At task boundaries, it resets its parameters to a fixed initialization distribution Q init Q_{\text{init}} , discarding prior knowledge to prevent negative transfer.

###### JTL SGD ( 𝒜 JTL \mathcal{A}_{\text{JTL}} ).

The JTL agent minimizes the average loss across all seen tasks. To maintain time-homogeneous dynamics within a task, we model experience replay as a fixed mixture approximation . The sampling distribution 𝒟 JTL k \mathcal{D}^{k}_{\text{JTL}} is a weighted mixture of the current and preceding task distributions: 𝒟 JTL k := ∑ j = 1 k w j k ​ 𝒫 j , where ​ w j k = N j ∑ i = 1 k N i . \mathcal{D}^{k}_{\text{JTL}}:=\sum_{j=1}^{k}w^{k}_{j}\,\mathcal{P}^{j},\quad\text{where }w^{k}_{j}=\frac{N_{j}}{\sum_{i=1}^{k}N_{i}}. (16) Unlike the ITL agent, the JTL agent inherits parameters from the previous task ( θ a k = θ b k − 1 \theta_{a_{k}}=\theta_{b_{k-1}} ), enabling forward transfer.

##### A.2.2 Why use reset ITL as the reference?

The ITL baseline used throughout the main comparison is intentionally reset at task boundaries. This choice isolates the value of the JTL objective as a prediction of the future : JTL assumes that the accumulated past remains predictive, whereas reset ITL assumes no such structure and starts each task from the same prior. The comparison therefore measures the benefit and cost of injecting historical bias into the optimization process.

A warm-start current-task learner is also natural: it would set 𝒟 k = 𝒫 k \mathcal{D}^{k}=\mathcal{P}^{k} as ITL does, but initialize task k k from the previous endpoint rather than from Q init Q_{\mathrm{init}} . In our decomposition, this variant has the same stationary distribution as ITL whenever the current-task optimizer is task-wise convergent, because both optimize only 𝒫 k \mathcal{P}^{k} . Thus its stationary gap against JTL is still the same instability term ℐ k \mathcal{I}^{k} . What changes is only the transient term: TE warm k = − ℐ k + δ k ​ ( 𝒜 warm ) − δ k ​ ( 𝒜 JTL ) . \mathrm{TE}^{k}_{\mathrm{warm}}=-\mathcal{I}^{k}+\delta^{k}(\mathcal{A}_{\mathrm{warm}})-\delta^{k}(\mathcal{A}_{\mathrm{JTL}}). Consequently, warm starts can shift the critical duration by reducing the current-task transient cost, but they do not remove the central trade-off: retaining past information helps only when the transient advantage it provides exceeds the stationary bias induced by optimizing against outdated tasks. The Window agent in Section 5 includes this case as W = 1 W=1 , making it a useful bridge between reset ITL, warm-start adaptation, and full JTL.

#### A.3 Decomposition: Instability and Transient Error

To rigorously analyze the source of the performance gap between ITL and JTL agents, we introduce a theoretical decomposition based on the asymptotic behavior of the learning process.

###### Definition 12 (Task-wise Convergence) .

An algorithm 𝒜 \mathcal{A} is Task-wise Convergent if, for any task k k , the associated Markov operator 𝒯 k \mathcal{T}^{k} admits a stationary probability measure Q ∞ k Q_{\infty}^{k} . Formally, if the task duration were infinite, the distribution of iterates Q n k Q^{k}_{n} would converge to Q ∞ k Q_{\infty}^{k} regardless of initialization: lim n → ∞ W 2 ​ ( Q n k , Q ∞ k ) = 0 , \lim_{n\to\infty}W_{2}(Q^{k}_{n},Q_{\infty}^{k})=0, (17) where W 2 W_{2} is the Wasserstein-2 distance ( Villani, 2009 ) , standard in the convergence analysis of stochastic optimization algorithms.

If convergence holds, the Average Task Error ( ALE k \mathrm{ALE}^{k} ) can be split into two components: the stationary component (error at equilibrium) and the transient component (error accumulated while moving toward equilibrium).

###### Definition 13 (Instability ℐ k \mathcal{I}^{k} ) .

Let 𝒜 ITL \mathcal{A}_{\text{ITL}} and 𝒜 JTL \mathcal{A}_{\text{JTL}} be task-wise convergent algorithms with stationary distributions Q ∞ ITL Q_{\infty}^{\text{ITL}} and Q ∞ JTL Q_{\infty}^{\text{JTL}} . The Instability measures the performance gap in the asymptotic limit: ℐ k := ℒ k ​ ( Q ∞ JTL ) − ℒ k ​ ( Q ∞ ITL ) . \mathcal{I}^{k}:=\mathcal{L}^{k}(Q_{\infty}^{\text{JTL}})-\mathcal{L}^{k}(Q_{\infty}^{\text{ITL}}). (18)

###### Definition 14 (Transient Error δ k \delta^{k} ) .

For a task-wise convergent algorithm 𝒜 \mathcal{A} , the Transient Error captures the excess loss accumulated during the mixing phase. It is defined as the average deviation from the stationary risk over the task duration S k S_{k} : δ k ​ ( 𝒜 ) := 1 N k ​ ∑ n ∈ S k ( ℒ k ​ ( Q n ) − ℒ k ​ ( Q ∞ 𝒜 ) ) . \delta^{k}(\mathcal{A}):=\frac{1}{N_{k}}\sum_{n\in S_{k}}\left(\mathcal{L}^{k}(Q_{n})-\mathcal{L}^{k}(Q_{\infty}^{\mathcal{A}})\right). (19)

This yields a rigorous decomposition of Transfer Efficiency, allowing us to treat the “quality of the final solution” and the “speed of learning” as distinct analytical objects.

###### Lemma 15 (Exact TE Decomposition) .

For any task k k , assuming the algorithms are task-wise convergent, the Transfer Efficiency decomposes exactly as: TE k = − ℐ k ⏟ Stationary Gap + ( δ k ​ ( 𝒜 ITL ) − δ k ​ ( 𝒜 JTL ) ) ⏟ Learning Speed Gap . \mathrm{TE}^{k}=\underbrace{-\mathcal{I}^{k}}_{\text{Stationary Gap}}+\underbrace{\left(\delta^{k}(\mathcal{A}_{\text{ITL}})-\delta^{k}(\mathcal{A}_{\text{JTL}})\right)}_{\text{Learning Speed Gap}}. (20)

###### Proof.

By definition, ALE k ​ ( 𝒜 ) = ℒ k ​ ( Q ∞ 𝒜 ) + δ k ​ ( 𝒜 ) \mathrm{ALE}^{k}(\mathcal{A})=\mathcal{L}^{k}(Q_{\infty}^{\mathcal{A}})+\delta^{k}(\mathcal{A}) . Substituting this into TE k = ALE k ​ ( 𝒜 ITL ) − ALE k ​ ( 𝒜 JTL ) \mathrm{TE}^{k}=\mathrm{ALE}^{k}(\mathcal{A}_{\text{ITL}})-\mathrm{ALE}^{k}(\mathcal{A}_{\text{JTL}}) and rearranging terms yields the result. ∎

When tasks have equal duration N N , we can state a general condition for when transfer is beneficial. To do so, we distinguish the average (per-step) Transient Error δ k \delta^{k} from the cumulative Transient Error, defined as Δ k ​ ( 𝒜 ) := N ⋅ δ k ​ ( 𝒜 ) = ∑ n ∈ S k ( ℒ k ​ ( Q n ) − ℒ k ​ ( Q ∞ 𝒜 ) ) \Delta^{k}(\mathcal{A}):=N\cdot\delta^{k}(\mathcal{A})=\sum_{n\in S_{k}}(\mathcal{L}^{k}(Q_{n})-\mathcal{L}^{k}(Q_{\infty}^{\mathcal{A}})) .

###### Lemma 16 (The Critical Task Duration) .

Assume a sequence of K K tasks of equal duration N N . Let Δ ¯ ​ ( 𝒜 ) = 1 K ​ ∑ k = 1 K Δ k ​ ( 𝒜 ) \bar{\Delta}(\mathcal{A})=\frac{1}{K}\sum_{k=1}^{K}\Delta^{k}(\mathcal{A}) denote the expected cumulative Transient Error, and ℐ ¯ = 1 K ​ ∑ k = 1 K ℐ k \bar{\mathcal{I}}=\frac{1}{K}\sum_{k=1}^{K}\mathcal{I}^{k} the average instability. Assuming the learning process converges such that Δ ¯ ​ ( 𝒜 ) ∈ o ​ ( N ) \bar{\Delta}(\mathcal{A})\in o(N) , and assuming ℐ ¯ > 0 \bar{\mathcal{I}}>0 , there exists a critical task horizon N max ∞ N^{\infty}_{\max} beyond which transfer becomes detrimental ( TE < 0 \text{TE}<0 ). The breakeven horizon is given by: N max ∞ = Δ ¯ ​ ( 𝒜 ITL ) − Δ ¯ ​ ( 𝒜 JTL ) ℐ ¯ . N^{\infty}_{\max}=\frac{\bar{\Delta}(\mathcal{A}_{\text{ITL}})-\bar{\Delta}(\mathcal{A}_{\text{JTL}})}{\bar{\mathcal{I}}}. (21)

###### Proof.

For equal-duration tasks, the average Transfer Efficiency is positive if: 1 K ​ ∑ k = 1 K ( δ k ​ ( 𝒜 ITL ) − δ k ​ ( 𝒜 JTL ) ) > 1 K ​ ∑ k = 1 K ℐ k . \frac{1}{K}\sum_{k=1}^{K}\left(\delta^{k}(\mathcal{A}_{\text{ITL}})-\delta^{k}(\mathcal{A}_{\text{JTL}})\right)>\frac{1}{K}\sum_{k=1}^{K}\mathcal{I}^{k}. By definition, the instability ℐ k \mathcal{I}^{k} relies on the stationary distribution and is independent of N N . Substituting δ k ​ ( 𝒜 ) = Δ k ​ ( 𝒜 ) / N \delta^{k}(\mathcal{A})=\Delta^{k}(\mathcal{A})/N , the condition for positive transfer becomes: Δ ¯ ​ ( 𝒜 ITL ) − Δ ¯ ​ ( 𝒜 JTL ) N > ℐ ¯ . \frac{\bar{\Delta}(\mathcal{A}_{\text{ITL}})-\bar{\Delta}(\mathcal{A}_{\text{JTL}})}{N}>\bar{\mathcal{I}}. If ℐ ¯ > 0 \bar{\mathcal{I}}>0 and the cumulative Transient Error is sub-linear ( Δ ¯ ​ ( 𝒜 ) ∈ o ​ ( N ) \bar{\Delta}(\mathcal{A})\in o(N) ), the left-hand transient advantage vanishes with N N , while the stationary gap remains. Solving the breakeven equality yields the critical threshold N max ∞ N_{\max}^{\infty} . If ℐ ¯ ≤ 0 \bar{\mathcal{I}}\leq 0 , the stationary component does not make JTL worse on average, so this argument does not imply an eventual detrimental-transfer horizon. ∎

###### On the Sub-Linear Transient Error Assumption.

The condition that the cumulative Transient Error grows sub-linearly ( Δ k ​ ( 𝒜 ) ∈ o ​ ( N ) \Delta^{k}(\mathcal{A})\in o(N) ) simply requires that the algorithm successfully converges toward its stationary risk, meaning the instantaneous error ℒ k ​ ( Q n ) − ℒ k ​ ( Q ∞ ) \mathcal{L}^{k}(Q_{n})-\mathcal{L}^{k}(Q_{\infty}) vanishes as n → ∞ n\to\infty . This is a mild assumption across standard optimization regimes: • Convex Optimization (Linear Models): In stochastic strongly convex optimization, sub-optimality decays at 𝒪 ⁡ ( 1 / n ) \mathcal{O}(1/n) . Integrating over N N steps yields a cumulative error bounded by 𝒪 ⁡ ( log ⁡ N ) ∈ o ⁡ ( N ) \mathcal{O}(\log N)\in o(N) . Even in strictly convex or non-smooth cases with a rate of 𝒪 ⁡ ( 1 / n ) \mathcal{O}(1/\sqrt{n}) , the cumulative error is 𝒪 ⁡ ( N ) ∈ o ⁡ ( N ) \mathcal{O}(\sqrt{N})\in o(N) ( Bubeck, 2015 ) .

• Non-Convex Optimization (Neural Networks): While deep networks optimize non-convex landscapes, they practically converge to local minima. Theoretically, under structural assumptions like the Polyak-Łojasiewicz (PL) condition ( Karimi et al., 2016 ) or the Neural Tangent Kernel regime ( Jacot et al., 2018 ) , SGD can achieve linear convergence rates, decaying exponentially as 𝒪 ⁡ ( e − λ ​ n ) \mathcal{O}(e^{-\lambda n}) . In such regimes, the cumulative Transient Error is bounded by a constant 𝒪 ⁡ ( 1 ) \mathcal{O}(1) , trivially satisfying the o ⁡ ( N ) o(N) condition.

#### A.4 Exact Analysis in the Quadratic Regime

We now specialize our analysis to the regime of quadratic loss functions. This setting allows for closed-form derivations of the learning dynamics, providing explicit expressions for both the asymptotic stability (Instability) and the convergence speed (Transient Error).

###### Assumption 17 (Quadratic Tasks) .

For every task j ∈ { 1 , … , K } j\in\{1,\dots,K\} , the loss function is defined as: ℒ j ​ ( θ ) = 1 2 ​ ( θ − θ ⋆ j ) ⊤ ​ H j ​ ( θ − θ ⋆ j ) + c j . \mathcal{L}^{j}(\theta)=\frac{1}{2}(\theta-\theta_{\star}^{j})^{\top}H_{j}(\theta-\theta_{\star}^{j})+c_{j}. (22) We assume the Hessian H j ≻ 0 H_{j}\succ 0 is symmetric positive definite. The scalar c j c_{j} represents the irreducible loss (or Bayes error) for task j j .

Under this assumption, the expected risk for task k k , 𝔼 θ ​ [ ℒ k ​ ( θ n ) ] \mathbb{E}_{\theta}[\mathcal{L}^{k}(\theta_{n})] , is completely determined by the first two moments of the parameter distribution: the mean μ n = 𝔼 ⁡ [ θ n ] \mu_{n}=\mathbb{E}[\theta_{n}] and the covariance Σ n = Cov ​ ( θ n ) \Sigma_{n}=\text{Cov}(\theta_{n}) : 𝔼 ⁡ [ ℒ k ​ ( θ n ) ] = 1 2 ​ ‖ μ n − θ ⋆ k ‖ H k 2 ⏟ Bias Contribution + 1 2 ​ Tr ​ ( H k ​ Σ n ) ⏟ Variance Contribution + c k . \mathbb{E}[\mathcal{L}^{k}(\theta_{n})]=\underbrace{\frac{1}{2}\|\mu_{n}-\theta_{\star}^{k}\|_{H_{k}}^{2}}_{\text{Bias Contribution}}+\underbrace{\frac{1}{2}\text{Tr}(H_{k}\Sigma_{n})}_{\text{Variance Contribution}}+c_{k}. (23)

##### A.4.1 SGD Dynamics on Quadratic Landscapes

Before analyzing the JTL/ITL trade-offs, we recall two fundamental results regarding the behavior of constant step-size SGD on quadratic objectives.

###### Lemma 18 (Convergence of the Mean) .

Consider the update θ n + 1 = θ n − η ​ g ^ n \theta_{n+1}=\theta_{n}-\eta\hat{g}_{n} where 𝔼 ⁡ [ g ^ n ∣ θ n ] = H ⁡ ( θ n − θ ⋆ ) \mathbb{E}[\hat{g}_{n}\mid\theta_{n}]=H(\theta_{n}-\theta_{\star}) . The expected parameter μ n \mu_{n} converges linearly to the optimum: μ n − θ ⋆ = ( I − η ​ H ) n ​ ( μ 0 − θ ⋆ ) . \mu_{n}-\theta_{\star}=(I-\eta H)^{n}(\mu_{0}-\theta_{\star}). (24) Convergence is guaranteed if the step-size satisfies η < 2 / λ max ​ ( H ) \eta<2/\lambda_{\max}(H) , where H H is the Hessian matrix of the objective being optimized.

###### Proof.

Taking expectations of the update rule yields μ n + 1 = μ n − η ​ H ​ ( μ n − θ ⋆ ) \mu_{n+1}=\mu_{n}-\eta H(\mu_{n}-\theta_{\star}) . Subtracting θ ⋆ \theta_{\star} from both sides gives the recurrence μ n + 1 − θ ⋆ = ( I − η ​ H ) ​ ( μ n − θ ⋆ ) \mu_{n+1}-\theta_{\star}=(I-\eta H)(\mu_{n}-\theta_{\star}) , which unrolls to the stated result. ∎

###### Lemma 19 (Parameter Covariance) .

Consider a learning process with constant step-size η \eta . The parameter covariance Σ n \Sigma_{n} evolves according to the linear recursion: Σ n + 1 = ( I − η ​ H ) ​ Σ n ​ ( I − η ​ H ) ⊤ + η 2 ​ C n , \Sigma_{n+1}=(I-\eta H)\Sigma_{n}(I-\eta H)^{\top}+\eta^{2}C_{n}, where C n = Cov ​ ( g ^ n ) C_{n}=\text{Cov}(\hat{g}_{n}) is the covariance of the batch gradient. For small step-sizes, the stationary covariance matrix Σ ∞ \Sigma_{\infty} satisfies the continuous-time Lyapunov equation: H ​ Σ ∞ + Σ ∞ ​ H = η ​ C ∞ . H\Sigma_{\infty}+\Sigma_{\infty}H=\eta C_{\infty}.

###### Proof.