To Retain or to Adapt? Generalizing Continual Learning — part 8/15

Cleaned-text code points [109369, 125351). Bibliography omitted here; source.md includes it.
Consider an SGD agent characterized by its deviation from stationarity: the mean displacement μ ~ n = μ n − μ ∞ \tilde{\mu}_{n}=\mu_{n}-\mu_{\infty} and the covariance deviation Σ ~ n = Σ n − Σ ∞ \tilde{\Sigma}_{n}=\Sigma_{n}-\Sigma_{\infty} . Let δ N \delta_{N} be the transient cost accumulated over a horizon N N . Under Assumption 22 , and for a sufficiently small step-size η \eta , δ N \delta_{N} is proportional to the drop in the weighted potential energy of the system from start to finish: δ N ≈ 1 N  τ  ( ℰ ⁡ ( μ ~ 0 , Σ ~ 0 ) − ℰ ⁡ ( μ ~ N , Σ ~ N ) ) , \delta_{N}\approx\frac{1}{N\tau}\left(\mathcal{E}(\tilde{\mu}_{0},\tilde{\Sigma}_{0})-\mathcal{E}(\tilde{\mu}_{N},\tilde{\Sigma}_{N})\right), (35) where τ = 4  η \tau=4\eta is the learning timescale. The potential energy function ℰ \mathcal{E} is defined by the Relative Curvature ℛ \mathcal{R} as: ℰ ⁡ ( μ ~ , Σ ~ ) = μ ~ ⊤  ℛ  μ ~ ⏟ Mean Potential + Tr  ( Σ ~  ℛ ) ⏟ Covariance Potential . \mathcal{E}(\tilde{\mu},\tilde{\Sigma})=\underbrace{\tilde{\mu}^{\top}\mathcal{R}\tilde{\mu}}_{\text{Mean Potential}}+\underbrace{\text{Tr}(\tilde{\Sigma}\mathcal{R})}_{\text{Covariance Potential}}. (36)

###### Geometric Interpretation.

This result provides a powerful physical intuition for the cost of learning: it is simply the difference between where the agent started and where it ended, measured on a quadratic potential surface warped by ℛ \mathcal{R} . This curvature takes different forms depending on the optimization strategy: • ITL: Here, H dyn = H eval H_{\text{dyn}}=H_{\text{eval}} , meaning ℛ = I \mathcal{R}=I . The potential is the standard Euclidean squared distance, and the transient cost is strictly proportional to the physical distance traveled.

• JTL: Here, ℛ ≠ I \mathcal{R}\neq I . The potential surface is stretched according to the task mixture. Directions where the mixture is steeper than the target task (eigenvalues of ℛ < 1 \mathcal{R}<1 ) correspond to discounted costs. Essentially, training on a diverse mixture acts as an implicit pre-conditioner, allowing the agent to "fall" into the target task’s basin more rapidly along those axes.

###### Proof.

The proof proceeds in four steps: establishing the dynamics of the second moment, summing the excess risk, simplifying the resulting geometric series, and relating the sum back to the state at step N N . First, recall that: δ N := 1 N  ∑ n = 0 N − 1 𝔼 ⁡ [ ℒ ⁡ ( θ n ) − ℒ ⁡ ( θ ⋆ ) ] . \delta_{N}:=\frac{1}{N}\sum_{n=0}^{N-1}\mathbb{E}\left[\mathcal{L}(\theta_{n})-\mathcal{L}(\theta_{\star})\right]. (37)

Step 1. Let θ ~ n = θ n − μ ∞ \tilde{\theta}_{n}=\theta_{n}-\mu_{\infty} be the error vector relative to the stationary mean. The dynamics are governed by the driving Hessian H dyn H_{\text{dyn}} . The SGD update implies that the second moment matrix S n = 𝔼 ⁡ [ θ ~ n  θ ~ n ⊤ ] S_{n}=\mathbb{E}[\tilde{\theta}_{n}\tilde{\theta}_{n}^{\top}] evolves according to the Lyapunov recursion ( Lemma 19 ): S n + 1 = ( I − η  H dyn )  S n  ( I − η  H dyn ) + η 2  C n , S_{n+1}=(I-\eta H_{\text{dyn}})S_{n}(I-\eta H_{\text{dyn}})+\eta^{2}C_{n}, (38) where C n C_{n} is the gradient noise covariance. The stationary covariance Σ ∞ \Sigma_{\infty} is the fixed point of this equation. To isolate the transient component, we consider the deviation Δ n = S n − Σ ∞ \Delta_{n}=S_{n}-\Sigma_{\infty} . Let C ∞ C_{\infty} denote the gradient noise covariance at stationarity. Then: Δ n + 1 = ( I − η  H dyn )  Δ n  ( I − η  H dyn ) + η 2  ( C n − C ∞ ) . \Delta_{n+1}=(I-\eta H_{\text{dyn}})\Delta_{n}(I-\eta H_{\text{dyn}})+\eta^{2}(C_{n}-C_{\infty}).

For the ITL agent, C n − C ∞ ≈ 0 C_{n}-C_{\infty}\approx 0 throughout training. For the JTL agent, C n − C ∞ ≈ 1 B  𝔼 j  [ H j  S ¯ n  H j ] C_{n}-C_{\infty}\approx\frac{1}{B}\mathbb{E}_{j}[H_{j}\bar{S}_{n}H_{j}] , where S ¯ n = ( μ n − θ ⋆ JTL , k )  ( μ n − θ ⋆ JTL , k ) ⊤ \bar{S}_{n}=(\mu_{n}-\theta_{\star}^{\text{JTL},k})(\mu_{n}-\theta_{\star}^{\text{JTL},k})^{\top} . Because S ¯ n \bar{S}_{n} is 𝒪 ⁡ ( 1 ) \mathcal{O}(1) at initialization and decays to zero, the source term η 2  ( C n − C ∞ ) \eta^{2}(C_{n}-C_{\infty}) is of order 𝒪 ⁡ ( η 2 ) \mathcal{O}(\eta^{2}) , whereas the homogeneous part of the deviation Δ n \Delta_{n} is 𝒪 ⁡ ( 1 ) \mathcal{O}(1) . Therefore, in the small step-size limit, the transient trajectory is dominated by the geometric decay: Δ n ≈ ( I − η  H dyn ) n  Δ 0  ( I − η  H dyn ) n . \Delta_{n}\approx(I-\eta H_{\text{dyn}})^{n}\Delta_{0}(I-\eta H_{\text{dyn}})^{n}. (39)

Step 2. Because the risk is determined by the evaluation Hessian H eval H_{\text{eval}} , the expected excess risk at step n n is 1 2  Tr  ( H eval  Δ n ) \frac{1}{2}\text{Tr}(H_{\text{eval}}\Delta_{n}) . Summing over the horizon yields δ N \delta_{N} : δ N = 1 2  N  ∑ n = 0 N − 1 Tr  ( H eval  Δ n ) . \delta_{N}=\frac{1}{2N}\sum_{n=0}^{N-1}\text{Tr}(H_{\text{eval}}\Delta_{n}). (40) Substituting the propagator form and utilizing the commutativity of H eval H_{\text{eval}} and H dyn H_{\text{dyn}} ( Assumption 22 ): δ N ≈ 1 2  N  Tr  ( Δ 0  H eval  ∑ n = 0 N − 1 ( I − η  H dyn ) 2  n ) . \delta_{N}\approx\frac{1}{2N}\text{Tr}\left(\Delta_{0}H_{\text{eval}}\sum_{n=0}^{N-1}(I-\eta H_{\text{dyn}})^{2n}\right). (41)

Step 3. The matrix geometric series ∑ n = 0 N − 1 A n \sum_{n=0}^{N-1}A^{n} with A = ( I − η  H dyn ) 2 A=(I-\eta H_{\text{dyn}})^{2} is ( I − A ) − 1  ( I − A N ) (I-A)^{-1}(I-A^{N}) . Expanding for small η \eta : I − ( I − η  H dyn ) 2 = 2  η  H dyn − η 2  H dyn 2 ≈ 2  η  H dyn . I-(I-\eta H_{\text{dyn}})^{2}=2\eta H_{\text{dyn}}-\eta^{2}H_{\text{dyn}}^{2}\approx 2\eta H_{\text{dyn}}. (42) Thus, ( I − A ) − 1 ≈ ( 2  η  H dyn ) − 1 (I-A)^{-1}\approx(2\eta H_{\text{dyn}})^{-1} . Introducing the Relative Curvature ℛ = H dyn − 1  H eval \mathcal{R}=H_{\text{dyn}}^{-1}H_{\text{eval}} : δ N ≈ 1 4  N  η  Tr  ( Δ 0  ℛ  [ I − ( I − η  H dyn ) 2  N ] ) . \delta_{N}\approx\frac{1}{4N\eta}\text{Tr}\left(\Delta_{0}\mathcal{R}\left[I-(I-\eta H_{\text{dyn}})^{2N}\right]\right). (43)

Step 4. Distributing the trace and using the commutativity of ℛ \mathcal{R} with the propagator: Tr  ( Δ 0  ℛ  ( I − η  H dyn ) 2  N ) = Tr  ( ℛ  ( I − η  H dyn ) N  Δ 0  ( I − η  H dyn ) N ) = Tr  ( Δ N  ℛ ) . \text{Tr}\left(\Delta_{0}\mathcal{R}(I-\eta H_{\text{dyn}})^{2N}\right)=\text{Tr}\left(\mathcal{R}(I-\eta H_{\text{dyn}})^{N}\Delta_{0}(I-\eta H_{\text{dyn}})^{N}\right)=\text{Tr}(\Delta_{N}\mathcal{R}). (44) This yields the potential energy conservation law: δ N ≈ 1 4  N  η  ( Tr  ( Δ 0  ℛ ) − Tr  ( Δ N  ℛ ) ) . \delta_{N}\approx\frac{1}{4N\eta}\left(\text{Tr}(\Delta_{0}\mathcal{R})-\text{Tr}(\Delta_{N}\mathcal{R})\right). (45) Finally, decomposing the deviation into mean and covariance components, Δ n = μ ~ n  μ ~ n ⊤ + Σ ~ n \Delta_{n}=\tilde{\mu}_{n}\tilde{\mu}_{n}^{\top}+\tilde{\Sigma}_{n} , and using Tr  ( x  x ⊤  ℛ ) = ‖ x ‖ ℛ 2 \text{Tr}(xx^{\top}\mathcal{R})=\|x\|^{2}_{\mathcal{R}} , we obtain: δ N ≈ 1 4  N  η  [ ( ‖ μ ~ 0 ‖ ℛ 2 + Tr  ( Σ ~ 0  ℛ ) ) − ( ‖ μ ~ N ‖ ℛ 2 + Tr  ( Σ ~ N  ℛ ) ) ] . \delta_{N}\approx\frac{1}{4N\eta}\left[\left(\|\tilde{\mu}_{0}\|^{2}_{\mathcal{R}}+\text{Tr}(\tilde{\Sigma}_{0}\mathcal{R})\right)-\left(\|\tilde{\mu}_{N}\|^{2}_{\mathcal{R}}+\text{Tr}(\tilde{\Sigma}_{N}\mathcal{R})\right)\right]. (46) ∎

#### A.5 Asymptotic Dynamics of the First and Second Order Moments

In Section A.4 , we derived closed-form expressions for the instability and transient costs by taking expectations over the stochasticity of the agent’s parameter trajectory. To analyze the asymptotic stability of the Joint-Task Learning (JTL) and Independent-Task Learning (ITL) agents as the number of tasks grows, we must now account for the randomness in the environment itself. Therefore, in this section, unless otherwise specified, the expectation operator 𝔼 ⁡ [ ⋅ ] \mathbb{E}[\cdot] is taken over the task generation process (i.e., the distribution of the task sequence).

To formalize how "concentrated" or "spread out" the tasks are, we introduce the notion of a Growth Rate ψ ⁡ ( k ) \psi(k) , which characterizes the asymptotic spread of the task optima as a function of the number of tasks k k .

###### Definition 26 (Growth Rate) .

Let ψ : ℕ → ℝ + \psi:\mathbb{N}\to\mathbb{R}^{+} be a monotonically increasing function such that the expected squared distance of the k k -th task optimum from the origin scales as: 𝔼 ⁡ [ ‖ θ ⋆ k ‖ 2 ] ∈ Θ ⁡ ( ψ ⁡ ( k ) ) . \mathbb{E}[\|\theta_{\star}^{k}\|^{2}]\in\Theta(\psi(k)). (47)

In the analysis that follows, we consider three distinct regimes of environment growth: Bounded ( ψ ⁡ ( k ) = 1 \psi(k)=1 ), Diffusive ( ψ ⁡ ( k ) = k \psi(k)=k ), and Linear ( ψ ⁡ ( k ) = k 2 \psi(k)=k^{2} ).

##### A.5.1 Geometric Drift

As the agent incorporates new tasks, the stationary targets of the mixture shift. We define the Geometric Drift of the first and second-order moments as the step-wise difference in the asymptotic JTL targets: Δ  μ k = θ ⋆ JTL , k − 1 − θ ⋆ JTL , k , Δ  Σ k = Σ ∞ JTL , k − 1 − Σ ∞ JTL , k . \displaystyle\Delta\mu^{k}=\theta_{\star}^{\text{JTL},k-1}-\theta_{\star}^{\text{JTL},k},\qquad\Delta\Sigma^{k}=\Sigma_{\infty}^{\text{JTL},k-1}-\Sigma_{\infty}^{\text{JTL},k}. (48) We proceed to characterize the asymptotic behavior of this drift depending on the growth rate function. Table 4 summarizes the convergence properties for each regime.

###### Lemma 27 (Geometric Mean Drift) .

If all tasks have finite training durations, the expected squared magnitude of the geometric mean shift scales as: 𝔼 ⁡ [ ‖ Δ  μ k ‖ 2 ] ∈ 𝒪 ⁡ ( ψ ⁡ ( k ) k 2 ) . \mathbb{E}[\|\Delta{\mu}^{k}\|^{2}]\in\mathcal{O}\left(\frac{\psi(k)}{k^{2}}\right). (49)

Implication: The mean transient vanishes asymptotically as long as the environment grows strictly slower than a linear trajectory ( ψ ⁡ ( k ) ≪ k 2 \psi(k)\ll k^{2} ).

###### Proof.

Let N j N_{j} denote the number of training steps for task j j , and b k = ∑ j = 1 k N j b_{k}=\sum_{j=1}^{k}N_{j} be the total training duration up to task k k . The effective weight of task j j in the mixture at stage k k is w j k = N j / b k w_{j}^{k}=N_{j}/b_{k} . Let α k = w k k := N k / b k \alpha_{k}=w_{k}^{k}:=N_{k}/b_{k} be the relative contribution of the newest task. If task durations are finite and bounded, α k ≈ 1 / k \alpha_{k}\approx 1/k asymptotically.

Recall the mixture optimum θ ⋆ JTL , k = ( H ¯ k ) − 1  c k \theta_{\star}^{\text{JTL},k}=(\bar{H}_{k})^{-1}c_{k} , where c k = ∑ j = 1 k w j k  H j  θ ⋆ j c_{k}=\sum_{j=1}^{k}w_{j}^{k}H_{j}\theta_{\star}^{j} . The recursive updates for the cumulative Hessian H ¯ k \bar{H}_{k} and target vector c k c_{k} are: H ¯ k \displaystyle\bar{H}_{k} = ( 1 − α k )  H ¯ k − 1 + α k  H k , \displaystyle=(1-\alpha_{k})\bar{H}_{k-1}+\alpha_{k}H_{k}, (50) c k \displaystyle c_{k} = ( 1 − α k )  c k − 1 + α k  H k  θ ⋆ k . \displaystyle=(1-\alpha_{k})c_{k-1}+\alpha_{k}H_{k}\theta_{\star}^{k}. (51) Substituting c k − 1 = H ¯ k − 1  θ ⋆ JTL , k − 1 c_{k-1}=\bar{H}_{k-1}\theta_{\star}^{\text{JTL},k-1} into the second equation: H ¯ k  θ ⋆ JTL , k = ( 1 − α k )  H ¯ k − 1  θ ⋆ JTL , k − 1 + α k  H k  θ ⋆ k . \bar{H}_{k}\theta_{\star}^{\text{JTL},k}=(1-\alpha_{k})\bar{H}_{k-1}\theta_{\star}^{\text{JTL},k-1}+\alpha_{k}H_{k}\theta_{\star}^{k}. (52) From the Hessian update, we write ( 1 − α k )  H ¯ k − 1 = H ¯ k − α k  H k (1-\alpha_{k})\bar{H}_{k-1}=\bar{H}_{k}-\alpha_{k}H_{k} . Substituting this back yields: H ¯ k  θ ⋆ JTL , k = ( H ¯ k − α k  H k )  θ ⋆ JTL , k − 1 + α k  H k  θ ⋆ k . \bar{H}_{k}\theta_{\star}^{\text{JTL},k}=(\bar{H}_{k}-\alpha_{k}H_{k})\theta_{\star}^{\text{JTL},k-1}+\alpha_{k}H_{k}\theta_{\star}^{k}. (53) Multiplying by H ¯ k − 1 \bar{H}_{k}^{-1} (which exists and commutes with H k H_{k} by Assumption 22 ): θ ⋆ JTL , k = θ ⋆ JTL , k − 1 + α k  H ¯ k − 1  H k  ( θ ⋆ k − θ ⋆ JTL , k − 1 ) . \theta_{\star}^{\text{JTL},k}=\theta_{\star}^{\text{JTL},k-1}+\alpha_{k}\bar{H}_{k}^{-1}H_{k}(\theta_{\star}^{k}-\theta_{\star}^{\text{JTL},k-1}). (54) The geometric drift is therefore: Δ  μ k = θ ⋆ JTL , k − 1 − θ ⋆ JTL , k = − α k  ( H ¯ k − 1  H k )  ( θ ⋆ k − θ ⋆ JTL , k − 1 ) . \Delta\mu^{k}=\theta_{\star}^{\text{JTL},k-1}-\theta_{\star}^{\text{JTL},k}=-\alpha_{k}(\bar{H}_{k}^{-1}H_{k})(\theta_{\star}^{k}-\theta_{\star}^{\text{JTL},k-1}). (55) Taking the squared norm and the expectation over the task sequence: 𝔼 ⁡ [ ‖ Δ  μ k ‖ 2 ] = α k 2  𝔼  [ ‖ ( H ¯ k − 1  H k )  ( θ ⋆ k − θ ⋆ JTL , k − 1 ) ‖ 2 ] . \mathbb{E}[\|\Delta\mu^{k}\|^{2}]=\alpha_{k}^{2}\,\mathbb{E}\left[\|(\bar{H}_{k}^{-1}H_{k})(\theta_{\star}^{k}-\theta_{\star}^{\text{JTL},k-1})\|^{2}\right]. (56) Under Assumption 22 , H ¯ k − 1  H k \bar{H}_{k}^{-1}H_{k} is a diagonal matrix of eigenvalue ratios. Let λ max \lambda_{\max} and λ min \lambda_{\min} bound the eigenvalues of all tasks. The spectral norm ‖ H ¯ k − 1  H k ‖ 2 \|\bar{H}_{k}^{-1}H_{k}\|_{2} is bounded by a constant κ ≈ λ max / λ min \kappa\approx\lambda_{\max}/\lambda_{\min} . Thus: 𝔼 ⁡ [ ‖ Δ  μ k ‖ 2 ] ≤ α k 2  κ 2  𝔼  [ ‖ θ ⋆ k − θ ⋆ JTL , k − 1 ‖ 2 ] . \mathbb{E}[\|\Delta\mu^{k}\|^{2}]\leq\alpha_{k}^{2}\kappa^{2}\mathbb{E}\left[\|\theta_{\star}^{k}-\theta_{\star}^{\text{JTL},k-1}\|^{2}\right]. (57) The difference θ ⋆ k − θ ⋆ JTL , k − 1 \theta_{\star}^{k}-\theta_{\star}^{\text{JTL},k-1} represents the deviation of the new task from the historical centroid. In expectation, the squared norm of this difference scales with the maximum variance of the task distribution, which is Θ ⁡ ( ψ ⁡ ( k ) ) \Theta(\psi(k)) . Because α k ∈ Θ ⁡ ( 1 / k ) \alpha_{k}\in\Theta(1/k) , we conclude: 𝔼 ⁡ [ ‖ Δ  μ k ‖ 2 ] ∈ 𝒪 ⁡ ( ψ ⁡ ( k ) k 2 ) . \mathbb{E}[\|\Delta{\mu}^{k}\|^{2}]\in\mathcal{O}\left(\frac{\psi(k)}{k^{2}}\right). (58) ∎

###### Lemma 28 (Geometric Covariance Drift) .

The expected magnitude of the geometric covariance shift scales with the finite difference (or derivative) of the growth function: 𝔼 ⁡ [ | Tr ⁡ ( Δ  Σ k ) | ] ∈ 𝒪 ⁡ ( ψ ⁡ ( k ) − ψ ⁡ ( k − 1 ) ) ≈ 𝒪 ⁡ ( ψ ′  ( k ) ) . \mathbb{E}[|\mathrm{Tr}(\Delta\Sigma^{k})|]\in\mathcal{O}(\psi(k)-\psi(k-1))\approx\mathcal{O}(\psi^{\prime}(k)). (59)

Implication: The covariance transient vanishes only if the environment grows sub-linearly.

###### Proof.

By Lemma 19 , the stationary covariance Σ ∞ k \Sigma_{\infty}^{k} scales linearly with the effective noise covariance C ∞ k C_{\infty}^{k} : H ¯ k  Σ ∞ k + Σ ∞ k  H ¯ k = η B  C ∞ k . \bar{H}_{k}\Sigma^{k}_{\infty}+\Sigma^{k}_{\infty}\bar{H}_{k}=\frac{\eta}{B}C^{k}_{\infty}. For a JTL objective, the noise covariance is dominated by the task diversity term ( Lemma 21 ): 𝔼 ⁡ [ 𝒱 div k ] = ∑ j = 1 k w j k  𝔼  [ ‖ ∇ ℒ j  ( θ ⋆ JTL , k ) ‖ 2 ] = ∑ j = 1 k w j k  𝔼  [ ‖ θ ⋆ j − θ ⋆ JTL , k ‖ H j 2 2 ] . \mathbb{E}[\mathcal{V}_{\text{div}}^{k}]=\sum_{j=1}^{k}w_{j}^{k}\,\,\mathbb{E}[\|\nabla\mathcal{L}^{j}(\theta_{\star}^{\text{JTL},k})\|^{2}]=\sum_{j=1}^{k}w_{j}^{k}\,\,\mathbb{E}[\|\theta_{\star}^{j}-\theta_{\star}^{\text{JTL},k}\|_{H_{j}^{2}}^{2}]. Because the Hessian spectrum is bounded ( Assumption 23 ), the expected task diversity scales directly with the spatial variance of the tasks: 𝔼 ⁡ [ 𝒱 div k ] ∈ Θ ⁡ ( ψ ⁡ ( k ) ) \mathbb{E}[\mathcal{V}_{\text{div}}^{k}]\in\Theta(\psi(k)) .

The covariance drift is the difference Δ  Σ k = Σ ∞ k − 1 − Σ ∞ k \Delta\Sigma^{k}=\Sigma_{\infty}^{k-1}-\Sigma_{\infty}^{k} . Because Σ ∞ k \Sigma_{\infty}^{k} is strictly proportional to C ∞ k C_{\infty}^{k} , the drift grows asymptotically with the derivative of the growth function: 𝔼 ⁡ [ | Tr ⁡ ( Δ  Σ k ) | ] ∝ ψ ⁡ ( k ) − ψ ⁡ ( k − 1 ) . \mathbb{E}[|\mathrm{Tr}(\Delta\Sigma^{k})|]\propto\psi(k)-\psi(k-1). Note that the sign of Tr ⁡ ( Δ  Σ k ) \mathrm{Tr}(\Delta\Sigma^{k}) might be negative, indicating the variance grows due to the addition of a new task. However, to analyze its asymptotic magnitude, we take the absolute value.