To Retain or to Adapt? Generalizing Continual Learning — part 13/15
Cleaned-text code points [185676, 201368). Bibliography omitted here; source.md includes it.
To determine the optimal window size, we minimize the total asymptotic risk contributed by the Window agent. This risk is composed of the diversity cost (the Instability), which grows with W W , and the Transient Error cost, which shrinks with W W : W ⋆ = arg min W [ 𝔼 [ ℐ W ] ⏟ Instability Cost + 𝔼 [ δ W ] ⏟ Transient Cost ] . W^{\star}=\arg\min_{W}\left[\underbrace{\mathbb{E}[\mathcal{I}^{W}]}_{\text{Instability Cost}}+\underbrace{\mathbb{E}[\delta^{W}]}_{\text{Transient Cost}}\right]. (119) Substituting the derived expressions and extracting the explicit dependence on W W : W ⋆ = arg min W [ W ⋅ ( Tr ( Σ x Σ Δ ) 6 + η Tr ( Σ x 2 Σ Δ ) 24 B ) ⏟ ℛ ℬ + 1 W ⋅ γ max c μ 2 σ Δ 2 τ N ⏟ ℛ 𝒯 ] . W^{\star}=\arg\min_{W}\left[W\cdot\underbrace{\left(\frac{\text{Tr}(\Sigma_{x}\Sigma_{\Delta})}{6}+\frac{\eta\text{Tr}(\Sigma_{x}^{2}\Sigma_{\Delta})}{24B}\right)}_{\mathcal{R_{B}}}+\frac{1}{W}\cdot\underbrace{\frac{\gamma_{\max}c_{\mu}^{2}\sigma^{2}_{\Delta}}{\tau N}}_{\mathcal{R_{T}}}\right]. (120) The objective function is of the form: f ( W ) = ℛ ℬ W + ℛ 𝒯 W − 1 , f(W)=\mathcal{R_{B}}W+\mathcal{R_{T}}W^{-1}, which is strictly convex. The minimum is achieved at the square-root balance: W ⋆ = ℛ 𝒯 / ℛ ℬ . W^{\star}=\sqrt{\mathcal{R_{T}}/\mathcal{R_{B}}}.
Written explicitly, the optimal window length is: W ⋆ \displaystyle W^{\star} = γ max c μ 2 σ Δ 2 / ( τ N ) Tr ( Σ x Σ Δ ) 6 + η Tr ( Σ x 2 Σ Δ ) 24 B \displaystyle=\sqrt{\frac{\gamma_{\max}c_{\mu}^{2}\sigma^{2}_{\Delta}/(\tau N)}{\frac{\text{Tr}(\Sigma_{x}\Sigma_{\Delta})}{6}+\frac{\eta\text{Tr}(\Sigma_{x}^{2}\Sigma_{\Delta})}{24B}}} (121) = c μ γ max τ N ( Tr ( Σ x Σ Δ ) 6 σ Δ 2 + η Tr ( Σ x 2 Σ Δ ) 24 B σ Δ 2 ) − 1 / 2 . \displaystyle=\frac{c_{\mu}\,\sqrt{{\gamma_{\max}}}}{\sqrt{\tau N}}\left(\frac{\text{Tr}(\Sigma_{x}\Sigma_{\Delta})}{6\,\sigma^{2}_{\Delta}}+\frac{\eta\text{Tr}(\Sigma_{x}^{2}\Sigma_{\Delta})}{24B\,\sigma^{2}_{\Delta}}\right)^{-1/2}. (122) This exact, non-asymptotic result perfectly aligns with the asymptotic scaling derived in Lemma 36 , confirming that W ⋆ ∝ N − 1 / 2 W^{\star}\propto N^{-1/2} .
### Appendix B Empirical Setup
#### B.1 Simulations
All simulations are implemented in JAX to heavily vectorize training over multiple random seeds and hyperparameter grids, ensuring statistically robust results.
##### B.1.1 Shallow Linear Experiments
To validate our exact analytical derivations for the continuous drift setting, we simulate a sequence of linear tasks tracking a random walk. The input dimension is d = 10 d=10 . Data inputs are sampled as x ∼ 𝒩 ( 0 , I ) x\sim\mathcal{N}(0,I) , and task labels are generated by a teacher θ ⋆ k \theta_{\star}^{k} as y = x ⊤ θ ⋆ k + ξ y=x^{\top}\theta_{\star}^{k}+\xi , where ξ ∼ 𝒩 ( 0 , σ ξ 2 ) \xi\sim\mathcal{N}(0,\sigma_{\xi}^{2}) . The teacher undergoes a random walk such that θ ⋆ k = θ ⋆ k − 1 + Δ k \theta_{\star}^{k}=\theta_{\star}^{k-1}+\Delta_{k} , with Δ k ∼ 𝒩 ( 0 , σ Δ 2 d I ) \Delta_{k}\sim\mathcal{N}(0,\frac{\sigma_{\Delta}^{2}}{d}I) .
We sweep over the two main axes of our phase diagram: • Task Duration ( N N ): Swept geometrically.
• Drift Magnitude ( σ Δ 2 \sigma_{\Delta}^{2} ): Swept logarithmically.
The models are trained using stochastic gradient descent (SGD) with a constant learning rate (e.g., η = 0.1 \eta=0.1 ). We compute the exact Transfer Efficiency between the Independent-Task Learning (ITL) and Joint-Task Learning (JTL) agents averaged over K K tasks and 40 40 random seeds, and overlay the theoretical breakeven boundary directly onto the empirical phase transitions to verify our non-asymptotic bounds.
##### B.1.2 Deep Linear Experiments
To analyze the scaling dynamics and the impact of the loss landscape, we extend the experiments to deep linear networks. We use a teacher-student setup with a depth-3 linear network, an input dimension of d = 10 d=10 , and an output dimension of 1 1 . The hidden width is swept across m ∈ { 32 , 64 , 128 , 256 , 512 , 1024 } m\in\{32,64,128,256,512,1024\} .
The teacher network’s weights are initialized randomly and undergo a continuous random walk to induce task drift. We fix the total drift magnitude to σ Δ 2 = 0.05 \sigma_{\Delta}^{2}=0.05 . Only three tasks are trained to isolate the initial transient and instability dynamics, with each task lasting for exactly N = 500 N=500 steps. We use a batch size of B = 32 B=32 and a label noise standard deviation of σ ξ = 0.01 \sigma_{\xi}=0.01 .
The Independent-Task Learning (ITL) agent simply continues training on the incoming data stream, while the Joint-Task Learning (JTL) agent utilizes exact experience replay, uniformly sampling its batches from the current and all previously seen tasks.
We evaluate two different initialization and scaling regimes. Let the hidden width be denoted by m m , and let the three trainable matrices be U ∈ ℝ d × m U\in\mathbb{R}^{d\times m} , V ∈ ℝ m × m V\in\mathbb{R}^{m\times m} , and a ∈ ℝ m × 1 a\in\mathbb{R}^{m\times 1} , with all entries initialized i.i.d. as 𝒩 ( 0 , 1 ) \mathcal{N}(0,1) . For an input x ∈ ℝ d x\in\mathbb{R}^{d} , the depth-3 linear network is implemented as h 1 ( x ) = x U d , h 2 ( x ) = h 1 ( x ) V m , f Θ ( x ) = h 2 ( x ) a m γ m . h_{1}(x)=\frac{xU}{\sqrt{d}},\qquad h_{2}(x)=\frac{h_{1}(x)V}{\sqrt{m}},\qquad f_{\Theta}(x)=\frac{h_{2}(x)a}{\sqrt{m}\,\gamma_{m}}. Thus the input and hidden matrix multiplications use fan-in normalization, while the scalar factor γ m \gamma_{m} controls the output scale of the parametrization. The same parametrization is used for the teacher and the student networks.
The two regimes differ only in the output multiplier γ m \gamma_{m} and the learning-rate multiplier s η ( m ) s_{\eta}(m) applied to the base learning rate: • NTK parametrization (NTP). We set γ m = 1 \gamma_{m}=1 and s η ( m ) = 1 s_{\eta}(m)=1 , so the effective SGD learning rate is independent of width. This is the neural-tangent scaling used to approach lazy kernel dynamics in the wide-network limit: as m m grows, the network function changes mainly through the linearization around initialization, and the hidden representation moves by a vanishing amount ( Jacot et al., 2018 ; Yang and Hu, 2021 ) .
• Maximal Update parametrization ( μ \mu P). We set γ m = m \gamma_{m}=\sqrt{m} and multiply the base learning rate by s η ( m ) = m / m 0 s_{\eta}(m)=m/m_{0} , where the base width is m 0 = 32 m_{0}=32 . Equivalently, compared with the NTK/NTP run, the output is scaled down by an additional factor m \sqrt{m} while the optimizer step size is scaled up linearly with width. This is the maximal-update parametrization: under width scaling, hidden-layer updates remain large enough to produce feature learning, and learning-rate choices transfer more stably across widths ( Yang and Hu, 2021 ; Yang et al., 2021 ) .
We sweep the base learning rate logarithmically from 10 − 3 10^{-3} to 10 1 10^{1} across 15 15 values. We observe hyperparameter transfer in μ \mu P as expected, meaning the optimal learning rate remains stable across widths, whereas it shifts significantly in NTP. To ensure a fair comparison, the learning rate for the ITL agent is chosen based on the first task’s performance, while the learning rate for the JTL agent is chosen based on the last task’s performance (lowest loss). All grid points are averaged over 5 5 random seeds.
The loss curves with the best selected learning rate are depicted in the figures below.
##### B.1.3 Window Predictive CL Algorithm Experiments
To empirically validate our non-asymptotic derivations for the Window Predictive CL algorithm ( Section A.7 ), we implement the Window agent in the shallow linear continuous drift environment. The Window agent optimizes a uniformly weighted mixture of the past W W tasks, 𝒟 W k = 1 W ∑ j = k − W + 1 k 𝒫 j \mathcal{D}^{k}_{W}=\frac{1}{W}\sum_{j=k-W+1}^{k}\mathcal{P}^{j} .
Our experiments for the Window agent are divided into two primary evaluations:
###### Real-Time Trajectory Evaluation with W ⋆ W^{\star} :
First, we test the predictive power of our theoretical optimal window formula ( Equation 121 ). For a given task duration N N , we compute the theoretical optimum W ⋆ W^{\star} using the environment and optimizer parameters (e.g., drift magnitude σ Δ 2 \sigma_{\Delta}^{2} , learning rate η \eta , batch size B B , and input covariance Σ x \Sigma_{x} ). We then simulate the full training trajectory of the Independent-Task Learning (ITL) agent, the Joint-Task Learning (JTL) agent, and the Window agent explicitly parameterized with W = W ⋆ W=W^{\star} . We average the test loss over multiple seeds and smooth the curves by averaging over the task duration N N to clearly visualize the transient and stationary phases of adaptation.
###### Transfer Efficiency Landscapes:
Second, we rigorously map the space of positive transfer to show how the environment’s non-stationarity limits the optimal memory horizon. We fix the task duration to distinct scales (e.g., N ∈ { 100,500 , 1000 } N\in\{100,500,1000\} ) and perform a 2D grid search over: • Window Size ( W W ): Swept across a discrete range of integer values.
• Drift Magnitude ( σ Δ 2 \sigma_{\Delta}^{2} ): Swept logarithmically.
For every point in this grid, we compute the exact Average Task Error ( ALE k \mathrm{ALE}^{k} ) over a long sequence of tasks (e.g., K = 100 K=100 ) to bypass initial task transient effects. We then calculate the Transfer Efficiency (TE) relative to the ITL baseline. This generates the 2D heatmaps (shown in Figure 15 ), which empirically verify our claims that (1) the area of positive transfer shrinks as task duration and drift increase, and (2) overly large window sizes introduce detrimental bias in highly non-stationary regimes, pulling the optimal W W toward 1 1 .
#### B.2 Real-World Benchmarks
##### B.2.1 CLEAR
The CLEAR dataset ( Lin et al., 2021 ) is a collection of images spanning the years 2004–2014, designed to evaluate models under natural temporal distribution shifts. We utilize the 100-class variant and split the collection into K = 10 K=10 distinct tasks, one for each year. The tasks are organized in their natural temporal ordering. All input images are resized to 224 × \times 224 pixels and normalized by subtracting the ImageNet mean μ = [ 0.485 , 0.456 , 0.406 ] \mu=[0.485,0.456,0.406] and dividing by the standard deviation σ = [ 0.229 , 0.224 , 0.225 ] \sigma=[0.229,0.224,0.225] .
##### B.2.2 MULTIDATASET (MD5)
The MULTIDATASET benchmark consists of a sequence of K = 5 K=5 distinct open-source classification datasets with no semantic overlap. Specifically, the tasks consist of classifying automobile models ( Krause et al., 2013 ) , aircraft models ( Maji et al., 2013 ) , textures ( Cimpoi et al., 2014 ) , dishes ( Bossard et al., 2014 ) , and pets ( Parkhi et al., 2012 ) . Each dataset originally features a different number of classes, samples, and input resolutions (detailed in Table 7 ).
To prevent the introduction of architectural biases or capacity imbalances, we standardize all tasks to have exactly 30 classes. Furthermore, we enforce identical batch sizes and total update steps across tasks, regardless of the original dataset’s size. All input images are resized to 224 × \times 224 pixels and normalized using the standard ImageNet statistics ( μ = [ 0.485 , 0.456 , 0.406 ] \mu=[0.485,0.456,0.406] , σ = [ 0.229 , 0.224 , 0.225 ] \sigma=[0.229,0.224,0.225] ). During training, the data is augmented with random crops, random horizontal flips, and random rotations of up to 15 degrees.
##### B.2.3 Meta-World
Meta-World ( Yu et al., 2020b ) is a suite of simulated robotic manipulation tasks built on the MuJoCo physics engine. Our experiments use the MT10 benchmark, a fixed collection of 10 10 distinct manipulation tasks ( reach , push , pick-place , door-open , drawer-open , drawer-close , button-press-topdown , peg-insert-side , window-open , window-close ) on the shared Sawyer robot. We use the v3 environment versions provided by the official Meta-World release. Unlike the meta-learning split (ML10), MT10 is designed for multi-task and continual evaluation on a common task set, with no held-out tasks.
At each step, the environment yields a 39 39 -dimensional observation vector that includes the robot’s joint positions, the gripper state, and the spatial coordinates of relevant objects, with the goal position zero-masked. For the JTL agent, this observation is concatenated with a 10 10 -dimensional one-hot task identifier, yielding a 49 49 -dimensional input. The agent emits a 4 4 -dimensional continuous action (end-effector delta plus gripper command), with components squashed to [ − 1 , 1 ] [-1,1] via a tanh \tanh . Each episode lasts at most 500 500 time steps and yields a binary success flag in addition to the shaped reward.
#### B.3 Synthetic Benchmarks
##### B.3.1 Permuted CIFAR-10
Permuted CIFAR-10 is a benchmark constructed from the standard CIFAR-10 dataset ( Krizhevsky and Hinton, 2009 ) by applying fixed random spatial permutations to the images. We utilize two different permutation severities in our experiments, defined by the spatial dimensions of the perturbed area: 16 × \times 16 and 32 × \times 32. These denote the side length of the square patch of pixels—centered in the image—that undergoes the random permutation (see Figure 19 and Figure 19 ). We refer to these respective benchmarks as ‘PC-16’ and ‘PC-32’. All input images are normalized by subtracting the mean μ = [ 0.507 , 0.486 , 0.441 ] \mu=[0.507,0.486,0.441] and dividing by σ = [ 0.267 , 0.256 , 0.276 ] \sigma=[0.267,0.256,0.276] .
##### B.3.2 Shuffled CIFAR-10
Shuffled CIFAR-10 is an alternative benchmark built from the CIFAR-10 dataset ( Krizhevsky and Hinton, 2009 ) , which isolates the effect of concept drift by applying targeted label resampling to a set of samples fixed at the beginning of training . We sweep the proportion of shuffled labels across three severities: 10%, 30%, and 50%. To guarantee that this injected label noise is strictly adversarial, we prevent the true label from being drawn during the reshuffling process. We refer to these benchmarks as ‘SC-10’, ‘SC-30’, and ‘SC-50’. Input images are normalized using the same statistics as Permuted CIFAR-10. Crucially, because the labels are irreversibly corrupted in the training set, we evaluate the resulting generalization penalty on the uncorrupted test set.
#### B.4 Training Procedures & Networks
###### Supervised Learning
All supervised learning agents consist of a deep neural network backbone, an optimizer, and a learning rate scheduler. Across all supervised experiments, the core architecture is a ResNet-18 ( RN18 ). The output dimension of the final linear classification head is set to the total number of classes in the respective benchmark (100 for CLEAR, 30 for MD5, and 10 for Permuted/Shuffled CIFAR-10), and this head is shared across all tasks. Both the Independent-Task Learning (ITL) and Joint-Task Learning (JTL) agents are trained for an identical number of gradient steps h h per task. General optimization configurations are detailed in Table 8 , with explicitly tuned hyperparameters listed in Table 9 .
ITL Agent. Given a sequence of K K tasks, the ITL agent is trained to minimize the cross-entropy loss exclusively on the training data of the current active task. To establish a strict ITL baseline and prevent any forward transfer, both the network weights and the optimizer state of the ITL agent are completely re-initialized to their random starting priors at the beginning of every task.