To Retain or to Adapt? Generalizing Continual Learning — part 14/15

Cleaned-text code points [201368, 216996). Bibliography omitted here; source.md includes it.

JTL Agent. The JTL agent minimizes the expected loss over the union of all currently and previously observed tasks. In practice, this is implemented using an exact experience replay buffer: at each gradient step, the optimizer processes a concatenated batch comprising data from the current task stream alongside data uniformly sampled from the buffer of previous tasks. Unlike the ITL agent, the JTL agent’s network parameters and optimizer state are preserved across task boundaries.

###### Reinforcement Learning

In our reinforcement learning experiments, we compare ITL and JTL agents using Soft Actor-Critic (SAC) ( Haarnoja et al., 2018a ) on the MT10 subset of the Meta-World benchmark ( Yu et al., 2020b ) , with automatic entropy temperature tuning ( Haarnoja et al., 2018b ) . SAC is the standard off-policy continuous-control algorithm used in the multi-task Meta-World literature ( Yu et al., 2020b ) , and its use of a single replay buffer makes the ITL and JTL agents differ only in (i) what data populates the buffer and (ii) whether the policy is conditioned on a task identifier; the underlying optimizer, network architecture, and per-epoch compute budget are otherwise identical.

Architecture (shared). For both agents, the actor and the twin critics are MLPs with two hidden layers of 400 400 units and ReLU activations. The actor outputs the mean and (clipped) log-standard-deviation of a diagonal Gaussian, squashed by tanh \tanh ; the critics output a scalar Q Q -value. The log-std is clipped to [ − 20 , 2 ] [-20,2] .

ITL Agent. For each task in MT10 we train a separate SAC agent from a random initialization, with its own replay buffer and its own learned entropy temperature α \alpha . The agent observes only the 39-dimensional Meta-World state of its task; no task identifier is appended. Each ITL run lasts 500 500 epochs; per epoch we collect 40 40 rollouts of length 500 500 ( 2 × 10 4 2\times 10^{4} environment steps) into the buffer and then perform 500 500 SAC gradient updates with mini-batch size 500 500 .

JTL Agent. We train a single SAC agent across all MT10 tasks under a curriculum : at epoch e e the active set is the first 1 + ⌊ e / N ⌋ 1+\lfloor e/N\rfloor tasks in a fixed ordering, with task duration N = 500 N=500 epochs. The actor and critics are conditioned on a 10 10 -dimensional one-hot task identifier concatenated to the observation. A single replay buffer of capacity 10 6 10^{6} is shared across all tasks and is never reset; whenever a new task becomes active we re-initialize the Adam states of the actor and critic optimizers (the entropy temperature is kept) so that the optimizer can adapt to the new objective. Per epoch we collect 200 200 rollouts of length 500 500 ( 10 5 10^{5} environment steps) by uniformly cycling over the currently active environments, then perform 500 500 SAC gradient updates with mini-batch size 500 500 sampled uniformly from the shared buffer. Evaluation each epoch is performed on the current task.

Task ordering. For the curriculum experiments shown in the main paper we use the easy_to_hard ordering, obtained by sorting MT10 tasks in decreasing order of the final single-task success rate of the ITL SAC agents under the same configuration. The resulting sequence is drawer-close , window-close , button-press-topdown , drawer-open , reach , window-open , door-open , push , peg-insert-side , pick-place . Each configuration (ITL and JTL) is replicated over 10 10 seeds; per-task figures aggregate seeds for which the entire 500 500 -epoch stage is logged, with a minimum of 7 7 seeds per relative epoch.

##### B.4.1 Hyperparameters

###### Supervised Learning

The key hyperparameters—learning rate and weight decay—are carefully tuned separately for each combination of agent and benchmark using a grid search. To ensure a fair comparison, the optimizer and scheduler dynamics are fixed across most supervised learning experiments. We employ Stochastic Gradient Descent (SGD) paired with a task-wise Cosine Annealing scheduler. Specifically, the learning rate is smoothly annealed toward zero over the course of each task’s duration ( h h steps) and undergoes a hard reset to its maximum initial value at the beginning of the next task, providing the necessary plasticity to rapidly minimize the changing objective.

###### Reinforcement Learning

For a fair comparison, ITL and JTL share the same SAC core hyperparameters (network width, learning rates, γ \gamma , τ \tau , entropy auto-tuning, mini-batch size, and the number of SAC gradient updates per epoch). The two agents differ only along axes that are intrinsic to the ITL/JTL distinction: (i) the size of the rollout collected per epoch ( 2 × 10 4 2\times 10^{4} environment steps for ITL on its single active task vs. 10 5 10^{5} for JTL split uniformly over the currently active tasks), (ii) the presence of a one-hot task conditioning for JTL, (iii) replay buffer scope (per-task vs. shared across the curriculum), and (iv) optimizer state at task boundaries (full re-initialization at every task for ITL vs. Adam-state reset of actor/critic only when a new task becomes active for JTL). All optimization is performed with Adam. Here compute-matched refers to the gradient-update budget—the number of SAC updates ( 500 500 ) and the mini-batch size ( 500 500 ) per epoch are identical for the two agents—rather than the environment-step count, which differs by construction ( 2 × 10 4 2\times 10^{4} steps per epoch for ITL on its single task vs. 10 5 10^{5} for JTL split over the active tasks); if anything, JTL is given more environment data per epoch, yet still underperforms.

We follow the MT-SAC configuration of Yu et al. (2020b) for the shared core, with the ITL-side cycle/rollout schedule taken from the single-task SAC appendix of the same work. The resulting hyperparameters are reported in Table 10 ; per-task and per-curriculum runtime budgets follow directly from the rollout schedule and the number of gradient updates.

Effect of the SAC mini-batch size on ITL agents. We fix the SAC mini-batch size to 500 500 for both ITL and JTL so that the two agents share an identical per-update compute budget ( Table 10 ). Since the quadratic analysis predicts that the optimization-induced ( Variance ) penalty scales as 1 / B 1/B in the batch size B B ( Section 4 ), it is natural to ask whether the absolute performance of the agents, and hence the task-difficulty ordering that drives the aggregate Transfer Efficiency, is sensitive to this choice. We conducted a batch-size ablation for the single-task (ITL) SAC agents, comparing B = 500 B=500 against a larger B = 5000 B=5000 under otherwise identical hyperparameters. Each of the 10 10 ITL agents is trained for 500 500 epochs ( 2 × 10 4 2\times 10^{4} environment steps per epoch, i.e. 10 7 10^{7} steps per task) in the easy_to_hard ordering; we compared the mean success rate over 10 10 seeds with a ± \pm SEM band, smoothed with a centered rolling average. Two observations stood out. First, the larger batch size yields a uniformly higher and lower-variance success rate on nearly every task, consistent with the 1 / B 1/B gradient-noise scaling: the reduced stochasticity in each SAC update translates into faster and more stable convergence (most visibly on the harder push , door-open , button-press-topdown , and window-open blocks). Second, and more importantly for our claims, the relative ordering of the tasks by difficulty is essentially unchanged: the tasks that are easy at B = 500 B=500 (e.g. drawer-close , reach , window-close ) remain easy at B = 5000 B=5000 , and the tasks that single-task SAC fails to solve within the budget (e.g. pick-place , peg-insert-side ) remain unsolved at both batch sizes. Because the per-task Transfer Efficiency decomposition is driven by this difficulty ordering and by the high-instability tasks, the qualitative conclusion—negative aggregate TE on MT10 dominated by a handful of high-instability tasks—is robust to the mini-batch size. We therefore report the compute-matched B = 500 B=500 setting for all quantitative results in the main text; the per-task trajectory illustration in Figure 4 instead shows the visually cleaner B = 5000 B=5000 runs, whose task-difficulty ordering and hence the qualitative transfer story is identical.

#### B.5 Additional Figures and Empirical Substantiation

This subsection collects supporting diagnostics for the empirical claims in the main text. The figures should be read as mechanism checks for the Transfer Efficiency (TE) decomposition rather than as separate benchmark claims: they expose task heterogeneity, instability proxies, width- and parametrization-dependent optimization bias, and the real-time trajectories that are averaged into the final TE curves.

##### B.5.1 Pairwise Transfer is Asymmetric, and Why Ordering Matters

To isolate the directional structure of transfer, we measure pairwise interference directly: for every ordered pair ( t i , t j ) (t_{i},t_{j}) of MT10 tasks we train a two-task joint (JTL) agent and compare the performance it achieves on t i t_{i} against the single-task (ITL) specialist for t i t_{i} . Figure 21 reports the resulting normalized gap Δ ⁡ ( t i ∣ t j ) = ITL ​ ( t i ) − JTL ​ ( t i ∣ t j ) ITL ​ ( t i ) , \Delta(t_{i}\mid t_{j})\;=\;\frac{\text{ITL}(t_{i})-\text{JTL}(t_{i}\mid t_{j})}{\text{ITL}(t_{i})}, where the row indexes the evaluated task t i t_{i} and the column indexes the co-trained task t j t_{j} . With the figure’s sign convention, Δ > 0 \Delta>0 (red) means joint training (JTL) hurts the row task relative to its ITL specialist (ITL is better), Δ < 0 \Delta<0 (blue) means JTL helps it (JTL is better), and the diagonal is 0 0 by construction.

The salient structural fact is that the matrix is not symmetric : in general Δ ⁡ ( t i ∣ t j ) ≠ Δ ⁡ ( t j ∣ t i ) \Delta(t_{i}\mid t_{j})\neq\Delta(t_{j}\mid t_{i}) . Transfer between two tasks is directional—one task can be substantially harmed by a partner that it, in turn, helps. Drawer-close and peg-insert-side , for instance, are improved by joint training with almost every partner (predominantly blue rows), whereas push , pick-place , and button-press-topdown are degraded by most partners (predominantly red rows), and the two directions of a given pair frequently carry opposite signs.

This asymmetry has a direct consequence for the lifelong objective. Because the quantity we ultimately optimize is the time-averaged error across the whole task sequence , a pair in which one direction is positive and the other negative need not produce net-negative transfer: the gains on the helped task can offset or outweigh the losses on the harmed task. More importantly, asymmetry turns task ordering into a usable degree of freedom. Concretely, suppose Δ ⁡ ( t 1 ∣ t 2 ) > 0 \Delta(t_{1}\mid t_{2})>0 (joint training would hurt t 1 t_{1} ) while Δ ⁡ ( t 2 ∣ t 1 ) < 0 \Delta(t_{2}\mid t_{1})<0 (joint training helps t 2 t_{2} ). Then presenting the tasks in the order t 1 → t 2 t_{1}\rightarrow t_{2} (training t 1 t_{1} alone first, and only introducing t 2 t_{2} afterwards under joint training) lets t 1 t_{1} bank its ITL-level performance during its first phase while t 2 t_{2} reaps the positive transfer during the joint phase. The lifelong error over the pair can then fall below the average of the two ITL specialists, even though the symmetric (simultaneous, from-scratch) joint agent would have shown mixed or negative transfer. In other words, the off-diagonal sign structure of Figure 21 is precisely the lever that the curriculum exploits, and it is the structural reason to adopt an explicit ordering.

We stress that the matrix is measured under simultaneous two-task training from a shared initialization, so it is a proxy for the directional interference. It motivates ordering as a mechanism and explains the observed sensitivity of aggregate TE to the task sequence. A systematic search over orderings that maximize lifelong performance using this asymmetry is a natural direction left to future work.

##### B.5.2 Proxy measures of instability

We consider two proxies to measure Instability. The first, ℐ C ​ U \mathcal{I}_{CU} , approximates the theoretical definition ( Definition 3 ) by training ITL and JTL models to convergence and measuring the difference in their average task-wise error. The second, ℐ t ​ r ​ a ​ n ​ s ​ f ​ e ​ r \mathcal{I}_{transfer} , is a heuristic that measures the average off-diagonal error in a task transfer matrix (i.e., the error of a model trained on task i i when evaluated on task j j ). In practice, ℐ t ​ r ​ a ​ n ​ s ​ f ​ e ​ r \mathcal{I}_{transfer} is computationally cheaper.

As shown in Table 11 , both proxies confirm our qualitative assessment: CLEAR exhibits low Instability, while MD5 and MT10 are high. The proxies also validate our methodology for the controlled benchmarks, showing that Instability increases with the permutation size (PC-16 vs. PC-32) and the label shuffle percentage (SC-10% to SC-50%). The two proxy columns are not expected to match numerically: ℐ C ​ U \mathcal{I}_{CU} compares the converged JTL and ITL solutions directly, while ℐ t ​ r ​ a ​ n ​ s ​ f ​ e ​ r \mathcal{I}_{transfer} measures off-diagonal transfer between separately trained task specialists. Their agreement at the level of ordering is the relevant point for our empirical use of Instability.

##### B.5.3 Scaling Deep Linear Networks Experiments

Figures 22 and 23 expand the deep-linear study by varying hidden width while holding the data-generating process fixed. These experiments isolate a key message from the optimization-bias analysis: even when the realizable input-output map is linear, the training parametrization changes the TE landscape. Increasing width does not simply make JTL and ITL equivalent. Instead, the lazy and rich regimes trace different families of TE frontiers because they induce different optimization paths, different effective conditioning, and different amounts of Instability after joint training.

The lazy-regime simulations in Figure 22 behave most like the fixed-feature or neural-tangent picture. Width increases capacity and smooths some finite-width effects, but the qualitative transfer boundary remains governed by the same tradeoff as in the shallow theory: JTL reduces transient optimization cost by reusing past data, while it pays an Instability cost when the current task has drifted too far from the historical average. Thus positive TE persists only in regions where the transient savings dominate the mismatch induced by jointly fitting previous tasks.

The rich-regime simulations in Figure 23 show that feature learning changes this balance. Because the representation can move during training, the model is not constrained to interpolate between tasks in the same way as in the lazy regime. This can improve transfer when the tasks share useful structure, but it can also amplify path dependence when the representation learned for earlier tasks is poorly aligned with later ones. The comparison between the two grids therefore supports the claim that TE is not determined only by task drift and sample count; it also depends on the optimizer-induced solution selected within an overparameterized model class.

##### B.5.4 Real-time training trajectories

The main text reports TE after averaging over tasks and seeds. The trajectories below show where those averages come from in time. In particular, they distinguish transient effects at the beginning of each task from persistent bias or interference accumulated across the sequence.