Metalearning Continual Learning Algorithms — part 3/5

Cleaned-text code points [30777, 46565). Bibliography omitted here; source.md includes it.

Unless otherwise indicated (e.g, later for Split-MNIST; Sec. 4.3 ), all tasks are configured to be a 5-way classification task. This is one of the classic configurations for few-shot learning tasks, and also allows us to evaluate the principle of ACL with reasonable computational costs—like any sequence learning-based metalearning methods, scaling up to many more classes is challenging; we further discuss this in Sec. 5 . For the standard datasets such as MNIST, we split the dataset into subsets of disjoint classes ( Srivastava et al., 2013 ) : for example for MNIST which is originally a 10-way classification task, we split it into two 5-way tasks, one consisting of images of class ‘0’ to ‘4’ (‘MNIST-04’), and another one made of class ‘5’ to ‘9’ images (‘MNIST-59’). When we refer to a dataset without specifying the class range, we refer to the first subset. Unless stated otherwise, we concatenate 15 examples from each class for each task in the context for both meta-training and meta-testing (resulting in the sequences of length 75 for each task). All images are resized to 32 × 32 32\times 32 3 3 -channel images, and normalized according to the original dataset statistics. We refer to Appendix A for further details.

Table 2 shows the results when the models are meta-tested on the test set of the datasets used for meta-training. We observe that for both pairs of meta-training datasets, the models meta-trained without the ACL loss catastrophically forget the first task after learning the second one: the accuracy on the first task is at the chance level of about 20% for 5-way classification after learning the second task in-context (see rows with “ACL/No”). In contrast, ACL-learned CL algorithms preserve the performance of the first task (“ACL/Yes”). This effect is particularly pronounced in the Omniglot/Mini-ImageNet case (involving two very different domains). Note that there is a slight performance degradation from the single task to two-task setting in the FC100/Mini-ImageNet case (Table 1, bottom block). This is not surprising as training a model that performs well on two tasks is inherently more challenging than the single-task case, depending on the specific set of tasks involved; in particular, in the domain-incremental setting where the output layer is shared between the two tasks, “similar” tasks are inherently more confusing (e.g., in certain sequences, FC100 label 1 may be similar to Mini-ImageNet label 3; while Omniglot examples are consistently very distinguishable from Mini-ImageNet examples).

Table 2 shows evaluations of the same models but using two standard datasets, 5-way MNIST and CIFAR-10, for meta-testing. Again, the ACL-trained models preserve memory of the first task after learning the second one. In the Omniglot/Mini-ImageNet case, we even observe certain positive backward tranfer effects: in particular, in the “MNIST-then-CIFAR10” continual learning case, the performance on MNIST noticeably improves after learning CIFAR10, which is also behavior encouraged by the backward term in the ACL objective.

### 4.2 Analysis: Emergence of In-Context Catastrophic Forgetting

Here we closely examine how “in-context catastrophic forgetting” emerges during meta-training of the baseline models without the backward transfer term (the last/third term in Eq. 6 ) in the ACL objective (corresponding to the ACL/No case in Tables 2 and 2 ). We focus on the Omniglot/Mini-ImageNet case, but similar trends can also be observed in the FC100/Mini-ImageNet case. Figures 2a and 2b show two representative scenarios we observe for different random seeds. These figures show an evolution of six individual meta-training loss terms (the lower the better): 4 out of 6 curves correspond to the metalearning progress reported separately for the cases where either Dataset A (here Omniglot) or B (here Mini-ImageNet) is used to sample the task at the first (1) or second (2) position in the 2-task CL meta-training demo sequences. The 2 remaining curves are the ACL backward transfer losses (“ ACL bwd ”), also reported for Datasets A and B separately.

Figure 2a shows the case where the two datasets are metalearned at the same time. We observe that when the metalearning curves go down, the backward transfer losses go up, indicating that more the model metalearns, more it tends to forget in-context. The trend is the same when one task is metalearned before the other one (Figure 2b ). Here Dataset A alone is metalearned first, when B is not metalearned yet; both metalearning and backward transfer curves first go down for A—as the model has not yet metalearned the second task at this stage, nothing causes forgetting. However, at around 2,800 steps, the model also starts becoming capable of learning tasks sampled from Dataset B in-context. From this point, the backward transfer loss for Dataset A begins to go up, indicating again “opposing forces” between learning a new task and remembering a past task in-context.

These observations clearly indicate that, without explicitly including the backward transfer loss as part of the metalearning objectives, gradient descent search tends to find solutions/CL algorithms that prefer to erase previously learned knowledge. This is rather intuitive; it seems easier to find such algorithms that ignore any impact of the current learning on past learning than those that additionally preserve prior knowledge. This indicates that the ACL objective is crucial for metalearning CL algorithms that overcome catastrophic forgetting.

84.5

96.0

84.3

### 4.3 General Evaluation

Evaluation on Standard Split-MNIST. Here we evaluate the ACL-learned algorithms (meta-trained as described in Sec. 4.1 ) on the standard Split-MNIST task in both domain-incremental and class-incremental settings ( Hsu et al., 2018 ; Van de Ven and Tolias, 2018b ) , and compare its performance with existing CL and meta-CL algorithms (see Appendix A.7 for the full list of references). Our comparison focuses on methods that do not rely on replay memory. Table 3 shows the results. Since the ACL models are general-purpose metalearners, they can be directly evaluated (meta-tested) on a new task, here Split-MNIST. The second-to-last row of Table 3 , “ACL (Out-of-the-box model)”, corresponds to our model from Sec. 4.1 meta-trained on Omniglot and Mini-ImageNet using the 2-task ACL objective. It performs very competitively against the best existing methods in the domain-incremental setting, while, in the 2-task class-incremental setting, it largely outperforms all of them except GeMCL, another meta-CL method. The same model can be further meta-finetuned using the 5-task version of the ACL loss (here we only used Omniglot as the meta-finetuning data). The resulting model (the last row of Table 3 ) outperforms all other methods in all settings studied here. We are not aware of any existing hand-crafted CL algorithms that can achieve ACL’s performance without any replay memory. We refer to Appendix A.7 / B for further discussions and ablation studies.

Evaluation on diverse task domains. Using the setting of Sec. 4.1 , we also evaluate ACL models for CL involving more tasks and domains, using meta-test sequences made of MNIST, CIFAR-10, and Fashion MNIST. We also vary the number of tasks in the ACL objective: in addition to the model meta-trained on Omniglot/Mini-ImageNet (Sec. 4.1 ), we also meta-train a model (with the same architecture and hyper-parameters) using 3 tasks, Omniglot, Mini-ImageNet, and FC100, using the 3-task ACL objective (see Appendix A.5 ), resulting in meta-training that not only involves longer CL sequences but also more data. The full results of this experiment can be found in Appendix B.4 . We find that both the 2-task and 3-task meta-trained models are capable of retaining the knowledge of multiple tasks during meta-testing without catastrophic forgetting; while the performance on prior tasks gradually degrades as the model learns new tasks, and performance on new tasks also becomes moderate (see also Sec. 5 on limitations). The 3-task one outperforms the 2-task one overall, encouragingly indicating a potential for further improvements even within a fixed parameter budget.

Going beyond: limitations and outlook. The experiments presented above effectively demonstrate that self-referential weight matrices can encode a continual learning algorithm that outperforms handcrafted learning algorithms and existing metalearning approaches for CL. While we consider this as an important result for metalearning and in-context learning in general, we note that current state-of-the-art CL methods use neither regularization-based CL algorithms nor meta-continual learning methods mentioned above, but the so-called learning to prompt (L2P)-family of methods ( Wang et al., 2022b ; Wang et al., 2022a ) that leverage pre-trained models, namely a vision Transformer (ViT) pre-trained on ImageNet ( Dosovitskiy et al., 2021 ) . Here we examine how ACL can leverage pre-trained models to potentially go beyond the experimental scale considered so far. To study this, we take a pre-trained (frozen) ViT model, and add self-referential layers on top of it to build a continual learner.

We use two datasets from the prior L2P work above ( Wang et al., 2022b ; Wang et al., 2022a ) : 5-datasets ( Ebrahimi et al., 2020 ) and Split-CIFAR-100 in the class-incremental setting, but we focus on our custom “ mini ” versions thereof by only using the two first classes within each task (i.e., 2-way version); and for Split-CIFAR100, we only use the 5 first tasks (instead of 10). As we’ll see, this simplified setting is enough to illustrate an important current limitation of in-context CL. Again following L2P ( Wang et al., 2022b ; Wang et al., 2022a ) , we use ViT-B/16 ( Dosovitskiy et al., 2021 ) (available via PyTorch) as the pre-trained vision model, which we keep frozen. The self-referential component uses the same configuration as in the Split-MNIST experiment. We meta-train the resulting model using Mini-ImageNet and Omniglot with the 5-task ACL loss.

Table 4 shows the results. Even in this “mini” setting, ACL’s performance is far behind that of L2P methods on the original setting. Notably, the frozen ImageNet-pre-trained features with the metalearner trained on Mini-ImageNet and Omniglot are not enough to perform well on the 5-th task of Split-CIFAR100; and SVHN and notMNIST of 5-datasets; even when these tasks are evaluated in isolation. This illustrates a general limitation of in-context learning, showing the necessity for further scaling, i.e., meta-training on more diverse datasets for in-context CL and ACL to be possibly successful in more general settings.

Ablations. We provide several additional ablation studies in Appendix B , including those on the choice of meta-validation datasets and the effect of varying the number of in-context examples.

## 5 Discussion

##### Other Limitations.

In addition to the limitations already mentioned above, here we discuss others. First of all, as an in-context/learned learning algorithm, there are challenges in terms of both domain and length generalization. We qualitatively observe these to some extent in Sec. 4.3 ; further discussion and experimental results are presented in Appendix B.3 & B.5 . Regarding the length generalization, we note that unlike the standard “quadratic" Transformers, linear Transformers/FWPs-like SRWMs can be trained by carrying over states across two consecutive batches for arbitrarily long sequences. Such an approach has been successfully applied to language modeling with FWPs ( Schlag et al., 2021 ) . This possibility, however, has not been investigated here, and is left for future work. Also, directly scaling ACL for real-world tasks involving many more classes does not seem straightforward: it would involve very long meta-training sequences. That said, it may be possible that ACL could be achieved without exactly following the process we proposed here; as we discuss below for the case of large language models (LLMs), certain real-world data may naturally give rise to an ACL-like objective. The scope of this work is also limited to image classification, which can be solved by feedforward NNs. Future work may investigate the possibility to extend ACL to continual learning of sequence learning tasks, such as continually learning new languages. Finally, ACL learns CL algorithms that are specific to the pre-specified model architecture; more general metalearning algorithms may aim at achieving learning algorithms that are applicable to any model, as is the case with many classic learning algorithms.

Interpretability & Extracting Novel Algorithm Design Principles? One potential application of metalearning is to discover novel learning algorithm design principles, and turn them into a human-interpretable, general learning algorithm. However, as in prior work on fast weight programmers ( Irie and Schmidhuber, 2023b ) , we found it very hard to interpret learned weight modification algorithms for CL. We provide example weight visualizations with the model used in the class-incremental setting of Split-MNIST (Sec. 4.3 ) while feeding meta-test demo examples to the model for two tasks from Split-MNIST (class 0 vs 1, and 2 vs 3, respectively), in Figure 3 and 4 (see Figure 5 in the appendix for presentation of the third task, 4 vs 5). One natural difficulty is to deal with a large number of weight matrices: given that our model has 2 SRWM layers with 16 heads each, and considering the 4 components of SRWM (“o”, “q”, “k”, “ β \beta ” parts; “ β \beta ” part is denoted with “b”), 128 matrices have to be visualized over time (here we selected 1 head in each layer as representative examples). Future work on interpretability may need to focus on smaller models to reduce this number.

Related work. There are several recent works that are catagorized as meta-continual learning or continual metalearning (see, e.g., Javed and White (2019) ; Beaulieu et al. (2020) ; Caccia et al. (2020) ; He et al. (2019) ; Yap et al. (2021) ; Munkhdalai and Yu (2017) ). For example, Javed and White (2019) ; Beaulieu et al. (2020) use “model-agnostic metalearning” (MAML; Finn et al. (2017) ; Finn and Levine (2018) ) to metalearn representations for CL while still making use of classic learning algorithms for CL; this requires tuning of the learning rate and number of iterations for optimal performance during CL at meta-test time (see, e.g., Appendix A.7 ). In contrast, our approach learn learning algorithms in the spirit of Hochreiter et al. (2001) ; Younger et al. (1999) ; this may be categorized as “in-context continual learning.” Several recent works (see, e.g., Irie and Schmidhuber (2023a) ; von Oswald et al. (2023b) ) mention the possibility of such in-context CL but existing works ( Irie et al., 2022c ; Coda-Forno et al., 2023 ; Lee et al., 2023 ) that learn multiple tasks sequentially in-context do not focus on catastrophic forgetting which is one of the central challenges of CL. Here we show that in-context learning also suffers from catastrophic forgetting in general (Sec. 4.1 - 4.2 ) and propose ACL to address this problem. We also note that the use of SRWM is particularly relevant to continual metalearning. In fact, with regular linear Transformers or FWPs, the question remains regarding how to continually learn the “slow weights” ( Schmidhuber, 1992b ) . In principle, recursive self-modification as in SRWM is an answer to this question as it collapses such meta-levels into a single self-referential loop ( Hofstadter, 1979 ; Schmidhuber, 1992a ) . We also refer to Schmidhuber (1994) ; Schmidhuber (1995) ; Schmidhuber et al. (1997) for other prior work on meta-continual learning.