Metalearning Continual Learning Algorithms — part 4/5
Cleaned-text code points [46565, 62141). Bibliography omitted here; source.md includes it.
Artificial v. Natural ACL in Large Language Models? Recently, the “on-the-fly” or in-context few-shot learning capability of sequence processing NNs has attracted broader interest in the context of LLMs ( Brown et al., 2020 ) . In fact, the task of language modeling itself has a form of sequence processing with error feedback —essential for metalearning ( Schmidhuber, 1990 ) : the correct label to be predicted is fed to the model with a delay of one time step in an auto-regressive manner ( Hochreiter et al., 2001 ) . Trained on a large amount of text covering a wide variety of credit assignment paths, LLMs exhibit certain sequential few-shot learning capabilities in practice ( Brown et al., 2020 ) . Here we explicitly/artificially constructed ACL meta-training sequences and objectives, but in modern LLMs trained on a large amount of data mixing a large diversity of dependencies using a large backpropagation span, it is conceivable that some ACL-like objectives may naturally appear in the data.
## 6 Conclusion
Our Automated Continual Learning (ACL) trains self-referential neural networks to metalearn their own in-context continual (meta)learning algorithms. ACL encodes the classic desiderata for continual learning (i.e., forward and backward transfer) into the objective function of the metalearner. ACL uses gradient descent to deal with the classic challenges of CL, to automatically discover CL algorithms with effective behavior; avoiding the need for manual, human-led algorithm design. Once trained, ACL-models autonomously run their own CL algorithms without requiring any human intervention. Our experiments reveal the problem of in-context catastrophic forgetting, and demonstrate the effectiveness of ACL to overcome it. We demonstrate promising results of ACL on the classic Split-MNIST benchmark where existing hand-crafted algorithms fail. We also highlight the need for further scaling ACL to succeed in more challenging scenarios. We believe this represents an important step towards developing open-ended continual metalearners based on neural networks.
#### Acknowledgements
This research was partially funded by ERC Advanced grant no: 742870, project AlgoRNN, and by Swiss National Science Foundation grant no: 200021_192356, project NEUSYM. We are thankful for hardware donations from NVIDIA and IBM. The resources used for this work were partially provided by Swiss National Supercomputing Centre (CSCS) projects s1145 and d123.
## Appendix A Experimental Details
### A.1 Continual and Metalearning Terminologies
Here we review the classic terminologies of continual learning and metalearning used in this paper.
##### Continual learning.
“Domain-incremental learning (DIL)” and “class-incremental learning (CIL)” are the two classic settings in continual learning [ Van de Ven and Tolias, 2018a , Van de Ven and Tolias, 2018b , Hsu et al., 2018 ] . They differ as follows. Let M M and N N denote positive integers. Consider continual learning of M M tasks where each task is an N N -way classification. In the DIL case, a model has an N N -way output classification layer, i.e., the class ‘0’ of the first task shares the same weights as the class ‘0’ of the second task, and so on. In the CIL case, the model’s output dimension is N ∗ M N*M ; the class indices of different tasks are not shared, neither are the corresponding weights in the output layer. In our experiments, all CIL models have the ( N ∗ M ) (N*M) -way output from the first task (instead of progressively increasing the output size). In this work, the third variant called “task-incremental learning” which assumes that we have access to the task identity as an extra input, is not considered as it is known to make the CL problem almost trivial. CIL is typically reported to be the hardest setting among them.
##### Metalearning.
Unlike standard learning, which simply involves “training” and “testing,” metalearning requires introducing the “meta-training” and “meta-testing” terminologies since each of these phases involves “training/test” processes within itself. Each of them requires “training” and “test” examples. In the main text, we referred to the training examples as demonstrations , and the test example as consisting of a query (input) and a target (output). An alternative terminology one can find in the literature is “meta-training training/test examples”, and “meta-test training/test examples” of Beaulieu et al. [2020] ; we opted for “demonstrations” and “queries/targets” to avoid this rather heavy terminology (except for “meta-test training iterations” of OML discussed in Appendix A.7 ). In both phases, a sequence-processing neural net observes a sequence of (meta-training or meta-test) training examples—each consisting of input features and a correct label, and the resulting states of the sequence processor (i.e., states of the weights in the case of SRWM) are used to make predictions on (meta-training or meta-test) test examples’ input features presented to the model without the label. During the meta-training phase, we modify the trainable parameters of the metalearner through gradient descent minimizing the metalearning loss function. During meta-testing, no human-designed optimization for weight modification is used anymore; the SRWMs modify their own weights following their own learning rules defined in their forward pass (Eqs. 1 - 3 ).
### A.2 Datasets
Here we provide more details about the datasets used in this work.
For the classic image classification datasets such as MNIST [ LeCun et al., 1998 ] , CIFAR10 [ Krizhevsky, 2009 ] , and FashionMNIST (FMNIST; Xiao et al. [2017] ) we refer to the original references for details.
For Omniglot [ Lake et al., 2015 ] , we use Vinyals et al. [2016] ’s 1028/172/432-split for the train/validation/test set, as well as their data augmentation methods using rotation of 90, 180, and 270 degrees. Original images are grayscale hand-written characters from 50 different alphabets. There are 1632 different classes with 20 examples for each class.
Mini-ImageNet contains color images from 100 classes with 600 examples for each class. We use the standard train/valid/test class splits of 64/16/20 following Ravi and Larochelle [2017] .
FC100 is based on CIFAR100 [ Krizhevsky, 2009 ] . 100 color image classes (600 images per class, each of size 32 × 32 32\times 32 ) are split into train/valid/test classes of 60/20/20 [ Oreshkin et al., 2018 ] .
The “5-datasets” dataset [ Ebrahimi et al., 2020 ] consists of 5 datasets: CIFAR10, MNIST, FashionMNST, SVNH [ Netzer et al., 2011 ] , and notMNIST [ Bulatov, 2011 ] .
Split-CIFAR100 is also based on CIFAR100. The standard setting splits the original 100-way classification task into a sequence of ten 10-way classification tasks.
We use torchmeta [ Deleu et al., 2019 ] which provides common experimental setups for few-shot/metalearning to sample and construct meta-train/test datasets.
### A.3 Training Details & Hyper-Parameters
We use the same model architecture and meta-training hyper-parameters in all our experiments. All hyper-parameters are summarized in Table 5 . We use the Adam optimizer with the standard Transformer learning rate warmup scheduling [ Vaswani et al., 2017 ] . The vision backend is the classic 4-layer convolutional NN of Vinyals et al. [2016] . Most configurations follow those of Irie et al. [2022c] ; except that we initialize the ‘query’ sub-matrix in the self-referential weight matrix using a normal distribution with a mean value of 0 and standard deviation of 0.01 / d head 0.01/\sqrt{d_{\text{head}}} while other sub-matrices use an std of 1 / d head 1/\sqrt{d_{\text{head}}} (motivated by the fact that a generated query vector is immediately multiplied with the same SRWM to produce a value vector). For further details, we refer readers to our public code (link provided on page 1). We conduct our experiments using a single V100-32GB, 2080-12GB or P100-16GB GPUs, and the longest single meta-training run takes about one day.
### A.4 Evaluation Procedure
For evaluation on the classic few-shot learning datasets (i.e., Omniglot, Mini-Imagenet and FC100), we use 5 different sets of 32 K random test episodes each, and report the mean and standard deviation.
For evaluation on other datasets, we use 5 different sets of randomly sampled demonstrations, and use the entire test set as the queries/targets. We report the corresponding mean and standard deviation across these 5 evaluation runs.
For the Split-MNIST (and other “Split-X”) experiments, we do 10 meta-testing runs to compute the mean and standard deviation as the baseline models are also trained for 10 runs in Hsu et al. [2018] ; see further details in Appendix A.7 .
### A.5 ACL Objectives with More Tasks
We can straightforwardly extend the 2-task version of ACL presented in Sec. 3 to more tasks. In the 3-task case (we denote the three tasks as 𝐀 {\mathbf{A}} , 𝐁 {\mathbf{B}} , and 𝐂 {\mathbf{C}} ) used in Sec. 4.3 and Appendix B.4 , the objective function contains six terms. The following three terms are added to Eq. 6 : − ( log ( p ( y target 𝒞 | 𝒙 query 𝒞 ; 𝑾 𝒜 , ℬ , 𝒞 ( θ ) ) ) + log ( p ( y target ℬ | 𝒙 query ℬ ; 𝑾 𝒜 , ℬ , 𝒞 ( θ ) ) ) + log ( p ( y target 𝒜 | 𝒙 query 𝒜 ; 𝑾 𝒜 , ℬ , 𝒞 ( θ ) ) ) ) \displaystyle-\Bigl(\log\big(p(y^{\mathcal{C}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{C}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A,B,C}}(\theta))\big)+\log\big(p(y^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A,B,C}}(\theta))\big)+\log\big(p(y^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A,B,C}}(\theta))\big)\Bigr)
This also naturally extends to the 5-task loss used in the Split-MNIST experiment (Table 3 ), and so on. As one can observe, the number of terms quadratically increases with the number of tasks. Nevertheless, computing these loss terms isn’t immediately impractical because they essentially just require forwarding the network for one step, for many independent queries. This can potentially be heavily parallelized as a batch operation. While this may still be a concern when scaling up much further, a natural open research question is whether we really need all these terms in the case we have many more tasks. Ideally, we want these models to “systematically generalize” to more tasks even when they are trained with only a handful of them [ Fodor and Pylyshyn, 1988 ] . This is an interesting research question on generalization to be studied in future work.
### A.6 Auxiliary 1-shot Learning Objective
In practice, instead of training the models only for the “15-shot learning” objective (as described in the main text, we use 15 demonstrations for each class), we also add an auxiliary loss for 1-shot learning. This incentivizes the models to learn as soon as the first demonstrations of the task become available; in practice, we found this to be generally useful for efficient meta-training.
### A.7 Details of the Split-MNIST experiment
Here we provide details of the Split-MNIST experiments presented in Sec. 4 and Table 3 .
Split-MNIST is obtained by transforming the original 10-way MNIST dataset into a sequence of five 2-way classification tasks by partitioning the 10 classes into 5 groups/pairs of two classes each, in a fixed order from 0 to 9 (i.e., grouping 0/1, 2/3, 4/5, 6/7, and 8/9). Regarding the difference between domain/class-incremental settings, we refer to Appendix A.1 .
For meta-finetuning using the 5-task ACL loss (last row of Table 3 ), we randomly sample 5 tasks from Omniglot (in principle, we should make sure that different tasks in the same sequence have no underlying class overlap; in practice, our current implementation simply randomly draws 5 independent tasks from Omniglot).
The baseline methods presented in Table 3 include: standard SGD and Adam optimizers, Adam with the L2 regularization, elastic weight consolidation [ Kirkpatrick et al., 2017 ] and its online variant [ Schwarz et al., 2018 ] , synaptic intelligence [ Zenke et al., 2017 ] , memory aware synapses [ Aljundi et al., 2018 ] , learning without forgetting (LwF; Li and Hoiem [2016] ). For these methods, we directly take the numbers reported in Hsu et al. [2018] for the 5-task domain/class-incremental settings. For the 2-task class-incremental learning case, we use Hsu et al. [2018] ’s code to train the corresponding models (the number for LwF is not included as it is not implemented in their code base).
Finally we also evaluate two classic meta-CL baselines: Online-aware Meta-Learning (OML; Javed and White [2019] ) and Generative Meta-Continual Learning (GeMCL; Banayeeanzade et al. [2021] ). OML is a MAML-based metalearning approach. We note that as reported by Javed and White [2019] in their public GitHub code repository; after some critical bug fix, the performance of their OML matches that of a followp work by Beaulieu et al. [2020] , which is a direct application of OML to another model architecture. Therefore, we focus on OML as our main MAML-based baseline. We utilize the publicly available, ready-to-use model checkpoint of Javed and White [2019] (meta-trained on Omniglot, with a 1000-way output layer).
We evaluate the corresponding OML model in two ways (Table 3 ). In the first, ‘out-of-the-box’ case, we take the meta/pre-trained model and only tune its meta-testing learning rate (which is also done by Javed and White [2019] even for meta-testing on Omniglot). We find that this approach does not perform very well on Split-MNIST (Table 3 ). In the other approach (denoted as ‘optimized number of meta-testing iterations’ in Table 3 ), we additionally tune the number of meta-test training iterations. We’ve done a grid search of the meta-test learning rate in 3 ∗ { 1 e − 2 , 1 e − 3 , 1 e − 4 , 1 e − 5 } 3*\{1e^{-2},1e^{-3},1e^{-4},1e^{-5}\} and the number of meta-test training steps in { 1 , 2 , 5 , 8 , 10 } \{1,2,5,8,10\} using a meta-validation set based on an MNIST validation set (5 K held-out images from the training set); we found the learning rate of 3 e − 4 3e^{-4} and meta-test training of 8 8 steps to consistently perform best in all our settings. We’ve also tried it ‘with’ and ‘without’ the standard mean/std normalization of the MNIST dataset; better performance was achieved without such normalization, which is consistent as they do not normalize the Omniglot dataset for their meta-training/testing.
The sensitivity of the MAML-based methods [ Javed and White, 2019 , Beaulieu et al., 2020 ] w.r.t. meta-test hyper-parameters has been also noted by Banayeeanzade et al. [2021] . Importantly, this is one of the characteristics of hand-crafted learning algorithms that we precisely aim to avoid using learned learning algorithms .
OML’s weak performance on the 5-task class-incremental setting is somewhat surprising, since genenralization from Omniglot to MNIST is typically straightforward in non-continual few-shot learning settings (see, e.g., Koch et al. [2015] , Vinyals et al. [2016] , Munkhdalai and Yu [2017] ). At the same time, to the best of our knowledge, OML-trained models have not been tested in such a condition in prior work. Based on our results, it may be the case that the publicly available out-of-the-box OML model is overtuned for Omniglot/Mini-ImageNet; or the frozen “representation network” may not be ideal for genenralization.