Metalearning Continual Learning Algorithms — part 5/5

Cleaned-text code points [62141, 71238). Bibliography omitted here; source.md includes it.

Regarding the GeMCL baseline, we use the code and a pre-trained model (meta-trained on Omniglot) made publicly available by Banayeeanzade et al. [2021] . Similarly to the ACL models, GeMCL also does not require any special tuning at meta-test time. Nevertheless, for both models, we conducted an ablation study on the effect of varying the number of meta-test training examples (5 vs. 15; 15 is the number used in meta-training). We find the consistent number, i.e., 15, to work better than 5. For the ACL version that is meta-finetuned using the 5-task ACL objective (using only the Omniglot dataset), we additionally tested the cases where we use 5 demonstrations for meta-training, and 5 or 15 for meta-testing. We find that again, the consistent number of demonstrations tends to yield the best performance. See Appendix B.3 and Table 6 for the full results.

More ablation studies can be found in Appendix B .

### A.8 Details of the Split-CIFAR100 and 5-datasets Experiment using ViT

As we described in Sec. 4.3 , for the experiments on Split-CIFAR100 and 5-datasets, following Wang et al. [2022b] , Wang et al. [2022a] , we use ViT-B/16 pre-trained on ImageNet [ Dosovitskiy et al., 2021 ] which is available through torchvision [ Paszke et al., 2019 ] . In this experiments, we resize all images to 3 × \times 224 × \times 224 and feed them to the ViT. We remove the output layer of the ViT, and use its 768-dimensional feature vector from the penultimate layer as the image encoding. The learnable self-referential component which is added on top of this frozen ViT encoder has the same architecture (2 layers, 16 heads) as in the rest of the paper (see all hyper-parameters in Table 5 ). All the ViT parameters are frozen throughout the experiment.

63.8

84.5

91.2

96.0

79.0

84.3

## Appendix B Extra Experimental Results

### B.1 Ablation Studies on the Choice of Meta-Validation Dataset

In general, when dealing with out-of-domain generalization, the choice of validation procedures to select final model checkpoints plays a crucial role in the evaluation of the corresponding method [ Csordás et al., 2021 , Irie et al., 2021b ] .

For the out-of-the-box model evaluation, the checkpoints are selected based on the average meta-validation performance on the validation set corresponding to the few-shot learning datasets used for meta-training: Omniglot and mini-ImageNet (or Omniglot, mini-ImageNet, and FC100 in the case of 3-task ACL), completely independently of the meta-test datasets used for evaluation. In contrast, in the meta-finetuning process of Table 3 , we selected our model checkpoints through meta-validation on the MNIST validation dataset (we held out 5 K images from the training set).

Here we conduct an ablation study of the choice of meta-validation set, using three Split-‘X’ tasks where ‘X’ is either MNIST, FashionMNIST (FMNIST) or CIFAR-10 (in each case, we isolate 5 K images from the corresponding training set to create a validation set). In addition, we also evaluate the effect of meta-finetuning datasets (Omniglot only vs. Omniglot and mini-ImageNet).

Table 7 shows the results (we use 15 meta-training and meta-testing demonstrations, except for the Omniglot-finedtuned/MNIST-validated model from Table 3 which happens to be configured with 5 demos). Effectively, we observe that meta-validation using the validation set matching the test domain is useful. Also, meta-finetuning only on Omniglot is beneficial for the performance on MNIST when meta-validated on MNIST or FMNIST.

However, importantly, our ultimate goal is not to obtain a model that is specifically tuned for certain datasets; we aim at building models that generally work well across a wide range of tasks (ideally on any tasks); in fact, several existing works in the few-shot learning literature evaluate their methods in such settings (see, e.g., Requeima et al. [2019] , Bronskill et al. [2020] , Triantafillou et al. [2020] ). This also goes hand-in-hand with the idea of scaling up ACL (our current model is tiny; see hyper-parameters in Table 5 ; the vision component is also a shallow ‘Conv-4’ net) as well as various other considerations on self-improving continual learners (see, e.g., Schmidhuber [2018] ), such as automated curriculum learning [ Graves et al., 2017 ] .

84.3

90.4

63.4

76.6

89.9

68.6

### B.2 Performance on Split-Omniglot

Here we report the performance of the ACL and GeMCL models used in the Split-MNIST experiment (Sec. 4.3 ) on “in-domain” 5-task 2-way Split-Omniglot. Table 8 shows the result. Performance is very similar between ACL and the baseline GeMCL on this task in the class incremental setting, unlike on Split-MNIST (Table 3 ) where we observe a larger performance gap between the same models. Here we also include an evaluation under the “domain incremental” setting for the sake of completeness but note that GeMCL is not originally meta-trained for this setting.

### B.3 Varying the Number of In-Context Examples/Demonstrations

Table 6 shows an ablation study on the number of demonstrations used for meta-training and meta-testing on the Split-MNIST task. We observe that for the ACL model trained only with 5 examples during meta-training, providing more examples (15 examples) during meta-testing is not beneficial. In fact, it even largely degrades performance in certain cases (see the last column); this is one form of the “length generalization” problem. When the number of meta-training examples is consistent with the one used during meta-testing, the 15-example case (i.e., providing more demonstrations to the model) consistently outperforms the 5-example one.

### B.4 Varying the Number of Tasks in the ACL Meta-Training Objective

Table 9 provides the complete results discussed in Sec. 4.3 under “Evaluation on diverse task domains”.

### B.5 Further Discussion on Limitations

Here we provide further discussion and experimental results on the limitations of learned learning algorithms.

##### Domain generalization.

As a data-driven learned algorithm, the domain generalization capability is a typical limitation as it depends on the diversity of meta-training data. Certain results we presented above are representative of this limitation. In particular, in Table 7 , the model meta-trained/finetuned on Omniglot using Split-MNIST as the meta-validation set does not perform well on Split-CIFAR10.

While meta-training and meta-validation on a larger/diverse set of datasets may be an immediate remedy to obtain more robust ACL models, we note that since ACL is also a “continual metalearning” algorithm (Sec. 5 ), an ideal ACL model should also continually incorporate and learn from more data during potentially lifelong meta-testing; we leave such an investigation for future work.

Comment on meta-generalization. We also note that in general, “unseen” datasets do not necessarily imply that they are harder tasks than “in-domain” test sets; when meta-trained on Omniglot and mini-ImageNet, meta-generalization on “unseen” MNIST is easier (the accuracy is higher) than on the “in-domain” test set of mini-ImageNet with heldout/unseen classes (compare Tables 2 and 2 ).

##### Length generalization.

We qualitatively observed the limited length generalization capability in Table 9 (meta-trained with up to 3 tasks and meta-tested with up to 4 tasks) and in Appendix B.3 (meta-trained using 5 demonstrations and meta-tested using 15 demos). Similarly, we observed that the performance on Split-Omniglot in the domain-incremental setting of Sec. B.2 degraded as we increased the number of tasks: accuracies for 5, 10 and 20 tasks are 92.3 % ± 0.4 92.3\%\pm 0.4 , 82.0 % ± 0.4 82.0\%\pm 0.4 and 67.6 % ± 1.1 67.6\%\pm 1.1 , respectively. As noted in Sec. 5 , this is a general limitation of sequence processing neural networks, and there is a potential remedy for this limitation (meta-training on more tasks and with “context carry-over”) which we leave for future work.

### B.6 More Visualizations

Figure 5 shows the continuation of Figures 3 and 4 , corresponding to the demonstrations of the third task (class ‘4’ vs. ‘5’).

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .