Metalearning Continual Learning Algorithms — part 1/5

Cleaned-text code points [0, 15592). Bibliography omitted here; source.md includes it.

# Metalearning Continual Learning Algorithms

###### Abstract

General-purpose learning systems should improve themselves in open-ended fashion in ever-changing environments. Conventional learning algorithms for neural networks, however, suffer from catastrophic forgetting (CF), i.e., previously acquired skills are forgotten when a new task is learned. Instead of hand-crafting new algorithms for avoiding CF, we propose Automated Continual Learning (ACL) to train self-referential neural networks to metalearn their own in-context continual (meta)learning algorithms. ACL encodes continual learning (CL) desiderata---good performance on both old and new tasks---into its metalearning objectives. Our experiments demonstrate that ACL effectively resolves ‘‘in-context catastrophic forgetting,’’ a problem that naive in-context learning algorithms suffer from; ACL-learned algorithms outperform both hand-crafted learning algorithms and popular meta-continual learning methods on the Split-MNIST benchmark in the replay-free setting, and enables continual learning of diverse tasks consisting of multiple standard image classification datasets. We also discuss the current limitations of in-context CL by comparing ACL with state-of-the-art CL methods that leverage pre-trained models. Overall, we bring several novel perspectives into the long-standing problem of CL. 1 1 1 This work was conducted while the authors were at the Swiss AI Lab, IDSIA (2023). An early version of this work, titled “Automating Continual Learning”, was made available online in September 2023. See also closely related concurrent work by Lee et al. (2023) and Vettoruzzo et al. (2024) ; ours builds on our prior work on “in-context sequential multi-task learning” ( Irie et al., 2022c ) . Our code is public: https://github.com/IDSIA/automated-cl .

Reviewed on OpenReview: https://openreview.net/forum?id=IaUh7CSD3k

## 1 Introduction

Enemies of memories are other memories ( Eagleman, 2020 ) . Continually learning artificial neural networks (NNs) are memory systems in which their weights store memories of task-solving skills or programs, and their learning algorithm is responsible for memory read/write operations. Conventional learning algorithms, which are used to train NNs in the standard scenarios where all training data is available at once , are known to be inadequate for continual learning (CL) of multiple tasks where data for each task is available sequentially and exclusively , one at a time. They suffer from catastrophic forgetting (CF; McCloskey and Cohen (1989) ; Ratcliff (1990) ; French (1999) ; McClelland et al. (1995) ): NNs forget, or rather, a learning algorithm erases previously acquired skills, in exchange for learning to solve a new task.

Naturally, a certain degree of forgetting is unavoidable when the memory capacity is limited, and the amount of things to remember exceeds such an upper bound. In general, however, capacity is not the fundamental cause of CF; typically, the same NNs that suffer from CF when trained sequentially on two tasks can perform well on both tasks when trained jointly on them instead (see, e.g., Hsu et al. (2018) ; Irie et al. (2022a) ).

The real root cause of CF lies in the learning algorithm as a memory mechanism. An effective CL algorithm should preserve previously acquired knowledge while also leveraging previous learning experiences to improve future learning, by maximally exploiting the limited memory space of model parameters. All of this is the decision-making problem of learning algorithms . In fact, we cannot blame conventional learning algorithms for causing CF, since they are not “aware” of such a problem. They are designed to train NNs for a given task at hand; they treat each learning experience independently (they are stationary up to certain momentum parameters in certain optimizers), and ignore any potential influence of current learning on past or future learning experiences. Effectively, more sophisticated algorithms previously proposed to combat CF ( Kortge, 1990 ; French, 1991 ) , such as elastic weight consolidation ( Kirkpatrick et al., 2017 ; Schwarz et al., 2018 ) or synaptic intelligence ( Zenke et al., 2017 ) , often introduce manually designed constraints as regularization terms to explicitly penalize certain modifications to the previously learned model parameters/weights.

Here, instead of hand-crafting learning algorithms for continual learning, we train sequence-learning self-referential neural networks ( Schmidhuber, 1992a ; Schmidhuber, 1987 ) to metalearn their own “in-context” continual learning algorithms. We train them through gradient descent on metalearning objectives that reflect desiderata of CL—good performance on both old and new tasks. In fact, by extending the standard settings of few-shot/metalearning based on sequence-processing NNs ( Hochreiter et al. (2001) ; Younger et al. (1999) ; Cotter and Conwell (1991) ; Cotter and Conwell (1990) ; Mishra et al. (2018) ; see Sec. 2.2 ), the continual learning problem can also be formulated as a long-span sequence processing task ( Irie et al., 2022c ) . We can obtain such CL sequences by concatenating multiple few-shot/metalearning “episodes,” where each episode is a sequence of input/target examples corresponding to a task to be learned. As we’ll see in Sec. 3 , this setting also allows us to seamlessly incorporate classic desiderata of CL into the objective functions of the metalearner.

Once formulated as a sequence learning problem, we let gradient descent search for CL algorithms achieving the desired CL behaviors in the program space of NN weights. In principle, all typical challenges of CL—such as the stability-plasticity dilemma ( Grossberg, 1982 ; Elsayed and Mahmood, 2024 ) —are automatically discovered and handled by the gradient-based program search process. Once meta-trained, CL is automated through recursive self-modification dynamics of the NN, without requiring any human intervention such as adding extra regularization or tuning hyper-parameters. Therefore, we call our method, Automated Continual Learning (ACL).

Our experiments focus on supervised image classification, making use of standard few-shot learning datasets for meta-training, namely, Mini-ImageNet ( Vinyals et al., 2016 ; Ravi and Larochelle, 2017 ) , Omniglot ( Lake et al., 2015 ) , and FC100 ( Oreshkin et al., 2018 ) , while we also meta-test on other datasets including MNIST ( LeCun et al., 1998 ) , FashionMNIST ( Xiao et al., 2017 ) and CIFAR-10 ( Krizhevsky, 2009 ) .

Our experiments reveal various facets of in-context CL: (1) we show that without ACL, naive in-context learners suffer from “in-context catastrophic forgetting” (Sec. 4.1 ); we illustrate its emergence (Sec. 4.2 ) using comprehensible two-task settings, (2) we show very promising practical results of ACL by successfully metalearning a CL algorithm that outperforms hand-crafted learning algorithms and prior meta-continual learning methods ( Javed and White, 2019 ; Beaulieu et al., 2020 ; Banayeeanzade et al., 2021 ) on the classic Split-MNIST benchmark ( Hsu et al. (2018) ; Van de Ven and Tolias (2018b) ; Sec. 4.3 ), and (3) we highlight the current limitations and the need for further scaling up ACL, through a comparison with the prompt-based CL methods ( Wang et al., 2022b ; Wang et al., 2022a ) that leverage pre-trained models, using Split-CIFAR100 and 5-datasets ( Ebrahimi et al., 2020 ) .

## 2 Background

Here we provide a brief review of background concepts essential for describing our method in Sec. 3 : continual learning and its desiderata (Sec. 2.1 ), few-shot/metalearning via sequence processing (Sec. 2.2 ), and linear transformer/fast weight programmer architectures (Sec. 2.3 ) which form the foundations of the sequence processing neural network we use in our experiments.

### 2.1 Continual Learning

The main scope of this work is continual learning ( Ring, 1994 ; Caruana, 1997 ; Thrun, 1998 ) in supervised learning settings, even though high-level principles we discuss here also transfer to reinforcement learning. In addition, we focus on CL methods that keep model sizes constant (unlike certain CL methods that incrementally add more parameters as more tasks are presented; see, e.g., Rusu et al. (2016) ), and we do not make use of any external replay memory (used in other CL methods; see, e.g., Robins (1995) ; Shin et al. (2017) ; Rolnick et al. (2019) ; Riemer et al. (2019) ; Zhang et al. (2022) ).

Classic desiderata for a CL system (see, e.g., Lopez-Paz and Ranzato (2017) ; Veniat et al. (2021) ) are typically summarized as good performance on three metrics: classification accuracies on each dataset (their average), backward transfer (i.e., impact of learning a new task on the model’s performance on previous tasks; e.g., catastrophic forgetting is a negative backward transfer), and forward transfer (impact of learning the current task on the model’s performance in a future task). From a broader perspective of metalearning systems, we may also want to measure learning acceleration (i.e., whether the system leverages previous learning experiences to accelerate future learning); here our primary focus is the classic CL metrics above.

### 2.2 Metalearning via Sequence Learning a.k.a. In-Context Learning

In Sec. 3 , we formulate continual learning as a long-span sequence processing task. This is a direct extension of the classic formulation of few-shot/metalearning as a sequence learning problem, which we briefly review here.

Unlike standard learning whose goal is to train a model on a fixed task, metalearning involves training a model on many tasks or learning episodes , where each task serves as a learning example for the model’s metalearning, so that the model learns its own algorithm to learn a new task. Each episode consists of a sequence of training examples (or demonstrations ), followed by a test example (or query ) whose label (or target ) is what the model is tasked to predict; such a sequence can be processed by a sequence processing neural network. More formally, let d d , N N , K K , P P be positive integers. Here we assume that each task is a N N -way classification task with K K demonstrations for each class (the so-called N N -way K K -shot classification settings). At each time step t ∈ { 1 , … , N ⋅ K } t\in\{1,...,N\cdot K\} , a sequence processing NN with a parameter vector θ ∈ ℝ P \theta\in\mathbb{R}^{P} observes a pair ( 𝒙 t {\bm{x}}_{t} , y t y_{t} ) where 𝒙 t ∈ ℝ d {\bm{x}}_{t}\in\mathbb{R}^{d} is the observation/data and y t ∈ { 1 , … , N } y_{t}\in\{1,...,N\} is its label. After presentation of these N ⋅ K N\cdot K examples (demonstrations consisting of K K examples for each one of N N classes), one extra input 𝒙 ∈ ℝ d {\bm{x}}\in\mathbb{R}^{d} (a query) is fed to the model without its true label but with an “unknown label” token ∅ \varnothing (thus, the model can accept up to N + 1 N+1 different labels). The model is meta-trained to predict its true label (a target); that is, the model parameters θ \theta are optimized to maximize the probability p ⁡ ( y | ( 𝒙 1 , y 1 ) , … , ( 𝒙 N ⋅ K , y N ⋅ K ) , ( 𝒙 , ∅ ) ; θ ) p(y|({\bm{x}}_{1},y_{1}),...,({\bm{x}}_{N\cdot K},y_{N\cdot K}),({\bm{x}},\varnothing);\theta) of the correct label y ∈ { 1 , … , N } y\in\{1,...,N\} of the input query 𝒙 {\bm{x}} .

Meta-training requires many such sequences/episodes, which can be constructed by using a regular dataset with C C classes; for each sequence, we sample N N random but distinct classes out of C C ( N < C N<C ). The resulting classes are re-labelled such that each class is assigned to one out of N N distinct random label index which is unique to the sequence. For each of these N N classes, we sample K K examples. We randomly order these N ∗ K N*K examples to obtain a unique demonstration sequence. Since class-to-label associations are randomized and unique to each sequence ( ( 𝒙 1 , y 1 ) , … , ( 𝒙 N ⋅ K , y N ⋅ K ) , ( 𝒙 , ∅ ) ({\bm{x}}_{1},y_{1}),...,({\bm{x}}_{N\cdot K},y_{N\cdot K}),({\bm{x}},\varnothing) ), each such a sequence represents a new learning example to meta-train the model. To be more specific, this is the synchronous label setting of Mishra et al. (2018) where the learning phase (during which the model observes demo examples, ( 𝒙 1 , y 1 ) ({\bm{x}}_{1},y_{1}) , etc.) is separated from the prediction phase (predicting label y y given ( 𝒙 , ∅ ) ({\bm{x}},\varnothing) ). We opted for this variant in our experiments as we empirically found this (at least in our specific settings) more stable than the delayed label setting ( Hochreiter et al., 2001 ) where the model has to make a prediction for every input, and the label is fed to the model with a delay of one time step. The process for meta-testing is the same, except that θ \theta is not updated .

Note that this formulation of metalearning as sequence processing is not new. Since the seminal works ( Cotter and Conwell, 1990 ; Cotter and Conwell, 1991 ; Younger et al., 1999 ; Hochreiter et al., 2001 ) , many sequence processing neural networks (see, e.g., Bosc (2015) ; Santoro et al. (2016) ; Duan et al. (2016) ; Wang et al. (2017) ; Munkhdalai and Yu (2017) ; Munkhdalai and Trischler (2018) ; Miconi et al. (2018) ; Miconi et al. (2019) ; Munkhdalai et al. (2019) ; Kirsch and Schmidhuber (2021) ; Sandler et al. (2021) ; Huisman et al. (2023) , including Transformers ( Vaswani et al., 2017 ; Mishra et al., 2018 ) ) have been trained as a metalearner ( Schmidhuber, 1987 ; Schmidhuber, 1992a ) that metalearn to learn by observing sequences of training examples (i.e., pairs of inputs and their labels). More recently, this was rebranded as in-context learning in the context of language modeling ( Brown et al., 2020 ) .

### 2.3 Self-Referential Weight Matrices and Recursive Self-Transformers

##### General description.

Our method (Sec. 3 ) can be applied to any sequence-processing NN architectures in principle. Nevertheless, certain architectures naturally fit better to parameterize a self-improving continual learner. Here we use the modern self-referential weight matrix (SRWM; Irie et al. (2022c) ; Irie et al. (2023) ) to build a generic self-modifying NN. An SRWM is a weight matrix that sequentially modifies itself as a response to a stream of input observations ( Schmidhuber, 1992a ; Schmidhuber, 1993 ) .

The modern SRWM belongs to the family of linear Transformers (LTs) a.k.a. Fast Weight Programmers (FWPs; Schmidhuber (1991) ; Schmidhuber (1992b) ; Katharopoulos et al. (2020) ; Choromanski et al. (2021) ; Peng et al. (2021) ; Schlag et al. (2021) ; Irie et al. (2021a) ). Linear Transformers and FWPs are an important class of the now popular Transformers ( Vaswani et al., 2017 ) ; unlike the standard ones whose computational requirements grow quadratically and whose state size grows linearly with the sequence length, LTs/FWPs’ complexity is linear and the state size is constant w.r.t. sequence length, similar to the standard recurrent neural network. This property is particularly relevant for in-context CL, as we ultimately aim for such a system to continue learning over an arbitrarily long, lifelong timespan. Moreover, the duality between linear attention and FWPs ( Schlag et al., 2021 ) —and likewise, between linear attention and gradient descent-trained linear layers ( Irie et al., 2022a ; Aizerman et al., 1964 ) —have played a key role in intuitively conceptualizing in-context learning capabilities of Transformers ( von Oswald et al., 2023a ; Dai et al., 2023 ) .