Metalearning Continual Learning Algorithms — part 2/5

Cleaned-text code points [15592, 30777). Bibliography omitted here; source.md includes it.

The dynamics of an SRWM layer ( Irie et al., 2022c ) are described as follows. Let d in d_{\text{in}} , d out d_{\text{out}} , t t be positive integers, and ⊗ \otimes denote outer product. At each time step t t , an SRWM 𝑾 t − 1 ∈ ℝ ( d out + 2 ∗ d in + 1 ) × d in {\bm{W}}_{t-1}\in\mathbb{R}^{(d_{\text{out}}+2*d_{\text{in}}+1)\times d_{\text{in}}} observes an input 𝒖 t ∈ ℝ d in {\bm{u}}_{t}\in\mathbb{R}^{d_{\text{in}}} , and outputs 𝒐 t ∈ ℝ d out {\bm{o}}_{t}\in\mathbb{R}^{d_{\text{out}}} , while also updating itself from 𝑾 t − 1 {\bm{W}}_{t-1} to 𝑾 t {\bm{W}}_{t} as: [ 𝒐 t , 𝒌 t , 𝒒 t , β t ] \displaystyle[{\bm{o}}_{t},{\bm{k}}_{t},{\bm{q}}_{t},\beta_{t}] = 𝑾 t − 1 ​ 𝒖 t \displaystyle={\bm{W}}_{t-1}{\bm{u}}_{t} (1) 𝒗 t = 𝑾 t − 1 ​ ϕ ​ ( 𝒒 t ) \displaystyle{\bm{v}}_{t}={\bm{W}}_{t-1}\phi({\bm{q}}_{t}) ; 𝒗 ¯ t = 𝑾 t − 1 ϕ ( 𝒌 t ) \displaystyle;\,\bar{{\bm{v}}}_{t}={\bm{W}}_{t-1}\phi({\bm{k}}_{t}) (2) 𝑾 t = 𝑾 t − 1 + σ ⁡ ( CLOSE \displaystyle{\bm{W}}_{t}={\bm{W}}_{t-1}+\sigma( OPEN β t ) ​ ( 𝒗 t − 𝒗 ¯ t ) ⊗ ϕ ⁡ ( 𝒌 t ) \displaystyle\beta_{t})({\bm{v}}_{t}-\bar{{\bm{v}}}_{t})\otimes\phi({\bm{k}}_{t}) (3) where 𝒗 t , 𝒗 ¯ t ∈ ℝ ( d out + 2 ∗ d in + 1 ) {\bm{v}}_{t},\bar{{\bm{v}}}_{t}\in\mathbb{R}^{(d_{\text{out}}+2*d_{\text{in}}+1)} are value vectors, 𝒒 t ∈ ℝ d in {\bm{q}}_{t}\in\mathbb{R}^{d_{\text{in}}} and 𝒌 t ∈ ℝ d in {\bm{k}}_{t}\in\mathbb{R}^{d_{\text{in}}} are query and key vectors, and σ ⁡ ( β t ) ∈ ℝ \sigma(\beta_{t})\in\mathbb{R} is the learning rate. σ \sigma and ϕ \phi denote sigmoid and softmax functions respectively. ϕ \phi is typically also applied to 𝒖 t {\bm{u}}_{t} in Eq. 1 ; here we follow Irie et al. (2022c) ’s few-shot image classification setting, and use the variant without it. Eq. 3 corresponds to a rank-one update of the SRWM, from 𝑾 t − 1 {\bm{W}}_{t-1} to 𝑾 t {\bm{W}}_{t} , through the delta learning rule ( Widrow and Hoff, 1960 ; Schlag et al., 2021 ) where the self-generated patterns, 𝒗 t {\bm{v}}_{t} , ϕ ⁡ ( 𝒌 t ) \phi({\bm{k}}_{t}) , and σ ⁡ ( β t ) \sigma(\beta_{t}) , play the role of target , input , and learning rate of the learning rule respectively. The delta rule is crucial for the performance of LTs ( Schlag et al., 2021 ; Irie et al., 2021a ; Irie et al., 2022b ; Irie and Schmidhuber, 2023b ; Yang et al., 2024b ; Yang et al., 2024a ) .

The initial weight matrix 𝑾 0 {\bm{W}}_{0} is the only set of trainable parameters of this layer, which encodes the seed self-modification algorithm. We use the multi-head version of the computation above ( Irie et al., 2022c ) to directly replace the multi-head self-attention layer in the Transformer, yielding a “Recursive Self-Transformer”.

4-learning-rate version. In practice, we use the “4-learning rate” version ( Irie et al., 2022c ) of SRWM that learns to use different learning rates for each of “o”, “k”, “q”, “ β \beta ” sub-blocks of 𝑾 t − 1 {\bm{W}}_{t-1} by splitting 𝑾 t − 1 {\bm{W}}_{t-1} into sub-matrices: 𝑾 t − 1 = [ 𝑾 t − 1 o , 𝑾 t − 1 k , 𝑾 t − 1 q , 𝑾 t − 1 b ] {\bm{W}}_{t-1}=[{\bm{W}}^{o}_{t-1},{\bm{W}}^{k}_{t-1},{\bm{W}}^{q}_{t-1},{\bm{W}}^{b}_{t-1}] that produce 𝒐 t {\bm{o}}_{t} , 𝒌 t {\bm{k}}_{t} , 𝒒 t {\bm{q}}_{t} , and β t \beta_{t} , respectively, in Eq. 1 . As we use 4 learning rates, the dimension of 𝑾 t − 1 {\bm{W}}_{t-1} becomes ℝ ( d out + 2 ∗ d in + 4 ) × d in \mathbb{R}^{(d_{\text{out}}+2*d_{\text{in}}+4)\times d_{\text{in}}} , and β t = [ β t o , β t k , β t q , β t b ] ∈ ℝ 4 \beta_{t}=[\beta^{o}_{t},\beta^{k}_{t},\beta^{q}_{t},\beta^{b}_{t}]\in\mathbb{R}^{4} . For example, the corresponding update equations for the “o”-part 𝑾 t − 1 o {\bm{W}}^{o}_{t-1} are: 𝒐 t q = 𝑾 t − 1 o ​ ϕ ​ ( 𝒒 t ) \displaystyle{\bm{o}}^{q}_{t}={\bm{W}}^{o}_{t-1}\phi({\bm{q}}_{t}) ; 𝒐 t k = 𝑾 t − 1 o ϕ ( 𝒌 t ) \displaystyle;\,{\bm{o}}^{k}_{t}={\bm{W}}^{o}_{t-1}\phi({\bm{k}}_{t}) (4) 𝑾 t o = 𝑾 t − 1 o + σ ⁡ ( CLOSE \displaystyle{\bm{W}}^{o}_{t}={\bm{W}}^{o}_{t-1}+\sigma( OPEN β t o ) ​ ( 𝒐 t q − 𝒐 t k ) ⊗ ϕ ⁡ ( 𝒌 t ) \displaystyle\beta^{o}_{t})({\bm{o}}^{q}_{t}-{\bm{o}}^{k}_{t})\otimes\phi({\bm{k}}_{t}) (5) where 𝒐 t q {\bm{o}}^{q}_{t} and 𝒐 t k {\bm{o}}^{k}_{t} denote the “o”-part of 𝒗 t {\bm{v}}_{t} and 𝒗 ¯ t \bar{{\bm{v}}}_{t} in Eq. 2 respectively, and β t o ∈ ℝ \beta^{o}_{t}\in\mathbb{R} is one of the four learning rates that is dedicated to the “o”-part. The update equations for 𝑾 t − 1 q {\bm{W}}^{q}_{t-1} , 𝑾 t − 1 k {\bm{W}}^{k}_{t-1} , 𝑾 t − 1 β {\bm{W}}^{\beta}_{t-1} are analogous.

## 3 Method

Here we describe the proposed approach, Automated Continual Learning (ACL).

Problem Formalization. Building on the formulation of learning as sequence processing (Sec. 2.2 ), we formulate continual learning also as a long-span sequence learning task. Let D D , N N , K K , L L denote positive integers. Consider two N N -way classification tasks 𝐀 {\mathbf{A}} and 𝐁 {\mathbf{B}} to be learned sequentially (this can be straightforwardly extended to more tasks). We denote the respective training datasets as 𝒜 \mathcal{A} and ℬ \mathcal{B} , and test sets as 𝒜 ′ \mathcal{A^{\prime}} and ℬ ′ \mathcal{B^{\prime}} . The formulation here applies to both “meta-training” and “meta-test” phases (see Appendix A.1 for more on this terminology). We assume that each datapoint in these datasets consists of one input feature 𝒙 ∈ ℝ D {\bm{x}}\in\mathbb{R}^{D} of dimension D D (generically denoted as vector 𝒙 {\bm{x}} , but it is an image in all our experiments) and one label y ∈ { 1 , … , N } y\in\{1,...,N\} . We consider two sequences of L L training examples ( ( 𝒙 1 𝒜 , y 1 𝒜 ) , … , ( 𝒙 L 𝒜 , y L 𝒜 ) ) \left(({\bm{x}}^{\mathcal{A}}_{1},y^{\mathcal{A}}_{1}),...,({\bm{x}}^{\mathcal{A}}_{L},y^{\mathcal{A}}_{L})\right) and ( ( 𝒙 1 ℬ , y 1 ℬ ) , … , ( 𝒙 L ℬ , y L ℬ ) ) \left(({\bm{x}}^{\mathcal{B}}_{1},y^{\mathcal{B}}_{1}),...,({\bm{x}}^{\mathcal{B}}_{L},y^{\mathcal{B}}_{L})\right) sampled from the respective training sets 𝒜 \mathcal{A} and ℬ \mathcal{B} . In practice, L = N ​ K L=NK where K K is the number of training examples for each class. By concatenating these two sequences, we obtain one long sequence representing CL examples to be presented to a (left-to-right processing) auto-regressive sequence model. At the end of the sequence, the model is tasked to make predictions on test/query examples sampled from both 𝒜 ′ \mathcal{A^{\prime}} and ℬ ′ \mathcal{B^{\prime}} ; we assume a single query example for each task (hence, without index): ( 𝒙 𝒜 ′ , y 𝒜 ′ ) ({\bm{x}}^{\mathcal{A^{\prime}}},y^{\mathcal{A^{\prime}}}) and ( 𝒙 ℬ ′ , y ℬ ′ ) ({\bm{x}}^{\mathcal{B^{\prime}}},y^{\mathcal{B^{\prime}}}) respectively; which we further denote as ( 𝒙 query 𝒜 , y target 𝒜 ) ({\bm{x}}^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}query}}},y^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}target}}}) and ( 𝒙 query ℬ , y target ℬ ) ({\bm{x}}^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}query}}},y^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}target}}}) for clarity.

Our model is a self-referential NN that modifies its own weight matrices as a function of input observations. To simplify the notation, we denote the state of our self-referential NN as a single SRWM 𝑾 ∗ {\bm{W}}_{*} (even though the model may contain many of them in practice) where we’ll replace ∗ * by various symbols representing the context/inputs fed to the model. Given a sequence ( ( 𝒙 1 𝒜 , y 1 𝒜 ) , … , ( 𝒙 L 𝒜 , y L 𝒜 ) , ( 𝒙 1 ℬ , y 1 ℬ ) , … , ( 𝒙 L ℬ , y L ℬ ) ) \left(({\bm{x}}^{\mathcal{A}}_{1},y^{\mathcal{A}}_{1}),...,({\bm{x}}^{\mathcal{A}}_{L},y^{\mathcal{A}}_{L}),({\bm{x}}^{\mathcal{B}}_{1},y^{\mathcal{B}}_{1}),...,({\bm{x}}^{\mathcal{B}}_{L},y^{\mathcal{B}}_{L})\right) , the model processes one x-y pair as input at a time, from left to right, in an auto-regressive manner. Let 𝑾 𝒜 {\bm{W}}_{\mathcal{A}} denote the state of the SRWM that has consumed the first part of the sequence, i.e., the examples from Task 𝐀 {\mathbf{A}} , ( 𝒙 1 𝒜 , y 1 𝒜 ) , … , ( 𝒙 L 𝒜 , y L 𝒜 ) ({\bm{x}}^{\mathcal{A}}_{1},y^{\mathcal{A}}_{1}),...,({\bm{x}}^{\mathcal{A}}_{L},y^{\mathcal{A}}_{L}) , and let 𝑾 𝒜 , ℬ {\bm{W}}_{\mathcal{A},\mathcal{B}} denote the state of the SRWM after having observed the entire sequence.

ACL Meta-Training Objectives. The ACL meta-training objective consists in correctly predicting the target for queries of all the tasks learned so far, at every task boundary. That is, in the case of two-task scenario described above (learning Task 𝐀 {\mathbf{A}} then Task 𝐁 {\mathbf{B}} ), we use the weight matrix 𝑾 𝒜 {\bm{W}}_{\mathcal{A}} to predict the label y target 𝒜 y^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}target}}} from input ( 𝒙 query 𝒜 , ∅ ) ({\bm{x}}^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}query}}},\varnothing) ; and we use the weight matrix 𝑾 𝒜 , ℬ {\bm{W}}_{\mathcal{A},\mathcal{B}} to predict the label y target ℬ y^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}target}}} from input ( 𝒙 query ℬ , ∅ ) ({\bm{x}}^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}query}}},\varnothing) as well as the label y target 𝒜 y^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}target}}} from input ( 𝒙 query 𝒜 , ∅ ) ({\bm{x}}^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}query}}},\varnothing) . Figure 1 provides an illustration.

Let p ⁡ ( y | 𝒙 ; 𝑾 ∗ ) p(y|{\bm{x}};{\bm{W}}_{*}) denote the model’s output probability for label y ∈ { 1 , . . , N } y\in\{1,..,N\} given input 𝒙 {\bm{x}} and model state 𝑾 ∗ {\bm{W}}_{*} . The ACL objective can be expressed as: ⁡ m ​ i ​ n ​ i ​ m ​ i ​ z ​ e θ − ( log ⁡ ( p ⁡ ( y target 𝒜 | 𝒙 query 𝒜 ; 𝑾 𝒜 ​ ( θ ) ) ) + log ⁡ ( p ⁡ ( y target ℬ | 𝒙 query ℬ ; 𝑾 𝒜 , ℬ ​ ( θ ) ) ) + log ⁡ ( p ⁡ ( y target 𝒜 | 𝒙 query 𝒜 ; 𝑾 𝒜 , ℬ ​ ( θ ) ) ) ) \displaystyle\mathop{\text{}}{minimize}_{\theta}-\left(\log(p(y^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A}}{\color[rgb]{0,0,0}(\theta)}))+\log(p(y^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A,B}}{\color[rgb]{0,0,0}(\theta)}))+\log(p(y^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A,B}}{\color[rgb]{0,0,0}(\theta)}))\right) (6) for an arbitrary meta-training sequence ( ( 𝒙 1 𝒜 , y 1 𝒜 ) , … , ( 𝒙 L 𝒜 , y L 𝒜 ) , ( 𝒙 1 ℬ , y 1 ℬ ) , … , ( 𝒙 L ℬ , y L ℬ ) ) \left(({\bm{x}}^{\mathcal{A}}_{1},y^{\mathcal{A}}_{1}),...,({\bm{x}}^{\mathcal{A}}_{L},y^{\mathcal{A}}_{L}),({\bm{x}}^{\mathcal{B}}_{1},y^{\mathcal{B}}_{1}),...,({\bm{x}}^{\mathcal{B}}_{L},y^{\mathcal{B}}_{L})\right) (extensible to mini-batches using multiple such sequences), where θ \theta denotes the model parameters (e.g., initial weights 𝑾 0 {\bm{W}}_{0} for an SRWM layer) which are trained using “memory-efficient” backpropagation through time of Irie et al. (2022c) .

The ACL objective function above (Eq. 6 ) is simple but encapsulates desiderata of continual learning (Sec. 2.1 ). The last term of Eq. 6 with p ⁡ ( y target 𝒜 | 𝒙 query 𝒜 ; 𝑾 𝒜 , ℬ ) p(y^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{A}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A,B}}) or schematically 𝒑 ⁡ ( 𝒜 ′ | 𝒜 , ℬ ) {\bm{p}}(\mathcal{A}^{\prime}|\mathcal{A},\mathcal{B}) , optimizes for backward transfer : (1) remembering the first task 𝐀 {\mathbf{A}} after learning 𝐁 {\mathbf{B}} (combatting catastrophic forgetting), and (2) leveraging learning of 𝐁 {\mathbf{B}} to improve performance on the past task 𝐀 {\mathbf{A}} . The second term of Eq. 6 , p ⁡ ( y target ℬ | 𝒙 query ℬ ; 𝑾 𝒜 , ℬ ) p(y^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}target}}}|{\bm{x}}^{\mathcal{B}}_{\text{{\color[rgb]{0,0,0}query}}};{\bm{W}}_{\mathcal{A,B}}) or 𝒑 ⁡ ( ℬ ′ | 𝒜 , ℬ ) {\bm{p}}(\mathcal{B}^{\prime}|\mathcal{A},\mathcal{B}) , optimizes forward transfer leveraging the past learning experience of 𝐀 {\mathbf{A}} to improve predictions in the second task 𝐁 {\mathbf{B}} , in addition to simply learning to solve Task 𝐁 {\mathbf{B}} from the corresponding demonstrations. Finally, the first term of Eq. 6 incentivizes the model to learn Task 𝐀 {\mathbf{A}} as soon as Task 𝐀 {\mathbf{A}} demos are observed.

Overall Model Architecture. As discussed in Sec. 2.3 , in our model, the core sequential dynamics of CL are learned by the self-referential layers. However, as an image-processing NN, our model makes use of a vision backend: We use the “Conv-4” architecture ( Vinyals et al., 2016 ) in all our experiments, except in the last one where we use a pre-trained vision Transformer ( Dosovitskiy et al., 2021 ) . Overall, the model takes an ( 𝒙 {\bm{x}} /image, y y /label)-pair as input; the image is processed through a feedforward vision NN to yield a feature vector, and the label is encoded as a one-hot vector. We concatenate these two vectors, and feed it to a linear projection layer to obtain the input ( 𝒖 t {\bm{u}}_{t} ; Eq. 1 ) for the first SRWM layer. Note that this is one of the limitations of this work: more general ACL should also learn to self-modify the vision components. 2 2 2 One “straightforward” architecture fitting the bill is the MLP-mixer architecture ( Tolstikhin et al. (2021) ; built of several linear layers), where all linear layers are replaced by the self-referential linear layers of Sec. 2.3 . While we implemented such a model, it turned out to be too slow for us to conduct corresponding experiments. Our code includes a “self-referential MLP-mixer” implementation, but for further experiments, future work on such an architecture may require a more efficient implementation.

Another crucial architectural choice that is specific to continual/multi-task image processing is normalization layers (see also related discussion in Bronskill et al. (2020) ). Typical NNs used in few-shot learning (e.g., Vinyals et al. (2016) ) contain batch normalization layers (BN; Ioffe and Szegedy (2015) ) in the vision backend. All our models use instance normalization (IN; Ulyanov et al. (2016) ) instead of BN because in our preliminary experiments, we expectably found IN to generalize much better than BN layers in the CL setting.

## 4 Experiments

98.2

54.6

46.2

53.0

81.5

51.8

68.2

52.5

### 4.1 Two-Task Setting: Comprehensible Study

Similar to how conventional learning algorithms suffer from catastrophic forgetting in the continual learning setting, we first show that in-context learning also suffers from “in-context catastrophic forgetting,” and that the ACL method (Sec. 3 ) can effectively help overcome it. As a minimal case to illustrate this, we focus on the two-task “domain-incremental” CL setting (see Appendix A.1 for details regarding this standard CL terminology). We consider two scenarios for meta-training datasets to be used to sample tasks : Omniglot ( Lake et al., 2015 ) and Mini-ImageNet ( Vinyals et al., 2016 ; Ravi and Larochelle, 2017 ) ; or FC100 ( Oreshkin et al. (2018) , based on CIFAR100; Krizhevsky (2009) ) and Mini-ImageNet (see Appendix A.2 for data details). The order of the datasets used to sample the two tasks within the meta-training sequences is alternated for every batch. We compare the models with and without the backward transfer term in the ACL loss (the last term in Eq. 6 ).