Continual Learning and Catastrophic Forgetting — part 3/5

Cleaned-text code points [31241, 46539). Bibliography omitted here; source.md includes it.

Instead of explicitly storing past observations, it is also possible to learn a generative model, which can then be used to generate the data to be replayed ( Mocanu et al., 2016 ; Shin et al., 2017 ; Wu et al., 2018 ; Rao et al., 2019 ; Cong et al., 2020 ; Khan et al., 2023 ) . An issue with such ‘generative replay’ is that it can be difficult to train generative models of decent quality, especially in an incremental setting or when data are complex ( Aljundi et al., 2019a ; Lesort et al., 2019a ) . This issue can sometimes be alleviated by replaying latent features instead of raw inputs ( Liu et al., 2020 ; Pellegrini et al., 2020 ; Ostapenko et al., 2022 ) , but this ‘internal replay’ approach needs some form of pretraining to work well.

When replay is used to train a deep neural network, the loss on the replayed data ( ℓ replay \ell_{\text{replay}} ) is usually added to the loss on the current data ( ℓ current \ell_{\text{current}} ), possibly with some weighting applied, and the objective is to optimize the combined loss: ℓ total ​ ( 𝜽 ) = ℓ current ​ ( 𝜽 ) + ℓ replay ​ ( 𝜽 ) \ell_{\text{total}}(\boldsymbol{\theta})=\ell_{\text{current}}(\boldsymbol{\theta})+\ell_{\text{replay}}(\boldsymbol{\theta}) (2) with 𝜽 \boldsymbol{\theta} the parameters of the network that can be updated during training.

An alternative way to use the loss on the replayed data is to construct one or more inequality constraints that should be respected when optimizing the loss on the current data ( Lopez-Paz and Ranzato, 2017 ; Chaudhry et al., 2019a ) . Although this strategy, which is referred to as ‘gradient episodic memory’, is sometimes categorized in the literature under replay, we consider it an optimization-based approach and discuss it in subsection 3.4 .

Especially for large and complex continual learning problems, replay seems to play an important and perhaps necessary role. However, as replay involves constantly retraining on past data, an important concern with this approach is the potentially high computational cost. Luckily, there is evidence that it is not necessary to fully (re)train on all past tasks whenever a new task is learned. One reason is that learning something new is typically more demanding than preventing its forgetting once it has been learned, which explains why it can be sufficient to replay only relatively small amounts of data ( van de Ven et al., 2020 ) . In the cognitive science literature, it has further been suggested that it might only be needed to replay old data that are similar to the new data ( McClelland et al., 2020 ) . The intuition for this is that interference between unrelated items should be small anyway (for example because such items are stored in separate parts of the network). This idea has inspired a series of continual learning studies asking how to adaptively select which data to replay given the currently observed data ( Riemer et al., 2019 ; Aljundi et al., 2019a ; Klasson et al., 2023 ; Krawczyk and Gepperth, 2023 ) . Other open questions with regards to replay are how to store and compress data, in what format to replay data (e.g., raw data or intermediate features), and how to best integrate replay with other methods.

### 3.2 Parameter Regularization

Another popular approach for continual learning is parameter regularization. When a new task is learned, parameter regularization discourages large changes to parameters of the network that are thought to be important for previous tasks (Fig. 4 b). From a neuroscience perspective, this approach can be linked to metaplasticity ( Abraham, 2008 ) , as it can be interpreted as equipping the network parameters with an internal state that modulates their level of plasticity ( Zenke et al., 2017 ) . Another motivation for parameter regularization comes from a Bayesian perspective, as instances of this approach can often be expressed or interpreted as performing sequential approximate Bayesian inference on the parameters of a neural network ( Kirkpatrick et al., 2017 ; Nguyen et al., 2018 ; Farquhar and Gal, 2019 ) .

We define parameter regularization as adding a regularization term to the loss function to penalize changes to the network’s parameters 𝜽 \boldsymbol{\theta} , whereby the applied penalty is usually weighted by an estimate of how important parameters are for previously learned tasks (e.g., Kirkpatrick et al., 2017 ; Zenke et al., 2017 ; Aljundi et al., 2018 ; Schwarz et al., 2018 ; Ritter et al., 2018 ): ℓ total ​ ( 𝜽 ) = ℓ current ​ ( 𝜽 ) + ‖ 𝜽 − 𝜽 ∗ ‖ Σ \ell_{\text{total}}(\boldsymbol{\theta})=\ell_{\text{current}}(\boldsymbol{\theta})+\left\lVert\boldsymbol{\theta}-\boldsymbol{\theta}^{*}\right\rVert_{\Sigma} (3) with 𝜽 ∗ \boldsymbol{\theta}^{*} the value of the parameters relative to which changes are penalized (often this is the value of 𝜽 \boldsymbol{\theta} at the end of the previous task), Σ \Sigma an estimate of how important the parameters are for past tasks and ‖ . ‖ Σ \left\lVert.\right\rVert_{\Sigma} a weighted norm. The most common choice is a weighted L 2 L^{2} -norm, in which case the regularization term becomes 1 2 ​ ( 𝜽 − 𝜽 ∗ ) T ​ Σ ​ ( 𝜽 − 𝜽 ∗ ) \frac{1}{2}\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{*}\right)^{T}\Sigma\left(\boldsymbol{\theta}-\boldsymbol{\theta}^{*}\right) . We note that it is also possible to use estimates of the importance of parameters for past tasks in other ways, for example to reduce the learning rate for relatively important parameters (e.g., Özgün et al., 2020 ) or to perform gradient projection during optimization (e.g., Kao et al., 2021 ). Although in the literature these approaches are sometimes categorized under parameter regularization, we consider them optimization-based approaches and discuss them in subsection 3.4 .

A pivotal aspect of parameter regularization is estimating the importance of the network’s parameters for past tasks. An often-used method is to leverage the Fisher Information matrix ( Kirkpatrick et al., 2017 ) , which, under certain assumptions, indicates how a small change to the parameters would impact the loss. The Fisher Information is typically approximated with a diagonal matrix, thus assuming independence among all parameters. However, this assumption can be relaxed, for example by instead using a Kronecker-factored approximation ( Martens and Grosse, 2015 ; Ritter et al., 2018 ; Kao et al., 2021 ) . An important disadvantage of the Fisher Information is that it can be costly to compute. Several other parameter regularization methods instead estimate parameter importance online throughout training, which often incurs substantially lower computational costs ( Zenke et al., 2017 ; Aljundi et al., 2018 ) .

Although parameter regularization methods have shown success in task- and domain-incremental learning problems, they often struggle to learn inter-task boundaries in class-incremental learning scenarios ( Lesort et al., 2019b ; van de Ven et al., 2022 ; Kessler et al., 2023 ) .

### 3.3 Functional Regularization

An inherent difficulty with parameter regularization is that correctly estimating the importance of parameters for past tasks is very hard, which is due to the complex relation between the behaviour of a deep neural network and its parameters. Instead of operating in the parameter space, a more effective approach might be applying regularization in the function space of a neural network ( Benjamin et al., 2019 ; Pan et al., 2020 ; Titsias et al., 2020 ) . The goal of such functional regularization is to prevent large changes to a network’s input-output mapping f 𝜽 f_{\boldsymbol{\theta}} at a set of specific inputs, which are termed ‘anchor points’ (Fig. 4 c). Similar to parameter regularization, functional regularization can be expressed as adding a penalty term to the loss function: ℓ total ​ ( 𝜽 ) = ℓ current ​ ( 𝜽 ) + ⟨ f 𝜽 , f 𝜽 ∗ ⟩ 𝒜 \ell_{\text{total}}(\boldsymbol{\theta})=\ell_{\text{current}}(\boldsymbol{\theta})+\left<f_{\boldsymbol{\theta}},f_{\boldsymbol{\theta}^{*}}\right>_{\mathcal{A}} (4) with f 𝜽 ∗ f_{\boldsymbol{\theta}^{*}} the input-output mapping relative to which changes are penalized (often this is the input-output mapping of the network at the end of the previous task) and 𝒜 \mathcal{A} the set of anchor points where the divergence between f 𝜽 f_{\boldsymbol{\theta}} and f 𝜽 ∗ f_{\boldsymbol{\theta}^{*}} is evaluated.

There are different ways in which the divergence between f 𝜽 f_{\boldsymbol{\theta}} and f 𝜽 ∗ f_{\boldsymbol{\theta}^{*}} can be measured. For classification-based problems, following Li and Hoiem (2017) , a popular choice is to use the knowledge distillation loss proposed by Hinton et al. (2015) , which involves the cross entropy between temperature-scaled logits of both networks. However, the divergence between f 𝜽 f_{\boldsymbol{\theta}} and f 𝜽 ∗ f_{\boldsymbol{\theta}^{*}} does not need to be measured at the output level. In recent years, functional regularization is increasingly applied at various levels of the representation of neural networks, in which case it is also called feature distillation ( Heo et al., 2019 ; Douillard et al., 2020 ; Gomez-Villa et al., 2022 ; Roy et al., 2023 ) .

Another important aspect of functional regularization is the selection of anchor points. For relatively simple problems, Robins (1995) demonstrated that functional regularization with random patterns as anchor points, which he called ‘pseudorehearsal’, can already work reasonably well. However, for more complex problems, it is important that the set of anchor points is representative of the inputs from previous tasks. A naive solution would be to use all inputs that have been seen so far as anchor points, but this requires storing those inputs and functional regularization with a large number of anchor points can incur high computational costs. An option that does not involve storing past samples is using the currently observed inputs as anchor points ( Li and Hoiem, 2017 ) . This approach, which is known as ‘learning without forgetting’, tends to work well when the inputs from different tasks have similar structure (e.g., as is the case with tasks that all consist of natural images), but in general there is no guarantee that the current inputs are suitable anchor points for previous tasks. Another option is to use as anchor points a small number of strategically selected inputs from previous tasks that represent those tasks well. One way to select such representative inputs is by formulating neural networks as Gaussian Processes ( Khan et al., 2019 ) , as this allows for summarizing the input distributions of previous tasks with inducing points ( Titsias et al., 2020 ) or memorable inputs ( Pan et al., 2020 ) . How to optimally select the anchor points for functional regularization is still largely an open question.

Functional regularization is closely related to replay. In fact, functional regularization can be interpreted as a form of replay, whereby the replayed data consist of the anchor points labelled with the predictions for those points made by (a previous version of) the network itself. The key difference between both approaches is that with replay inputs and targets of past tasks are stored externally to the network (e.g., in a memory buffer or in the form of a separate generative model), while with functional regularization only inputs of past tasks are stored externally. With functional regularization, the input-output mapping of past tasks is therefore only stored internally in the network itself (or in its copies), while with replay at least part of this mapping is stored externally. However, in the continual learning literature, the distinction between replay and functional regularization is sometimes blurred. For example, with generative replay there is often only a generative model for the input distribution, and the generated inputs that are replayed are labelled based on predictions made for them by a previous version of the network (e.g., Shin et al., 2017 ). Consequently, despite its name, such generative replay is actually a form of functional regularization. Moreover, the replay of stored data from a memory buffer has also been combined with distillation (e.g., Buzzega et al., 2020 ; Boschini et al., 2023 ), making those instances a form of functional regularization as well.

### 3.4 Optimization-based Approaches

The three approaches discussed so far – replay, parameter regularization and functional regularization – operate by making changes to the loss function that is optimized (as shown by equations 2 - 4 ). An alternative approach to continual learning is to change how the loss function is optimized (Fig. 4 d). The standard optimization routines that are used in deep learning, such as stochastic gradient descent (SGD) and its variants (e.g., AdaGrad, Duchi et al., 2011 ; Adam, Kingma and Ba, 2014 ), have been developed for stationary settings. In non-stationary settings, there are typically no guarantees for their behavior, yet these standard optimization routines are the default choice in most work on continual learning. However, in the last few years there has been an increasing attention in the continual learning literature for the role of optimization, and there have been several attempts to develop novel optimization routines specific for continual learning. It has even been argued that an wholistic solution for continual learning must consist of both changes to the loss function and changes to how that loss function is optimized ( Hess et al., 2023 ) .

Already about six years ago it was empirically found that the type of optimizer (e.g., plain SGD, AdaGrad or Adam) can have a large effect on continual learning performance ( Hsu et al., 2018 ) . Later work explored a wider range of factors that influence optimization (e.g., learning rate, batch size) and concluded that the best optimization routines for continual learning are the ones that tend to find wider or flatter minima ( Mirzadeh et al., 2020 ) . This can be explained because with a wider minimum, larger changes to the parameters are needed to ‘get out of the minimum’. This insight has motivated several continual learning works to modify optimization routines to encourage finding such wider minima ( Deng et al., 2021 ; Yang et al., 2023 ; Tran Tung et al., 2023 ) .

A popular way to control how a given loss function is optimized is by using adaptive learning rates. For example, one strategy might be to reduce the learning rate for either parameters or units that are estimated to be important for past tasks ( Ahn et al., 2019 ; Jung et al., 2020 ; Özgün et al., 2020 ; Paik et al., 2020 ; Laborieux et al., 2021 ; Soures et al., 2021 ; Malviya et al., 2022 ) . This approach is related to parameter regularization, but it is different because the use of adaptive learning rates does not change the loss function, while parameter regularization does. Similar to parameter regularization, the use of adaptive learning rates can be related to the neuroscience concept of metaplasticity ( Abraham, 2008 ) . A different metaplasticity-inspired way to control the optimization trajectory is through probabilistic parameter updates ( Zohora et al., 2020 ; Schug et al., 2021 ) rather than adjusting the learning rate.