Continual Learning and Catastrophic Forgetting — part 1/5

Cleaned-text code points [0, 15739). Bibliography omitted here; source.md includes it.

# Continual Learning and Catastrophic Forgetting

###### Abstract

This book chapter delves into the dynamics of continual learning, which is the process of incrementally learning from a non-stationary stream of data. Although continual learning is a natural skill for the human brain, it is very challenging for artificial neural networks. An important reason is that, when learning something new, these networks tend to quickly and drastically forget what they had learned before, a phenomenon known as catastrophic forgetting. Especially in the last decade, continual learning has become an extensively studied topic in deep learning. This book chapter reviews the insights that this field has generated.

## Key Points

• Incrementally learning from a non-stationary stream of data, referred to as continual learning, is a key aspect of intelligence.

• Artificial neural networks tend to rapidly and drastically forget previously learned information when learning something new, a phenomenon referred to as catastrophic forgetting.

• Catastrophic forgetting is an important reason why continual learning is so challenging for deep neural networks, but solving the continual learning problem requires more than preventing catastrophic forgetting.

• Two distinctions often made in the deep learning literature on continual learning are between task-based and task-free continual learning, and between task-, domain- and class-incremental learning. These two distinctions are orthogonal to each other, and can be captured in a single framework.

• Deep learning methods for continual learning are evaluated using metrics covering performance, diagnostic analysis and resource efficiency.

• Six main computational approaches for continual learning with deep neural networks are (i) replay, (ii) parameter regularization, (iii) functional regularization, (iv) optimization-based approaches, (v) context-dependent processing and (vi) template-based classification.

• Establishing further connections between deep learning and cognitive science in the context of continual learning could benefit both fields.

## 1 Introduction

Continual learning is a key aspect of intelligence. The ability to accumulate knowledge by incrementally learning from one’s experiences is an important skill for any agent, natural or artificial, that operates in a non-stationary world. Humans are excellent continual learners. The human brain can incrementally learn new skills without compromising those that were learned before, and it is able to integrate and contrast new information with previously acquired knowledge ( Flesch et al., 2018 ; Kudithipudi et al., 2022 ) . Intriguingly, artificial deep neural networks, although rivaling human intelligence in other ways, almost completely lack this ability to learn continually. Most strikingly, when these networks are trained on something new, they tend to “catastrophically” forget what they had learned before ( McCloskey and Cohen, 1989 ; Ratcliff, 1990 ) .

The inability of deep neural networks to continually learn has important practical implications. Due to its powerful representation learning capabilities, deep learning has become a major driving force behind many recent advances in artificial intelligence. However, to achieve their strong performance, deep neural networks must be trained for extended periods of time on large amounts of data (e.g., Radford et al., 2021 ). This resource-intensive training makes the lack of continual learning abilities of these networks particularly costly. For example, if after an expensive training process has finished, relevant new data become available, rapidly updating the network by training only on the new data does not work. To avoid adapting too strongly to the new data, the network must be trained on both old and new data together. However, even such continued joint training often does not yield satisfactory results. Instead, practitioners in industry tend to periodically retrain the entire network from scratch on all data, despite the large computational costs ( Huyen, 2022 ) . Therefore, developing successful continual learning methods for deep learning could result in significant efficiency gains and a substantial reduction in the required resources. Another important potential application for continual learning is correcting errors or biases. After being trained, deep neural networks are often found to make mistakes or have certain biases (e.g., a subset of demographics is underrepresented in the training set), but updating a network to correct for these is difficult ( Mitchell et al., 2022 ) . Another practical use case of continual learning is in edge applications ( Deng et al., 2020 ) . These applications require the ability to learn in real-time on-device to reduce their reliance on pre-deployed solutions. For these and other reasons (e.g., see Verwimp et al., 2023 ), continual learning has become an intensively studied topic in deep learning, and is seen as one of the main open challenges in the field ( Hadsell et al., 2020 ; Bubeck et al., 2023 ) .

In addition to the practical arguments above, another motivation for studying continual learning in deep neural networks is to gain insight into the computational principles that might underlie the cognitive processes supporting continual learning in the brain. Artificial deep neural networks are a popular class of computational models that can account for many aspects of information processing in the brain (e.g., van Gerven and Bohte, 2017 ; Perconti and Plebe, 2020 ; Doerig et al., 2023 ). Yet, in terms of continual learning, this class of models has clear insufficiencies. If we can understand the reasons for this failure and how to fix it, this might give clues as to which computational processes underlie the cognitive skill of continual learning. At the same time, but in the reverse direction, the brain’s exceptional ability to continually learn can serve as a source of inspiration for the development of novel continual learning algorithms for deep learning.

The structure of this book chapter is as follows. Section 2 introduces the problem of continual learning, illustrates why it is so challenging for deep neural networks, and discusses different variants of the continual learning problem. Section 3 covers various computational strategies that have been proposed to improve the continual learning capabilities of neural networks. Section 4 compares how continual learning is studied in cognitive science versus in deep learning, and section 5 concludes.

## 2 The Continual Learning Problem

Continual learning is the skill of incrementally learning from a non-stationary stream of data. In this definition, the term “non-stationary” indicates that the distribution of the data from which is learned changes over time. The term “incrementally” signals that new learning should not overwrite what was learned before, but that knowledge should be accumulated. In this section, we discuss why continual learning is so challenging for artificial deep neural networks (subsections 2.1 and 2.2 ), we review different types of continual learning (subsections 2.3 and 2.4 ), and we briefly discuss how continual learning is evaluated (subsection 2.5 ).

### 2.1 Catastrophic Forgetting

Central to continual learning is the concept of catastrophic forgetting, also referred to as catastrophic interference. Catastrophic forgetting is the phenomenon that artificial neural networks tend to rapidly and drastically forget previously learned information when learning new information (see Fig. 1 ).

When a neural network is sequentially trained on multiple tasks, forgetting of earlier tasks can be expected because the network parameters are adjusted to optimize the loss on the new task, which likely pushes these parameters away from their optimum value that was found for the earlier tasks. It was appreciated early on that such forgetting would occur when incrementally training a neural network on multiple tasks, but initially it was speculated that this forgetting might be relatively mild: Hinton et al. (1986) hypothesized that the many small parameter updates that work together to optimize the new task, might mostly cancel each other out in terms of their effect on previous tasks. This however turned out not to be the case. McCloskey and Cohen (1989) and Ratcliff (1990) were the first to demonstrate that the sequential training of simple neural networks on disjoint sets of data results in drastic forgetting, even with only small amounts of training on the new data distribution. McCloskey and Cohen also noted that this forgetting is substantially worse than that observed in humans, which prompted them to describe it as “catastrophic”.

Inspired by the observations from McCloskey and Cohen and Ratcliff , in the 1990s several research groups began exploring the problem of catastrophic forgetting in early connectionist models. The proposed solutions incorporated concepts such as reducing network overlap (e.g., sparse or orthogonal representations) and the rehearsal of prior data (see Robins, 1995 and French, 1999 for reviews). Following these works, the early concepts of continual learning began to emerge ( Thrun and Mitchell, 1995 ) . More recently, Srivastava et al. (2013) and Goodfellow et al. (2013) demonstrated that catastrophic forgetting remains a problem in modern deep learning architectures, which had just begun gaining popularity. Since then there has been a rapid rise in the number of deep learning studies addressing the problem of continual learning.

### 2.2 Other Features Important for Continual Learning

While catastrophic forgetting is the root cause for why continual learning is so challenging for neural networks, it is not the case that preventing catastrophic forgetting is enough to solve the continual learning problem. One reason for this is that approaches that lower forgetting often introduce or exacerbate other issues. For example, reduced forgetting often comes at the cost of an impaired ability to learn new information, a trade-off known as the stability-plasticity dilemma ( Grossberg, 1982 ) . Another example is that catastrophic forgetting could be avoided by always training on all data seen so far, but this approach presents significant challenges in terms of memory and computation usage. Among others, Díaz-Rodríguez et al. (2018) , Mundt et al. (2022) , Prabhu et al. (2023) and Verwimp et al. (2023) have stressed that continual learning should focus on more than just preventing forgetting.

In a recent perspective article, Kudithipudi et al. (2022) argued that, in addition to avoiding catastrophic forgetting, successful continual learning methods should exhibit the following features for full-scale operation:

1. Adaptation - Continual learning models must be able to quickly adapt to new situations or surroundings, without requiring extensive offline (re)training. Such rapid adaptation is essential when models are deployed in the real world, where conditions may vary considerably and change quickly. It was recently shown that incremental training of deep neural networks on a sequence of tasks can lead to a substantial loss of plasticity ( Dohare et al., 2023 ) , highlighting that rapid adaptation is still an open problem in continual learning.

2. Exploiting task similarity - When the different tasks (or contexts) that must be learned are related, it should be possible to exploit their similarity to achieve ‘positive transfer’ between tasks. Positive transfer means that due to learning one task, the network also becomes better at another task – either directly in terms of performance improvement, or indirectly by making (re)learning of that task easier. For example, once a human has learned to play a first musical instrument (e.g., the piano), it is typically easier for them to master a second one (e.g., the violin). In continual learning, there are two types of transfer: forward transfer , whereby learning a new task facilitates future tasks; and backward transfer , whereby learning a new task benefits previously learned tasks. Especially positive backward transfer has proven to be a challenging feature to achieve with artificial deep neural networks ( Lopez-Paz and Ranzato, 2017 ) . A promising route to enabling improved knowledge transfer between tasks is learning compositional representations ( Mendez and Eaton, 2023 ) .

3. Task agnostic - In many real world problem settings, continual learning models cannot rely on an oracle to tell them for each encountered example to which task (or domain / context) it belongs. It is often a desirable property for continual learning models to be task agnostic . The term task agnostic can refer to multiple different things: not knowing task identity during testing, not knowing task identity during training, not being informed about task switches, or even that there is no discrete set of underlying tasks at all. Various forms of being task agnostic are covered in subsections 2.3 and 2.4 .

4. Noise tolerance - Deep learning models are usually trained on datasets that are curated, cleaned, and annotated to optimize training (e.g., ImageNet; Deng et al., 2009 ). Training instead on non-curated datasets can lead to substantially lower performance and generalization capabilities ( Tian et al., 2021 ) . For practical applications of continual learning, it is important that models are able to deal with data in the form they arrive – raw, noisy and uncleaned; as well as with situations in which the noise level or distribution changes over time, for example due to variability in the environment or in the sensors.

5. Resource efficiency and sustainability - When deep neural networks are expected to continue learning for extended periods of time, it is critical that they do so in a sustainable and resource efficient manner. Existing continual learning methods often violate this desideratum ( Vogelstein et al., 2020 ; Prabhu et al., 2023 ) . For example, storing all data points that are encountered (e.g., in a memory buffer) can be problematic in terms of required storage space, while constantly retraining on all previous tasks can quickly become infeasible in terms of computational costs. Verwimp et al. (2023) argued that he goal of continual learning can be interpreted as finding the best approach in terms of some trade-off between performance and resource efficiency (e.g., memory and compute), whereby the relative importances in the trade-off differ depending on the problem.

### 2.3 Task-based versus Task-free Continual Learning

A common assumption in continual learning research is that there is a discrete set of tasks that are presented to the network one after the other, often with marked boundaries between tasks. This task-based continual learning setting is illustrated in Fig. 2 a. A popular way to set up a task set for continual learning is to take an existing dataset (e.g., MNIST, LeCun et al., 1998 ; CIFAR-100, Krizhevsky, 2009 ; MiniImageNet, Vinyals et al., 2016 ) and split it into tasks based on the class labels (e.g., Split MNIST; see Fig. 2 ). Another common way to create multiple tasks from a single dataset is to use task-specific transformations, for example rotations, permutations or the addition of noise. An alternative approach is to interweave multiple datasets (e.g., one task contains the ten MNIST-digits, another task contains the ten digits from the SVHN dataset; Netzer et al., 2011 ). Task-based continual learning is an important and frequently used setting in the literature because it provides a convenient way to study various aspects of continual learning in a controlled and isolated manner.