Continual Learning and Catastrophic Forgetting — part 2/5
Cleaned-text code points [15739, 31241). Bibliography omitted here; source.md includes it.
However, not all aspects of the rich pattern of non-stationarity in the real world are well captured by the task-based continual learning setting. For example, continual learning methods developed in the task-based setting can be reliant on the presence of hard boundaries between tasks (to perform certain consolidation operations, such as updating the memory buffer or replacing a stored copy of the model) ( Aljundi et al., 2019b ; Zeno et al., 2019 ) , and for many methods it is unclear how they could benefit if a previously seen task is encountered again ( Stojanov et al., 2019 ; Hemati et al., 2023 ) . This has motivated the emergence of task-free continual learning (Fig. 2 b), which allows for gradual transitions between tasks and repetition of tasks. With task-free continual learning, there is usually still an underlying set of discrete tasks (e.g., Lee et al., 2020 ; Jin et al., 2021 ; Shanahan et al., 2021 ), but the transitions between those tasks are continuous, in the sense that the probabilities of observing each task gradually change over time. A description of how the probabilities of observing each task change over time has been referred to as a schedule ( Shanahan et al., 2021 ; Wang et al., 2022a ) .
Following van de Ven et al. (2022) , the above descriptions of task-based and task-free continual learning can be formalized by letting 𝒯 \mathcal{T} be the set of underlying tasks and defining the data stream as a sequence of experiences: { e 1 , e 2 , … , e N } \{e_{1},e_{2},...,e_{N}\} . These experiences e n e_{n} are the ‘incremental steps’ of a continual learning problem, in the sense that they are presented one after the other and the network has free access to the data of the current experience but not to the data from other experiences (except possibly to data stored in a memory buffer). In the standard task-based continual learning setting, each experience contains all training data for its corresponding task: e t = 𝒟 t train e_{t}=\mathcal{D}^{\text{train}}_{t} for all t ∈ 𝒯 t\in\mathcal{T} . In the task-free continual learning setting, each data point within every experience can be sampled from any combination of underlying tasks: e n [ i ] ∼ ∑ t ∈ 𝒯 π t n , i 𝒟 t e_{n}[i]\sim\sum_{t\in\mathcal{T}}\pi^{n,i}_{t}\mathcal{D}_{t} (1) whereby e n [ i ] e_{n}[i] is data point i i of experience n n and π t n , i \pi^{n,i}_{t} is the probability that this data point is sampled from 𝒟 t \mathcal{D}_{t} , the data distribution of task t t . In this notation, the schedule is given by { π t n , i } n ∈ ℕ ≤ N , i ∈ ℕ ≤ I n , t ∈ 𝒯 \{\pi^{n,i}_{t}\}_{n\in\mathbb{N}_{\leq N},i\in\mathbb{N}_{\leq I_{n}},t\in\mathcal{T}} , where I n I_{n} denotes the number of data points in experience n n . In this framework for continual learning, the task set 𝒯 \mathcal{T} describes what part of the data change over time (i.e., the non-stationary aspect of the data) and the schedule π \pi describes how it changes over time (i.e., the temporal correlation structure). Finally, we note that this framework can be generalized to allow for a continuous task set (this requires changing the summation in equation 1 to an integral), but as far as we are aware this option has not yet been systematically explored in the literature.
In addition to ‘task-based’ and ‘task-free’, other labels that have been used to describe variants of continual learning are ‘streaming’ and ‘online’. Although the precise definitions of these terms vary, streaming continual learning generally means that only a single training example is presented to the network at a time (i.e., in the above framework, each experience e n e_{n} contains only one data point) ( Hayes et al., 2020 ; Banerjee et al., 2021 ) , and online continual learning typically refers to that the network encounters each training sample only once, also referred to as the ‘single-pass-through-data’ setting ( Aljundi et al., 2019a ; Chen et al., 2020 ) .
### 2.4 Three Continual Learning Scenarios
Another important way in which continual learning problems can differ from each other is described by van de Ven and Tolias (2018) . They distinguished three scenarios that have been widely used in the literature: task-incremental learning (sometimes abbreviated as Task-IL or TIL), domain-incremental learning (Domain-IL or DIL) and class-incremental learning (Class-IL or CIL). Formally, these three scenarios can be distinguished based on whether, at test time, task identity is provided and, if not, whether task identity must be inferred. This means that, in theory, any sequence of tasks could be performed according to all three scenarios. Fig. 3 illustrates what it means for the Split MNIST toy problem to be performed according to each scenario. These three scenarios were initially described for task sequences with clear boundaries (i.e., task-based continual learning), but it has since been pointed out that they also generalize to task-free continual learning ( van de Ven et al., 2022 ) . The key to generalizing these three scenarios is defining them based on how the non-stationary aspect of the data (i.e., the aspect of the data that changes over time, or the ‘task set’) relates to the function or mapping that must be learned by the network.
Informally, with task-incremental learning a network must incrementally learn a set of distinct tasks. The use of ‘distinct tasks’ here indicates that the network is always aware of which task it is presented with. Task identity might, for example, be known because it is explicitly provided, because it is clear from context which task must be performed, or because the inputs from different tasks are easily distinguishable from each other. Thanks to the availability of task identity information, with task-incremental learning it is possible to use networks with task-specific components (e.g., a separate output layer per task), or even to have a completely separate network per task – in which case there is no forgetting at all. Thus, simply preventing catastrophic forgetting is not difficult in this scenario. Instead, the challenge with task-incremental learning is to do better (in terms of a trade-off between performance and resource efficiency) than the naive solution in which there is a separate network per task. To realize this, it is necessary to achieve a positive transfer between tasks by sharing learned representations across tasks. An illustrative example of task-incremental learning is learning to play different musical instruments (e.g., first the piano, then the violin), since it should typically be clear which instrument must be played.
Domain-incremental learning can be described as the structure of the problem to be learned being always the same, but the context or input-distribution changes (e.g., there are domain shifts). In contrast to task-incremental learning, where the network always knows from which task an example is, with domain-incremental learning it is not necessarily clear to which domain a presented example belongs. As a result, in this scenario it is not possible to use networks with ‘domain’-specific components, unless the network also infers to which domain an example belongs (e.g., as done by Heald et al., 2021 ; Verma et al., 2021 ). However, inferring domain identity can be challenging and is not necessarily the most effective way to solve a domain-incremental learning problem. Illustrative examples of domain-incremental learning are learning to drive in various weather conditions or learning to classify objects under different lighting conditions (e.g., indoors versus outdoors).
Finally, class-incremental learning is the problem of incrementally learning to distinguish between an increasing number of objects or classes. In the literature, this scenario is often implemented in a ‘task-based manner’, meaning that there is a sequence of classification-based ‘tasks’, with each task containing a distinct set of classes, and the goal is to learn to discriminate between the classes of all tasks. Let us illustrate this with an example. Imagine that a network is first presented with a task consisting of pianos and guitars, and later with one containing saxophones and violins. In the class-incremental learning scenario, after seeing both tasks, the network is expected to be able to discriminate between all four musical instruments (i.e., it should have learned a four-way classifier). On the other hand, if this same task sequence were performed according to the task-incremental learning scenario, the network would only be expected to be able to distinguish between instruments in the same task, but not between those from different tasks. In this task-based setting, class-incremental learning can be decomposed into solving each individual task (or prediction within the task) and inference of task identity (or prediction across tasks) ( Soutif-Cormerais et al., 2021 ; Guo et al., 2023 ) . Especially inferring task identity, which has links to out-of-distribution detection ( Henning et al., 2021 ; Kim et al., 2022 ) , is often found to be particularly difficult. Beyond the task-based setting, this can be generalized by saying that a challenging aspect of class-incremental learning is learning to discriminate between classes that are not observed together.
#### Clarifying note.
It is useful to point out that the distinction between task-, domain- and class-incremental learning is orthogonal to the distinction between task-based and task-free continual learning, in the sense that each of the three scenarios can occur in both a task-based and a task-free continual learning setting. In the literature, there is sometimes confusion about the difference between ‘task-based continual learning’ and ‘task-incremental learning’. We therefore clarify that task-based continual learning refers to that training data change over time in a task-by-task manner (and with clear boundaries between tasks), while task-incremental learning signals that task identity is known to the network at test time.
### 2.5 Evaluation
With the rapid increase in interest in continual learning, numerous approaches have been proposed for evaluating and comparing different continual learning methods (e.g., Lopez-Paz and Ranzato, 2017 ; Díaz-Rodríguez et al., 2018 ; New et al., 2022 ; De Lange et al., 2023 ; Kudithipudi et al., 2023 ). Typically, metrics for continual learning cover one of three areas: i) performance, ii) diagnostic analysis, and iii) resource efficiency.
With regards to evaluating performance in a continual learning setting, two important questions are: (a) how to evaluate it, and (b) when to evaluate it. Regarding the first question, continual learning performance is typically evaluated using the average of a certain performance metric (e.g., test accuracy in case of classification) over a family of tasks. Usually this family of tasks consists of all tasks trained on so far, or sometimes all tasks that will be trained on, and o the performance on all evaluated tasks is weighted equally. Regarding the question when to evaluate performance, one approach is to only do so at the end of training on all tasks. Another common approach is to evaluate performance periodically throughout training by interleaving training and evaluation blocks. Performance can for example be evaluated after finishing training on each task, or after a fixed number of training steps.
A disadvantage of comparing continual learning methods only based on their average performance is that this does not provide much insight into how each method addresses continual learning. For example, a plastic model that achieves 0% accuracy on the first task and 100% accuracy on the second task, has the same average performance as a completely rigid model with 100% accuracy on the first task and 0% accuracy on the second. To provide greater insight into the dynamics of continual learning, there are several diagnostic metrics. One such diagnostic metric is learning accuracy. This measures well a model can learn from the current task, which, when compared to a baseline model, can provide a measure of plasticity. Another popular diagnostic metric is backward transfer, which measures how the performance on previous tasks changes when training on new ones. This metric provides insight into the degree of catastrophic forgetting that occurs and the stability of the model. Another diagnostic metric is forward transfer, which quantifies how much training on previous tasks improves the performance of a model on, or its ability to learn, future tasks.
A third set of metrics evaluates the resource efficiency of continual learning methods. One way to quantify resource efficiency is through the computational overhead and energy consumption of a method. The number of operations and the complexity are important factors when deploying continual learning methods to real-world problems, especially when targeting edge applications. It is important to note that although computational complexity is often only computed for the training phase, a portion of continual learning methods introduce additional complexity during inference, which can be measured as well and is not always identical to the training costs. Another key aspect of resource efficiency is sustainability, which refers to how quickly a model grows in terms of parameters, or in terms of memory for storing information about prior experiences (e.g., data samples, model copies).
## 3 Continual Learning Approaches
In this section we focus on approaches that have been proposed to address the continual learning problem. We do not provide an extensive review of individual continual learning methods (for this, we refer to Belouadah et al., 2021 ; De Lange et al., 2022 ; Masana et al., 2023 ; Wang et al., 2023b ), but rather we discuss the main computational strategies that underlie these methods. Individual continual learning methods often combine multiple of these strategies.
### 3.1 Replay
Perhaps the most widely used approach for continual learning is replay. The idea behind replay, which is also referred to as rehearsal, is to approximate interleaved learning by complementing the training data of the current task or experience with data that are representative of previous ones (Fig. 4 a). Replay has close links to neuroscience. In the brain, the re-occurence of neuronal activity patterns that represent previous experiences is believed to be important for the stabilization and consolidation of new memories ( Wilson and McNaughton, 1994 ; Rasch and Born, 2007 ) .
In the deep learning literature, a common way to add replay to a neural network is to store previously seen data in a memory buffer and revisit them later on. Such ‘experience replay’ can substantially speed up training in reinforcement learning ( Lin, 1992 ; Mnih et al., 2015 ) , and it has proven to be an effective way to reduce catastrophic forgetting in continual learning ( Robins, 1995 ; Rolnick et al., 2019 ; Chaudhry et al., 2019b ; Buzzega et al., 2020 ) . A common assumption in benchmarks for continual learning is that only a limited amount of data can be stored (but see Prabhu et al., 2023 ; Verwimp et al., 2023 for recent perspectives on this assumption), and a research question that has received a lot of attention is how to best pick the samples to store in the memory buffer ( Rebuffi et al., 2017 ; Chaudhry et al., 2019b ; Aljundi et al., 2019c ; Mundt et al., 2023 ) .