Continual Learning and Catastrophic Forgetting — part 4/5
Cleaned-text code points [46539, 61992). Bibliography omitted here; source.md includes it.
Another optimization-based tool that has been explored in continual learning is gradient projection. With gradient projection, rather than basing parameter updates on the original gradient g = ∇ 𝜽 ℓ ( 𝜽 ) g=\nabla_{\boldsymbol{\theta}}\ell\left(\boldsymbol{\theta}\right) , they are based on a projected version g ¯ \bar{g} of that gradient. A first popular approach is ‘orthogonal gradient projection’ ( Zeng et al., 2019 ; Farajtabar et al., 2020 ; Saha et al., 2021 ) . To restrict parameter updates to directions that do not interfere with the performance on old tasks, this approach projects gradients to the subspace orthogonal to the gradients of old tasks. A less restrictive version of this approach instead projects gradients using the inverse of the Fisher Information as projector matrix ( Kao et al., 2021 ) . Another gradient projection-based approach is ‘gradient episodic memory’ ( Lopez-Paz and Ranzato, 2017 ; Chaudhry et al., 2019a ) . The projection mechanism of this approach is derived from a constrained optimization problem which aims to optimize the loss on a new task without increasing the loss on old tasks.
Another method that falls into the category of optimization-based techniques is based on the biological process of synaptic consolidation, which consolidates information within an individual synapse over multiple timescales ( Sossin, 2008 ; Morris, 2003 ) . Such synaptic consolidation can be modelled with a complex synaptic model consisting of a rapidly adapting weight 𝜽 f \boldsymbol{\theta}_{f} and a slowly evolving weight 𝜽 s \boldsymbol{\theta}_{s} . The rapidly adapting weight, 𝜽 f \boldsymbol{\theta}_{f} , is driven by a combination of the loss function and a regularization-like function that attempts to prevent divergence from 𝜽 s \boldsymbol{\theta}_{s} . The main distinction between parameter regularization and this approach lies in the dynamic nature of 𝜽 s \boldsymbol{\theta}_{s} , which slowly evolves to consolidate the information learned by 𝜽 f \boldsymbol{\theta}_{f} . Several studies have used this approach to reduce catastrophic forgetting in continual learning ( Zenke et al., 2015 ; Leimer et al., 2019 ; Soures et al., 2021 ) .
### 3.5 Context-dependent Processing
Another popular approach for continual learning is context-dependent processing. The idea behind this approach is to use certain parts of the network only for specific tasks or contexts, in order to reduce the interference that can occur between them (Fig. 4 e). It is worth noting that when taken to the extreme, this approach corresponds to having a completely separate network per task or context. In this case, there would be no interference or forgetting at all, but there would also no longer be any possibility of positive transfer between tasks or contexts. It could therefore be argued that continual learning methods should aim to only segregate information that is unrelated to each other (as there is likely no positive transfer to be gained between them anyway), while storing related information in the same part of the network.
A widespread example of context-dependent processing in continual learning is the use of a separate linear output layer for each task to be learned. The use of such a ‘multi-headed output layer’ has become the default setup for task-incremental learning experiments. But there are other ways in which context-dependent processing is used as well.
One other way to induce context-dependent processing in a neural network is by gating either its units or its parameters in a different way for each task that must be learned. It is possible to specify such task-specific gates a priori and randomly ( Masse et al., 2018 ) , but it is also possible to learn them, for example using gradient descent ( Serra et al., 2018 ) , Hebbian plasticity ( Flesch et al., 2023 ) or evolutionary algorithms ( Ellefsen et al., 2015 ; Fernando et al., 2017 ) . The use of task- or context-specific gates has been linked to the concept of neuromodulation in the brain, which refers to chemical signals that locally modify the way inputs are processed ( Marder and Thirumalai, 2002 ) .
Another way to realize context-dependent processing is by periodically adding new components, and thus dynamically expanding the network ( Zhou et al., 2012 ; Terekhov et al., 2015 ; Draelos et al., 2017 ) . Expanding the network can be based on when extra capacity is needed ( Hung et al., 2019 ; Mitchell et al., 2023 ) , but in many cases new components are simply added whenever a new task must be learned ( Rusu et al., 2016 ; Yoon et al., 2018 ) . Often, this approach is combined with freezing parts of the network after a task has been learned. To control the growth of the network, many model expansion methods make use of sparse training (e.g., prompt learning or the use of adaptors; Wang et al., 2022b ; Gao et al., 2023 ) or pruning techniques ( Golkar et al., 2019 ; Pandit and Kudithipudi, 2020 ; Yan et al., 2021 ) . The approach of dynamically expanding the network can be linked to the process of neurogenesis, which refers to the addition of neurons to certain brain regions throughout life ( Aimone et al., 2014 ) . It has indeed been hypothesized that neurogenesis plays a role in the brain’s ability to mitigate catastrophic forgetting ( Wiskott et al., 2006 ) .
An important assumption that often underlies the use of context-dependent processing in continual learning is that the context (or task) is always clear to the network. As a result of this, when used by itself, this approach is limited to task-incremental learning type of problems. To be applicable to domain- or class-incremental learning problems, context-dependent processing must be combined with an algorithm for context identification (e.g., as done by Aljundi et al., 2017 ; von Oswald et al., 2020 ; Wortsman et al., 2020 ; Henning et al., 2021 ; Verma et al., 2021 ; Heald et al., 2021 ). In this regard it is important to realize that context identification itself is a class-incremental learning problem as well, as it consists of distinguishing categories (in this case ‘contexts’) that are not observed together. Inferring context information has indeed proven to be a difficult problem, and presents a promising avenue for future research.
### 3.6 Template-based Classification
An approach to class-incremental learning that is often used in continual learning is template-based classification. With this approach, a ‘class template’ is learned for every class, and classification is performed based on which class template is closest or most suitable for the sample to be classified (Fig. 4 f). In this description, a class template can be thought of as a representation or a model of that particular class. In the context of class-incremental learning, an important advantage of template-based classification is that it avoids the need to make comparisons between classes during training . Standard softmax-based classifiers have to learn decision boundaries between all classes during their training, but this is challenging with class-incremental learning because not all classes are observed together. Template-based classifiers instead only have to learn a template per class during their training, and the comparison between classes is deferred to test time. Importantly, while the original problem is a class-incremental learning problem, learning these class templates is a task-incremental learning problem, whereby each ‘task’ is to learn a template for a specific class. This means that with this approach it is possible to use ‘template-specific components’, or other context-dependent processing approaches.
A popular way to implement template-based classification is to use ‘prototypes’ as class templates. The use of such prototypes has roots in cognitive science, where it is an influential model for how humans make categorization decisions ( Nosofsky, 1986 ) . In deep learning, a prototype is often taken to be the mean vector of examples from a class in an embedding space defined by a neural network ( Snell et al., 2017 ; Yang et al., 2018 ) . Classification is then done by putting samples through the embedding network and assigning them to the class of the prototype they are closest to. When a suitable pretrained embedding network is available and it is kept frozen throughout training, storing raw data is not needed and prototype-based classification only requires storing a single prototype per class (e.g., Hayes and Kanan, 2020 ). If the embedding network requires updates, for example to better distinguish new classes, one option is to store a few well-chosen examples per class to update the prototypes following changes to the embedding network ( Rebuffi et al., 2017 ; De Lange and Tuytelaars, 2021 ) . Alternatively, prototype drift can be addressed without storing past examples by estimating and correcting drift based on the observed drift for data from the current task ( Yu et al., 2020 ; Wei et al., 2021 ) .
Another example of template-based classification that has shown promise for class-incremental learning is generative classification ( van de Ven et al., 2021 ; Banayeeanzade et al., 2021 ) . With generative classification, the class templates that are learned are generative models, and their suitability for a sample to be classified is computed as the sample’s likelihood under each of the generative models. An advantage of generative classification is that it does not require storing samples, but a drawback is that learning decent generative models can be challenging, especially in the case of limited training data or complex distributions. Moreover, inference (i.e., making classification decisions) can be computationally costly as it requires computing likelihoods under the generative model of each class. Several more lightweight and efficient alternatives for generative classification have been proposed. One option is to train an energy-based model and compute energy values per class or context instead of likelihoods ( Li et al., 2022 ; Joseph et al., 2022 ) . Another option is to train class-specific models to replicate the outputs of a frozen random network and perform classification based on each model’s prediction error ( Zając et al., 2024 ) .
## 4 Continual Learning in Deep Learning versus in Cognitive Science
So far, this book chapter has mostly reviewed the deep learning literature on continual learning. In this section, we briefly ask how continual learning is studied in the cognitive science literature. When it comes to continual learning, the goals of deep learning and cognitive science are different, but clearly related. Deep learning aims to engineer artificial neural networks so that they can learn continually, while cognitive science is concerned with understanding how this already present skill is implemented by the brain.
One central aspect of continual learning that is widely studied in cognitive science is forgetting. Forgetting refers to the loss or decay of previously acquired information and can result from various factors in biological organisms ( Hardt et al., 2013 ; Davis and Zhong, 2017 ) . There exist passive mechanisms, such as decay over time through natural aging and transient forgetting in which forgetting is often reversible, and there are active mechanisms of forgetting, such as intentional forgetting, retrieval-induced forgetting and interference-based forgetting. An example of interference-based forgetting is retroactive interference, where acquisition of new memories during consolidation leads to forgetting of old ones ( Wixted, 2004 ; Alves and Bueno, 2017 ) , which is probably most akin to the forgetting that happens when deep neural networks are continually trained. Studies into how biological systems mitigate this retroactive interference have already inspired many approaches for continual learning with deep neural networks (e.g., Kaplanis et al., 2018 ; Masse et al., 2018 ; Tadros et al., 2022 ; Kudithipudi et al., 2022 ; Arani et al., 2022 ; Wang et al., 2023a ; Jeeveswaran et al., 2023 ), and it seems likely that deep learning can still gain further benefits from the rich characterization and understanding of forgetting that has been accrued in the cognitive science literature. For example, something not often acknowledged in deep learning is that with bounded resources, forgetting could be beneficial or even necessary. Indeed, some deep learning applications might benefit from controlled or intentional forgetting, similar to how forgetting is thought to be important for the brain’s ability to continue learning new information (e.g., Nørby, 2015 ; Richards and Frankland, 2017 ; Bjork and Bjork, 2019 ).
While forgetting is well studied in cognitive science, other aspects of continual learning (cf. subsection 2.2 ) appear to receive less attention; or at least the way in which these other aspects are studied in deep learning is different and mostly unconnected from the way they are studied in cognitive science. However, there are some efforts to bridge this gap between both fields. In particular, Flesch et al. (2018) designed several experiments to compare the continual learning performance of humans and artificial neural networks. They demonstrate that, unlike deep learning models that suffer from blocked training, humans actually perform better with blocked training (and worse with interleaved training). In another work, Flesch et al. (2023) show how, through the incorporation of different mechanisms, the learning dynamics of artificial neural networks can be made to more closely match those observed in humans. Establishing more such connections between cognitive science and deep learning in the context of continual learning, especially if they go beyond just forgetting, might substantially benefit both fields.
## 5 Conclusion
In this book chapter we have reviewed the challenges of continual learning with deep neural networks, which is a topic of great interest in the artificial intelligence field. The current inept ability of deep learning models to continually learn from a stream of incoming data is a major obstacle to the development of truly intelligent artificial agents that can accumulate knowledge by incrementally learning from their experiences.
Catastrophic forgetting is the most well-known issue in continual learning. However, in addition to overcoming catastrophic forgetting, a complete solution to continual learning requires the development of models capable of quickly adapting to new situations, exploiting similarities between tasks, operating in a task agnostic manner, being tolerant to noise, and using resources in an efficient and sustainable manner. Continual learning is thus not a unitary problem, and this is further illustrated by the observation that the main challenges of continual learning can differ substantially depending on how exactly the problem is set up. To fully capture the gradations and complexities of continual learning, a variety of settings and benchmarks needs to be considered. Among these are task-, domain- and class-incremental learning, as well as task-based and task-free continual learning. Similarly, comprehensively evaluating continual learning approaches cannot be done using a single metric; in this book chapter we have discussed a variety of continual learning metrics covering performance, diagnostics and resource efficiency.