Measuring Catastrophic Forgetting in Neural Networks — part 1/4

Cleaned-text code points [0, 15928). Bibliography omitted here; source.md includes it.

# Measuring Catastrophic Forgetting in Neural Networks

###### Abstract

Deep neural networks are used in many state-of-the-art systems for machine perception. Once a network is trained to do a specific task, e.g., bird classification, it cannot easily be trained to do new tasks, e.g., incrementally learning to recognize additional bird species or learning an entirely different task such as flower recognition. When new tasks are added, typical deep neural networks are prone to catastrophically forgetting previous tasks. Networks that are capable of assimilating new information incrementally, much like how humans form new memories over time, will be more efficient than re-training the model from scratch each time a new task needs to be learned. There have been multiple attempts to develop schemes that mitigate catastrophic forgetting, but these methods have not been directly compared, the tests used to evaluate them vary considerably, and these methods have only been evaluated on small-scale problems (e.g., MNIST). In this paper, we introduce new metrics and benchmarks for directly comparing five different mechanisms designed to mitigate catastrophic forgetting in neural networks: regularization, ensembling, rehearsal, dual-memory, and sparse-coding. Our experiments on real-world images and sounds show that the mechanism(s) that are critical for optimal performance vary based on the incremental training paradigm and type of data being used, but they all demonstrate that the catastrophic forgetting problem has yet to be solved.

## Introduction

While the basic architecture and training algorithms behind deep neural networks (DNNs) are over 30 years old, interest in them has never been greater in both industry and the artificial intelligence research community. Owing to far larger datasets, increases in computational power, and innovations in activation functions, DNNs have achieved near-human or super-human abilities on a number of problems, including image classification [ 2016 ] , speech-to-text [ 2015 ] , and face identification [ 2015 ] . These algorithms power most of the recent advances in semantic segmentation [ 2015 ] , visual question answering [ 2017 ] , and reinforcement learning [ 2013 ] . While these systems have become more capable, the standard multi-layer perceptron (MLP) architecture and typical training algorithms cannot handle incrementally learning new tasks or categories without catastrophically forgetting previously learned training data. Fixing this problem is critical to making agents that incrementally improve after deployment. For non-embedded or personalized systems, catastrophic forgetting is often overcome simply by storing new training examples and then re-training either the entire network from scratch or possibly only the last few layers. In both cases, retraining uses both the previously learned examples and the new examples, randomly shuffling them so that they are independent and identically distributed (iid). Retraining can be slow, especially if a dataset has millions or billions of instances.

Catastrophic forgetting was first recognized in MLPs almost 30 years ago [ 1989 ] . Since then, there have been multiple attempts to mitigate this phenomenon [ 1987 , 1995 , 2014 , 2016 , 2017 , 2017 , 2017 ] . However, these methods vary considerably in how they train and evaluate their models and they focus on small datasets, e.g., MNIST. It is not clear if these methods will scale to larger datasets containing hundreds of categories. In this paper, we remedy this problem by providing a comprehensive empirical review of methods to mitigate catastrophic forgetting across a variety of new metrics. While catastrophic forgetting occurs in unsupervised frameworks [ 2016 , 2014 , 2017 ] , we focus on supervised classification. Our major contributions are: • We demonstrate that despite popular claims [ 2017 ] , catastrophic forgetting is not solved.

• We establish new benchmarks with novel metrics for measuring catastrophic forgetting. Previous work has focused on MNIST, which contains low-resolution images and only 10 classes. Instead, we use real-world image/audio classification datasets containing 100-200 classes. We show that, although existing models perform well on MNIST for a variety of different incremental learning problems, performance drops significantly with more challenging datasets.

• We identified five common mechanisms for mitigating catastrophic forgetting: 1) regularization, 2) ensembling, 3) rehearsal, 4) dual-memory models, and 5) sparse-coding. Unlike previous work, we directly compare these distinct approaches.

### Problem Formulation

In this paper, we study catastrophic forgetting in MLP-based neural networks that are incrementally trained for classification tasks. In our setup, the labeled training dataset D D is organized into T T study sessions (batches), i.e., D = { B t } t = 1 T D=\left\{{B_{t}}\right\}_{t=1}^{T} . Each study session B t B_{t} consists of N t N_{t} labeled training data points, i.e., B t = { ( 𝐱 j , y j ) } j = 1 N t B_{t}=\left\{{\left({{\mathbf{x}}_{j},y_{j}}\right)}\right\}_{j=1}^{N_{t}} , where 𝐱 j ∈ ℝ d \mathbf{x}_{j}\in\mathbb{R}^{d} and y j y_{j} is a discrete label. N t N_{t} is variable across sessions. The model is only permitted to learn sessions sequentially, in order. At time t t the network can only learn from study session B t B_{t} ; however, models may use auxiliary memory to store previously observed sessions, but this memory use must be reported. We do not assume sessions are iid, e.g., some sessions may contain data from only a single category. In between sessions, the model may be evaluated on test data. Because this paper’s focus is catastrophic forgetting, we focus less on representation learning and obtain feature vectors using embeddings from pre-trained networks. Note that in some other papers, new sessions are called new ‘tasks.’ We refer to the first study session as the model’s ‘base set knowledge.’

## Why Does Catastrophic Forgetting Occur?

Catastrophic forgetting in neural networks occurs because of the stability-plasticity dilemma [ 2005 ] . The model requires sufficient plasticity to acquire new tasks, but large weight changes will cause forgetting by disrupting previously learned representations. Keeping the network’s weights stable prevents previously learned tasks from being forgotten, but too much stability prevents the model from learning new tasks. Prior research has tried to solve this problem using two broad approaches. The first is to try to keep new and old representations separate, which can be done using distributed models, regularization, and ensembling. The second is to prevent the forgetting of prior knowledge simply by training on the old tasks (or some facsimile of them) as well as new tasks, thereby preventing the old tasks from being forgotten. Besides requiring costly re-learning of previous examples and additional storage, this scheme is still not as effective as simply combining the new data with the old data and completely re-training the model from scratch. This solution is inefficient as it prevents the development of deployable systems that are capable of learning new tasks over the course of their lifetime.

## Previous Surveys

? ( ? ) exhaustively reviewed mechanisms for preventing catastrophic forgetting that were explored in the 1980s and 1990s. ? ( ? ) compared different activation functions and learning algorithms to see how they affected catastrophic forgetting, but these methods were not explicitly designed to mitigate catastrophic forgetting. The authors concluded that the learning algorithms have a larger impact, which is what we focus on in our paper. They sequentially trained a network on two separate tasks using three different scenarios: 1) identical tasks with different forms of input, 2) similar tasks, and 3) dissimilar tasks. We adopt a similar paradigm, but our experiments involve a much larger number of tasks. We also focus on methods explicitly designed to mitigate catastrophic forgetting.

? ( ? ) reviewed neural networks that can adapt their plasticity over time, which they called Evolved Plastic Artificial Neural Networks. Their review covered a wide-range of brain-inspired algorithms and also identified that the field lacks appropriate benchmarks. However, they did not conduct any experiments or establish benchmarks for measuring catastrophic forgetting. We remedy this gap in the literature by establishing large-scale benchmarks for evaluating catastrophic forgetting in neural networks, and we compare methods that use five distinct mechanisms for mitigating it.

## Mitigating Catastrophic Forgetting

While not exhaustive, we have identified five main approaches that have been pursued for mitigating catastrophic forgetting in MLP-like architectures, which we describe in the next subsections. We describe the models we have selected in greater detail in the Experimental Setup section.

### Regularization Methods

Regularization methods add constraints to the network’s weight updates, so that a new session is learned without interfering with prior memories. ? ( ? ) implemented a network that had both ‘fast’ and ‘slow’ training weights. The fast weights had high plasticity and were easily affected by changes to the network, and the ‘slow’ weights had high stability and were harder to adapt. This kind of dual-weight architecture is similar in idea to dual-network models, but has not been proven to be sufficiently powerful to learn a large number of new tasks. Elastic weight consolidation (EWC) [ 2017 ] adds a constraint to the loss function that directs plasticity away from weights that contribute the most to previous tasks. We use EWC to evaluate the regularization mechanism.

### Ensemble Methods

Ensemble methods attempt to mitigate catastrophic forgetting either by explicitly or implicitly training multiple classifiers together and then combining them to generate the final prediction. For the explicit methods, such as Learn++ and TradaBoost, this prevents forgetting because an entirely new sub-network is trained for a new session [ 2001 , 2007 ] . However, memory usage will scale with the number of sessions, which is highly non-desirable. Moreover, this prevents portions of the network from being re-used for the new session. Two methods that try to alleviate the memory usage problem are Accuracy Weighted Ensembles and Life-long Machine Learning [ 2003 , 2017 ] . These methods automatically decide whether a sub-network should be removed or added to the ensemble.

PathNet can be considered as an implicit ensemble method [ 2017 ] . It uses a genetic algorithm to find an optimal path through a fixed-size neural network for each study session. The weights in this path are then frozen; so that when new sessions are learned, the knowledge is not lost. In contrast to the explicit ensembles, the base network’s size is fixed and it is possible for learned representations to be re-used which allows for smaller, more deployable models. The authors showed that PathNet learned subsequent tasks more quickly, but not how well earlier tasks were retained. We have selected PathNet to evaluate the ensembling mechanism, and we show how well it retains pre-trained information.

### Rehearsal Methods

Rehearsal methods try to mitigate catastrophic forgetting by mixing data from earlier sessions with the current session being learned [ 1995 ] . The cost is that this requires storing past data, which is not resource efficient. Pseudorehearsal methods use the network to generate pseudopatterns [ 1995 ] that are combined with the session currently being learned. Pseudopatterns allow the network to stabilize older memories without the requirement for storing all previously observed training data points. ? ( ? ) used this approach to incrementally train an autoencoder, where each session contained images from a specific category. After the autoencoder learned a particular session, they passed the session’s data through the encoder and stored the output statistics. During replay, they used these statistics and the decoder network to generate the appropriate pseudopatterns for each class.

The GeppNet model proposed by ? ( ? ) reserves its training data to replay after each new class was trained. This model used a self-organizing map (SOM) as a hidden-layer to topologically reorganize the data from the input layer (i.e., clustering the input onto a 2-D lattice). We use this model to explore the value of rehearsal.

### Dual-Memory Models

Dual-memory models are inspired by memory consolidation in the mammalian brain, which is thought to store memories in two distinct neural networks. Newly formed memories are stored in a brain region known as the hippocampus. These memories are then slowly transferred/consolidated to the pre-frontal cortex during sleep. Several algorithms based on these ideas have been created. Early work used fast (hippocampal) and slow (cortical) training networks to separate pattern-processing areas, and they passed pseudopatterns back and forth to consolidate recent and remote memories [ 1997 ] . In general, dual-memory models incorporate rehearsal, but not all rehearsal-based models are dual-memory models.

Another model proposed by ? ( ? ), which we denote GeppNet+STM, stores new inputs that yield a highly uncertain prediction into a short-term memory (STM) buffer. This model then seeks to consolidate the new memories into the entire network during a separate sleep phase. They showed that GeppNet+STM could incrementally learn MNIST classes without forgetting previously trained ones. We use GeppNet and GeppNet+STM to evaluate the dual-memory approach.

### Sparse-Coding Methods

Catastrophic forgetting occurs when new internal representations interfere with previously learned ones [ 1999 ] . Sparse representations can reduce the chance of this interference; however, sparsity can impair generalization and ability to learn new tasks [ 1995 ] .

Two models that implicitly use sparsity are CALM and ALCOVE. To learn new data, CALM searches among competing nodes to see which nodes have not been committed to another representation [ 2014 ] . ALCOVE is a shallow neural network that uses a sparse distance-based representation, which allows the weights assigned to older tasks to be largely unchanged when the network is presented with new data [ 1992 ] . The Sparse Distributed Memory (SDM) is a convolution-correlation model that uses sparsity to reduce the overlap between internal representations [ 1988 ] . CHARM and TODAM are also convolution-correlation models that use internal codings to ensure that new input representations remain orthogonal to one another [ 1983 , 1982 ] .

The Fixed Expansion Layer (FEL) model creates sparse representations by fixing the network’s weights and specifying neuron triggering conditions [ 2013 ] . FEL uses excitatory and inhibitory fixed weights to sparsify the input, which gates the weight updates throughout the network. This enables the network to retain prior learned mappings and reduce representational overlap. We use FEL to evaluate the sparsity mechanism.

## Experimental Setup

We explore how well methods to mitigate catastrophic forgetting scale on hard datasets involving fine-grained image and audio classification. These datasets were chosen because they contain 1) different data modalities (image and audio), 2) a large number of classes, and 3) a small number of samples per class. These datasets are more meaningful (real-world problems) and more practical than MNIST. We also use MNIST to showcase the value of these real-world datasets. See Table 1 for dataset statistics.

### Dataset Description

#### MNIST

MNIST is a classic dataset in machine learning containing 10 digit classes. Its grayscale images are 28 × 28 28\times 28 .

#### CUB-200