Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy — part 2/9

Cleaned-text code points [15291, 31008). Bibliography omitted here; source.md includes it.

Pseudo-rehearsal is based on the assumption that storing previously seen data is not required to deal with CF. It is sufficient to generate new data synthetically, known as pseudo-samples, in order to represent old classes on demand. As illustrated in Figure 3 , the generated data does not have to be identical to the original. In this figure, the first row shows noisy images generated by ? (?) to represent classes from the MNIST (?) data set, namely 0, 1, 2, 3, and 4. Although the samples in the first row cannot be identified as digits by a human, they are intended to activate the same set of neural network connections as the original ones in the second row. Therefore, they can be used to retrain the model from time to time, remembering previously learned concepts.

To perform a new training session and acquire new knowledge, a copy of the existing model is created. This is called a clone of the model. Then, some noisy generated images are labeled by this cloned model. The pairs of noisy images and their labels which have high confidence by the cloned model are concatenated to the current task data set as synthetic data. Thus, the original model is trained on this extended data set. This aims to grant the model to remember classes from an old task. The key problem with synthetically produced samples is that they don’t actually belong to the class to which they were assigned. This is known as the ambiguities problem in Incremental Learning (?). Therefore, they can conflict with the newest classes the model is learning. Just as a playful example, assume that a neural network was trained to classify images of cats and dogs. Then a new class is added to this neural network. Now assume that the random image generated by the algorithm depicts a sample from the new class (a bird, for example). This image will be labeled as a dog or a cat because the last trained model only knows these two classes.

Bayesian Learning and Stochastic Langevin Dynamics were used by ? (?) to improve the generation of such synthetic data samples. These two techniques were used to generate samples that would maximize the neuron activation, instead of random generation. They forced the generated samples to increase the node output that represents their classes. However, these techniques can lead to the generation of synthetic data with low diversity. To handle this problem the authors proposed a Brownian motion (?) technique to avoid the selection of similar samples to the data set (using the distance among them in input space). Therefore, they can produce a diverse synthetic data set, discarding the most similar. This technique proved to be resilient to CF in toy data sets, like MNIST, where the dimension of the input data, (28, 28) pixels, is not as big as it could be in real applications, such as facial recognition (?). In applications where the input space is larger, the performance of this technique tends to quickly decrease.

The above technique aims to generate synthetic data that will activate specific neurons that represent certain classes. Nevertheless, there is no assurance that the most elevated ones will be generated. To force this, ? (?) propose to use the inverted model trained on the previous task to generate pseudo-samples passing the class label as input. However, the synthetic data generated still does not seem like the original samples. The strategy was used again by ? (?), however, different from the previous, they demonstrated that using the Knowledge Distillation (KD) (?) improves the final result. They call this strategy a memory recovery paradigm. And, recently, ? (?) used the DeepInversion (?) model to generate the pseudo-samples in the Few-shot Incremental Learning setup.

Generative networks were also adopted to create synthetic samples that were more likely to be similar to the original ones. For instance, ? (?) used a Recurrent Neural Network (RNN) called DRAW (?) to generate synthetic data. Different from traditional RNNs, their model has 2 heads: i) a fully connected NN with a softmax activation to classify the input; and, ii) a decoder to re-create the input. It showed that Generative Adversarial Networks (?) (GANs) could play an important role in helping to mitigate the CF effects for pseudo-rehearsal. In the same year, ? (?) propose the Deep Generative Replay framework which uses a GAN as an auxiliary model used at the start of each training session to generate synthetic data from already known classes. The GAN is trained at the end of each training session using the data set of the current task and the self-generated one.

GANs were also used to produce synthetic data to avoid CF in more general scenarios: i) label-conditioned image generation, where the input is the label of class and the output is an image; and, ii) image-conditioned generation, where the input is an image of a class and the output is an image from the same class, but in a different domain. For instance, a cartoon cat image as input can result in a sketch cat image as output (?). Although generative models can generate synthetic data for your own incremental training, these data are not always high quality, problems like blur usually occur. Therefore, ? (?) propose to generate feature maps that are passed to the classifier’s internal layers instead of samples in the original input space. The use of feature maps instead of pseudo-samples to do the classifications is an easier task due to the size of the feature map in comparison to the raw image.

Learning using Encoded Experience Replay (CLEER) (?) is an Autoencoder (AE) trained to make all tasks share the same distribution in the embedding space. For this, the model is divided into three parts: i) an encoder; ii) a shared classifier; and, iii) a decoder to produce pseudo-samples. The encoder is trained to generate embedding with the same distribution, regardless of tasks. For this, it models the embedding space as a multi-modal distribution using the Gaussian Mixture Model (GMM). When a new task needs to be learned, the model uses the decoder with the GMM to generate pseudo-samples that are merged with the samples of the current task. So, with this expanded data set, the model goes through a new learning phase in these three components. By doing this, the encoder encourages all samples to be grouped together in a single embedding region. As a result, the categorization task is simplified. It works as a form of normalization.

Variational Autoencoder (VAE) is naturally more resilient to CF (?). Due to this, VAE is used in Incremental Learning both to generate images and classification tasks. The choice of the prior distribution over the latent space is crucial to a VAE present accurate results (?). However, in an Incremental Learning process, choosing this distribution is not trivial, because the data changes over a lifelong term. ? (?) propose to use the gradient in an end-to-end training to find a prior that is a mixture of current data distribution and old distributions. However, it also presents image quality problems produced by the decoder.

Another way used to generate examples of known classes is by using class prototypes. The model can remember the class since it serves as an anchor. A prototype is a representation of the samples of a class and is usually determined by averaging all available samples. This is the strategy that requires less storage memory overhead, however, it requires that all samples from a class are clustered and separated from other classes. In these scenarios, simply add Gaussian noise to the prototype to get an ephemeral collection that can be used together with the training set in each training session (?, ?).

Instead of forcing the approximation of samples in the embedding space, ? (?) propose to adapt the value of the model’s weights to handle instances from different classes. They did this by employing the model together with three auxiliary models: i) a Solver , which predicts the response of a task; ii) a Dynamic Parameter Generator (DPG), which creates a set of parameters to be merged in the Solver for each sample; iii) an Encoder that creates an embedding to be used as input by the DPG and the Decoder; and, iv) a Decoder which creates realistic data samples of tasks that are no longer available. This strategy allows mutating the weights based on the current processed sample. It helps the model to mitigate the semantic drift problem (?). This is also considered pseudo-rehearsal (and not a Dynamic Network) as the model maintains its structure but modifies its parameter values by utilizing data derived from prior knowledge.

Figure 4 shows how these models interact with each other during inference (left side) and training (right side) time. Given an image of a new class, the Encoder generates an embedding vector. This vector is given to the DPG which outputs a set of weights. These weights are combined with those provided by the Solver to predict input class y 𝑦 y italic_y , along with the samples of the new task. The models are trained with samples generated by the Decoder . Data Generator , a macro component composed of the Encoder and Decoder , can also suffer from CF. To mitigate this, the authors propose to use a KD loss as illustrated on the right side of Figure 4 . A frozen version of the last training session is used as a “teacher” in the current training session to remember the old behavior of the models. For this, a random embedding vector from a normal distribution is used to generate a synthetic image. Next, the objective is to minimize the difference between that synthetic image and the one generated by the current Decoder . In addition, the current Encoder must generate an embedding from the synthetic image and minimize the difference between its embedding and that used by the frozen Decoder .

While the results indicate strong performance in a two-task sequence, further evaluation is necessary to validate the method’s efficacy in handling arbitrary task sequences and more complex tasks beyond MNIST. Future investigations should focus on expanding the evaluation to encompass diverse scenarios, ensuring the method’s scalability and robustness. Nevertheless, the use of synthetic data sets that are more similar to the current samples shows more promising results.

Hypernetworks are also used to generate values for the model’s parameters (?) based on a task identifier. This approach passes the problem of CF to the hypernetwork. However, ? (?) advocate this is less affected because the pair of input-output is less than these relations between data of classes from tasks. Although it still happens, they use a VAE to apply pseudo-rehearsal in the hypernetwork.

? (?) explain the CF problem as a causal effect of the old data lost in new training. They argue that replaying a feature representation of data does not achieve the causal information of replaying the original data. However, they showed that it is possible to distill the colliding effect between the old and the new data on embedding space.

Other learning algorithms were also tested with pseudo-rehearsal.For instance, Contrastive Hebbian Learning (?) and No-Prop (?) were tested by ? (?).However, they only tested on toy data sets and with shallow networks. Future work could perform tests with deep architectures and more data sets and also use GANs to generate synthetic data. Nevertheless, in our work, we focus on methods based on the gradient descent learning algorithm, so this work is out of our scope. Figure 5 depicts an overview of this category. It shows that the pseudo-examples can be used to change the parameter values of the model or distill the causal information. In addition, it shows the main ways to generate pseudo-samples. In the next section, we present works that allow for the storage of certain outdated data to prevent CF.

#### 3.1.2 Mini-rehearsal

The idea behind mini-rehearsal is that a subset of the original data must be retained to prevent CF while training a model on a new task. The objective is to minimize the amount of old data that needs to be kept in order to allow the model to learn the new task without experiencing CF. This can be accomplished through selective data sampling. This subset is known as coreset. The main objective is to identify a representative subset of the training set. Therefore, with this subset stored, CF can be handled by the model.

The Greedy Sampler and Dumb Learner (GDumb), an established baseline of this category that demonstrates its strength, is a naive model that questions the advancements of contemporary approaches to address the CF suggested in the literature (?). A memory manager and a learner make up GDumb’s two main modules. For each new class that is learned, a new bucket is made to hold its samples, according to the naive management practiced by the previous. The sample from the bucket containing the most samples is removed when memory is full. The learner uses samples stored in memory to train a DNN from scratch. Several studies that have been published in the literature are surpassed by this model.

There is a lot of research on how to improve the sample selection for the coreset (?, ?, ?, ?, ?). It is possible to select the coreset that reduce the forgetting of knowledge using a bilevel optimization based on gradients (?). However, it has a high computational cost despite the optimizations proposed by the authors. Using a GeForce GTX 1080 Ti GPU to select a coreset of 100 samples in a set of 1000, takes about 20 seconds, while selecting 400 samples in the same set takes more than three minutes. This shows that the time does not increase linearly. In addition, it only works with a static set of samples for the task, therefore it cannot be used in an Online Incremental Learning setup.

Most researchers advocate that the size of coreset has to be limited, but there are two ways to think about this: i) M c subscript 𝑀 𝑐 M_{c} italic_M start_POSTSUBSCRIPT italic_c end_POSTSUBSCRIPT samples per class; or ii) M 𝑀 M italic_M samples to be managed to all classes. The former has the problem of total memory increase as the tasks learned increase, due to this, some researchers believe that compressing the data is a way to overcome this problem (?, ?). Exemplar Streaming (ExStream) algorithm compresses the bucket data and is compared to other two compression algorithms: i) Online k-means; ii) and, CluStream; and two replacement algorithms: i) reservoir sampling ii) and, first-in-first-out queue. The results showed that when we have a bucket size above 256, all algorithms behaved similarly. However, when the bucket size is small, like four or eight, the correct choice of samples can improve the results. For instance, ExStrem showed an improvement of more than 50%, in some cases, compared to the reservoir algorithm (?).

The sample compression causes a tradeoff between quality and quantity, however, it is possible to have gained in accuracy, even with lower quality samples (?). It has also been noted in the literature to apply mutations to coreset samples in order to maximize memory retention. Mnemonics (?) consider each bucket as a variable and apply a post-training phase, where the samples stored in the memory mutate their values to be considered more representative. Despite it being more computationally efficient than the above methods, it still requires to be done in the offline phase. ? (?) apply gradient-based editing on samples from coreset (in an online fashion), instead of replacing them, however, they were looking to unbounded task incremental learning setup, where the data distribution changes without a sense of task identifier.