Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy — part 3/9

Cleaned-text code points [31008, 46529). Bibliography omitted here; source.md includes it.

Another way to reduce the amount of memory required is to consider storing the representations (feature maps) instead of raw samples to perform the rehearsal process. REMIND (?) showed that it does not present better results in scenarios where each task is learned sequentially, however in scenarios where classes came in any order (online learning), it presents the state-of-the-art results. To achieve this, they used the MixUp (?) technique in the stored samples to get more robust and representative sets in the replay process.

Given that the coreset has a fixed size of M 𝑀 M italic_M , we need a method to replace, update, or remove samples from the classes’ bucket when memory is full. Applying the herding selection algorithm, ICaRL (?), detailed in Section 3.5 as it is a hybrid technique, was a pioneer in this field. Herding selection chooses the samples closest to the centroid of the samples of each class. Reservoir sampling is the most used strategy to manage the coreset. This is a naïve strategy that gives each sample the same chance of being selected to be removed from the coreset once it is full. As a result, it cannot be assumed that the coreset will always be balanced, however, empirical research suggests an approximation of this (?).

Metrics can be employed to determine the selection of samples that will be retained within the coreset. Shapley value was used to estimate the individual contribution of each sample to the performance of the model (?). Maximally Interfered Retrieval (MIR) uses the loss to check if the incoming sample is more negatively impacted by updating the model parameters than the already stored data. Information theory was used to manage the most relevant samples to keep in the coreset using the Bayesian model to compute the criteria efficiently by exploiting rank-one matrix structures (?). Classifier’s confidence is another metric that was used in the literature. It is possible to select the elements with the largest or worst confidence. In the former case, it is selecting samples that the model has no problem hitting, while the latter are samples that the model has difficulty inferring correctly. According to empirical studies, choosing the most trustworthy ones produces better results (?).

An RL agent was trained to decide what proportion of memory should be allocated to each class and also which samples it should keep (?). As the agent can also suffer from CF, it is only trained on the first task, splitting the data into N 𝑁 N italic_N groups to simulate various tasks. In an Online Incremental Learning setup, the problem of an imbalanced and noisy data set is more predominant. Online Coreset Selection (?) manage the memory following three selection objectives: i) minibatch similarity, to select representative samples to the current task; ii) cross-batch diversity, to reduce the redundancy among the samples of the current task; and, iii) coreset affinity, to minimize the interference between the selected samples and knowledge of the previous tasks.

All pretrained models can use the coreset in an Incremental Learning scenario, transforming it into a Memory Augmented Neural Network (MANN) (?). The memory of MANN is a table in which each row is a tuple { l m subscript 𝑙 𝑚 l_{m} italic_l start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT , v m subscript 𝑣 𝑚 v_{m} italic_v start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT , α m subscript 𝛼 𝑚 \alpha_{m} italic_α start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT }, where l m subscript 𝑙 𝑚 l_{m} italic_l start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT is the instance label, v m subscript 𝑣 𝑚 v_{m} italic_v start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT is the embedding generated by the last layer of the model (just before softmax activation) and α m subscript 𝛼 𝑚 \alpha_{m} italic_α start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT is the weight attached to the row. During inference, they interpolate the prediction of the pretrained model and prediction of the memory, using the top N 𝑁 N italic_N most similar tuples comparing v m subscript 𝑣 𝑚 v_{m} italic_v start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT with the embedding generated by the last layer’s model. These two values are interpolated by means of a set of weights, which is generated by a recurrent neural network (RNN) in each prediction.

At training time, the truth label is used to update the memory if a mistake is done by the model. In addition, only tuples where l m subscript 𝑙 𝑚 l_{m} italic_l start_POSTSUBSCRIPT italic_m end_POSTSUBSCRIPT is equal to the truth label are updated, so that rare classes are not forgotten. The RNN is also trained in an online fashion to decide the relevance of the prediction done by the memory and the pretrained model. Although the idea of the work implies that CF is avoided, there are no evaluations in this regard. Instead, the authors only evaluate the forward transfer knowledge without checking if older classes are still remembered by the model. Nevertheless, it is evident that this memory aids in addressing the issue of imbalanced data sets by leveraging the top N 𝑁 N italic_N similar tuples (?).

Following the approach based on feature embeddings (last layer’s output), ? (?) propose the adoption of a hash table using the embedding as key and the truth label as value. The model is divided into three components: i) the embedding generator; ii) the hash table memory; and, iii) the classifier. During training time, all models are updated. In the hash table, samples are appended as a circular buffer, while the embedding generator and the classifier are updated with backpropagation. In inference time, a K-Nearest Neighbors (KNN) is applied to the hash table keys, and the result is used to modify the classifier weights on demand. This is an online adaptation, as done by ? (?), however, instead of using a model to generate the drift in the weights, a little training session is performed to find this Δ Δ \Delta roman_Δ drift. As the memory has a fixed size and acts as a circular buffer, the model has a tendency on the most recently trained classes, making it difficult to remember rare classes or old tasks.

The literature makes numerous and varied uses of the coreset to reduce the CF. MeRGAN (?) analyzes two ways: i) joint training with replay; and ii) replay alignment. In the first, the samples are generated conditioned in the label of past tasks to be used together with the samples of the current task. While in the second, it is the application of KD in the Generator model, making the generator of the current task a student of the generator learned in the last task. Gradient Episodic Memory (GEM) (?) uses it to add a hard constraint in the updating step of the model. It only accepts changes in the parameters of the model if the error on coreset does not increase. This avoids forgetting in the coreset and allows backward knowledge transfer. However, the general error in the class can still increase. Moreover, the model can lose its ability to learn new tasks due to this hard restriction. ? (?) proposes to relax this constraint and start to accept updates despite increasing the error by a threshold. They called this method A-GEM. It outperforms GEM in accuracy and in the number of tasks that the model can learn.

To regularize the training in a different approach, ? (?) divide the gradient into two pieces and use the coreset to do this: i) relative to all tasks; and, ii) relative to the specific task. Then, they apply one constraint to each part. In the first, the gradients have to be in the same direction. In the last, they have to be orthogonal to another task-specific gradient from other tasks. Another regularization method adopted was the Elastic Weight Consolidation (?), used in the Natural Language Generation (NLG) domain (?). This constraint is a technique from the Sub-networks category that will be discussed in Section 3.3 .

? (?) and ? (?) hypothesized three main reasons why methods that use mini-rehearsal techniques present in the problem of forgetting: i) imbalanced magnitudes; ii) semantic drift; and, iii) ambiguities. ? (?) create LUCIR to prevent these three points. For the former reason, they noticed that the value of logits referring to classes of the current task tends to have a magnitude much higher than that of the classes of old tasks due to data imbalance, where the samples of the classes of the current task are abundant while the samples of the old classes are scarce. Therefore, they apply cosine normalization to the logits to make them all of the same magnitude. To face the semantic drift, they use KD in the embedding layer. Finally, to handle ambiguities among the classes, they use a margin ranking loss.

BiC appends a final layer to the model responsible to equalize the bias of the logits in relation to the new classes (?). For this, the samples of the current task are divided into two sets, validation, and training. The training set, which is still abundant, is used to train the model down to the logits layer, this is called stage 1 of the training session. Then, the validation set, which is balanced against the in-memory set, is used to fine-tune the last linear layer, which is used to make predictions. In the first stage, they also use KD to prevent semantic drift. They advocate that, in the second stage, the fact of training in a balanced data set, the model can handle the problem of ambiguities by itself. BiC is not categorized in dynamic networks because despite it appending a final layer on the model, it only happens on the model’s creation, not in its lifelong term.

The problem of imbalanced magnitudes was also observed by ? (?). To handle this issue, they use extra memory to store the statistics of classifier nodes of classes in the current training session when the data of these classes are abundant. In the next training sessions, when the data of these classes are sparse, the magnitude of their nodes tends to drop. Therefore, these statistics are used to rescale the weights of those parameters. ? (?) demonstrates that in an Online Incremental Learning setup, employing random transformations on the coreset to increase its size helps the model generalize knowledge and mitigate CF. Their work serves as a strong baseline for this particular setup.

? (?) propose to use a stream of unlabeled data from the internet to overcome the problem of unbalanced data. They advocate that this alleviates the three problems above. Moreover, they indicate this approach is better than fine-tuning the model in a post-training with a restricted amount of data. The data that comes from the stream are labeled by the model itself versioned on the last training session, in such a way as to distill the knowledge of old classes. In the context of object detection, ? (?) use unlabeled data to get a sampling of objects that are not presented in the current task anymore. Moreover, they use a scheme of dual networks to apply KD of this old knowledge.

? (?) argue that this bias is mainly caused by computing the softmax probabilities by combining the output scores for all the old and new classes. Thus, it is proposed to separate the softmax layer, leaving a head for each task and applying KD with the coreset during training. However, ? (?) show that the models also create a bias in the embedding space for the most recent classes. To address this, it is proposed to perform a semantic augmentation (semanAug) in the latent space with the old samples.

? (?) go in this direction and defend that forgetting mainly affects the classification layer. They proposed that vanilla fine-tuning can mitigate CF since classifiers learned in old tasks can be kept to standardize the weights against all classifiers learned in each task. They argue that this is necessary due to the bias towards the new classes. To make the weights fairer after learning a new class, the standardization of initial weights is done.

? (?) propose the Adaptation by Distillation (AD) method to show the advantages of KD in terms of Incremental Learning. In the training process, the lifelong term model does not directly learn from the truth labels, instead, an auxiliary model, called ” expert ”, is trained with original data. After that, the main model is trained by this expert, using its soft labels. They showed that this process gives the model a more generalized knowledge, reducing the effects of CF. Despite the KD being widely used to mitigate CF, ? (?) show, in their experiments, that it is worst to model accuracy when the model has access to a representative coreset.

The Meta-learning approach was proposed to create a model agnostic (?). This model is not a specialist in any task; however, its solution is near all tasks. Therefore, it can use the coreset with the one-shot learning strategy to solve any task in its lifelong term. ? (?) find that sparse learning emerges, due to a large fraction of learning rates dropping to zero in this type of model (MAML). They showed that it can be used to focus the learning on the most appropriate regions of the model. ? (?) show that this approach also works in the context of object detection; and, XtarNet (?) uses meta-learning to train a MetaCNN to produce an embedding to new classes. These new embeddings are combined with the ones generated by a pretrained network. Then, the combined embedding is used for the final classification. Part of the data from each class is maintained in memory to be used as a query set and support set in meta-learning.

Generative classifiers are less prone to CF because the decision boundaries of n 𝑛 n italic_n classes learned sequentially are the same as when trained together (?). GeMCL extends OML (?) replacing its discriminative classifier with a generative one (Bayesian classifier). ? (?) propose a Bayesian framework where task-conditioned posterior parameter distributions are continually learned and compressed in a Hypernetwork. They reported almost no forgetting and show it scales to modern architectures such as ResNets.

Recently, with the increasing attention on Transformers, researchers have begun investigating their potential benefits in addressing the issue of CF (?, ?). They propose the hypothesis that by identifying the appropriate Prompt, the model can effectively prevent CF. However, a challenge arises in determining the ideal Prompt for each instance. In their study, ? (?) propose a method to learn and store the Prompts associated with each class within the coreset. Each Prompt is linked to a key, which is the embedding of a sample from the corresponding class. During inference, the current sample’s embedding is utilized to select the most suitable Prompt. This approach opens up a new research direction, as it shifts the focus from CF avoidance to discovering the optimal Prompt.

Figure 6 depicts an overview of this category. Although rehearsal methods show good results, they present an inherent problem of overfitting in the coreset (?). ? (?) advocate that this result is observed because the models that use coreset stay in the same low-loss region after a task has finished. However, they point out that more research in this direction is required. There are many spaces in research to evaluate the best approach to select the best samples to maintain in memory. In the next section, we present the works that use techniques focused on learning representations, that we classify as Distance-based methods.

### 3.2 Distance-based