Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy — part 7/9

Cleaned-text code points [93041, 108678). Bibliography omitted here; source.md includes it.

An even more resource-efficient proposal was done by ? (?). They only store the statistic (mean and std) of the last layers, so in inference time, the weights are generated according to the task. To decide which task the sample belongs to, they use an auto-encoder for each task. While saving resources on the neural network, it creates new structures and information to store. It is important to point out here that when tasks share the first layers, the model presents favoritism in the first learned task. When the tasks have a similar domain it works fine, however, to a different domain it can deliver poor results.

Several studies have explored the utilization of methods and strategies within the Dynamic Networks category to tackle CF in more challenging problem setups and contexts. For instance, ? (?) work on Unbounded Task Incremental Learning proposing a framework that creates a new network, called Expert, to handle the unknown classes. As the data arrives in the stream, when the sample is not recognized by any Expert that already exists, the sample is stored in an ephemeral memory. When this memory is full, a new Expert is created and trained with this data, after this, the data is discarded. In the context of GANs, ? (?) propose to integrate each parameter from two parts: i) shared one; and, ii) task-specific one. The former is fixed after the first task or gotten from a pretrained model. The latter is created for each new task, in such a way that, by adding the two parts, the model produces good results. To create distinguished task-specific parts, they apply the Kullback-leibler divergence constraints.

A typical challenge faced by Dynamic Network techniques is the requirement of knowing the task that a given sample belongs to. This problem also happens in some models of the Sub-network group too. Some works propose models equipped with their own task selector, but not all. Therefore, some authors focus on research methods to be used as oracles to decide which task each sample belongs to. ? (?) propose to use a KNN as oracle and just create a new output layer head to each task. This is resource-efficient; however, they only present tests on permutation MNIST data sets. Consequently, each task exhibits highly similar input statistics, i.e., there is no significant distribution shift across the tasks.

In summary, models within the Dynamic Networks category exhibit the ability to increase their size to accommodate new knowledge, with some models altering their structures while others create auxiliaries structures. Figure 13 provides an overview of this category. The primary objective of these models is to learn new tasks while minimizing the increase in model size. However, some authors accept a polynomial increase in size to achieve more efficient learning of new tasks. Similar to the Sub-Networks category, distinguishing the task to which a sample belongs remains a challenge in Dynamic Networks approaches. This area of research is still actively evolving and holds promise for improving existing methods. However, a notable drawback of this category is the space required by the model, making it less suitable for scenarios such as edge computing or mobile applications where space constraints are critical. In the subsequent section, we will discuss methods that combine strategies from multiple categories, aiming to leverage the strengths of each approach.

### 3.5 Hybrid Approaches

Researchers who often employ a hybrid approach combine two or more techniques described in previous sections to deal with CF effectively, thus leveraging the strengths of different strategies to achieve better performance. Certain combinations are relatively straightforward to implement, such as incorporating rehearsal methods with other techniques, and often yield enhancements in accuracy for previously learned tasks. The flexibility of hybrid approaches also allows them to tailor their solutions to specific requirements and achieve a balance between mitigating forgetting and acquiring new knowledge.

#### 3.5.1 Enhancing Distance-based using Rehearsal

Distance-based approaches encounter a significant challenge in effectively tracking class prototypes when the feature extractor undergoes weight modifications during training. These changes in weights result in shifts of the prototype vectors to different regions in the feature space, which can lead to confusion in the final classifier’s decision-making process. To address this issue, rehearsal techniques can be employed by storing a subset of samples and their corresponding feature vectors. By incorporating rehearsal, the feature vectors of these stored samples can be closely monitored during the learning of new tasks, enabling the detection of any distance shifts caused by weight modifications. ICaRL pioneered the combination of rehearsal and distance-based methods to tackle CF, effectively maintaining accurate class representations despite weight modifications and prototype shifts (?).

Rehearsal, as demonstrated by ? (?), can be effectively combined with a Distance-based and Knowledge Distillation (KD), in a meta-learning context, to enable adaption to new tasks while minimizing the interference with previously learned tasks. This integration is complemented by the use of a coreset, which helps maintain consistent embeddings of stored samples. The proposed model, presented by ? (?), consists of two main components: an extractor and a classifier. At the start of the training session, a temporary copy of the extractor is created with its parameters frozen. Throughout the training, both the frozen extractor and the active one produce identical embeddings for the same input data. This alignment is crucial to ensure that the classifier achieves high accuracy when predicting new classes during the fine-tuning process. The entire process takes place within the meta-learning inner loop.

Furthermore, in an Online Incremental Learning setup where the model lacks task ID knowledge and must learn in a single pass, a Distance-based approach augmented with a Mini-rehearsal technique can be employed to mitigate CF. To address the challenge of limited samples per class, the MixUp technique introduced by (?) can be utilized for data augmentation. By combining these strategies, including Distance-based, KD, Mini-rehearsal, and MixUp, a model can adapt to new tasks, retain past knowledge, and mitigate CF while overcoming limitations associated with this setup.

Distance-based approaches benefit from using cosine distance classifiers, which have shown greater resilience to CF compared to softmax classifiers (?, ?). However, this advantage holds true only when the embeddings belonging to the same class can be effectively clustered in the latent space and remain sufficiently distant from other class clusters. To address this requirement, ? (?) propose a solution that involves storing a coreset of samples and applying Contrastive loss during training. By leveraging the coreset and Contrastive loss, the aim is to encourage the embeddings to automatically cluster in a way that facilitates better separation between classes. Thus, the model can better organize the embeddings in the latent space, resulting in improved class separation and enhanced resilience to CF.

In a highly challenging setup, such as Online Incremental Learning combined with an Unbounded Task scenario, the utilization of rehearsal techniques becomes crucial for preserving previous knowledge in Distance-based methods. To address this, ? (?) propose a memory-based approach that consists of two components: samples and prototypes. Memory storage effectively handles the unbalanced data stream that arises in such setups. The assumption is that prototypes will change over time as new information is received. Therefore, a novel loss function is introduced to guide the feature extractor in generating embeddings that are closer to the class prototype while being farther away from other prototypes. This encourages better separation between classes and improves the model’s ability to retain previous knowledge. As the samples in the memory are replaced with new ones, the prototypes evolve accordingly, adapting to the changing data distribution. This dynamic updating of prototypes ensures that the model remains up-to-date and capable of handling evolving tasks and unbounded learning scenarios effectively.

The analysis of geometric relations among classes can be examined from multiple perspectives through the utilization of Mixed-curvature Spaces, as proposed by ? (?). Their approach involves creating distinct spaces for each task and projecting samples onto all relevant spaces during inference. By leveraging a coreset, their objective is to ensure that new projections do not introduce confusion with classification from previous spaces. Similarly, ? (?) employ a projection technique for sample embeddings. However, they adopt a Voronoi Diagram as the designated space. Specifically, when a new class emerges, they partition a cell within the diagram. One portion of the split is dedicated to retaining existing knowledge, while the other part accommodates the new class. This process, facilitated by the utilization of a coreset, entails strategic decisions on cell selection for partitioning and ensuring the preservation of a structured framework that mitigates the challenges posed by CF.

In scenarios where the total number of classes is known in advance, yet the classes must be learned sequentially, it is feasible to predefine a distinct region for each class and solely train a projection layer, as proposed by ? (?). It is important to note that the underlying embedding generation model remains fixed throughout this process, and it can either be a pretrained model or be frozen after the initial task. Since the projection layer necessitates updates with each new task, the inclusion of a coreset becomes imperative to retain the knowledge on how to appropriately project each class into its respective space, thereby effectively addressing the challenge of CF.

#### 3.5.2 Enhancing Distance-based using Dynamic Networks

Hybrid approaches that combine Dynamic Networks with Distance-based methods offer another effective solution. The best example is the Encoder-Based Lifelong Learning approach, which creates a new classification layer for each task and tracks changes in class embeddings (?). In this method, an auto-encoder is trained for each task, allowing the preservation of embeddings and enabling the fixation of the classification layer for old tasks through knowledge distillation.

The use of auto-encoders in this approach serves two key purposes. Firstly, it allows the embeddings of one class to be stationary over the model lifespan. Secondly, the auto-encoders can serve as oracles, providing guidance to the classifier in the model’s lifelong learning process. This selection is based on the distances between the preserved embeddings, ensuring accurate and task-specific classification.

By combining Dynamic Networks with Distance-based methods and leveraging the Encoder-Based Lifelong Learning approach, a model can adapt to new tasks while retaining knowledge from previous tasks. This hybrid approach offers a comprehensive solution that effectively handles lifelong learning scenarios mitigating CF. However, since the approach relies on training separate auto-encoder for each task, the memory footprint of the model grows as new tasks are introduced. Therefore, the auto-encoder structure has to be much lower than the original model to be a viable approach.

#### 3.5.3 Enhancing Dynamic Networks using Rehearsal

One significant limitation of Dynamic Networks methods lies in their elevated growth rate. To mitigate and control this growth, a prudent approach involves maintaining a coreset, which enables the model to modify certain parameters while preserving knowledge of previous classes. Consequently, the model can effectively acquire new knowledge with reduced resource requirements, such as fewer neurons and layers.

? (?) propose to use an Evolvable Neural Turing Machine (?) (ENTM). In this model, a tape is added to store contextual information. This tape has two heads, one to read and the other to write, which are controlled by a neural network. They used an approach called Neuroevolution of Augmenting Topologies (NEAT) (?) to produce this neural network. NEAT uses evolutionary algorithms to increase the size of the neural network, so it starts with a shallow network and evolves it to a more robust one until it finds an acceptable solution. In addition, ENTM stores the contextual information that allows the neural network to remember things. The left side of Figure 14 illustrates the proposed architecture. This model is hybrid, ENTM is its complementary learning system, and the NEAT module makes its neural network grow dynamically. Unfortunately, they only tested this proposal in an RL context and requires a handcraft-designed input and output by task. The input needs to receive three pieces of information: i) the new state; ii) the reward (environment output); and, iii) the contextual information in the reader’s head tape. The network’s output produces: i) the action (environment input); ii) a context representation to be written in the tape; and, iii) controllers to the tape’s heads. Besides that, they do not compare their results with other works, just demonstrating that this model can solve the RL problem defined by ? (?).

? (?) expand the model adding a new classifier to each task and using an approach based on herding selection to maintain samples on the coreset. In contrast to ICaRL (presented in Section 3.5.1 ), they train a parametric classifier together with the feature extractor. The right side of Figure 14 presents the process of training. For each new task, a new head classifier is added to the model and trained with cross-entropy loss, while the oldest ones are trained with distillation loss using KD. They store a part of the older task data but also grow the network structure by adding a new head to each task. To handle the problem of the imbalanced data set, they apply a post-training fine tune with samples from memory and a subset of samples from the current task such that all classes have the same number of samples.

? (?) extend DGR (?) adding the capability of expanding the generator model, creating a hybrid model from Pseudo-rehearsal and Dynamic Networks. After a training session, the generator parameters are frozen, then in the next session, more resources (layers and neurons) are added. Due to this, it requires storing a mask to identify which parameters belong to each task. Instead of generating pseudo-samples, ? (?) propose to use a Variational Autoencoder (VAE) to generate weights to create an ensemble of models, all of them with the same structure. These weights are considered a kind of rehearsal of old knowledge. Although the method proposed by ? (?) also has a weight generator module, the technique proposed by ? (?) differs from them because they create an ensemble of models.

RL agent was used as a neural structure search method to expand the model in a Task Incremental Learning setup (?). It considers as the environment the current task data, a continual learner, a knowledge repository, and the coreset. The knowledge repository contains the parameter’s value of old tasks. Thus, the agent uses the environment to create a new learner, minimizing the forgetting ratio (validating it with the coreset) and maximizing the accuracy of the current task.