Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy — part 8/9

Cleaned-text code points [108678, 123878). Bibliography omitted here; source.md includes it.

An additional prevalent challenge encountered in Dynamic Networks methods pertains to the determination of the task to which a sample belongs during inference. The application of a coreset can serve as a valuable aid in alleviating this predicament as well. Dynamically Expandable Representation ? (?) expands the model creating a new feature extractor for each task and having a unique classifier. Once the extractor is learned, it is frozen. The classifier is always modifying its parameters to learn new tasks, so it needs to store a coreset to avoid the CF. The classifier uses as input the concatenation of features produced by all extractors. Therefore, the representation size of each sample increases when a new task is learned.

An alternative approach to address the challenge of identifying the task associated with a sample in Dynamic Networks methods is to employ multiple classifier heads, each dedicated to a specific task. By assessing the confidence levels of each classifier, the one exhibiting the highest confidence can be selected. ? (?) propose to use the coreset, containing data from previous tasks, as a negative class within the current head classifier. Consequently, each classifier can function as an effective out-of-distribution detector, enabling the appropriate head selection during inference.

Both Dynamic Networks and Rehearsal techniques suffer from the disadvantage of increased storage requirements compared to the original model. Dynamic Networks necessitate storing additional parameters, while Rehearsal involves storing samples from a coreset. The increased space demands should be carefully considered in terms of resource utilization and memory constraints. ? (?) introduces a baseline framework for this hybrid category, which takes into account the overall memory consumption, additional parameters, and the presence of samples within the coreset. This framework is designed to strike a delicate balance between the number of retained samples in the coreset and the expansion of the model itself. Through their empirical analysis, the authors note that expanding the model at the deeper layers yields substantial advantages in terms of both accuracy and space requirements. Notably, this proposed baseline outperforms, in accuracy and space requirements, numerous techniques discussed in this section.

#### 3.5.4 Enhancing Sub-networks using Rehearsal

The application of Incremental Learning using Sub-networks techniques holds particular significance in scenarios characterized by limited computing resources, such as edge computing. Notably, the integration of a coreset, which retains select samples from previous tasks, can play a crucial role in facilitating the discovery of improved and more shareable sub-structures between tasks (?). Recent studies have highlighted that it may not be necessary to regularize all model weights to mitigate CF, as focusing on regularization solely in the last layer can suffice. This approach, known as functional regularization, has been effectively employed by ? (?) and ? (?), utilizing a Gaussian Processes (GP).

One drawback of functional regularization using GPs is its reliance on storing a large number of samples from the most recent tasks. Consequently, a hybrid approach combining regularization (identifying sub-networks within the last layer) with mini-rehearsal techniques should be favorable. It is important to note that the choice of mini-rehearsal technique varies across different studies. The results achieved through this approach exhibit similarities to methods that adopt regularization across all model weights while being computationally more efficient.

Another noteworthy technique is Neural Calibration for Online Continual Learning (NCCL) proposed by ? (?). NCCL extends the concept of Elastic Weight Consolidation (EWC) by introducing two mechanisms to modulate the input and output signals of DNNs, referred to as neuron calibration. The parameters of this module are learned to strike a balance in the stability-plasticity dilemma, aiming to discover sub-modules within the model that can effectively solve each task. However, it still requires storing a portion of the training data from each learned class to aid in the learning process.

? (?) employ two key techniques to encourage the formation of Sub-networks using Rehearsal techniques: Sparse Activations and Semantic Dropout. Sparse Activations utilize the k-Winner-Take-All (k-WTA) activation function, enabling the selective propagation of the top k 𝑘 k italic_k units with the highest activations to subsequent layers. On the other hand, Semantic Dropout promotes dropout in different neurons when learning new tasks, leveraging the activation patterns of neurons. This approach proves particularly advantageous when dealing with uncorrelated tasks.

Incorporating Sub-networks and Rehearsal techniques offer promising avenues for continual learning in environments of fixed space resources. They provide mechanisms to identify and leverage task-specific sub-structures within the model, thereby facilitating better retention of previously acquired knowledge while adapting to new tasks. Nevertheless, it is crucial to address the challenge of storing training data to ensure the effectiveness of these techniques.

#### 3.5.5 Growth Sub-networks using Dynamic Networks

The combination of Dynamic Networks and Sub-networks offers a solution to the limited resource challenge encountered by Sub-networks methods. One method in this regard is the Dynamically Expandable Network (DEN) introduced by ? (?). DEN aims to determine the minimum set of connections required by a Neural Network to learn a new task. If the existing connections are insufficient, DEN dynamically expands the model’s resources.

DEN operates through three distinct phases: training, expanding, and avoiding semantic drift. In the training phase, the model undergoes initial training using L 1 subscript 𝐿 1 L_{1} italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT regularization, which promotes weight sparseness. When a new task is introduced, an output node corresponding to the task is added. Subsequently, a pre-training step takes place, where only the last layer remains unfrozen, adjusting only the weights of the newly added node. Since the remaining parts of the model also maintain sparse connections, a Breadth-First Search (BFS) algorithm is employed to establish a path S 𝑆 S italic_S that connects every output node to the model inputs. Afterward, all neurons except those belonging to path S 𝑆 S italic_S are frozen, and the model undergoes conventional training, minimizing the loss associated with the specific task.

During the expanding phase, the model checks if the loss for the new task has reached a pre-defined acceptable level. If not, k 𝑘 k italic_k neurons are added to all layers and another round of training takes place using L 1 subscript 𝐿 1 L_{1} italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT regularization to promote sparsity and discard unnecessary neurons.

Finally, in the avoiding semantic drift phase, the model examines if any neuron in the original path S 𝑆 S italic_S has undergone significant changes. If so, it indicates that the neuron is prone to semantic drift with respect to some previous task. Then, the neuron is duplicated: one of them preserves the old value and is removed from path S 𝑆 S italic_S , while the other remains for the current task.

By dynamically adjusting the model’s architecture and selectively expanding and modifying connections, DEN addresses the resource limitations faced by Sub-networks methods. This adaptive approach allows the model to accommodate new tasks while mitigating interference with prior knowledge and ensuring efficient resource utilization.

Another hybrid approach, proposed by ? (?), also addresses the challenge of limited resources by identifying sub-structures for each task and expanding them when necessary. Their method involves an iterative process consisting of three phases. In the first phase, the model learns the task by utilizing unfrozen neurons while applying a mask to the frozen ones. The second phase involves iterative pruning of the non-frozen neurons to maintain the model’s capacity. This pruning process helps optimize the resource allocation and ensures that only the most relevant connections are retained. Lastly, if the model fails to achieve the pre-defined minimum accuracy, it undergoes expansion and the training process recommences for the specific task. This expansion allows the model to acquire additional resources and adapt to the task’s requirements. Once the pre-defined accuracy threshold is reached, the mask utilized during the learning phase is saved, and the corresponding weights are frozen.

By iteratively learning, pruning, and expanding the model’s sub-structures, this method provides a hybrid solution that effectively manages resource limitations. This approach allows the model to dynamically allocate resources based on task-specific requirements while preserving the knowledge gained from previous tasks.

This combination takes the advantage of Sub-networks category of encapsulating the maximum number of tasks in the same network structure while avoiding the resource limitation. On the other hand, it only grows when it is really necessary, handling the problem in the Dynamic Networks category of increasing the size in a linear way. Therefore, it is considered an excellent choice, however, there are several environments and setups that it does not work, for instance, in Unbound Task Incremental setup or edge computing environment. In the general case, the accuracy of techniques in this category is worst than the techniques from the Rehearsal category.

#### 3.5.6 Combining Sub-networks, Dynamic Networks, and Rehearsal

The Else-Net model proposed by ? (?) combines features of Sub-networks, Dynamic Networks, and Rehearsal. In this model, each layer consists of multiple knowledge blocks that process the input. A gating mechanism selects the most important block, generating an embedding for the next layer. When a new task needs to be learned, a temporary block is created. If the gate identifies it as important, it is retained for future tasks; otherwise, it is discarded, and other modules have their parameters adjusted. Additionally, a small portion (10 percent) of samples from each learned class is stored in the coreset to mitigate forgetting.

A notable difference from other methods discussed earlier is that the presence of a coreset allows for weight adaptation in a portion of the model without causing significant forgetting. This flexibility makes the Else-Net model applicable to setups beyond Task Incremental Learning, where the task identifier is known during both training and evaluation phases.

Figure 15 provides an overview of approaches that combine multiple categories to address CF. Each category has its own strengths and limitations. It is evident that no model can learn an infinite number of tasks with a fixed parameter count. Conversely, creating a separate model for each new task is impractical. Storing all previously trained samples is also infeasible due to storage space and retraining time constraints, but preserving key samples appears to enhance the retention capacity. Comparing these approaches is challenging due to the lack of consensus regarding metrics, datasets, and testing methodologies.

## 4 Final Considerations

Catastrophic forgetting, initially identified as a problem in machine learning by ? (?), refers to the phenomenon where a model loses previously acquired knowledge when learning new tasks. ? (?) further investigated this issue and found that it is particularly pronounced in connectionist models like Neural Networks. Nowadays, despite the significant advancements in DL, addressing the challenge of CF remains an ongoing problem in the field, as evident from our comprehensive review. While pretrained networks in Computer Vision, and more recently in Natural Language Processing, exhibit reduced susceptibility to CF, relying solely on pretraining can lead to poor generalization for new tasks that differ significantly from the initial task. To overcome this limitation, it is crucial to ensure the diversity and heterogeneity of the dataset used for the initial task.

In our review, we explored various methods and techniques to mitigate the effects of Catastrophic Forgetting in DL. To facilitate the comprehension and analysis of these approaches, we introduced a taxonomy that classifies them into four main categories. This classification framework enhances our understanding of the different strategies proposed in the literature, enabling future researchers to categorize and compare their effectiveness in addressing CF.

The first category we reviewed is Rehearsal, which was originally introduced by ? (?). Rehearsal involves reprocessing previously encountered data when learning a new class. However, in practice, this approach is often infeasible due to the accumulation of a large volume of data over time. Within this category, we further divided it into two groups: pseudo-rehearsal and mini-rehearsal. Pseudo-rehearsal techniques aim to overcome the storage issue by generating simulated data on demand instead of storing the actual samples. This approach allows for more efficient use of resources while still providing the model with relevant information from previous tasks. On the other hand, mini-rehearsal approaches address the problem by storing a limited amount of samples from each class. These stored samples are carefully selected to ensure that the model does not suffer from catastrophic forgetting, even with this restricted subset of data.

During our review, we observed that there is room for improvement in sample selection algorithms for generating simulated samples on demand. By enhancing the accuracy and informativeness of these simulated samples, we can potentially enhance the effectiveness of pseudo-rehearsal techniques. Additionally, we encourage the utilization of Generative Adversarial Networks (GANs) for sample generation, as they have shown promise in producing more informative and diverse samples. Overall, the Rehearsal category presents interesting avenues for addressing catastrophic forgetting, and further advancements in sample selection algorithms and the use of GANs can contribute to more effective and efficient rehearsal-based techniques.

The second category we reviewed is distance-based approaches. This category focuses on non-parametric classifiers, such as cosine classifiers, which utilize class representations generated by deep models, as demonstrated by ? (?). This approach helps address the issue of bias in the latest classes by leveraging the class representations. However, it necessitates the use of an effective embedding generator that minimizes overlap among representations of different classes while accurately approximating the representations of the same class. Notably, unsupervised learning and ranking losses have emerged as the most effective strategies within this category, as evidenced by recent research.