Overcoming catastrophic forgetting in neural networks — part 1/4

Cleaned-text code points [0, 12480). Bibliography omitted here; source.md includes it.

Overcoming catastrophic forgetting in neural
networks
James Kirkpatricka, Razvan Pascanua, Neil Rabinowitza, Joel Venessa, Guillaume Desjardinsa,
Andrei A. Rusua, Kieran Milana, John Quana, Tiago Ramalhoa, Agnieszka Grabska-Barwinska a,
Demis Hassabisa, Claudia Clopathb, Dharshan Kumarana, and Raia Hadsella
aDeepMind, London, N1C 4AG, United Kingdom
bBioengineering department, Imperial College London, SW7 2AZ, London, United Kingdom
Abstract
The ability to learn tasks in a sequential fashion is crucial to the development of
artiﬁcial intelligence. Neural networks are not, in general, capable of this and it
has been widely thought that catastrophic forgetting is an inevitable feature of
connectionist models. We show that it is possible to overcome this limitation and
train networks that can maintain expertise on tasks which they have not experienced
for a long time. Our approach remembers old tasks by selectively slowing down
learning on the weights important for those tasks. We demonstrate our approach is
scalable and effective by solving a set of classiﬁcation tasks based on the MNIST
hand written digit dataset and by learning several Atari 2600 games sequentially.
1 Introduction
Achieving artiﬁcial general intelligence requires that agents are able to learn and remember many
different tasks Legg and Hutter [2007]. This is particularly difﬁcult in real-world settings: the
sequence of tasks may not be explicitly labelled, tasks may switch unpredictably, and any individual
task may not recur for long time intervals. Critically, therefore, intelligent agents must demonstrate a
capacity for continual learning: that is, the ability to learn consecutive tasks without forgetting how
to perform previously trained tasks.
Continual learning poses particular challenges for artiﬁcial neural networks due to the tendency for
knowledge of previously learnt task(s) (e.g. task A) to be abruptly lost as information relevant to the
current task (e.g. task B) is incorporated. This phenomenon, termed catastrophic forgetting [French,
1999, McCloskey and Cohen, 1989, McClelland et al., 1995, Ratcliff, 1990], occurs speciﬁcally when
the network is trained sequentially on multiple tasks because the weights in the network that are
important for task A are changed to meet the objectives of task B. Whilst recent advances in machine
learning and in particular deep neural networks have resulted in impressive gains in performance
across a variety of domains (e.g. [Krizhevsky et al., 2012, LeCun et al., 2015]), little progress has
been made in achieving continual learning. Current approaches have typically ensured that data from
all tasks are simultaneously available during training. By interleaving data from multiple tasks during
learning, forgetting does not occur because the weights of the network can be jointly optimized for
performance on all tasks. In this regime—often referred to as the multitask learning paradigm—deep
learning techniques have been used to train single agents that can successfully play multiple Atari
games [Rusu et al., 2015, Parisotto et al., 2015]. If tasks are presented sequentially, multitask learning
can only be used if the data are recorded by an episodic memory system and replayed to the network
during training. This approach (often called system-level consolidation [McClelland et al., 1995]),
is impractical for learning large numbers of tasks, as in our setting it would require the amount of
memories being stored and replayed to be proportional to the number of tasks. The lack of algorithms
arXiv:1612.00796v2 [cs.LG] 25 Jan 2017

to support continual learning thus remains a key barrier to the development of artiﬁcial general
intelligence.
In marked contrast to artiﬁcial neural networks, humans and other animals appear to be able to learn
in a continual fashion [Cichon and Gan, 2015]. Recent evidence suggests that the mammalian brain
may avoid catastrophic forgetting by protecting previously-acquired knowledge in neocortical circuits
[Cichon and Gan, 2015, Hayashi-Takagi et al., 2015, Yang et al., 2009, 2014]. When a mouse acquires
a new skill, a proportion of excitatory synapses are strengthened; this manifests as an increase in
the volume of individual dendritic spines of neurons [Yang et al., 2009]. Critically, these enlarged
dendritic spines persist despite the subsequent learning of other tasks, accounting for retention of
performance several months later [Yang et al., 2009]. When these spines are selectively “erased”, the
corresponding skill is forgotten [Hayashi-Takagi et al., 2015, Cichon and Gan, 2015]. This provides
causal evidence that neural mechanisms supporting the protection of these strengthened synapses
are critical to retention of task performance. Together, these experimental ﬁndings—together with
neurobiological models [Fusi et al., 2005, Benna and Fusi, 2016]—suggest that continual learning
in the mammalian neocortex relies on a process of task-speciﬁc synaptic consolidation, whereby
knowledge about how to perform a previously acquired task is durably encoded in a proportion of
synapses that are rendered less plastic and therefore stable over long timescales.
In this work, we demonstrate that task-speciﬁc synaptic consolidation offers a novel solution to the
continual learning problem for artiﬁcial intelligence. We develop an algorithm analogous to synaptic
consolidation for artiﬁcial neural networks, which we refer to as elastic weight consolidation (EWC
for short). This algorithm slows down learning on certain weights based on how important they are to
previously seen tasks. We show how EWC can be used in supervised learning and reinforcement
learning problems to train several tasks sequentially without forgetting older ones, in marked contrast
to previous deep-learning techniques.
2 Elastic weight consolidation
In brains, synaptic consolidation enables continual learning by reducing the plasticity of synapses that
are vital to previously learned tasks. We implement an algorithm that performs a similar operation in
artiﬁcial neural networks by constraining important parameters to stay close to their old values. In
this section we explain why we expect to ﬁnd a solution to a new task in the neighbourhood of an
older one, how we implement the constraint, and ﬁnally how we determine which parameters are
important.
A deep neural network consists of multiple layers of linear projection followed by element-wise
non-linearities. Learning a task consists of adjusting the set of weights and biases θ of the linear
projections, to optimize performance. Many conﬁgurations of θ will result in the same performance
[Nielsen, 1989, Sussmann, 1992]; this is relevant for EWC: over-parameterization makes it likely that
there is a solution for task B,θ∗
B, that is close to the previously found solution for task A,θ∗
A. While
learning task B, EWC therefore protects the performance in task A by constraining the parameters to
stay in a region of low error for task A centered aroundθ∗
A, as shown schematically in Figure 1. This
constraint is implemented as a quadratic penalty, and can therefore be imagined as a spring anchoring
the parameters to the previous solution, hence the name elastic. Importantly, the stiffness of this
spring should not be the same for all parameters; rather, it should be greater for those parameters that
matter most to the performance during task A.
In order to justify this choice of constraint and to deﬁne which weights are most important for a task,
it is useful to consider neural network training from a probabilistic perspective. From this point of
view, optimizing the parameters is tantamount to ﬁnding their most probable values given some data
D. We can compute this conditional probabilityp(θ|D) from the prior probability of the parameters
p(θ) and the probability of the datap(D|θ) by using Bayes’ rule:
logp(θ|D) = logp(D|θ) + logp(θ)− logp(D) (1)
Note that the log probability of the data given the parameters logp(D|θ) is simply the negative of the
loss function for the problem at hand−L(θ). Assume that the data is split into two independent parts,
one deﬁning task A (DA) and the other task B (DB). Then, we can re-arrange equation 1:
logp(θ|D) = logp(DB|θ) + logp(θ|DA)− logp(DB) (2)
Note that the left hand side is still describing the posterior probability of the parameters given the
entire dataset, while the right hand side only depends on the loss function for task B logp(DB|θ).
2

Figure 1: elastic weight consolidation (EWC) ensures task A is remembered whilst training on task B. Training
trajectories are illustrated in a schematic parameter space, with parameter regions leading to good performance
on task A (gray) and on task B (cream). After learning the ﬁrst task, the parameters are atθ∗
A. If we take gradient
steps according to task B alone (blue arrow), we will minimize the loss of task B but destroy what we have learnt
for task A. On the other hand, if we constrain each weight with the same coefﬁcient (green arrow) the restriction
imposed is too severe and we can only remember task A at the expense of not learning task B. EWC, conversely,
ﬁnds a solution for task B without incurring a signiﬁcant loss on task A (red arrow) by explicitly computing how
important weights are for task A.
All the information about task A must therefore have been absorbed into the posterior distribution
p(θ|DA). This posterior probability must contain information about which parameters were important
to taskA and is therefore the key to implementing EWC. The true posterior probability is intractable,
so, following the work on the Laplace approximation by Mackay [MacKay, 1992], we approximate
the posterior as a Gaussian distribution with mean given by the parametersθ∗
A and a diagonal precision
given by the diagonal of the Fisher information matrixF .F has three key properties [Pascanu and
Bengio, 2013]: (a) it is equivalent to the second derivative of the loss near a minimum, (b) it can
be computed from ﬁrst-order derivatives alone and is thus easy to calculate even for large models,
and (c) it is guaranteed to be positive semi-deﬁnite. Note that this approach is similar to expectation
propagation where each subtask is seen as a factor of the posterior [Eskin et al., 2004]. Given this
approximation, the functionL that we minimize in EWC is:
L(θ) =LB(θ) +
∑
i
λ
2Fi(θi−θ∗
A,i)2 (3)
whereLB(θ) is the loss for task B only,λ sets how important the old task is compared to the new one
andi labels each parameter.
When moving to a third task, task C, EWC will try to keep the network parameters close to the
learned parameters of both task A and B. This can be enforced either with two separate penalties, or
as one by noting that the sum of two quadratic penalties is itself a quadratic penalty.
2.1 EWC allows continual learning in a supervised learning context
We start by addressing the problem of whether elastic weight consolidation could allow deep neural
networks to learn a set of complex tasks without catastrophic forgetting. In particular, we trained a
fully connected multilayer neural network on several supervised learning tasks in sequence. Within
each task, we trained the neural network in the traditional way, namely by shufﬂing the data and
processing it in small batches. After a ﬁxed amount of training on each task, however, we allowed no
further training on that task’s dataset.
We constructed the set of tasks from the problem of classifying hand written digits from the MNIST
[LeCun et al., 1998] dataset, according to a scheme previously used in the continual learning literature
[Srivastava et al., 2013, Goodfellow et al., 2014]. For each task, we generated a ﬁxed, random
permutation by which the input pixels of all images would be shufﬂed. Each task was thus of equal
difﬁculty to the original MNIST problem, though a different solution would be required for each.
Detailed description of the settings used can be found in Appendix 4.1.
Training on this sequence of tasks with plain stochastic gradient descent (SGD) incurs catastrophic
forgetting, as demonstrated in Figure 2A. The blue curves show performance on the testing sets of
two different tasks. At the point at which the training regime switches from training on the ﬁrst
task (A) to training on the second (B), the performance for task B falls rapidly, while for task A it
climbs steeply. The forgetting of task A compounds further with more training time, and the addition
3