Overcoming catastrophic forgetting in neural networks — part 2/4

Cleaned-text code points [12480, 25992). Bibliography omitted here; source.md includes it.

Figure 2: Results on the permuted MNIST task. A: Training curves for three random permutations A, B and C
using EWC(red),L2 regularization (green) and plain SGD(blue). Note that only EWC is capable of mantaining
a high performance on old tasks, while retaining the ability to learn new tasks. B: Average performance across
all tasks using EWC (red) or SGD with dropout regularization (blue). The dashed line shows the performance
on a single task only. C: Similarity between the Fisher information matrices as a function of network depth for
two different amounts of permutation. Either a small square of 8x8 pixels in the middle of the image is permuted
(grey) or a large square of 26x26 pixels is permuted (black). Note how the more different the tasks are, the
smaller the overlap in Fisher information matrices in early layers.
of subsequent tasks. This problem cannot be countered by regularizing the network with a ﬁxed
quadratic constraint for each weight (green curves, L2 regularization): here, the performance in task
A degrades much less severely, but task B cannot be learned properly as the constraint protects all
weights equally, leaving little spare capacity for learning on B. However, when we use EWC, and thus
take into account how important each weight is to task A, the network can learn task B well without
forgetting task A (red curves). This is exactly the expected behaviour described diagrammatically in
Figure 1.
Previous attempts to solve the continual learning problem for deep neural networks have relied upon
careful choice of network hyperparameters, together with other standard regularization methods, in
order to mitigate catastrophic forgetting. However, on this task, they have only achieved reasonable
results on up to two random permutations [Srivastava et al., 2013, Goodfellow et al., 2014]. Using a
similar cross-validated hyperparameter search as [Goodfellow et al., 2014], we compared traditional
dropout regularization to EWC. We ﬁnd that stochastic gradient descent with dropout regularization
alone is limited, and that it does not scale to more tasks (Figure 2B). In contrast, EWC allows a large
number of tasks to be learned in sequence, with only modest growth in the error rates.
Given that EWC allows the network to effectively squeeze in more functionality into a network with
ﬁxed capacity, we might ask whether it allocates completely separate parts of the network for each
task, or whether capacity is used in a more efﬁcient fashion by sharing representation. To assess this,
we determined whether each task depends on the same sets of weights, by measuring the overlap
between pairs of tasks’ respective Fisher information matrices (see Appendix 4.3). A small overlap
means that the two tasks depend on different sets of weights (i.e. EWC subdivides the network’s
weights for different tasks); a large overlap indicates that weights are being used for both the two tasks
(i.e. EWC enables sharing of representations). Figure 2C shows the overlap as a function of depth.
As a simple control, when a network is trained on two tasks which are very similar to each other
(two versions of MNIST where only a few pixels are permutated), the tasks depend on similar sets of
weights throughout the whole network (grey curve). When then the two tasks are more dissimilar
from each other, the network begins to allocate separate capacity (i.e. weights) for the two tasks
(black line). Nevertheless, even for the large permutations, the layers of the network closer to the
output are indeed being reused for both tasks. This reﬂects the fact that the permutations make the
input domain very different, but the output domain (i.e. the class labels) is shared.
2.2 EWC allows continual learning in a reinforcement learning context
We next tested whether elastic weight consolidation could support continual learning in the far
more demanding reinforcement learning (RL) domain. In RL, agents dynamically interact with
the environment in order to develop a policy that maximizes cumulative future reward. We asked
whether Deep Q Networks (DQNs)—an architecture that has achieved impressive successes in such
challenging RL settings [Mnih et al., 2015]—could be harnessed with EWC to successfully support
continual learning in the classic Atari 2600 task set [Bellemare et al., 2013]. Speciﬁcally, each
4

experiment consisted of ten games chosen randomly from those that are played at human level or
above by DQN. At training time, the agent was exposed to experiences from each game for extended
periods of time. The order of presentation of the games was randomized and allowed for returning to
the same games several times. At regular intervals we would also test the agent’s score on each of the
ten games, without allowing the agent to train on them (Figure 3A).
Notably, previous reinforcement learning approaches to continual learning have either relied on either
adding capacity to the network [Ring, 1998, Rusu et al., 2016] or on learning each task in separate
networks, which are then used to train a single network that can play all games[Rusu et al., 2015,
Parisotto et al., 2015]. In contrast, the EWC approach presented here makes use of a single network
with ﬁxed resources (i.e. network capacity) and has minimal computational overhead.
In addition to using EWC to protect previously-acquired knowledge, we used the RL domain to
address a broader set of requirements that are needed for successful continual learning systems: in
particular, higher-level mechanisms are needed to infer which task is currently being performed,
detect and incorporate novel tasks as they are encountered, and allow for rapid and ﬂexible switching
between tasks [Collins and Frank, 2013]. In the primate brain, the prefrontal cortex is widely viewed
as supporting these capabilities by sustaining neural representations of task context that exert top-
down gating inﬂuences on sensory processing, working memory, and action selection in lower-level
regions [O’Reilly and Frank, 2006, Mante et al., 2013, Miller and Cohen, 2001, Doya et al., 2002].
Inspired by this evidence, we used an agent very similar to that described in [van Hasselt et al.,
2016] with few differences: (a) a network with more parameters, (b) a smaller transition table, (c)
task-speciﬁc bias and gains at each layer, (d) the full action set in Atari, (e) a task-recognition model,
and (e) the EWC penalty. Full details of hyper-parameters are described in Appendix app:atari. Here
we brieﬂy describe the two most important modiﬁcations to the agent: the task-recognition module,
and the implementation of the EWC penalty.
We treat the task context as the latent variable of a Hidden Markov Model. Each task is therefore
associated to an underlying generative model of the observations. The main distinguishing feature of
our approach is that we allow for the addition of new generative models if they explain recent data
better than the existing pool of models by using a training procedure inspired by the forget me not
process[Kieran et al., 2016] (see Appendix 4.2).
In order to apply EWC, we compute the Fisher information matrix at each task switch. For each
task, a penalty is added with anchor point given by the current value of the parameters and with
weights given by the Fisher information matrix times a scaling factor λ which was optimized by
hyperparameter search. We only added an EWC penalty to games which had experienced at least 20
million frames.
We also allowed the DQN agents to maintain separate short-term memory buffers for each inferred
task: these allow action values for each task to be learned off-policy using an experience replay
mechanism [Mnih et al., 2015]. As such, the overall system has memory on two time-scales: over
short time-scales, the experience replay mechanism allows learning in DQN to be based on the
interleaved and uncorrelated experiences [Mnih et al., 2015]. At longer time scales, know-how across
tasks is consolidated by using EWC. Finally, we allowed a small number of network parameters to be
game-speciﬁc, rather than shared across games. In particular, we allowed each layer of the network
to have biases and per element multiplicative gains that were speciﬁc to each game.
We compare the performance of agents which use EWC (red) with ones that do not (blue) over sets
of ten games in Figure 3. We measure the performance as the total human-normalized score across
all ten games. We average across random seeds and over the choice of which ten games were played
(see Appendix 4.2). We also clip the human-normalized score for each game to 1. Our measure of
performance is therefore a number with a maximum of 10 (at least at human level on all games)
where 0 means the agent is as good as a random agent. If we rely on plain gradient descent methods
as in [Mnih et al., 2015], the agent never learns to play more than one game and the harm inﬂicted
by forgetting the old games means that the total human-normalized score remains below one. By
using EWC, however, the agents do indeed learn to play multiple games. As a control, we also
considered the beneﬁt to the agent if we explicitly provided the agent with the true task label (Figure
3B, brown), rather than relying on the learned task recognition through the FMN algorithm (red). The
improvement here was only modest.
5

Figure 3: Results on Atari task. A: Schedule of games. Black bars indicate the sequential training periods
(segments) for each game. After each training segment, performance on all games is measured. The EWC
constraint is only activated to protect an agent’s performance on each game once the agent has experienced 20
million frames in that game. B: Total scores for each method across all games. Red curve denotes the network
which infers the task labels using the Forget Me Not algorithm; brown curve is the network provided with the
task labels. The EWC and SGD curves start diverging when games start being played again that have been
protected by EWC. C: Sensitivity of a single-game DQN, trained on Breakout, to noise added to its weights. The
performance on Breakout is shown as a function of the magnitude (standard deviation) of the weight perturbation.
The weight perturbation is drawn from a zero mean Gaussian with covariance that is either uniform (black; i.e.
targets all weights equally), the inverse Fisher ((F +λI)−1; blue; i.e. mimicking weight changes allowed by
EWC), or uniform within the nullspace of the Fisher (orange; i.e. targets weights that the Fisher estimates that
the network output is entirely invariant to). To evaluate the score, we ran the agent for ten full game episodes,
drawing a new random weight perturbation for every timestep.
While augmenting the DQN agent with EWC allows it to learn many games in sequence without
suffering from catastrophic forgetting, it does not reach the score that would have been obtained by
training ten separate DQNs (see Figure 1 in Appendix 4.2). One possible reason for this is that we
consolidated weights for each game based on a tractable approximation of parameter uncertainty, the
Fisher Information. We therefore sought to test the quality of our estimates empirically. To do so, we
trained an agent on a single game, and measured how perturbing the network parameters affected the
agent’s score. Regardless of which game the agent was trained on, we observed the same patterns,
shown in Figure 3C. First, the agent was always more robust to parameter perturbations shaped by the
inverse of the diagonal of the Fisher Information (blue), as opposed to uniform perturbations (black).
This validates that the diagonal of the Fisher is a good estimate of how important a certain parameter
is. Within our approximation, perturbing in the nullspace should have no effect on performance at
all on performance. Empirically, however, we observe that perturbing in this space (orange) has the
same effect as perturbing in the inverse Fisher space. This suggests that we are over-conﬁdent about
certain parameters being unimportant: it is therefore likely that the chief limitation of the current
implementation is that it under-estimates parameter uncertainty.
3 Discussion
We present a novel algorithm, elastic weight consolidation, that addresses the signiﬁcant problem
continual learning poses for neural networks. EWC allows knowledge of previous tasks to be
protected during new learning, thereby avoiding catastrophic forgetting of old abilities. It does so by
selectively decreasing the plasticity of weights, and thus has parallels with neurobiological models of
synaptic consolidation. We implement EWC as a soft, quadratic constraint whereby each weight is
pulled back towards its old values by an amount proportional to its importance for performance on
previously-learnt tasks. To the extent that tasks share structure, networks trained with EWC reuse
shared components of the network. We further show that EWC can be effectively combined with
deep neural networks to support continual learning in challenging reinforcement learning scenarios,
such as Atari 2600 games.
The EWC algorithm can be grounded in Bayesian approaches to learning. Formally, when there
is a new task to be learnt, the network parameters are tempered by a prior which is the posterior
distribution on the parameters given data from previous task(s). This enables fast learning rates on
parameters that are poorly constrained by the previous tasks, and slow learning rates for those which
are crucial.
6