Overcoming catastrophic forgetting in neural networks — part 3/4

Cleaned-text code points [25992, 40986). Bibliography omitted here; source.md includes it.

There has been previous work [French and Chater, 2002, Eaton and Ruvolo, 2013] using a quadratic
penalty to approximate old parts of the dataset, but these applications have been limited to small
models. Speciﬁcally, [French and Chater, 2002] used random inputs to compute a quadratic approxi-
mation to the energy surface. Their approach is slow, as it requires re-computing the curvature at
each sample. The ELLA algorithm described in [Eaton and Ruvolo, 2013] requires computing and
inverting matrices with a dimensionality equal to the number of parameters being optimized, therefore
it has been mainly applied to linear and logistic regressions. In contrast, EWC has a run time which is
linear in both the number of parameters and the number of training examples. We could only achieve
this low computational complexity by making several simpliﬁcations, most notably by approximating
the posterior distribution of the parameters on a task (i.e. the weight uncertainties) by a factorized
Gaussian, and by computing its variance using a point-estimate of the parameters, via the diagonal of
the Fisher Information matrix. Despite its low computational cost and empirical successes—even in
the setting of challenging RL domains—our use of a point estimate of the posterior’s variance (as in a
Laplace approximation) does constitute a signiﬁcant weakness (see Fig 4C). Our initial explorations
suggest that one might improve on this local estimate by using Bayesian neural networks [Blundell
et al., 2015].
While this paper has primarily focused on building an algorithm out of neurobiological observations, it
is also instructive to consider whether the algorithm’s successes can feed back into our understanding
of the brain. In particular, we see considerable parallels between EWC and two computational
theories of synaptic plasticity.
In this respect, the perspective we offer here aligns with a recent proposal that each synapse not
only stores its current weight, but also an implicit representation of its uncertainty about that weight
[Aitchison and Latham, 2015]. This idea is grounded in observations that post-synaptic potentials are
highly variable in amplitude (suggestive of sampling from the weight posterior during computation),
and that those synapses which are more variable are more amenable to potentiation or depression
(suggestive of updating the weight posterior). While we do not explore the computational beneﬁts
of sampling from a posterior here, our work aligns with the notion that weight uncertainty should
inform learning rates. We take this one step further, to emphasize that consolidating the high precision
weights enables continual learning over long time scales. With EWC, three values have to be stored
for each synapse: the weight itself, its variance and its mean. Interestingly, synapses in the brain also
carry more than one piece of information. For example, the state of the short-term plasticity could
carry information on the variance [Aitchison and Latham, 2015, Pﬁster et al., 2010]. The weight for
the early phase of plasticity [Clopath et al., 2008] could encode the current synaptic strength, whereas
the weight associated with the late-phase of plasticity or the consolidated phase could encode the
mean weight.
The ability to learn tasks in succession without forgetting is a core component of biological and
artiﬁcial intelligence. In this work we show that an algorithm that supports continual learning—which
takes inspiration from neurobiological models of synaptic consolidation—can be combined with deep
neural networks to achieve successful performance in a range of challenging domains. In doing so,
we demonstrate that current neurobiological theories concerning synaptic consolidation do indeed
scale to large-scale learning systems. This provides prima facie evidence that these principles may be
fundamental aspects of learning and memory in the brain.
Acknowledgements. We would like to thank P. Dayan, D. Wierstra, S. Mohamed, Yee Whye Teh
and K. Kavukcuoglu.
References
Laurence Aitchison and Peter E Latham. Synaptic sampling: A connection between psp variability
and uncertainty explains neurophysiological observations. arXiv preprint arXiv:1505.04544, 2015.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ-
ment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 47:
253–279, 2013.
Marcus K Benna and Stefano Fusi. Computational principles of synaptic memory consolidation.
Nature neuroscience, 2016.
7

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty
in neural network. In Proceedings of The 32nd International Conference on Machine Learning,
pages 1613–1622, 2015.
Joseph Cichon and Wen-Biao Gan. Branch-speciﬁc dendritic ca2+ spikes cause persistent synaptic
plasticity. Nature, 520(7546):180–185, 2015.
Claudia Clopath, Lorric Ziegler, Eleni Vasilaki, Lars Büsing, and Wulfram Gerstner. Tag-trigger-
consolidation: a model of early and late long-term-potentiation and depression. PLoS Comput Biol,
4(12):e1000248, 2008.
Anne GE Collins and Michael J Frank. Cognitive control over learning: creating, clustering, and
generalizing task-set structure. Psychological review, 120(1):190, 2013.
DC Dowson and BV Landau. The fréchet distance between multivariate normal distributions.Journal
of multivariate analysis, 12(3):450–455, 1982.
Kenji Doya, Kazuyuki Samejima, Ken-ichi Katagiri, and Mitsuo Kawato. Multiple model-based
reinforcement learning. Neural computation, 14(6):1347–1369, 2002.
Eric Eaton and Paul L Ruvolo. Ella: An efﬁcient lifelong learning algorithm. In International
Conference on Machine Learning, pages 507–515, 2013.
Eleazar Eskin, Alex J. Smola, and S.v.n. Vishwanathan. Laplace propagation. In Advances in Neural
Information Processing Systems 16, pages 441–448. MIT Press, 2004. URL http://papers.
nips.cc/paper/2444-laplace-propagation.pdf .
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3
(4):128–135, 1999.
Robert M French and Nick Chater. Using noise to compute error surfaces in connectionist networks:
a novel means of reducing catastrophic forgetting. Neural computation, 14(7):1755–1769, 2002.
Stefano Fusi, Patrick J Drew, and LF Abbott. Cascade models of synaptically stored memories.
Neuron, 45(4):599–611, 2005.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical
investigation of catastrophic forgeting in gradient-based neural networks. Int’l Conf. on Learning
Representations (ICLR), 2014.
Akiko Hayashi-Takagi, Sho Yagishita, Mayumi Nakamura, Fukutoshi Shirai, Yi I. Wu, Amanda L.
Loshbaugh, Brian Kuhlman, Klaus M. Hahn, and Haruo Kasai. Labelling and optical erasure
of synaptic memory traces in the motor cortex. Nature, 525(7569):333–338, 09 2015. URL
http://dx.doi.org/10.1038/nature15257.
Milan Kieran, Joel Veness, Michael Bowling, James Kirkpatrick, Anna Koop, and Demis Hassabis.
The forget me not process. In Advances in Neural Information Processing Systems 26 , page
accepted for publication, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classiﬁcation with deep convolu-
tional neural networks. In NIPS, pages 1097–1105, 2012.
Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database of handwritten digits,
1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444,
2015.
Shane Legg and Marcus Hutter. Universal intelligence: A deﬁnition of machine intelligence. Minds
and Machines, 17(4):391–444, 2007.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa-
tion, 4(3):448–472, 1992.
8

Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent
computation by recurrent dynamics in prefrontal cortex. Nature, 503(7474):78–84, 2013.
James L McClelland, Bruce L McNaughton, and Randall C O’Reilly. Why there are complementary
learning systems in the hippocampus and neocortex: insights from the successes and failures of
connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The
sequential learning problem. The psychology of learning and motivation, 24(109-165):92, 1989.
Earl K Miller and Jonathan D Cohen. An integrative theory of prefrontal cortex function. Annual
review of neuroscience, 24(1):167–202, 2001.
V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare,
Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control
through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
Robert H. Nielsen. Theory of the backpropagation neural network. InProceedings of the International
Joint Conference on Neural Networks, volume I, pages 593–605. Piscataway, NJ: IEEE, 1989.
Randall C O’Reilly and Michael J Frank. Making working memory work: a computational model of
learning in the prefrontal cortex and basal ganglia. Neural computation, 18(2):283–328, 2006.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and
transfer reinforcement learning. arXiv preprint arXiv:1511.06342, 2015.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint
arXiv:1301.3584, 2013.
Jean-Pascal Pﬁster, Peter Dayan, and Máté Lengyel. Synapses with short-term plasticity are optimal
estimators of presynaptic membrane potentials. Nature neuroscience, 13(10):1271–1275, 2010.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and
forgetting functions. Psychological review, 97(2):285, 1990.
Mark B Ring. Child: A ﬁrst step towards continual learning. In Learning to learn, pages 261–292.
Springer, 1998.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk-
patrick, Razvan Pascanu, V olodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy
distillation. arXiv preprint arXiv:1511.06295, 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray
Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint
arXiv:1606.04671, 2016.
Rupesh K Srivastava, Jonathan Masci, Sohrob Kazerounian, Faustino Gomez, and Juergen Schmid-
huber. Compete to compute. In Advances in Neural Information Processing Systems 26 ,
pages 2310–2318. Curran Associates, Inc., 2013. URL http://papers.nips.cc/paper/
5059-compete-to-compute.pdf .
Héctor J. Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input-
output map. Neural Networks, 5:589–593, 1992.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning.
Proceedings of the Thirtieth AAAI Conference on Artiﬁcial Intelligence, pages 2094–2100, 2016.
Joel Veness, Kee Siong Ng, Marcus Hutter, and Michael Bowling. Context tree switching. In 2012
Data compression conference., pages 327–336. IEEE, 2012.
Guang Yang, Feng Pan, and Wen-Biao Gan. Stably maintained dendritic spines are associated with
lifelong memories. Nature, 462(7275):920–924, 2009.
Guang Yang, Cora Sau Wan Lai, Joseph Cichon, Lei Ma, Wei Li, and Wen-Biao Gan. Sleep promotes
branch-speciﬁc formation of dendritic spines after learning. Science, 344(6188):1173–1178, 2014.
9

Hyperparameter Reference ﬁgure
3A 3B 3C
learning rate 10−3 10−5-10−3 10−3
dropout no yes no
early stopping no yes no
n. hidden layers 2 2 6
width hidden layers 400 400-2000 100
epochs / dataset 20 100 100
Table 1: Hyperparameters for each of the MNIST ﬁgures
4 Appendix
4.1 MNIST experiments
We carried out all MNIST experiments with fully-connected networks with rectiﬁed linear units.
In order to replicate the results of [Goodfellow et al., 2014], we compared to results obtained
using dropout regularization. As suggested in [Goodfellow et al., 2014], we applied dropout with a
probability of 0.2 to the input and of 0.5 to the other hidden layers. In order to give SGD with dropout
the best possible chance, we also used early stopping. Early stopping was implemented by computing
the test error on the validation set for all pixel permutations seen to date. Here, if the validation error
was observed to increase for more than ﬁve subsequent steps, we terminated this training segment and
proceeded to the next dataset; at this point, we reset the network weights to the values that had the
lowest average validation error on all previous datasets. Table 1 shows a list of all hyperparameters
used to produce the three graphs in Figure 3 of the main text. Where a range is present, the parameter
was randomly varied and the reported results were obtained using the best hyperparameter setting.
When random hyperparameter search was used, 50 combinations of parameters were attempted for
each number experiment.
4.2 Atari experiments
The agent architecture used is almost identical to that used in [van Hasselt et al., 2016]. In this section
we provide details on all the parameters used.
Images are preprocessed in the same way as in [Mnih et al., 2015], namely the 210x160 images from
the Atari emulator are downsampled to 84x84 using bilinear interpolation. We then convert the RGB
images to YUV and use the grayscale channel alone. The state used by the agent consists of the four
latest downsampled, grayscale observations concatenated together.
The network structure used is similar to the one from [Mnih et al., 2015], namely three convolutional
layers followed by a fully connected layer. The ﬁrst convolution had kernel size 8, stride 4 and 32
ﬁlters. The second convolution had kernel size 4, stride 2 and 64 ﬁlters. The ﬁnal convolution had
kernels size 3, stride 1 and 128 ﬁlters. The fully connected layer had 1024 units. Note that this
network has approximately four times as many parameters as the standard network, due to having
twice as many fully connected units and twice as many ﬁlters in the ﬁnal convolution. The other
departure from the standard network is that each layer was allowed to have task-speciﬁc gains and
biases. For each layer, the transformationx→y computed by the network is therefore:
yi =

∑
j
Wijxj +bc
i

gc
i (4)
where the biasesb and the gainsg. The network weights and biases where initialized by setting them
randomly with a uniform number between−σ andσ, withσ set to the square root of the incoming
hidden units (for a linear layer) or set to the area of the kernel times the number of incoming ﬁlters
(for convolutional layers). Biases and gains were initialized to 0 and 1 respectively.
We used an an ϵ-greedy exploration policy, where the probability of selecting random action, ϵ,
decayed with training time. We kept a different timer for each of the tasks. We setϵ = 1 for 5× 104
time steps, and then decayed this linearly to a value of 0.01 for the next 106.
10