Overcoming catastrophic forgetting in neural networks — part 4/4

Cleaned-text code points [40986, 49810). Bibliography omitted here; source.md includes it.

We trained the networks with the Double Q-learning algorithm [van Hasselt et al., 2016]. A training
step is carried out on a minibatch of 32 experiences every four steps. The target network is updated
every 3× 104 time steps. We trained with RMSProp, with a momentum of 0., a decay of 0.95, a
learning rate of 2.5× 10−4, and a maximum learning rate of 2.5× 10−3.
Other hyperparameters that we changed from the reference implementation were: 1) using a smaller
replay buffer (5× 105 past experiences), and 2) a scaling factor for the EWC penalty of 400. Another
subtle difference is that we used the full action set in the Atari emulator. In fact, although many
games only support a small subset of the 18 possible actions, in order to have a unified network
structure for all games we used 18 actions in each game.
We randomly chose the 10 games for each experiment from a pool of 19 Atari games for which the
standalone DQN could reach human-level performance in 50× 106 frames. The scores for each of
these games for the baseline algorithm, for EWC and for plain SGD training, as a function of the
number steps played in that game are shown in Figure 4. In order to get an averaged performance,
we chose 10 sets of 10 games, and ran 4 different random seeds for each set.
The most significant departure from the published models is the automatic determination of the task.
We model each task by a generative model of the environment. In this work, for simplicity, we only
model the current observation. The current task is modelled as a categorical contextc which is treated
as the hidden variable in an Hidden Markov Model that explain observations. In such a model the
probabilty of being in a particular contextc at timet evolves according to:
p(c,t + 1) =
∑
c′
p(c′,t )Γ(c,c′)
Γ(c,c′) =δ(c,c′)(1−α) + (1−δ(c,c′))α
whereδ is the Kronecker delta function andα is the probability of switching context. The task context
then conditions a generative model predicting the observation probability p(o|c,t ). Given such
generative models, the probability of being in a task set at timet can be inferred by the observations
seen so far as:
p(c|o1...ot)∝
∑
c′
Γ(c,c′)p(c′,t− 1)p(o|c,t )
The maximal probability context is then taken to be the current task label.
In our implementation, the generative models consist of factored multinomial distributions explaining
the probability of the state of each pixel in the observation space. The model is a parametrized
Dirichlet distribution, which summarizes the data seen so far using Bayesian updates. In order to
encourage each model to specialize, we train the models as follows. We partition time into windows
of a particular widthW . During each window, all the Dirichlet priors are updated with the evidence
seen so far. At the end of the window, the model best corresponding to the current task set is selected.
Since this model was the most useful to explain the current data, it keeps its prior, while all other
priors are reverted to their state at the beginning of the time window. We ensure that one hold-out
uniform (i.e. uninitialized) Dirichlet-multinomial is always available. Whenever the hold-out model
is selected a new generative model is created and a new task context is therefore created. This model
is Bayesian, in the sense that data is used to maintain beliefs over priors on the generative models,
and is non-parametric, in the sense that the model can grow in function of the observed data. It can
be seen as an implementation of the flat forget me not algorithm described in [Kieran et al., 2016].
The parameterα is not learnt. Instead we use the result from [Veness et al., 2012] where it is shown
that a time decaying switch rateα = 1/t guarantees good worst case asymptotic perfmance provided
the number of tasks grows aso
(
n
log n
)
.
Table 2 summarizes all hyper-parameters used for the Atari experiments. Except for the parameters
pertaining the EWC algorithm (Fisher multiplier, num. samples Fisher, EWC start) or pertaining
the task recognition models (model update period, model downscaling and size window), all the
parameters values are the same as from [van Hasselt et al., 2016] and have not been tuned for these
experiments.
11

Hyperparameter value brief description
action repeat 4 Repeat the same action for four frames. Each agent step will
occur every fourth frame.
discount factor 0.99 Discount factor used in the Q-learning algorithm.
no-op max 30 Maximum number of do nothing operations carried out at the
beginning of each training episode to provide a varied training set.
max. reward 1 Rewards are clipped to 1.
scaled input 84x84 Input images are scaled to 84x84 with bilinear interpolation.
optimization algorithm RMSprop Optimization algorithm used.
learning rate 0.00025 The learning rate in RMSprop.
max. learning rate 0.0025 The maximum learning rate that RMSprop will apply.
momentum 0. The momentum used in RMSprop.
decay 0.95 The decay used in RMSProp.
clipδ 1. Each gradient from Q-learning is clipped to ± 1.
max. norm 50. After clipping, if the norm of the gradient is greater
than 50., the gradient is renormalized to 50.
history length 4 The four most recently experienced frames are taken to
form a state for Q-learning
minibatch size 32 The number of elements taken from the replay buffer to form
a mini-batch training example.
replay period 4 A mini-batch is loaded from the replay buffer every 4
steps (16 frames including action repeat).
memory size 50000 The replay memory stores the last fify thousand
transitions experienced.
target update period 7500 The target network in Q-learning is updated to the policy
network every 7500 step.
min. history 50000 The agent will only start learning after fifty thousand
transitions have been stored into memory.
initial exploration 1. The value of the initial exploration rate.
exploration decay start 50000 The exploration rate will start decaying after fifty
thousand frames.
exploration decay end 1050000 The exploration rate will decay over one million frames.
final exploration 0.01 The value of the final exploration rate.
model update period 4 The Dirichlet model is updated every fourth step.
model downscaling 2 The Dirichlet model is downscaled by a factor of 2, that
is an image of size 42x42 is being modelled.
size window 4 The size of the window for the task recognition model learning.
num. samples Fisher 100 Whenever the diagonal of the Fisher is recomputed for a
task, one hundred mini-batches are drawn from the replay buffer.
Fisher multiplier 400 The Fisher is scaled by this number to form the EWC penalty.
start EWC 20E6 The EWC penalty is only applied after 5 million steps
(20 million frames).
Table 2: Hyperparameters for each of the MNIST figures
12

0
20000
40000
60000
80000
100000
120000
140000 crazy_climber
EWC
SGD
single game
0
2000
4000
6000
8000
10000
12000 riverraid
EWC
SGD
single game
0
50
100
150
200
250
300
350
400 jamesbond
EWC
SGD
single game
0
50
100
150
200
250
300 breakout
EWC
SGD
single game
0
2000
4000
6000
8000
10000 gopher
EWC
SGD
single game
0
1000
2000
3000
4000
5000
6000
7000
8000 kangaroo
EWC
SGD
single game
0
5000
10000
15000
20000
25000
30000
35000
40000 kung_fu_master
EWC
SGD
single game
100
80
60
40
20
0
20 fishing_derby
EWC
SGD
single game
0
200
400
600
800
1000
1200
1400 enduro
EWC
SGD
single game
30
20
10
0
10
20
30 pong
EWC
SGD
single game
0
5000
10000
15000
20000
25000
30000
35000
40000
45000 star_gunner
EWC
SGD
single game
0
2000
4000
6000
8000
10000
12000 demon_attack
EWC
SGD
single game
100
50
0
50
100 boxing
EWC
SGD
single game
0
1000
2000
3000
4000
5000
6000
7000 asterix
EWC
SGD
single game
0
10000
20000
30000
40000
50000 road_runner
EWC
SGD
single game
0
5000
10000
15000
20000
25000 defender
EWC
SGD
single game
0
5
10
15
20
25
30
35 freeway
EWC
SGD
single game
0
2000
4000
6000
8000
10000 krull
EWC
SGD
single game
0
200
400
600
800
1000
1200
1400
1600 space_invaders
EWC
SGD
single game
Figure 4: Score in the individual games as a function of steps played in that game. The black baseline curves
show learning on individual games alone.
4.3 Fisher overlap
To assess whether different tasks solved in the same network use similar sets of weights (Figure
3C in the mains text), we measured the degree of overlap between the two tasks’ Fisher matrices.
Precisely, we computed two tasks’ Fishers,F1 andF2, normalized these to each have unit trace, ˆF1
and ˆF2, then computed their Fréchet distance, a metric on the space of positive-semidefinite matrices
[Dowson and Landau, 1982]:
d2( ˆF1, ˆF2) = 1
2tr
(
ˆF1 + ˆF2− 2( ˆF1 ˆF2)1/2
)
= 1
2|| ˆF 1/2
1 − ˆF 1/2
2 ||F
which is bounded between zero and one. We then define the overlap as 1−d2, with a value of zero
indicating that the two tasks depend on non-overlapping sets of weights, and a value of one indicating
thatF1 =αF2 for someα> 0.
13