Sleep prevents catastrophic forgetting in spiking neural networks by forming joint synaptic weight representations — part 4/5

Cleaned-text code points [45876, 61189). Bibliography omitted here; source.md includes it.

Although classical interleaved training showed similar performance results in our model as interleaving training with sleep, we believe the latter to be superior on the following theoretical grounds. Classical interleaved training will necessarily cause the system to oscillate about the optimal location in synaptic weight space which can support both tasks because each training cycle uses a cost function specific to only a single task. While this can be ameliorated with a learning rate decay schedule, the system is never actually optimizing for the desired dualtask state. Sleep, on the other hand, can support not only replays of the old task, but also support replays which are a mixture of both tasks 42 , 45 , 46 . Thus, through unsupervised learning during sleep replay, the system is able to perform approximate optimization for the desired dual-task state.

While our model represents a dramatic simplification of any biological system, we believe that it captures some important processing steps of how animal and human brains interact with the external world. The primary visual system is believed to employ a sequence of processing steps when visual information is increasingly represented by neurons encoding higher level features 28 – 30 . This processing step was reduced to very simple convolution from input to hidden layer in our model. Subsequently, in the brain, associative areas and motor cortex are trained to make decisions based on reward signals released by neuromodulatory centers 8 , 47 – 49 . This was reduced in our model to synaptic projections from the hidden to output (decision making) layer implementing rewarded STDP to learn a task 24 – 26 .

Our results are in line with a large body of literature suggesting that interleaved training is capable of mitigating catastrophic forgetting in ANNs 4 , 8 , 9 and SNNs 10 , 11 . The novel contribution from this study is that the data intensive process of interleaved training can be avoided in SNNs by inserting periods of noise-induced spontaneous reactivation – unsupervised learning – during new task training; similar to how brains undergo offline consolidation periods during sleep resulting in reduced retroactive interference to previously learned tasks 21 , 43 . In fact, our results are in line with previous work done in humans showing that perceptual learning tasks are subject to retroactive interference by competing memories without an intervening period of REM sleep 37 , 38 . Moreover, performance on visual discrimination tasks in particular have been shown to steadily improve over successive nights of sleep 38 , consistent with our findings that interleaving multiple periods of sleep with novel task learning leads to optimal performance on each task.

Our study predicts synaptic level mechanisms of how sleep-based memory reactivation can protect old memory traces during training of a new interfering memory task. It suggests the apparent loss of recall performance for older tasks in ANNs and SNNs after new training does not necessarily imply a complete erasure of the old task, but instead indicates that the old tasks decision states became unreachable by the associated inputs. Sleep can reverse the damage to synaptic connectivity by replaying the old memory traces without explicit usage of the old training data.

## Methods

#### Environment

Foraging behavior took place in a virtual environment consisting of a 50×50 grid with randomly distributed “food” particles. Each particle was two pixels in length and could be classified into one of four types depending on its orientation: vertical, horizontal, positively sloped diagonal, or negatively sloped diagonal. During the initial unsupervised training period, the particles are distributed at random with the constraints that each of the four types are equally represented and no two particles can be directly adjacent. During training and testing periods only the task-relevant particles were present. When a particle was acquired as a result of the virtual agent moving, it was removed from its current location (simulating consumption) and randomly assigned to a new location on the grid, again with the constraint that it not be directly adjacent to another particle. This ensures a continuously changing environment with a constant particle density. The density of particles in the environment was set to 10%. The virtual agent can see a 7×7 grid of squares (the “visual field”) centered on its current location and it could move to any adjacent square, including diagonally, for a total of eight directions.

#### Network structure

The network was composed of 842 spiking map-based neurons (see Methods: Map-based neuron model below) 50 , 51 , arranged into three feed-forward layers to mimic a basic biological circuit: a 7×7 input layer (I), a 28×28 hidden layer (H), and a 3×3 output layer (O) with a nonfunctional center neuron ( Fig 1 ). Input to the network was simulated as a set of suprathreshold inputs to the neurons in layer I, equivalent to the lower levels of the visual system, which represent the position of particles in an egocentric reference frame relative to the virtual agent (positioned in the center of the 7×7 visual field). The most active neuron in layer O, playing the role of biological motor cortex, determined the direction of the subsequent movement. Each neuron in layer H, which can be loosely defined as higher levels of the visual system or associative cortex, received excitatory synapses from 9 randomly selected neurons in layer I. These connections initially had random strengths drawn from a normal distribution. Each neuron in layer H connected to every neuron in layer O with both an excitatory ( W ij ) and an inhibitory ( WI ij ) synapse. This provided an all-to-all connectivity pattern between these two layers and accomplished a balanced feed-forward inhibition 52 found in many biological structures 52 – 57 . Initially, all these connections had uniform strengths and the responses in layer O were due to the random synaptic variability. Random variability was a property of all synaptic interactions between neurons and was implemented as variability in the magnitude of the individual synaptic events.

#### Policy

Simulation time was divided up into epochs of 600 timesteps, each roughly equivalent to 300 ms. At the start of each epoch the virtual agent received input corresponding to locations of nearby particles within the 7×7 “visual field”. Thus 48 of the 49 neurons in layer I received input from a unique location relative to the virtual agent. At the end of the epoch the virtual agent made a single move based on the activity in layer O. If the virtual agent moved to a grid location with a “food” particle present, the particle was removed and assigned to a randomly selected new location.

Each epoch was of sufficient duration for the network to receive inputs, propagate activity forward, produce outputs, and return to a resting state. Neurons in layer I which represent locations in the visual field containing particles received a brief pulse of excitatory stimulation sufficient to trigger a spike; this stimulation was applied at the start of each movement cycle (epoch). At the end of each epoch the virtual agent moved according to the activity which has occurred in layer O.

The activity in layer O controlled the direction of the virtual agent’s movement. Each of the neurons in layer O mapped onto a specific direction (i.e. one of the eight adjacent locations or the current location). The neuron in layer O which spiked the greatest number of times during the first half of the epoch defined the direction of movement for that epoch. If there was a tie, the direction was chosen at random from the set of tied directions. If no neurons in layer O spiked, the virtual agent continued in the direction it had moved during the previous epoch.

There was a 1% chance on every move that the virtual agent would ignore the activity in layer O and instead move in a random direction. Moreover, for every movement cycle that passed without the virtual agent acquiring a particle, this probability was increased by 1%. The random variability promoted exploration vs exploitation dynamics and essentially prevented the virtual agent from getting stuck in movement patterns corresponding to infinite loops. While biological systems could utilize various different mechanisms to achieve the same goal, the method we implemented was efficient and effective for the scope of our study.

#### Neuron models

For all neurons we used spiking model identical to the model used in 12 , 13 that can be described by the following set of difference equations 51 , 58 , 59 : where V n is the membrane potential, I n is a slow dynamical variable describing the effects of slow conductances, and n is a discrete time-step (0.5 ms). Slow temporal evolution of I n was achieved by using small values of the parameter μ << 1. Input variables β n and σ n were used to incorporate external current (e.g. background synaptic input): . Parameter values were set to σ = 0.06, β e = 0.133, σ e = 1, and μ = 0.0005. The nonlinearity f α ( V n , I n ) was defined in the form of the piece-wise continuous function: where α = 3.65.

This model is very computationally efficient, and, despite its intrinsic low dimensionality, produces a rich repertoire of dynamics capable of mimicking the dynamics of Hodgkin-Huxley type neurons both at the single neuron level and in the context of network dynamics 51 , 58 , 60 .

To model the synaptic interactions, we used the following piece-wise difference equation:

Here g syn is the strength of the synaptic coupling, modulated by the target rate W j of receiving neuron j . Indices pre and post stand for the pre- and post-synaptic variables, respectively. The first condition, spike pre , is satisfied when the pre-synaptic spikes are generated. Parameter γ controls the relaxation rate of synaptic current after a presynaptic spike is received (0 ≤ γ < 1). The parameter R is the coefficient of variability in synaptic release. The standard value of R is 0.12. X is a random variable sampled from a uniform distribution with range [-1, 1]. Parameter V rp defines the reversal potential and, therefore, the type of synapse (i.e. excitatory or inhibitory). The term (1- R +2 XR ) introduces a variability in synaptic release such that the effect of any synaptic interaction has an amplitude that is pulled from a uniform distribution with range [1- R , 1+ R ] multiplied by the average value of the synapse.

#### Synaptic plasticity

Synaptic plasticity closely followed the rules introduced in 12 , 13 . A rewarded STDP rule 24 – 27 was operated on synapses between layers H and O while a standard STDP rule operated on synapses between layers I and H. A spike in a post-synaptic neuron that directly followed a spike in a pre-synaptic neuron created a pre before post event while the converse created a post before pre event. Each new post-synaptic (pre-synaptic) spike was compared to all pre-synaptic (post-synaptic) spikes with a time window of 120 iterations.

The value of an STDP event (trace) was calculated using the following equation 22 , 23 : where t r and t p are the times at which the pre- and post-synaptic spike events occurred respectively, T c is the time constant and is set to 40 ms, and K is maximum value of the trace tr k and is set to −0.04 for a post before pre event and 0.04 for a pre before post event.

A trace was immediately applied to synapse between neurons in layers I and H. However, for synapses between neurons in layers H and O the traces were stored for 6 epochs after its creation before being erased. During storage, a trace had an effect whenever there was a rewarding or punishing event. In such a case, the synaptic weights are updated as follows: where t is the current timestep, S rp is a scaling factor for reward/punishment, tr k is the magnitude of the trace, t k is the time of the trace event, c is a constant (=1 epoch) used for decreasing sensitivity to very recent spikes, W i = Σ j W ij is the total synaptic strength of all connections from the neuron i in layer H to all neurons in layer O, W i0 is a constant that is set to the initial value (target value) of W i at the beginning of the simulation. The term W i0 /W i helped to keep the output weight sum close to the initial target value. The effect of these rules was that neurons with lower total output strength could increase their output strength more easily.

The network was rewarded when the virtual agent moved to a location which contained a particle from a “food” pattern (horizontal in Task 1, vertical in Task 2) and S rp = 1, and received a punishment of S rp = −0.001 when it moved to a location with a particle from a neutral pattern (negative/positive diagonal in Task 1/2). A small punishment of S rp = −0.0001 was applied if the agent moved to a location without a particle present to help the virtual agent learn to acquire “food” as rapidly as possible. During periods of sleep the network received a constant reward of S rp = 0.5 on each movement cycle.

To ensure that neurons in layer O maintained a relatively constant long-term firing rate, the model incorporated homeostatic synaptic scaling which was applied every epoch. Each timestep, the total strength of synaptic inputs W j = Σ i W ij to a given neuron in layer O was set equal to the target synaptic input W j0 – a slow variable which varied over many epochs depending on the activity of the given neuron in layer O – which was updated according to:

To ensure that the net synaptic input W j to any neuron was unaffected by plasticity events at the individual synapses at distinct timesteps and equal to W j0 , we implemented a scaling process akin to heterosynaptic plasticity which occurs after each STDP event. When any excitatory synapse of neuron in layer O changed in strength, all other excitatory synapses received by that neuron were updated according to:

#### Simulated Sleep

To simulate the sleep phase, we inactive the sensory receptors (i.e. the input layer of network), cut off all sensory signals (i.e. remove all particles from the environment), and decouple output layer activity from motor control (i.e. the output layer can spike but no longer causes the agent to move). We also change the learning rule between the hidden and output layer from rewarded to unsupervised STDP (see Methods: Synaptic Plasticity for details) as there is no way to evaluate decision-making without sensory input or motor output.

To simulate the spontaneous activity observed during REM sleep, we provided noise to each neuron in the hidden layer in a way which ensured that the spiking statistics of each neuron was conserved across awake and sleep phases. To determine these spiking rates, we recorded average spiking rates of neurons in the hidden layer H during preceding training of both Task 1 and Task 2; these task specific spiking rates were then averaged to generate target spiking rates for hidden layer neurons. Interleaved S,T1 training consisted of alternating intervals of this sleep phase and training on Task 1, with each interval lasting 100 movement cycles (although no movement occurred).

#### Support Vector Machine Training