Sleep prevents catastrophic forgetting in spiking neural networks by forming joint synaptic weight representations — part 2/5

Cleaned-text code points [14753, 30634). Bibliography omitted here; source.md includes it.

(A) Mean performance (red line) and standard deviation (blue lines) over time: 100,000 aeons of unsupervised training (white), 50,000 aeons of Task 1 training (blue), 10,000 aeons of Task 1 (green) and Task 2 (yellow) testing, 50,000 aeons of Task 2 training (red), 10,000 aeons of Task 1 (green) and Task 2 (yellow) testing, 50,000 aeons of Interleaved T1,T2 training (purple), 10,000 aeons of Task 1 (green) and Task 2 (yellow) testing. (B) Mean and standard deviation of performance during testing on Task 1 (blue) and Task 2 (red) after each training period. Following Task 1 training, mean performance on Task 1 was 0.69 ± 0.02 while Task 2 was 0.53 ± 0.02. Conversely, following Task 2 training, mean performance on Task 1 was 0.52 ± 0.02 while Task 2 was 0.69 ± 0.04. Following Interleaved T1,T2 training, mean performance on Task 1 was 0.65 ± 0.03 while Task 2 was 0.67 ± 0.04. (C) Distributions of task-relevant synaptic weights. The distributional structure of Task 1-relevant synapses following Task 1 training (top-left) is destroyed following Task 2 training (top-middle), but partially recovered following Interleaved T1,T2 training (top-right). Similarly, the distributional structure of Task 2-relevant synapses following Task 2 training (bottom-middle), which was not present following Task 1 training (bottom-left), was partially preserved following Interleaved T1,T2 training (bottom-right). (D) Box plots with mean (dashed green line) and median (dashed orange line) of the distance to the decision boundary found by an SVM trained to classify Task 1 and Task 2 synaptic weight matrices for Task 1, Task 2, and Interleaved T1,T2 training across trials. Task 1 and Task 2 synaptic weight matrices had mean classification values of −0.069 and 0.069 respectively, while that of Interleaved T1,T2 training was 0.016. (E) Trajectory of H to O layer synaptic weights through PC space. Synaptic weights which evolved during Interleaved T1,T2 training (green dots) clustered in a location of PC space intermediary between the clusters of synaptic weights which evolved during training on Task 1 (red dots) and Task 2 (blue dots).

Interleaved training was proposed as a solution for catastrophic forgetting 4 , 8 , 9 , so we added an Interleaved Task 1 and Task 2 (Interleaved T1,T2 ) training phase to our simulation ( Figure 3A ) to test whether it was a capable of learning Task 1 (now new task) without overwriting Task 2 (old task). For interleaved training we alternated short presentations of Task 1 and Task 2 every 100 movement cycles. Figure 3B shows that, following Interleaved T1,T2 training, the network achieved a performance of 0.65 ± 0.03 on Task 1 and a performance of 0.67 ± 0.04 on Task 2. Therefore, Interleaved T1,T2 training allowed the network to relearn Task 1 without forgetting what the network had just learned during training on Task 2. Note, we also tested Interleaved T1,T2 training right after the unsupervised phase and found the same high performance for both Task 1 and Task 2 (not shown).

We identified task-relevant synapses after training on a given task (top 10% of synapses), and we traced the same set of synapses after training on the opposing task or after Interleavedψ1,T2 training. The structure in the distribution of Task 1-relevant synapses following Task 1 training ( Figure 3C , top-left) was destroyed following Task 2 training (top-middle; i.e., majority of Task 1-relevant synapses were reduced to zero after Task 2 training) but partially recovered following Interleaved T1,T2 training (top-right). Similarly, the structure in the distribution of Task 2-relevant synapses following Task 2 training (bottom-middle) was not present following Task 1 training (bottom-left) and was partially retained following Interleaved T1,T2 training (bottom-right).

To better understand the effect of Interleaved T1,T2 training on the synaptic weights, we trained a support vector machine (SVM; see Method: Support Vector Machine Training for details) with a radial basis function kernel to classify the synaptic weight configurations between layers H and O (i.e. those responsible for decision making) according to whether they serve to perform Task 1 or Task 2. Figure 3D shows the average distance from the decision boundary across trials for synaptic weights associated with Task 1, Task 2, and Interleaved T1,T2 training. While the SVM robustly classified the synaptic weight matrices from Task 1 and Task 2, the weight states after Interleaved T1,T2 training were significantly closer to the decision boundary (typically on the task 2 side). This indicates that the synaptic weight matrices from Interleaved T1,T2 training are a mixture of Task 1 and Task 2 states.

Figure 3E shows the trajectory of the synaptic weight distribution for the experiment in Figure 3A projected to 3-dimensions using principal components analysis (PCA). It can be seen that while synaptic weight matrices associated with Task 1 and Task 2 training cluster in distinct regions of PC space, Interleaved T1,T2 training pushes the synaptic weights to an intermediate location between Task 1 and Task 2.

### Periods of sleep allow for sequential training without catastrophic forgetting

Sleep is believed to be an off-line processing period when recent memories are replayed to avoid damage by new learning. Particularly for procedural (hippocampal-independent) memories, rapid-eye-movement (REM) sleep may organize neuronal activity to replay memory traces 31 . Can we implement a sleep like phase to our model to protect an old task and still accomplish new task learning without explicit re-training of the old task (e.g., without doing explicit interleaved training of Task 1 and Task 2)?

Again, we first trained the network on Task 1 and Task 2 sequentially to illustrate occurrence of catastrophic forgetting ( Figure 4A ). At this point the network remembered the most recent task (i.e. Task 2) but Task 1 was forgotten. Next, we implemented a training phase consisted of alternating periods of training on Task 1 (considered to be a new task here) lasting 100 movement cycles and periods of “sleep” of the same duration (we will refer to this training phase as Interleaved S,T1 ). To simulate sleep, the rewarded STDP rule was replaced by unsupervised STDP, ensuring a truly offline learning period, and hidden layer neurons were artificially stimulated by Poisson distributed spike trains in order to maintain spiking rates similar to that during task training (indeed, in vivo, activity of the neocortical neurons during REM sleep is similar to awake 32 ; see Methods: Simulated Sleep for details). Importantly, no training on Task 2 (old task here) was performed at any time during Interleaved S,T1 . Figure 4B shows that following Interleaved S,T1 the network achieved a performance of 0.69 ± 0.02 on Task 1 and a performance of 0.67 ± 0.03 on Task 2, comparable to both single task performances following sequential training on Task 1 (0.70 ± 002) and Task 2 (0.69 ± 0.03) ( Figure 1B/C ) and exceeding those achieved through Interleaved T1,T2 training ( Figure 3B ). When durations of Task 1 individual training episodes was increased significantly beyond 100 cycles during Interleaved S,T1 , the network was only able to perform well on the new Task 1 while performance on the old Task 2 dropped to the chance level (not shown).

Download figure

Open in new tab

(A) Task paradigm similar to that shown in (3A) but with 50,000 aeons of Interleaved S,T1 training (gray) instead of Interleaved T1,T2 training. Note that performance for Task 2 remains high despite no Task 2 training during this period. (B) Mean and standard deviation of performance during testing on Task 1 (blue) and Task 2 (red) after each training period. Following Task 1 training, mean performance on Task 1 was 0.70 ± 0.02 while Task 2 was 0.53 ± 0.02. Conversely, following Task 2 training, mean performance on Task 1 was 0.52 ± 0.02 while Task 2 was 0.69 ± 0.04. Following Interleaved S,T1 training, mean performance on Task 1 was 0.69 ± 0.02 while Task 2 was 0.67 ± 0.03. (C) Distributions of task-relevant synaptic weights. The distributional structure of Task 1-relevant synapses following Task 1 training (topleft) is destroyed following Task 2 training (top-middle), but partially recovered following Interleaved S,T1 training (top-right). Similarly, the distributional structure of Task 2-relevant synapses following Task 2 training (bottom-middle), which was not present following Task 1 training (bottom-left), was partially preserved following Interleaved S,T1 training (bottom-right). Task-relevant synapses were considered to be those which had a synaptic weight of at least 0.1 following training on that task. (B) Box plots with mean (dashed green line) and median (dashed orange line) of the distance to the decision boundary found by an SVM trained to classify Task 1 and Task 2 synaptic weight matrices for Task 1, Task 2, and Interleaved S,T1 training across trials. Task 1 and Task 2 synaptic weight matrices had mean classification values of −0.069 and 0.069 respectively, while that of Interleaved S,T1 training was −0.0047. (C) Trajectory of H to O layer synaptic weights through PC space. Synaptic weights which evolved during Interleaved S,T1 training (green dots) clustered in a location of PC space intermediary between the clusters of synaptic weights which evolved during training on Task 1 (red dots) and Task 2 (blue dots).

We interpret these results as follows (see sections below for detailed synaptic connectivity analysis). Each episode of new Task 1 training improves Task 1 performance but damages synaptic connectivity responsible for old Task 2. If continuous Task 1 training is long enough, the damage to Task 2 becomes irreversible. Having a sleep phase after a short period of Task 1 training enables spontaneous forward (H->O) replay that preferentially benefits the strongest synapses. Thus, if Task 2 synapses are still strong enough, they are replayed and increase. To keep the protocol consistent with our previous experiments on Interleaved T1,T2 training, we used a combination of sleep and Task 1 training – the same task that was initially trained to naïve network but overwritten during Task 2 training (i.e., entire sequence of events was T1 -> T2 -> Interleaved S,T1 ). However, we obtained the same results in an experiment when, after initial Task 1 training, Task 2 training was interleaved with sleep (i.e., T1 -> InterleavedS,T2), which prevented forgetting Task 1 while Task 2 was learned (see Extended Data Figure 1 ).

Download figure

Open in new tab

(A). Mean performance (red line) and standard deviation (blue lines) over time: 100,000 aeons of unsupervised training (white), 50,000 aeons of Task 1 training (blue), 10,000 aeons of Task 1 (green) and Task 2 (yellow) testing, 50,000 aeons of Interleaved T1,T2 training (pink), 10,000 aeons of Task 1 (green) and Task 2 (yellow) testing. (B) Mean and standard deviation of performance during testing on Task 1 (blue) and Task 2 (red) after each training period. Following Task 1 training, mean performance on Task 1 was 0.69 ± 0.02 while Task 2 was 0.53 ± 0.02. Following Interleaved T1,T2 training, mean performance on Task 1 was 0.68 ± 0.03 while Task 2 was 0.64 ± 0.04. (C) Task paradigm similar to that shown in (A) but with 50,000 aeons of InterleavedS,T2 training (gray) instead of Interleaved T1,T2 training. (D) Mean and standard deviation of performance during testing on Task 1 (blue) and Task 2 (red) after each training period. Following Task 1 training, mean performance on Task 1 was 0.70 ± 0.02 while Task 2 was 0.53 ± 0.02. Following Interleaved S,T2 training, mean performance on Task 1 was 0.68 ± 0.05 while Task 2 was 0.70 ± 0.03.

We next traced “task-relevant” synapses, i.e. synapses identified in the top 10% distribution following training on that specific task ( Figure 4C ; compare to Figure 3C for Interleaved T1,T2 training). The structure in the distribution of Task 1-relevant synapses following Task 1 training ( Figure 4C ; top-left) was destroyed following Task 2 training (top-middle) but partially recovered following Interleaved S,T1 training (top-right). The structure in the distribution of Task 2-relevant synapses following Task 2 training (bottom-middle) was not present following Task 1 training (bottom-left) and was partially retained following Interleaved S,T1 training (bottom-right). Thus, sleep can preserve important synapses while incorporating new ones.

Figure 4D shows that the SVM robustly classified the synaptic weight states from Task 1 and Task 2 while those from Interleaved S,T1 weight states fell significantly closer to the decision boundary. This indicates that, similar to Interleaved T1,T2 , the synaptic weight matrices which result from Interleaved S,T1 training are a mixture of Task 1 and Task 2 states. The trajectory of the synaptic weights in PC space shown in Figure 4E provides a visualization of these dynamics. Importantly, the smoothness of this trajectory to its steady state suggests that Task 2 information is never completely erased during this evolution. We take this as evidence that Interleaved S,T1 training is capable of integrating synaptic information relevant to Task 1 while preserving Task 2 information.

### Receptive fields of decision-making neurons after sleep represent multiple tasks

To observe that the network has learned both tasks after Interleaved S,T1 training, we mapped the receptive fields of decision-making neurons in layer O ( Figure 5 ; see Figure 2 for comparison). Figure 5A shows the receptive field for the neuron in layer O which controls movement in the upper-left direction. This neuron responds to both horizontal (rewarded for Task 1) and vertical (rewarded for Task 2) orientations in the upper-left quadrant of the visual field. Although it initially appears that this layer O neuron may also be responsive to diagonal patterns in this region, analysis of the receptive fields of neurons in layer H ( Figure 5B ) revealed that these receptive fields are selective to either horizontal food particles (left; rewarded for Task 1) or vertical food particles (right; rewarded for Task 2) in the upper-left quadrant of the visual field. Other receptive fields were responsible for avoidance of punished particles for both tasks (see examples in Figure 5B , bottom-middle-right and bottom-middle-left). Thus, the network will utilize one of two distinct sets of layer H neurons, selective for either Task 1 or Task 2, depending on which food particles are present in the environment.

Download figure

Open in new tab

(A) Left, Receptive field of the output layer neuron controlling movement to the upper-left direction following interleaved sleep and Task 1 training. This neuron has a complex receptive field capable of responding to horizontal and vertical orientations in the upper-left quadrant of the visual field. Right, Schematic of the connectivity between layers. (B) Examples of receptive fields of hidden layer neurons which synapse strongly onto the output neuron from (A) after interleaved Sleep and Task 1 training. The majority of these neurons selectively respond to horizontal food particles (left half) or vertical food particles (right half) in the upper-left quadrant of the visual field, promoting movement in that direction and acquisition of the rewarded patters. A few neurons (bottom-middle-left/right) can be seen to selectively respond to the presence of positive/negative diagonal food particles in the bottom-right quadrant of the visual field. Activation of these neurons will promote avoidance movement to the upper-left direction away from the punished patterns.

### Periods of sleep allow reintegration of new task without interference through renormalization of task-relevant synapses