Sleep prevents catastrophic forgetting in spiking neural networks by forming joint synaptic weight representations — part 3/5

Cleaned-text code points [30634, 45876). Bibliography omitted here; source.md includes it.

To visualize synaptic weight dynamics during Interleaved S,T1 training, traces of all synapses projecting to a single representative output layer neuron were plotted ( figure 6A ). At the onset of Interleaved S,T1 training (i.e. 240,000 aeons), the network was only able to perform on Task 2, meaning the strong synapses in the network were specific to this task. These synapses were represented by a cluster ranging from ~0.08 to ~0.4; the rest of synapses grouped near 0. As Interleaved S,T1 training progressed, Task 1 specific synapses moved to the strong cluster and some, presumably less important, Task 2 synapses moved to the weak cluster. After a period of time the rate of transfer decreased and the total number of synapses in each group stabilized, showing that the network is approaching equilibrium ( Figure 6B ).

Download figure

Open in new tab

(A) Dynamics of all incoming synapses to a single output layer neuron during Interleaved S,T1 training shows the synapses separate into two clusters. (B) Number of synapses in the strong (red) and weak (blue) clusters during Interleaved S,T1 . (C) Two-dimensional histograms illustrating synaptic weights dynamics. For each plot, the x-axis represents synaptic weight after Task 1 training and the y-axis represents the synaptic weight at a different point in time(Scale bar: brown - 50 synapses/bin, blue - 1 synapse/bin. One-dimensional projections along top and right sides show the global distribution of synapses at the time slices for a given plot. If no training occurred between the time slices, a diagonal plot depicts that synaptic weights have not changed (left). After a small amount of Task 2 training, all points lie near the diagonal (middle) indicating minimal changes to synaptic weights. Once Task 2 is fully trained (right), many synapses move far away from their original values. In particular, a red cluster along the x-axis indicates synapses which were strong after Task 1 training but were erased after Task 2 training. (D) Same as (C) except the x-axis refers to the end of Task 2 training. Again, a diagonal plot is attained when no training takes place between the time slices (left), and points lie near the diagonal when only a small amount of Interleaved S,T1 training occurs (middle). After a full period of Interleaved S,T1 training (right), weak synapses were recruited to support Task 1 (red cluster along the y-axis) and many Task 2 specific synapses remained moderately strong (blue cluster along x-axis).

To visualize how sleep renormalizes task relevant synapses, we plotted two-dimensional weight distributions for Task 1->Task2 ( Figure 6C ) and Task 2 -> Interleaved S,T1 ( Figure 6D ) experiments (see Methods: 2-D Synaptic Weight Distributions for details). To establish a baseline, in Figure 6C (left) the weight state at the end of Task 1 training (X-axis) (see overall timeline of this experiment in Figure 4A ) was compared to itself (Y-axis). This formed a perfectly diagonal plot. Most synapses were weak (red dots) with stronger synapses forming a tail in the distribution. The next comparison ( Figure 6C , middle) was between the weight state after Task 1 training (X-axis) and a time early on Task 2 training (Y-axis). At that time, synapses were only able to modify their strength slightly, causing most points to lie close to the diagonal. As training on Task 2 continued until maximum performance was reached, synapses moved far away from the diagonal ( Figure 6C , right). Two trends were observed. A set of synapses that had a strength near zero following Task 1 training increased strength following Task 2 training ( Figure 6D , right, red dots along Y-axis). At the same time, many strongly trained by Task 1 synapses were depressed down to zero ( Figure 6C , right, red dots along X-axis). The latter illustrates the effect of catastrophic forgetting - complete overwriting of the synaptic weight matrix caused performance of Task 1 to return to baseline after training on Task 2.

Does sleep prevent overwriting of the synaptic weight matrix? The Figure 6D plots use the weight state at the end of training Task 2 as a reference that is compared to different times during Interleaved S,T1 training. The first two plots ( Figure 6D , left/middle) are similar to those in Figure 6C . However, after Interleaved S,T1 training ( Figure 6D , right) many synapses that were strong following Task 2 training were not depressed to zero but rather were pushed to an intermediate strength where they are still functional (note cluster of points parallel to X-axis; see also projection to 1D on the right side of the graph). Thus, Interleaved S,T1 training, combining new training on Task 1 with periods of unsupervised sleep, moved synapses in a way that preserved strong synapses from a previously learned task while also introducing new strong synapses to perform a new task. Since a significant fraction of the strong synapses from training on Task 2 were preserved (due to the sleep periods), performance on Task 2 remained high following Interleaved S,T1 training despite the fact that the networks received no new training examples of Task 2.

### Periods of sleep push the network towards the intersection of the solution manifolds representing Task 1 and Task 2 specific weight configurations

To visualize the approximate task-specific solution manifolds ( M T1 and M T2 ) and their intersection ( M T1∩T2 ) in synaptic weight space, we used multiple trials (with different initialization) of Task 1 and Task 2 training to sample the manifolds. Figure 7A shows (in kPCA space) that multiple different configurations of synaptic weights can provide high performance for a given task. For example, all red dots in Figure 7A represent the states with the same high level of performance for Task 1 (but not Task 2). In addition, cyan and green dots represent states with high level of performance for both Task 1 and Task2. We interpret these results as evidence that synaptic weight space includes a manifold, M T1 , where different configurations of weights (red, green, cyan dots) all allow for Task 1 to perform well. This manifold intersects with another one, M T2 , where different weights configurations (blue, green, cyan dots) are all suitable for Task 2. Figures 7B and 7C show 2D projections of this space onto PCs 1 and 2 and PCs 1 and 3, respectively. From these projections, we can see that PC 1 seems to capture the extent to which a synaptic weight configuration is associated with Task 1 (positive values) or Task 2 (negative values), while PC 2 and PC 3 capture the variance in synaptic weight configurations associated with Task 1 and Task 2, respectively. Note, the trajectories through this space (red/blue lines) during Interleaved T1,T2 and Interleaved S,T1/T2 training would also belong to the respective task manifolds as performance on the old tasks was never lost in these training scenarios.

Download figure

Open in new tab

(A-C) Low-dimensional visualizations of the synaptic weight configurations of 10 networks obtained through kPCA for 3-dimensions (A), 2-dimensions using PC 1 and PC 3 (B), and 2-dimensions using PC 1 and PC 3 (C). Synaptic weight configurations taken from the last fifth of Task 1 (red dots), Task 2 (blue dots), Interleaved T1,T2 (green dots), and Interleaved S,T1/T2 (cyan dots) training are shown. PC 1 characterizes good performance on Task 1 (positively valued) or Task 2 (negatively valued) training. PC 2 (PC 3) characterizes the variability in Task 1 (Task 2) training. Trajectories resulting from Interleaved T1,T2 and Interleaved S,T1/T2 training following Task 1 (Task 2) training are shown in red (blue). (D-F) Average (solid lines) and standard deviation (shaded regions) of the n-dimensional Euclidean distances between the current synaptic weight configuration and M T1 (D), M T2 (E), and M T1∩T2 (F) during Sequential (orange), Interleaved T1,T2 (purple), and Interleaved S,T1/T2 (black) training. Following Task 2 (D) or Task 1 (E) training, Sequential training on the opposite task causes the synaptic weight configuration to diverge from the initial solution manifold, while Interleaved T1,T2 and Interleaved S,T1/T2 training do not. (F) Interleaved T1,T2 and Interleaved S,T1/T2 training cause the synaptic weight configuration to converge to M T1∩T2 while Sequential training avoids this intersection. (G) Authors’ interpretation of the task-specific point-sets shown in (A-C) as solution manifolds M T1 (red) and M T2 (blue). M T1 and M T2 can be thought of as two oppositely oriented elliptic paraboloids which intersect orthogonally near the origin (M T1∩T2 ; dark green). (H,I) Sequential training (pink arrow) causes the network to jump from one solution manifold to the other while avoiding M T1∩T2 , while Interleaved S,T1/T2 training (light green arrow) keep the network close to the initial solution manifold as it converges towards M T1∩T2 .

We calculated the distance from the current synaptic weight configurations to M T1 ( Figure 7D ), M T2 ( Figure 7E ), and M T1∩T2 ( Figure 7F ; see Methods: Distance from Solution Manifolds for details). Figures 7D and 7E show that while Sequential (T1->T2 or T2->T1) training causes synaptic weight configurations to diverge quickly from its initial solution manifold (i.e. M T1 or M T2 ), both Interleaved T1,T2 and Interleaved S,T1/T2 training cause synaptic weight configurations to stay close to the initial solution manifold as the new task was learned. (Note, that we under sampled M T1 and M T2 , which explains initial distance increase.) Importantly, Figure 7F shows that while both Interleaved T1,T2 and Interleaved S,T1/T2 training cause synaptic weight configurations to smoothly converge towards M T1⋂T2 , Sequential training avoids this intersection entirely.

In Figure 7G we show a schematic depiction of these results. The task-specific manifolds, M T1 and M T2 , are roughly defined in 3D projection by two orthogonal elliptic paraboloids with opposite orientation, with an approximately ellipsoidal intersection, M T1∩T2 . Figures 7H and 7I depict the trajectories the network takes in this space following Task 2 and Task 1 training, respectively. Sequential training causes the network to jump directly from one task-specific solution manifold to the other, resulting in catastrophic forgetting. In contrast, interleaving new task training with sleep (Interleaved S,T1/T2 ) prevents catastrophic forgetting by keeping the network close to the old task solution manifold as it converges towards M T1⋂T2 – a region capable a supporting both tasks simultaneously.

## Discussion

In this study we report that a multi-layer SNN utilizing reinforcement learning may exhibit catastrophic forgetting upon sequential training of two complementary complex foraging tasks, but the problem is mitigated if the network is allowed, during new task training, to undergo intervening periods of spontaneous reactivation which we consider to be equivalent to the replay observed during periods of sleep in biological systems. This scenario was effectively equivalent to explicit interleaved training of both tasks, however, no training data for the old task were required during “sleep”. At the synaptic level, training a new task alone led to complete overwriting of synaptic weights responsible for the previous task. In contrast, interleaving periods of reinforcement learning on a new task with periods of unsupervised learning during sleep preserved old task synapses damaged by new task training to avoid forgetting and enhanced new task synapses to allow new task learning. Thus, the network was pushed towards the intersection of the solution manifolds representing synaptic weight configurations associated with each task - an optimal compromise for performing both tasks.

The critical role that sleep plays in learning and memory is supported by a vast, interdisciplinary literature spanning both psychology and neuroscience 21 , 33 – 36 . Specifically, it has been suggested that REM sleep supports the consolidation of non-declarative or procedural memories while non-REM sleep supports the consolidation of declarative memories 21 , 35 , 37 . In particular, REM sleep has been shown to be important for the consolidation of memories of tasks involving perceptual pattern separation, such as the texture discrimination task 21 , 38 . Despite the difference in the cellular and network dynamics during these two stages of sleep 21 , 35 , both are thought to contribute to memory consolidation through repeated reactivation, or replay, of specific memory traces acquired during learning 19 – 21 , 33 , 34 , 37 , 39 . These studies suggest that through replay, sleep can support the process of off-line memory consolidation to circumvent the problem of catastrophic forgetting.

From mechanistic perspective, the sleep phase in our model protects old memories by enabling unsupervised learning - spontaneous replay of synapses responsible for previously learned tasks. We previously reported that in the thalamocortical models, a sleep phase may enable replay of spike sequences learned in awake to improve post-sleep performance 39 , 40 and to protect old memories from catastrophic forgetting 41 . Although in this work we model sleep and noise with spiking statistics similar to awake training, theoretical work from another group has also shown that noise causes implicit rehearsal of older memories which protects against interference 42 . Here we found, however, that a single episode of new task training using reinforcement learning could quickly erase an old memory to the point that it cannot be recovered by subsequent sleep. The solution was similar to how brain slowly learns procedural (hippocampal-independent) memories 21 , 35 , 37 , 38 , 43 . Each episode of new task training improves this task performance only slightly but also damages slightly synaptic connectivity responsible for the older task. Subsequent sleep phases enable replay that preferentially benefits the strongest synapses, such as those from old memory traces, to allow them to recover.

We found that multiple distinct configurations of synaptic weights can support each task in our model, suggesting the existence of task specific solution manifolds in synaptic weight space. Sequential training of new tasks makes the network to jump from one solution manifold to another, enabling memory for the most recent task but erasing memories of the previous tasks. Interleaving new task training with sleep phases enables the system to evolve towards intersection of these manifolds where synaptic weight configurations can support multiple tasks (a similar idea was recently proposed in the machine learning literature to minimize catastrophic interference by learning representations that accelerate future learning 44 ). From this point of view having multiple episodes of new task training interleaved with multiple sleep episodes allows gradual convergence to the intersection of the manifolds representing old and new tasks, while a single long episode of new task learning would push the network far away from the old task manifold making it impossible to recover by subsequent sleep.