Continual Backprop: Stochastic Gradient Descent with Persistent Randomness — part 3/3

Cleaned-text code points [31652, 44252). Bibliography omitted here; source.md includes it.

Prior works on the importance of initialization have focused on finding the correct weight magnitude to initialize the weights. Glorot et al. (2010) showed that it is essential to initialize the weights so that the gradients do not become exponentially small in the initial layers of a network with sigmoid activations. Sutskever et al. (2013) showed that initialization with small weights is critical for sigmoid activations as they may saturate if the weights are too large. More recent work (He et al., 2015) has gone into ensuring that the input signal’s scale of magnitude is preserved across the layers in a deep network. Despite all these works on the importance of initialization, the fact that its benefits are only present in the beginning but not continually has been overlooked as most of these works focused on offline learning, where learning has to be done just once.

## 6 Conclusion and limitations

In this work, we studied learning problems that have a large number of non-stationarities. We showed that in these problems, the performance of Backprop degrades over time, and it loses its ability to adapt. In other words, we showed that neural networks have a “decaying plasticity” problem. Ours is the first result showing this degradation in Backprop’s ability to adapt. This result means that Backprop can be unsuitable for continual adaptation.

We proposed the Continual Backprop (CBP) algorithm to get a robust continual learning algorithm where both components of Backprop are present continually. CBP extends the benefits of special random initialization throughout learning using a generate-and-test process. Our generate-and-test algorithm is the first one that is compatible with modern deep learning. We showed that CBP could continually adapt even after thousands of non-stationarities. Additionally, we showed that Backprop with L2 regularization can adapt continually in some problems, but it fails in many others.

The main limitation of our work is that our tester is based on heuristics. Even though it performs well, future work on more principled testers will improve the foundations of Continual Backprop.

Continual learning is becoming more and more relevant in machine learning. These systems can not use Backprop or its variants in their current form, as they are not stable in non-stationary problems. We expect that by establishing Backprop’s decaying plasticity, our work will invite many solutions similar to how many works have addressed “catastrophic forgetting”. We foresee that our work will open a path for better Continual Backprop algorithms.

## Appendix

## Appendix A More experiments with Backprop on the Bit-Flipping Problem

In Figure 2 we showed that the performance of Backprop for ReLU and tanh activations gets worse over time. In Figure 10 , we show that the performance of Backprop also degrades for other activation functions.

In Figure 2 and 10 we ran the experiment for 1M examples. For this length of experiment it may seem that the error for smaller step-size of 0.001 reduced or stayed the same for some activation functions. In Figure 11 , we show that even for a step-size of 0.001, if we let Backprop track for longer (5M), the error for all activations increases significantly or stays at a high level. Again, in Figure 11 , we bin the online prediction error into bins of size 20,000 and the shaded region in the figure shows the standard error of the binned error.

Next, we evaluate the performance of BP when using Adam instead of SGD. In all the experiments, we used Adam with betas (0.9, 0.999). First, in Figure 12 , we show that the performance of BP gets worse over time for a wide range of step-sizes for all activation functions. Then in Figure 13 , we show that even for a smaller step-size of 0.001, after tracking for longer (5M examples), the error for all activations increases significantly.

## Appendix B More experiments with CBP on the Bit-Flipping Problem

First, we look at the performance of CBP with the remaining activation functions to complement the results shown in Figure 5 . The first half of Figure 14 shows the online performance of various algorithms, and the second half shows the average loss over the experiment’s full length (1M examples) for various values of hyperparameter replacement-rate (for CBP) and weight-decay (for BP+L2).

To properly use Adam with CBP, we need to modify Adam. Adam maintains estimates of the average gradient and the average squared gradient. CBP resets some of the weights whenever a feature is replaced. Whenever CBP resets a weight, we set its estimated gradient and squared gradient to zero. Adam also maintains a ’timestep’ parameter to get an unbiased estimate of the average gradient. Again, whenever CBP resets a weight, we set its timestep parameter to zero.

In Figures 15 and 16 , we show that CBP with Adam performs significantly better than BP with Adam for a wide range of replacement-rates.

## Appendix C Ablation Study for the utility measure

The final utility measure that we used in this work consists of two parts, the contribution-utility and the adaptation-utility. Now, we will compare various parts of the utility measure on the Bit-Flipping problem. We use a learning network with tanh activation, and we use the Adam optimizer with a step-size of 0.01. Additionally, we also compare our utility measure with random utility and weight-magnitude-based utility (Mahmood and Sutton, 2013). The results for this comparison are presented in Figure 17(a) We compare the following utility measures.

• Random utility: Utility, r r at every time-step is uniformly randomly sampled from U ⁡ [ 0 , 1 ] U[0,1] r l , i , t = r ​ a ​ n ​ d ​ ( 0 , 1 ) r_{l,i,t}=rand(0,1)

• Weight-magnitude based utility: w ​ m wm at every time-step is updated as: w ​ m l , i , t = ( 1 − η ) ∗ ∑ k = 1 n l + 1 | w l , i , k , t | + η ∗ w ​ m l , i , t − 1 wm_{l,i,t}=(1-\eta)*\sum_{k=1}^{n_{l+1}}|w_{l,i,k,t}|+\eta*wm_{l,i,t-1}

• Contribution utility, c c , at every time-step is updated as described in Equation 1

• Mean corrected contribution utility, z z , at every time-step is updated as described in Equation 2

• Adaptation utility, a a , at every time-step is updated as: a l , i , t = ( 1 − η ) ∗ 1 ∑ j = 1 n l − 1 | w l , j , i , t | + η ∗ a l , i , t − 1 a_{l,i,t}=(1-\eta)*\frac{1}{\sum_{j=1}^{n_{l-1}}|w_{l,j,i,t}|}+\eta*a_{l,i,t-1}

• Overall utility, u u ,at every time-step is updated as described in Equation 5

Next we compared the two best performing utility measures, overall and mean-corrected contribution, on Slippery Ant. The results are presented in Figure 17(b) . The results show that both utility measures perform significantly better than PPO. But, the overall utility performs significantly better than the other utility measure. This difference in more pronounced near the end, when Continual PPO with the overall utility measure performs almost as well as it did at the beginning.

## Appendix D Continual PPO

We used the following values of the parameters for PPO, PPO+L2 and CPPO in our experiments on our non-stationary RL problem. For CPPO and PPO+L2 specific parameters, we chose the best performing values. • Policy Network: (256, tanh, 256, tanh, Linear) + Standard Deviation variable

• Value Network (256, tanh, 256, tanh, linear)

• iteration size: 4096

• num epochs: 10

• mini-batch size: 128

• GAE, λ \lambda : 0.95

• Discount factor, γ \gamma : 0.99

• clip parameter: 0.2

• Optimizer: Adam

• Optimizer step-size: 1 ​ e − 4 1e-4

• Optimizer β \beta s: (0.9, 0.999)

• weight-decay (for L2): { 10 − 3 , 10 − 4 , 10 − 5 , 10 − 6 } \{10^{-3},10^{-4},10^{-5},10^{-6}\}

• replacement-rate, maturity-threshold (for CPPO): { ( 10 − 3 , 1 e 2 ) , ( 10 − 3 , 5 e 2 ) , ( 10 − 4 , 5 e 3 ) \{(10^{-3},1e2),(10^{-3},5e2),(10^{-4},5e3) , ( 10 − 4 , 5 e 2 ) , ( 10 − 5 , 1 e 4 ) , ( 10 − 5 , 5 e 4 ) } (10^{-4},5e2),(10^{-5},1e4),(10^{-5},5e4)\}

## Appendix E Compute Used

All the experiments in this work were performed on Intel Gold 6148 Skylake CPUs or NVIDIA Tesla P100 GPUs. On the Bit-Flipping problem, all the algorithms took around 30 minutes for a single run with 1M examples. The total compute used for all the experiments on the Bit-Flipping problem was around 12960 CPU-days or 36 CPU-years. On the Permuted-MNIST problem, each took about 8 GPU hours, which totals 110 GPU days for all experiments. Finally, the experiments for non-stationary RL task took up to 90hours for 100M time-steps, the total compute used for all experiments on the non-stationary RL problem was 1980 CPU-days or 5.4 CPU years.

## Appendix F Affect of the speed of the distribution change on decaying plasticity

Figure 18 shows the online classification accuracy of BP on a deep ReLU-network with different speeds of distribution change. In the fastest changing distribution, the distribution changes after every 10k examples, while in the slowest changing distribution, it changes after every 1M examples. We used SGD with a step-size of 0.01. The network has three hidden layers with 2000 units each.

The Figure 18 shows that the drop in performance is fastest for the fastest changing distribution.

## Appendix G A deeper look in the networks: Feature Saturation and Vanishing Gradients

Figure 19(a) shows the number of saturated features for tanh-network on the Bit-Flipping problem. We call a feature is saturated when the magnitude of its output is more than 0.9. Figure 19(b) shows the average magnitude of gradients for the weights in the input layer. For different algorithms, the same values of hyper-parameters are used as we did in Figure 5 . In both Figure 19(a) and Figure 19(b) , the data is binned into bins of 20,000 examples.

Figure 19(a) shows that an increasingly large fraction of features saturate over time for Backprop. When a feature gets saturated, the network loses some of its representation power which causes worse performance. With Backprop, almost 90% of the features are saturated. Figure 19(b) shows that with BP, the magnitude of the gradient of the input weights becomes significantly smaller after some time. A small gradient means that the weights change less, and the network loses its ability to adapt. This increase in saturated features and the small gradient of the input weights partially explain why BP performs worse after extended tracking. However, the evolution of the level of feature saturation does not correlate with the performance of all algorithms. Figure 5 shows that CBP and L2 have the least online error, but the online error with of OnlineNorm is much larger than CBP or L2. However, in Figure 19(a) , OnlineNorm has the least number of saturated features. Similarly, in Figure 19(b) , OnlineNorm has a higher input gradient than CBP, but its performance is worse than that of CBP. This means that neither feature saturation nor the magnitude of input gradients are fully explaining the behavior of all algorithms.

## Appendix H Affect of network size on decaying plasticity

Figure 20 shows the online classification accuracy of BP with different deep ReLU-networks. Both networks had three hidden layers. The smaller network had just 100 hidden units, while the wider network had 1000 hidden units. We used SGD with a step-size of 0.003. Figure 20 shows that the drop in performance is fastest for the smaller network.

{contributions} Briefly list author contributions. This is a nice way of making clear who did what and to give proper credit.

H. Q. Bovik conceived the idea and wrote the paper. Coauthor One created the code. Coauthor Two created the figures.

###### Acknowledgements.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .