Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates — part 3/3

Cleaned-text code points [31057, 44761). Bibliography omitted here; source.md includes it.

There are several papers on the generalization gap between small and large mini-batches and the relationship between gradient noise, learning rate, and batch size. Our results here supplements this other work by illustrating the possibility of time varying high noise levels during training. As mentioned above, Jastrzębski et al. [2017] showed that SGD noise is proportional to the learning rate, the variance of the loss gradients, divided by the batch size. Similarly Smith and Le [ Smith and Le, 2017 ] derived the noise scale as g ≈ ϵ ​ N / B ⁡ ( 1 − m ) g\approx\epsilon N/B(1-m) , where g g is the gradient noise, ϵ \epsilon the learning rate, N N the number of training samples, and m m is the momentum coefficient. Furthermore, Smith et al. [2017] (also Jastrzębski et al. [2017] independently) showed an equivalence of increasing batch sizes instead of a decreasing learning rate schedule. Importantly, these authors demonstrated that the noise scale g g is relevant to training and not the learning rate or batch size. Our paper proposes that the noise scale is a part of training regularization and that all forms of regularization must be balanced during training.

Keskar et al. [2016] study the generalization gap between small and large mini-batches, stating that small mini-batch sizes lead to wide, flat minima and large batch sizes lead to sharp minima. They also suggest a batch size warm start for the first few epochs, then using a large batch size, which amounts to training with large gradient noise for a few epochs and then removing it. Our results contradicts this suggestion as we found it preferable to start training with little noise/regularization and let it increase (i.e., curriculum training), reach a noise peak, and reduce the noise level in the last part of training (i.e., simulated annealing).

Goyal et al. [2017] use a very large mini-batch size of up to 8,192 and adjust the learning rate linearly with the batch size. They also suggest a gradual warmup of the learning rate, which is a discretized version of CLR and matches our experience with an increasing learning rate. They make a point relevant to adjusting the batch size; if the network uses batch normalization, different mini-batch sizes leads to different statistics, which must be handled. Hoffer et al. [2017] made a similar point about batch norm and suggested using ghost statistics. Also, Hoffer et al. [2017] show that longer training lengths is a form of regularization that improves generalization. On the other hand, we undertook a more comprehensive approach to all forms of regularization and our results demonstrate that regularization from training with very large learning rates permits much shorter training lengths.

Furthermore, this paper points the way towards new research directions, such as the following three: 1. Comprehensive investigations of regularization: Characterizing “good noise” that improves the trained network’s ability to generalize versus “bad noise” that interferes with finding a good solution (i.e., Zhang et al. [2016] ). We find that there is a lack of a unified framework for treating SGD noise/diversity, such as architectural noise (e.g., dropout [ Srivastava et al., 2014 ] , dropconnect [ Wan et al., 2013 ] ), noise from hyper-parameter settings (e.g., large learning rates and mini-batch sizes), adding gradient noise, adding noise to weights [ Fortunato et al., 2017 ] , and input diversity (e.g., data augmentation, noise). Smith and Le [2017] took a step in this direction and Smith [2018] took another step by studying the combined effects of learning rates, batch sizes, momentum, and weight decay. Gradient diversity has been shown to lead to flatter local minimum and better generalization. A unified framework should resolve conflicting claims in the literature on the value of each of these, such as for architectural noise ( Srivastava et al. [2014] versus Hoffer et al. [2017] ). Furthermore, many papers study each of these factors independently and by focusing on the trees, one might miss the forest.

2. Time dependent application of good noise during training: As described above, combining curriculum learning with simulated annealing leads to cyclical application. To the best of our knowledge, this has only been applied sporadically in a few methods such as CLR [ Smith, 2017 ] or cyclical batch sizes [ Jastrzębski et al., 2017 ] but these all fall under a single umbrella of time dependent gradient diversity (i.e., noise). Also, a method can be developed to learn an adaptive noise level while training the network.

3. Discovering new ways to stabilize optimization (i.e., SGD) with large noise levels: Our evidence indicates that normalization (and batch normalization in particular) is a catalyst enabling super-convergence in the face of potentially destabilizing noise from the large learning rates. Normalization methods (batch norm, layer normalization Ba et al. [2016] , streaming normalization Liao et al. [2016] ) and new techniques (i.e., cyclical gradient clipping) to stabilize training need further investigation to discover better ways to keep SGD stable in the presence of enormous good noise.

### A.4 Hardware, software, and architectures

All of the experiments were run with Caffe (downloaded October 16, 2016) using CUDA 7.0 and Nvidia’s CuDNN. Most of these experiments were run on a 64 node cluster with 8 Nvidia Titan Black GPUs, 128 GB memory, and dual Intel Xeon E5-2620 v2 CPUs per node and we utilized the multi-gpu implementation of Caffe. The other experiments were run on an IBM Power8, 32 compute nodes, 20 cores/node, 4 Tesla P100 GPUs/node with 255 GB available memory/node.

The Resnet-56 architecture consists of three stages. Within each stage, the same residual block structure is sequentially repeated. This structure is given in Table 3 . Between stages, a different residual block structure is used to reduce the spatial dimension of the channels. Table 4 shows this structure. The overall architecture is described in Table 5 . Following the Caffe convention, each Batch Norm layer was followed by a scaling layer to achieve true batch normalization behavior. This and the other architectures necessary to replicate this work will be made available upon publication.

Cifar-10 and Cifar-100 were downloaded from https://www.cs.toronto.edu/~kriz/cifar.html . The MNIST database of handwritten digits was obtained from http://yann.lecun.com/exdb/mnist/ . The resnet architectures are available at https://github.com/yihui-he/resnet-cifar10-caffe . The wide resnet was created from a resnet with 32 layers by increasing the number of channels by a factor of 4 instead of the factor of 2 used by resnet. A 40 layer densenet architecture was create from the code at https://github.com/liuzhuang13/DenseNetCaffe .

Imagenet, which is a large image database with 1.24 million training images for 1,000 classes, was downloaded from http://image-net.org/download-images . The resnet-50 and inception-resnet-v2 architectures used for these experiments were obtained from https://github.com/soeaver/caffe-model .

### A.5 Hyper-parameters and adaptive learning rate methods

We tested the effect of batch normalization on super-convergence. Initially, we found that having u ​ s ​ e ​ _ ​ g ​ l ​ o ​ b ​ a ​ l ​ _ ​ s ​ t ​ a ​ t ​ s : t ​ r ​ u ​ e use\_global\_stats:true in the test phase prevents super-convergence. However, we realized this was due to using the default value of m ​ o ​ v ​ i ​ n ​ g ​ _ ​ a ​ v ​ e ​ r ​ a ​ g ​ e ​ _ ​ f ​ r ​ a ​ c ​ t ​ i ​ o ​ n = 0.999 moving\_average\_fraction=0.999 that is only appropriate for the typical, long training times. However, when a network is trained very quickly, as with super-convergence, a value of 0.999 0.999 does not update the accumulated global statistics quickly enough. Hence, we found that the smaller values listed in Table 6 (column BN MAF) were more appropriate. We found that these smaller values of m ​ o ​ v ​ i ​ n ​ g ​ _ ​ a ​ v ​ e ​ r ​ a ​ g ​ e ​ _ ​ f ​ r ​ a ​ c ​ t ​ i ​ o ​ n moving\_average\_fraction worked well and we didn’t need to resort to ghost statistics [ Hoffer et al., 2017 ] .

We ran a series of experiments with a variety of ranges for CLR. Table 6 shows the results for maximum learning rate bounds from 1.0 to 3.5. These experiments show that a maximum learning rate of approximately 3 performed well.

We also investigated whether adaptive learning rate methods in a piecewise constant training regime would learn to adaptively use large learning rates to improve training speed and performance. We tested Nesterov momentum [ Sutskever et al., 2013 , Nesterov, 1983 ] , AdaDelta [ Duchi et al., 2011 ] , AdaGrad [ Zeiler, 2012 ] , and Adam [ Kingma and Ba, 2014 ] on Cifar-10 with the Resnet-56 architecture but none of these methods speed up the training process in a similar fashion to super-convergence. Specifically, Figure 9(a) shows the results of this training for Nesterov momentum [ Sutskever et al., 2013 , Nesterov, 1983 ] , AdaDelta [ Duchi et al., 2011 ] , AdaGrad [ Zeiler, 2012 ] , and Adam [ Kingma and Ba, 2014 ] . We found no sign that any of these methods discovered the utility of large learning rates nor any indication of super-convergence-like behavior. This is important because this lack is indicative of the failing in the theory behind these approaches and imply that there is a need for further work on adaptive methods.

We also ran these adaptive learning methods with the 1cycle learning rate policy and found that Nesterov, AdaDelta, and AdaGrad allowed super-convergence to occur, but we were unable to create this phenomenon with Adam. For example, Figure 9(b) shows a comparison of super-convergence to a piecewise constant training regime with the Nesterov momentum method. Here super-convergence yields a final test accuracy after 10,000 iterations of 92.1% while the piecewise constant training regime at iteration 80,000 has an accuracy of 90.9%.

### A.6 Experimental details and additional results

We ran a wide variety of experiments and due to space limitations in the main article, we report results here that did not fit in the main article.

The main paper shows final test accuracy results with Cifar-100, MNIST, and Imagenet, both with and without super-convergence. Figure 10(a) shows the results of the LR range test for Resnet-56 with the Cifar-100 training data. The curve is smooth and accuracy remains high over the entire range from 0 to 3 indicating a potential for super-convergence. An example of super-convergence with Cifar-100 with Resnet-56 is given in Figure 10(b) , where there is also a comparison to the results from a piecewise constant training regime. Furthermore, the final test accuracy for the super-convergence curve in this experiment is 68.6%, while the accuracy for the piecewise constant method is 59.8%, which is an 8.8% improvement.

Figure 11 shows a comparison of super-convergence, both with and without dropout. The LR range test with and without dropout is shown in Figure 11(a) . Figure 11(b) shows the results of training for 10,000 iterations. In both cases, the dropout ratio was set to 0.2 and the Figure shows a small improvement with dropout.

The effect of mini-batch size is discussed in the main paper but here we present a table containing the final accuracies of super-convergence training with various mini-batch sizes. One can see in Table 7 the final test accuracy results and this table shows that larger mini-batch sizes 5 5 5 GPU memory limitations prevented our testing a total batch size greater than 1,530 produced better final accuracies, which differs from the results shown in the literature. Most of results reported in the main paper are with a total mini-batch size of 1,000.

In addition, we ran experiments on Resnet-56 on Cifar-10 with modified values for momentum and weight decay to determine if they might hinder the super-convergence phenomenon. Figure 12(a) shows the results for momentum set to 0.8, 0.85, 0.9, and 0.95 and the final test accuracies are listed in Table 8 . These results indicate only a small change in the results, with a setting of 0.9 being a bit better than the other values. In Figure 12(b) are the results for weight decay values of 10 − 3 , 10 − 4 , 10 − 5 , 10^{-3},10^{-4},10^{-5}, and 10 − 6 10^{-6} . In this case, a weight decay value of 10 − 3 10^{-3} prevents super-convergence, while the smaller values do not. This demonstrates the principle in the main paper that regularization by weight decay must be reduced to compensate for the increased regularization by large learning rates. This Figure also shows that a weight decay value of 10 − 4 10^{-4} performs well.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .