Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates — part 2/3
Cleaned-text code points [15808, 31057). Bibliography omitted here; source.md includes it.
In this paper we do not perform a full evaluation of the effectiveness of this technique as it is tangential to the theme of this work. We only use this method here to demonstrate that training with large learning rates are indicated by this approximation. We leave a full assessment and tests of this method to estimate optimal adaptive learning rates as future work.
## 5 Experiments and analysis
This section highlights a few of our more significant experiments. Additional experiments and details of our architecture are illustrated in the Supplemental Materials.
Figure 4(a) provides a comparison of super-convergence with a reduced number of training samples. When the amount of training data is limited, the gap in performance between the result of standard training and super-convergence increases. Specifically, with a piecewise constant learning rate schedule the training encounters difficulties and diverges along the way. On the other hand, a network trained with specific CLR parameters exhibits the super-convergence training curve and trains without difficulties. The highest accuracies attained using standard learning rate schedules are listed in Table 6 and super-convergence test accuracy is 1.2%, 5.2%, and 9.2% better for 50,000, 20,000, and 10,000 training cases, respectively. Hence, super-convergence becomes more beneficial when training data is more limited.
We also ran experiments with Resnets with a number of layers in the range of 20 to 110 layers; that is, we ran experiments on residual networks with l l layers, where l = 20 + 9 n ; l=20+9n; for n = 0 , 1 , … 10 n=0,1,...10 . Figure 4(b) illustrates the results for Resnet-20 and Resnet-110, for both a typical (piecewise constant) training regime with a standard initial learning rate of 0.35 and for CLR with a stepsize of 10,000 iterations. The accuracy increase due to super-convergence is greater for the shallower architectures (Resnet-20: CLR 90.4% versus piecewise constant LR schedule 88.6%) than for the deeper architectures (Resnet-110: CLR 92.1% versus piecewise constant LR schedule 91.0%).
There are discussions in the deep learning literature on the effects of larger batch size and the generalization gap ( Keskar et al., 2016 ; Jastrzębski et al., 2017 ; Chaudhari and Soatto, 2017 ; Hoffer et al., 2017 ) . Hence, we investigated the effects total mini-batch size 2 2 2 Most of our reported results are with a total mini-batch size of 1,000 and we primarily used 8 GPUs and split the total mini-batch size 8 ways over the GPUs. used in super-convergence training and found a small improvement in performance with larger batch sizes, as can be seen in Figure 5(a) . In addition, Figure 5(b) shows that the generalization gap (the difference between the training and test accuracies) are approximately equivalent for small and large mini-batch sizes. This result differs than results reported elsewhere ( Keskar et al., 2016 ) and illustrates that a consequence of training with large batch sizes is the ability to use large learning rates.
### 5.1 Other datasets and architectures
The wide resnet was created from a resnet with 32 layers by increasing the number of channels by a factor of 4 instead of the factor of 2 used by resnet. Table 2 provides the final result of training using the 1cycle learning rate schedule with learning rate bounds from 0.1 to 1.0. In 100 epochs, the wide32 network converges and provides a test accuracy of 91.9 % ± 0.2 91.9\%\pm 0.2 while the standard training method achieves an accuracy of only 90.3 ± 1.0 90.3\pm 1.0 in 800 epochs. This demonstrates super-convergence for wide resnets.
A 40 layer densenet architecture was create from the code at https://github.com/liuzhuang13/DenseNetCaffe . Table 2 provides the final accuracy results of training using the 1cycle learning rate schedule with learning rate bounds from 0.1 to 4.0 and cyclical momentum in the range of 0.9 to 0.85. This Table also shows the effects of longer training lengths, where the final accuracy improves from 91.7% for a quick 50 epoch (4,882 iterations) training to 92.8% with a longer 150 epoch (14,648 iterations) training. The step learning rate policy attains an equivalent accuracy of 92.7% but requires 400 epochs to do so.
The MNIST database of handwritten digits from 0 to 9 (i.e., 10 classes) has a training set of 60,000 examples, and a test set of 10,000 examples. It is simpler than Cifar-10, so the shallow, 3-layer LeNet architecture was used for the tests. Caffe https://github.com/BVLC/caffe provides the LeNet architecture and its associated hyper-parameters with the download of the Caffe framework. Table 2 lists the results of training the MNIST dataset with the LeNet architecture. Using the hyper-parameters provided with the Caffe download, an accuracy of 99.03% is obtained in 85 epochs. Switching from an inv learning rate policy to a step policy produces equivalent results. However, switching to the 1cycle policy produces an accuracy near 99.3%, even in as few as 12 epochs.
Cifar-100 is similar to CIFAR-10, except it has 100 classes and instead of 5,000 training images per class, there are only 500 training images and 100 testing images per class. Table 2 compares the final accuracies of training with a step learning rate policy to training with a 1cycle learning rate policy. The training results from the 1cycle learning rate policy are significantly higher than than the results from the step learning rate policy and the number of epochs required for training is greatly reduced (i.e., at only 25 epochs the accuracy is higher than any with a step learning rate policy).
### 5.2 Imagenet
Our experiments show that reducing regularization in the form of weight decay when training Imagenet allows the use of larger learning rates and produces much faster convergence and higher final accuracies.
Figure 6(a) presents the comparison of training a resnet-50 architecture on Imagenet with the current standard training methodology versus the super-convergence method. The hyper-parameters choices for the original training is set to the recommended values in Szegedy et al. (2017) (i.e., momentum = 0.9, LR = 0.045 decaying every 2 epochs using an exponential rate of 0.94, WD = 10 − 4 10^{-4} ). This produced the familiar training curve indicated by the blue lines in Figure 6 3 3 3 Curves in the Figures for Imagenet are the average of only two runs. . The final top-1 test accuracy in Figure 6(a) is 63.7% but the blue curve appears to be headed to an final accuracy near 65%.
The red and yellow training curves in Figure 6(a) use the 1cycle learning rate schedule for 20 epochs, with the learning rate varying from 0.05 to 1.0, then down to 0.00005. In order to use such large learning rates, it was necessary to reduce the value for weight decay. Our tests found that weight decay values in the range from 3 × 10 − 6 3\times 10^{-6} to 10 − 5 10^{-5} provided the best top-1 accuracy of 67.6 % 67.6\% . Smaller values of weight decay showed signs of underfitting, which hurt performance It is noteworthy that the two curves (especially for a weight decay value of 3 × 10 − 6 3\times 10^{-6} ) display small amounts of overfitting. Empirically this implies that small amounts of overfitting is a good indicator of the best value and helps the search for the optimal weight decay values early in the training.
Figure 6(b) presents the training a inception-resnet-v2 architecture on Imagenet. The blue curve is the current standard way while the red and yellow curves use the 1cycle learning rate policy. The inception architecture tells a similar story as the resnet architecture. Using the same step learning rate policy, momentum, and weight decay as in Szegedy et al. (2017) produces the familiar blue curve in Figure 6(b) . After 100 epochs the accuracy is 67.6% and the curve can be extrapolated to an accuracy in the range of 69-70%.
On the other hand, reducing the weight decay permitted use of the 1cycle learning rate policy with the learning rate varying from 0.05 to 1.0, then down to 0.00005 in 20 epochs. The weight decay values in the range from 3 × 10 − 6 3\times 10^{-6} to 10 − 6 10^{-6} work well and using a weight decay of 3 × 10 − 6 3\times 10^{-6} provides the best accuracy of 74.0 % 74.0\% . As with resnet-50, there appears to be a small amount of overfitting with this weight decay value.
The lesson from these experiments is that deep neural networks can be trained much faster by super-convergence methodology than by the standard training methods.
## 6 Conclusion
We presented empirical evidence for a previously unknown phenomenon that we name super-convergence. In super-convergence, networks are trained with large learning rates in an order of magnitude fewer iterations and to a higher final test accuracy than when using a piecewise constant training regime. Particularly noteworthy is the observation that the gains from super-convergence increase as the available labeled training data becomes more limited. Furthermore, this paper describes a simplification of the Hessian-free optimization method that we used for estimating learning rates. We demonstrate that super-convergence is possible with a variety of datasets and architectures, provided the regularization effects of large learning rates are balanced by reducing other forms of regularization.
We believe that a deeper study of super-convergence will lead to a better understanding of deep networks, their optimization, and their loss function landscape.
## Appendix A Supplemental material
This Appendix contains information that does not fit within the main paper due to space restrictions. This includes an intuitive explanation for super-convergence, a discussion relating the empirical evidence from super-convergence to discussions in the literature, and the details of the experiments that were carried out in support of this research.
### A.1 Intuitive explanation for super-convergence
Figure 7(a) provides an example of transversing the loss function topology. 4 4 4 Figure reproduced from Goodfellow et al. [2014] with permission. This figure helps give an intuitive understanding of how super-convergence happens. The blue line in the Figure represents the trajectory of the training while converging and the x’s indicate the location of the solution at each iteration and indicates the progress made during the training. In early iterations, the learning rate must be small in order for the training to make progress in an appropriate direction. The Figure also shows that significant progress is made in those early iterations. However, as the slope decreases so does the amount of progress per iteration and little improvement occurs over the bulk of the iterations. Figure 7(b) shows a close up of the final parts of the training where the solution maneuvers through a valley to the local minimum within a trough.
Cyclical learning rates are well suited for training when the loss topology takes this form. The learning rate initially starts small to allow convergence to begin. As the network traverses the flat valley, the learning rate is large, allowing for faster progress through the valley. In the final stages of the training, when the training needs to settle into the local minimum (as seen in Figure 7(b) ), the learning rate is once again reduced to a small value.
### A.2 Large learning rate regularization
Super-convergence requires using a large maximum learning rate value. The LR range test reveals evidence of regularization through results shown in Figure 8(a) . Figure 8(a) shows an increasing training loss and decreasing test loss while the learning rate increases from approximately 0.2 to 2.0 when training with the Cifar-10 dataset and a Resnet-56 architecture, which implies that regularization is occurring while training with these large learning rates. Similarly, Figure 8(b) presents the training and test loss curves for Resnet-20 and Resnet-110, where one can see the same decreasing generalization error. In addition, we ran the LR range test on residual networks with l l layers, where l = 20 + 9 n ; l=20+9n; for n = 0 , 1 , … 10 n=0,1,...10 and obtained similar results.
There is additional evidence that large learning rates are regularizing the training. As shown in the main paper, the final test accuracy results from a super-convergence training is demonstrably better than the accuracy results from a typical training method. Since a definition of regularization is “any modification we make to a learning algorithm that is intended to reduce its generalization error” [ Goodfellow et al., 2016 ] , large learning rates should be considered as regularizing. Furthermore, others show that large learning rates leads to larger gradient noise, which leads to better generalization (i.e., Jastrzębski et al. [2017] , Smith et al. [2017] ).
### A.3 Relationship of super-convergence to SGD and generalization
There is substantial discussion in the literature on stochastic gradient descent (SGD) and understanding why solutions generalize so well (i.e, Chaudhari et al. [2016] , Chaudhari and Soatto [2017] , Im et al. [2016] , Jastrzębski et al. [2017] , Smith and Le [2017] , Kawaguchi et al. [2017] ). Super-convergence provides empirical evidence that supports some theories, contradicts some others, and points to the need for further theoretical understanding. We hope the response to super-convergence is similar to the reaction to the initial report of network memorization [ Zhang et al., 2016 ] , which sparked an active discussion within the deep learning research community on better understanding of the factors in SGD leading to solutions that generalize well (i.e., [ Arpit et al., 2017 ] ).
Our work impacts the line of research on SGD and the importance of noise for generalization. In this paper we focused on the use of CLR with very large learning rates, where large learning rates add noise/regularization in the middle of training. Jastrzębski et al. [2017] stated that higher levels of noise lead SGD to solutions with better generalization. Specifically, they showed that the ratio of the learning rate to the batch size, along with the variance of the gradients, controlled the width of the local minima found by SGD. Independently, Chaudhari and Soatto [2017] show that SGD performs regularization by causing SGD to be out of equilibrium, which is crucial to obtain good generalization performance, and derive that the ratio of the learning rate to batch size alone controls the entropic regularization term. They also state that data augmentation increases the diversity of SGD’s gradients, leading to better generalization. The super-convergence phenomenon provides empirical support for these two papers.
In addition there are several other papers in the literature which state that wide, flat local minima produce solutions that generalize better than sharp minima [ Hochreiter and Schmidhuber, 1997 , Keskar et al., 2016 , Wu et al., 2017 ] . Our super-convergence results align with these results in the middle of training, yet a small learning rate is necessary at the end of training, implying that the minima of a local minimum is narrow. This needs to be reconciled.