Cyclical Learning Rates for Training Neural Networks — part 2/3

Cleaned-text code points [15809, 31433). Bibliography omitted here; source.md includes it.

Figure 1 shows the result of running with the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy with the parameter setting in Table 2 . As shown in Table 1 , one obtains the same test classification accuracy of 81.4 % 81.4\% after only 25,000 25,000 iterations with the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy as obtained by running the standard hyper-parameter settings for 70,000 70,000 iterations.

One might speculate that the benefits from the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r triangular policy derive from reducing the learning rate because this is when the accuracy climbs the most. As a test, a d ​ e ​ c ​ a ​ y decay policy was implemented where the learning rate starts at the m ​ a ​ x ​ _ ​ l ​ r max\_lr value and then is linearly reduced to the b ​ a ​ s ​ e ​ _ ​ l ​ r base\_lr value for s ​ t ​ e ​ p ​ s ​ i ​ z ​ e stepsize number of iterations. After that, the learning rate is fixed to b ​ a ​ s ​ e ​ _ ​ l ​ r base\_lr . For the d ​ e ​ c ​ a ​ y decay policy, m ​ a ​ x ​ _ ​ l ​ r = 0.007 max\_lr=0.007 , b ​ a ​ s ​ e ​ _ ​ l ​ r = 0.001 base\_lr=0.001 , and s ​ t ​ e ​ p ​ s ​ i ​ z ​ e = 4000 stepsize=4000 . Table 1 shows that the final accuracy is only 78.5 % 78.5\% , providing evidence that both increasing and decreasing the learning rate are essential for the benefits of the CLR method.

Figure 4 compares the e ​ x ​ p exp learning rate policy in Caffe with the new e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy using g ​ a ​ m ​ m ​ a = 0.99994 gamma=0.99994 for both policies. The result is that when using the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy one can stop training at iteration 42,000 42,000 with a test accuracy of 82.2 % 82.2\% (going to iteration 70,000 70,000 does not improve on this result). This is substantially better than the best test accuracy of 79.1 % 79.1\% one obtains from using the e ​ x ​ p exp learning rate policy.

The current Caffe download contains additional architectures and hyper-parameters for CIFAR-10 and in particular there is one with sigmoid non-linearities and batch normalization. Figure 6 compares the training accuracy using the downloaded hyper-parameters with a fixed learning rate (blue curve) to using a cyclical learning rate (red curve). As can be seen in this Figure, the final accuracy for the fixed learning rate (60.8%) is substantially lower than the cyclical learning rate final accuracy (72.2%). There is clear performance improvement when using CLR with this architecture containing sigmoids and batch normalization.

Experiments were carried out with architectures featuring both adaptive learning rate methods and CLR. Table 3 lists the final accuracy values from various adaptive learning rate methods, run with and without CLR. All of the adaptive methods in Table 3 were run by invoking the respective option in Caffe. The learning rate boundaries are given in Table 3 (just below the method’s name), which were determined by using the technique described in Section 3.3 . Just the lower bound was used for b ​ a ​ s ​ e ​ _ ​ l ​ r base\_lr for the f ​ i ​ x ​ e ​ d fixed policy.

Table 3 shows that for some adaptive learning rate methods combined with CLR, the final accuracy after only 25,000 iterations is equivalent to the accuracy obtained without CLR after 70,000 iterations. For others, it was necessary (even with CLR) to run until 70,000 iterations to obtain similar results. Figure 5 shows the curves from running the Nesterov method with CLR (reached 81.3% accuracy in only 25,000 iterations) and the Adam method both with and without CLR (both needed 70,000 iterations). When using adaptive learning rate methods, the benefits from CLR are sometimes reduced, but CLR can still valuable as it sometimes provides benefit at essentially no cost.

#### 4.1.2 ResNets, Stochastic Depth, and DenseNets

Residual networks [ 10 , 11 ] , and the family of variations that have subsequently emerged, achieve state-of-the-art results on a variety of tasks. Here we provide comparison experiments between the original implementations and versions with CLR for three members of this residual network family: the original ResNet [ 10 ] , Stochastic Depth networks [ 13 ] , and the recent DenseNets [ 12 ] . Our experiments can be readily replicated because the authors of these papers make their Torch code available 3 3 3 https://github.com/facebook/fb.resnet.torch, https://github.com/yueatsprograms/Stochastic_Depth, https://github.com/liuzhuang13/DenseNet . Since all three implementation are available using the Torch 7 framework, the experiments in this section were performed using Torch. In addition to the experiment in the previous Section, these networks also incorporate batch normalization [ 15 ] and demonstrate the value of CLR for architectures with batch normalization.

Both CIFAR-10 and the CIFAR-100 datasets were used in these experiments. The CIFAR-100 dataset is similar to the CIFAR-10 data but it has 100 classes instead of 10 and each class has 600 labeled examples.

The results for these two datasets on these three architectures are summarized in Table 4 . The left column give the architecture and whether CLR was used in the experiments. The other two columns gives the average final accuracy from five runs and the initial learning rate or range used in parenthesis, which are reduced (for both the fixed learning rate and the range) during the training according to the same schedule used in the original implementation. For all three architectures, the original implementation uses an initial LR of 0.1 which we use as a baseline.

The accuracy results in Table 4 in the right two columns are the average final test accuracies of five runs. The Stochastic Depth implementation was slightly different than the ResNet and DenseNet implementation in that the authors split the 50,000 training images into 45,000 training images and 5,000 validation images. However, the reported results in Table 4 for the SD architecture is only test accuracies for the five runs. The learning rate range used by CLR was determined by the LR range test method and the cycle length was choosen as a tenth of the maximum number of epochs that was specified in the original implementation.

In addition to the accuracy results shown in Table 4 , similar results were obtained in Caffe for DenseNets [ 12 ] on CIFAR-10 using the prototxt files provided by the authors. The average accuracy of five runs with learning rates of 0.1 , 0.2 , 0.3 0.1,0.2,0.3 was 91.67 % , 92.17 % , 92.46 % , 91.67\%,92.17\%,92.46\%, respectively, but running with CLR within the range of 0.1 to 0.3, the average accuracy was 93.33 % 93.33\% .

The results from all of these experiments show similar or better accuracy performance when using CLR versus using a fixed learning rate, even though the performance drops at some of the learning rate values within this range. These experiments confirm that it is beneficial to use CLR for a variety of residual architectures and for both CIFAR-10 and CIFAR-100.

### 4.2 ImageNet

The ImageNet dataset [ 21 ] is often used in deep learning literature as a standard for comparison. The ImageNet classification challenge provides about 1,000 1,000 training images for each of the 1,000 1,000 classes, giving a total of 1,281,167 1,281,167 labeled training images.

#### 4.2.1 AlexNet

The Caffe website provides the architecture and hyper-parameter files for a slightly modified AlexNet [ 17 ] . These were downloaded from the website and used as a baseline. In the training results reported in this section, all weights were initialized the same so as to avoid differences due to different random initializations.

Since the b ​ a ​ t ​ c ​ h ​ s ​ i ​ z ​ e batchsize in the architecture file is 256 256 , an epoch is equal to 1,281,167 / 256 = 5,005 1,281,167/256=5,005 iterations. Hence, a reasonable setting for s ​ t ​ e ​ p ​ s ​ i ​ z ​ e stepsize is 6 6 epochs or 30,000 30,000 iterations.

Next, one can estimate reasonable minimum and maximum boundaries for the learning rate from Figure 7 . It can be seen from this figure that the training doesn’t start converging until at least 0.006 0.006 so setting b ​ a ​ s ​ e ​ _ ​ l ​ r = 0.006 base\_lr=0.006 is reasonable. However, for a fair comparison to the baseline where b ​ a ​ s ​ e ​ _ ​ l ​ r = 0.01 base\_lr=0.01 , it is necessary to set the b ​ a ​ s ​ e ​ _ ​ l ​ r base\_lr to 0.01 0.01 for the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r triangular and t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policies or else the majority of the apparent improvement in the accuracy will be from the smaller learning rate. As for the maximum boundary value, the training peaks and drops above a learning rate of 0.015 0.015 so m ​ a ​ x ​ _ ​ l ​ r = 0.015 max\_lr=0.015 is reasonable. For comparing the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy to the e ​ x ​ p exp policy, setting b ​ a ​ s ​ e ​ _ ​ l ​ r = 0.006 base\_lr=0.006 and m ​ a ​ x ​ _ ​ l ​ r = 0.014 max\_lr=0.014 is reasonable and in this case one expects that the average accuracy of the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy to be equal to the accuracy from the e ​ x ​ p exp policy.

Figure 9 compares the results of running with the f ​ i ​ x ​ e ​ d fixed versus the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy for the AlexNet architecture. Here, the peaks at iterations that are multiples of 60,000 should produce a classification accuracy that corresponds to the f ​ i ​ x ​ e ​ d fixed policy. Indeed, the accuracy peaks at the end of a cycle for the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy are similar to the accuracies from the standard f ​ i ​ x ​ e ​ d fixed policy, which implies that the baseline learning rates are set quite well (this is also implied by Figure 7 ). As shown in Table 1 , the final accuracies from the CLR training run are only 0.4 % 0.4\% better than the accuracies from the f ​ i ​ x ​ e ​ d fixed policy.

Figure 10 compares the results of running with the e ​ x ​ p exp versus the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy for the AlexNet architecture with g ​ a ​ m ​ m ​ a = 0.999995 gamma=0.999995 for both policies. As expected, Figure 10 shows that the accuracies from the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy do oscillate around the e ​ x ​ p exp policy accuracies. The advantage of the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy is that the accuracy of 56.5 % 56.5\% is already obtained at iteration 300,000 300,000 whereas the e ​ x ​ p exp policy takes until iteration 460,000 460,000 to reach 56.5 % 56.5\% .

Finally, a comparison between the f ​ i ​ x ​ e ​ d fixed and e ​ x ​ p exp policies in Table 1 shows the f ​ i ​ x ​ e ​ d fixed and t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policies produce accuracies that are almost 2 % 2\% better than their exponentially decreasing counterparts, but this difference is probably due to not having tuned g ​ a ​ m ​ m ​ a gamma .

#### 4.2.2 GoogLeNet/Inception Architecture

The GoogLeNet architecture was a winning entry to the ImageNet 2014 image classification competition. Szegedy et al . [ 25 ] describe the architecture in detail but did not provide the architecture file. The architecture file publicly available from Princeton 4 4 4 vision.princeton.edu/pvt/GoogLeNet/ was used in the following experiments. The GoogLeNet paper does not state the learning rate values and the hyper-parameter solver file is not available for a baseline but not having these hyper-parameters is a typical situation when one is developing a new architecture or applying a network to a new dataset. This is a situation that CLR readily handles. Instead of running numerous experiments to find optimal learning rates, the b ​ a ​ s ​ e ​ _ ​ l ​ r base\_lr was set to a best guess value of 0.01 0.01 .

The first step is to estimate the s ​ t ​ e ​ p ​ s ​ i ​ z ​ e stepsize setting. Since the architecture uses a batchsize of 128 128 an epoch is equal to 1,281,167 / 128 = 10,009 1,281,167/128=10,009 iterations. Hence, good settings for s ​ t ​ e ​ p ​ s ​ i ​ z ​ e stepsize would be 20,000 20,000 , 30,000 30,000 , or possibly 40,000 40,000 . The results in this section are based on s ​ t ​ e ​ p ​ s ​ i ​ z ​ e = 30000 stepsize=30000 .

The next step is to estimate the bounds for the learning rate, which is found with the LR range test by making a run for 4 epochs where the learning rate linearly increases from 0.001 0.001 to 0.065 0.065 (Figure 11 ). This figure shows that one can use bounds between 0.01 0.01 and 0.04 0.04 and still have the model reach convergence. However, learning rates above 0.025 0.025 cause the training to converge erratically. For both t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 and the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policies, the b ​ a ​ s ​ e ​ _ ​ l ​ r base\_lr was set to 0.01 0.01 and m ​ a ​ x ​ _ ​ l ​ r max\_lr was set to 0.026 0.026 . As above, the accuracy peaks for both these learning rate policies correspond to the same learning rate value as the f ​ i ​ x ​ e ​ d fixed and e ​ x ​ p exp policies. Hence, the comparisons below will focus on the peak accuracies from the LCR methods.

Figure 12 compares the results of running with the f ​ i ​ x ​ e ​ d fixed versus the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy for this architecture (due to time limitations, each training stage was not run until it fully plateaued). In this case, the peaks at the end of each cycle for the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy produce better accuracies than the f ​ i ​ x ​ e ​ d fixed policy. The final accuracy shows an improvement from the network trained by the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy (Table 1 ) to be 1.4 % 1.4\% better than the accuracy from the f ​ i ​ x ​ e ​ d fixed policy. This demonstrates that the t ​ r ​ i ​ a ​ n ​ g ​ u ​ l ​ a ​ r ​ 2 triangular2 policy improves on a “best guess” for a fixed learning rate.

Figure 13 compares the results of running with the e ​ x ​ p exp versus the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy with g ​ a ​ m ​ m ​ a = 0.99998 gamma=0.99998 . Once again, the peaks at the end of each cycle for the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy produce better validation accuracies than the e ​ x ​ p exp policy. The final accuracy from the e ​ x ​ p ​ _ ​ r ​ a ​ n ​ g ​ e exp\_range policy (Table 1 ) is 2% better than from the e ​ x ​ p exp policy.

## 5 Conclusions

The results presented in this paper demonstrate the benefits of the cyclic learning rate (CLR) methods. A short run of only a few epochs where the learning rate linearly increases is sufficient to estimate boundary learning rates for the CLR policies. Then a policy where the learning rate cyclically varies between these bounds is sufficient to obtain near optimal classification results, often with fewer iterations. This policy is easy to implement and unlike adaptive learning rate methods, incurs essentially no additional computational expense.

This paper shows that use of cyclic functions as a learning rate policy provides substantial improvements in performance for a range of architectures. In addition, the cyclic nature of these methods provides guidance as to times to drop the learning rate values (after 3 - 5 cycles) and when to stop the the training. All of these factors reduce the guesswork in setting the learning rates and make these methods practical tools for everyone who trains neural networks.