Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates

arxiv:1708.07120

Original source · Paper JSON

Data, training, optimization · 2017-08-23

Cites (1) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
• Abstract
• 1 Introduction
• 2 Background
• 3 Super-convergence
• 4 Estimating optimal learning rates
• 5 Experiments and analysis
• 5.1 Other datasets and architectures
• 5.2 Imagenet
• 6 Conclusion
• Appendix A Supplemental material
• A.1 Intuitive explanation for super-convergence
• A.2 Large learning rate regularization
• A.3 Relationship of super-convergence to SGD and generalization
• A.4 Hardware, software, and architectures
• A.5 Hyper-parameters and adaptive learning rate methods
• A.6 Experimental details and additional results
• Instructions for reporting errors