Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

arxiv:1904.00962

Original source · Paper JSON

Data, training, optimization · 2019-04-01

Cites (0) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Abstract
• 1 Introduction
• 1.1 Related Work
• 2 Preliminaries
• Notation.
• Theorem 1 ( ( Ghadimi & Lan, 2013b ) ) .
• 3 Algorithms
• 3.1 Lars Algorithm
• Theorem 2 .
• 3.2 Lamb Algorithm
• Theorem 3 .
• 4 Experiments
• 4.1 Bert Training
• Comparison with AdamW and Lars .
• 4.2 ImageNet Training with ResNet-50.
• 4.3 Hyperparameters for scaling the batch size
• 5 Conclusion
• 6 Acknowledgement
• Appendix
• Appendix A Proof of Theorem 2
• Proof.
• Appendix B Proof of Theorem 3
• Proof.
• Appendix C Comparison of Convergence Rates of Lars and Sgd
• Appendix D N-LAMB: Nesterov Momentum for LAMB
• Appendix E LAMB with learning rate correction
• Appendix F LAMB with different norms
• Appendix G Regular Batch Sizes for Small Datasets: MNIST and CIFAR-10.
• Appendix H Implementation Details and Additional Results
• H.0.1 BERT
• H.0.2 ImageNet
• H.1 Baseline tuning details for ImageNet training with ResNet-50
• Instructions for reporting errors