Scaling Laws for Neural Language Models

arxiv:2001.08361

Original source · Paper JSON

Data, training, optimization · 2020-01-23

Cites (1) · Cited by (42)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Scaling Laws for Neural Language Models
• Abstract
• 1 Introduction
• 1.1 Summary
• Performance depends strongly on scale, weakly on model shape:
• Smooth power laws:
• Universality of overfitting:
• Universality of training:
• Transfer improves with test performance:
• Sample efficiency:
• Convergence is inefficient:
• Optimal batch size:
• 1.2 Summary of Scaling Laws
• 1.3 Notation
• 2 Background and Methods
• 2.1 Parameter and Compute Scaling of Transformers
• 2.2 Training Procedures
• 2.3 Datasets
• 3 Empirical Results and Basic Power Laws
• 3.1 Approximate Transformer Shape and Hyperparameter Independence
• 3.2 Performance with Non-Embedding Parameter Count N N
• 3.2.1 Comparing to LSTMs and Universal Transformers
• 3.2.2 Generalization Among Data Distributions
• 3.3 Performance with Dataset Size and Compute
• 4 Charting the Infinite Data Limit and Overfitting
• 4.1 Proposed L ⁡ ( N , D ) L(N,D) Equation
• 4.2 Results
• 5 Scaling Laws with Model Size and Training Time
• 5.1 Adjustment for Training at B crit ​ ( L ) B_{\rm crit}(L)
• 5.2 Results for L ⁡ ( N , S min ) L(N,S_{\rm min}) and Performance with Model Size and Compute
• 5.3 Lower Bound on Early Stopping Step
• 6 Optimal Allocation of the Compute Budget
• 6.1 Optimal Performance and Allocations
• 6.2 Predictions from L ⁡ ( N , S min ) L(N,S_{\rm min})
• 6.3 Contradictions and a Conjecture
• 7 Related Work
• 8 Discussion
• Acknowledgements
• Appendix A Summary of Power Laws
• Appendix B Empirical Model of Compute-Efficient Frontier
• B.1 Defining Equations
• B.2 Efficient Training
• B.3 Comparison to Inefficient
• B.4 Suboptimal Model Sizes
• Appendix C Caveats
• Appendix D Supplemental Figures
• D.1 Early Stopping and Test vs Train
• D.2 Universal Transformers
• D.3 Batch Size
• D.4 Sample Efficiency vs Model Size
• D.5 Context Dependence
• D.6 Learning Rate Schedules and Error Analysis
• D.7 Fit Details and Power Law Quality
• D.8 Generalization and Architecture