Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

arxiv:2405.18392

Original source · Paper JSON

Data, training, optimization · 2024-05-28

Cites (6) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
• Abstract
• 1 Introduction
• 2 Background: Cosine Learning Rate Schedule for LLMs
• 3 A Different Route: Constant Learning Rate with Cooldown
• 3.1 The Advantages
• 3.2 Experimental Comparison
• 3.3 Scaling Up: 1B and 8B Models
• 4 Do We Even Need to Cooldown?
• 4.1 Stochastic Weight Averaging (SWA)
• 4.2 Schedule-Free Optimizer (SFO)
• 5 The Implications for Scaling Law Research
• 6 Limitations
• 7 Related work
• 8 Conclusion
• Acknowledgements
• Appendix A Experimental Details
• A.1 Overview
• A.2 Setup for the 1B and 8B Runs
• A.3 FLOPs computation
• Appendix B Additional Results
• B.1 More Results on Cooldown
• B.2 Additional Results and Compute Savings
• B.3 Learning Curves for All Models of Scaling Experiments
• B.4 Additional Experiments on OpenWebText2
• B.5 Full Results of Large Model Runs
• Instructions for reporting errors