Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

arxiv:2305.14342

Original source · Paper JSON

Data, training, optimization · 2023-05-23

Cites (3) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7

Contents

• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• Abstract
• 1 Introduction
• 2 Method
• 2.1 Motivations
• 2.2 Sophia: S econd- o rder Cli p ped Stoc h astic Opt i miz a tion
• 2.3 Diagonal Hessian Estimators
• 3 Experiments
• 3.1 Experimental Setup
• Hyperparamter tuning strategy.
• 3.2 Evaluation
• Methodology for comparing the optimizers for LLMs.
• Technical details.
• 3.3 Results
• 3.4 Analysis
• 3.5 Ablation Study
• 4 Theoretical Analysis
• Assumption 4.1 .
• Assumption 4.2 .
• Theorem 4.3 .
• 5 Related work
• 6 Conclusion
• Acknowledgements
• Appendix A Additional Experiment Results
• Appendix B Additional Experiment Details
• B.1 Hyperparamter Tuning
• B.2 Model and Implementation Details
• B.3 Downtream Evaluation
• Appendix C Limitations
• Appendix D Theoretical Analyses: Details of Section 4
• Lemma D.1 .
• Proof of Lemma D.1 .
• Lemma D.2 .
• Proof of Lemma D.2 .
• Lemma D.3 .
• Proof of Lemma D.3 .
• Lemma D.4 .
• Proof of Lemma D.4 .
• Lemma D.5 .
• Proof of Lemma D.5 .
• Lemma D.6 .
• Proof of Lemma D.6 .
• Lemma D.7 .
• Proof of Lemma D.7 .
• Lemma D.8 .
• Proof of Lemma D.8 .
• Lemma D.9 .
• Proof of Lemma D.9 .
• Lemma D.10 (Descent Lemma) .
• Proof of Lemma D.10 .
• Lemma D.11 .
• Proof of Lemma D.11 .
• D.1 Lower bound for SignGD on 2-dimensional quadratic loss
• Theorem D.12 .
• Proof of Theorem D.12 .
• Instructions for reporting errors