The Road Less Scheduled

arxiv:2405.15682

Original source · Paper JSON

Data, training, optimization · 2024-05-24

Cites (0) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• The Road Less Scheduled
• Abstract
• 1 Introduction
• 1.1 Summary of Results
• 1.2 Notation
• 2 Method
• Theorem 1 .
• 2.1 General Theory
• Theorem 2 .
• 2.2 On Large Learning Rates
• Theorem 3 .
• 2.3 How Schedule-Free Replaces a Schedule
• 3 Related Work
• 4 Experiments
• 4.1 Deep Learning Problems
• 4.2 MLCommons Algorithmic Efficiency benchmark
• 4.3 Convex Problems
• 4.4 Implementation Concerns
• 5 Parameter Sensitivity
• 6 Conclusion
• 7 Contributions
• Appendix A Proof of Theorem 2
• Proof.
• Appendix B Recovering Prior Conversions, and Connections to Momentum
• Theorem 4 .
• Proof.
• B.1 Recovering Linear Decay
• Appendix C Generalizing Theorem 2 via Bregman Divergences
• Theorem 5 .
• Proof.
• Appendix D Acceleration
• Corollary 1 .
• Proof.
• Corollary 2 .
• Proof.
• D.1 An Optimistic Regret Bound
• Lemma 1 .
• Proof.
• Appendix E Strongly Convex Losses
• Theorem 6 .
• Proof.
• Appendix F Large Step size convergence
• Proof.
• Appendix G Experimental Setup
• G.1 Convex experiments
• G.2 CIFAR-10
• G.3 CIFAR-100
• G.4 SVHN
• G.5 ImageNet
• G.6 IWSLT14
• G.7 NanoGPT
• G.8 MAE
• G.9 DLRM
• G.10 MRI
• G.11 Algoperf
• Appendix H Polyak and Primal Averaging Runs
• Appendix I Additional LR Sensitivity Plots
• Instructions for reporting errors