Grokfast: Accelerated Grokking by Amplifying Slow Gradients

arxiv:2405.20233

Original source · Paper JSON

Data, training, optimization · 2024-05-30

Cites (1) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Grokfast : Accelerated Grokking by Amplifying Slow Gradients
• Abstract
• 1 Introduction
• 2 Amplifying the Low-Frequencies of the Stochastic Gradients
• 2.1 Filter Design
• 2.2 Experiment
• 2.3 Discussion
• Q1. Are both slow and fast gradients necessary?
• Q2. Exploiting state transition in the training of a model under grokking.
• Q3. Synergistic effect with weight decay.
• 2.4 Limitations
• 3 Grokfast with Exponential Moving Average Gradient Filter
• 4 Experiment
• 4.1 Algorithmic Data
• 4.2 MNIST
• 4.3 QM9
• 4.4 IMDb
• 5 More Discussion
• 5.1 Difference between Algorithm 2 and the Momentum in Typical Optimizers
• 5.2 Visualizing Trajectories
• 6 Related Work
• Grokking.
• Optimization techniques.
• 7 Conclusion
• Appendix A Frequency Responses of the Parameter Updates under Grokfast
• Theorem A.1 .
• Proof of Theorem A.1 .
• Proposition A.2 (SGD with momentum) .
• Proof of Proposition A.2 .
• Proposition A.3 (SGD with Nesterov’s momentum) .
• Proof of Proposition A.3 .
• Appendix B Task Details
• B.1 Binary Operation (Algorithmic Data)
• B.2 MNIST
• B.3 QM9
• B.4 IMDb
• Appendix C Autograd Implementation
• Appendix D Time and Memory Requirements
• Appendix E More Visualization
• Instructions for reporting errors