MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters

arxiv:2402.02342

Original source · Paper JSON

Data, training, optimization · 2024-02-04

Cites (4) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10

Contents

• MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
• Abstract
• Keywords:
• 1 Introduction
• 2 Problem Setting
• 3 Forward and Backward Views
• 4 MetaOptimize
• Remark 4.1 .
• 5 Reducing Complexity
• 2 × \times 2 approximation.
• L-approximation.
• Intuition.
• Containing some prior methods as special cases.
• 6 Hessian-Free MetaOptimize
• 7 Experiments
• 7.1 CIFAR10 dataset
• 7.2 Non-stationary CIFAR100
• 7.3 ImageNet dataset
• 7.4 Language modeling
• 7.5 Sensitivity analysis
• 8 Related Works
• 9 Limitations and Future Works
• Impact Statement
• Appendix A Step-size Optimization for Different Choices of Base and Meta updates
• A.1 Derivation of G meta G^{\textrm{meta}} for Different Meta Updates
• A.1.1 Meta SGD
• A.1.2 Meta Adam
• A.1.3 Meta Lion
• A.2 Derivation of G base G^{\textrm{base}} for Different Base Updates
• A.3 Base SGD
• A.3.1 Base AdamW
• A.3.2 Base Lion
• Appendix B Exiting Step-size Optimization Algorithms as Special Cases of MetaOptimize
• B.1 IDBD and Its Extensions
• B.2 Hyper-gradient Descent
• Appendix C Experiment Details
• Appendix D Further Experimental Results
• Instructions for reporting errors