Symbolic Discovery of Optimization Algorithms

arxiv:2302.06675

Original source · Paper JSON

Data, training, optimization · 2023-02-13

Cites (1) · Cited by (12)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Symbolic Discovery of Optimization Algorithms
• Abstract
• 1 Introduction
• 2 Symbolic Discovery of Algorithms
• 2.1 Program Search Space
• 2.2 Efficient Search Techniques
• 2.3 Generalization: Program Selection and Simplification
• 3 Derivation and Analysis of Lion
• 3.1 Derivation
• 3.2 Analysis
• 4 Evaluation of Lion
• 4.1 Image Classification
• 4.2 Vision-Language Contrastive Learning
• 4.3 Diffusion Model
• 4.4 Language Modeling and Fine-tuning
• 4.5 Comparison with Other Popular Optimizers
• 4.6 Ablations
• 5 Hyperparameter Tuning
• 6 Limitations
• 7 Related Work
• 8 Conclusion
• Acknowledgements
• Appendix A Pseudocode for AdamW and Lion
• Appendix B Image Classification Tasks
• Appendix C NLP Tasks
• Appendix D Other Discovered Programs
• Appendix E Architecture Details for Language Modeling
• Appendix F Details of Proxy Tasks
• Appendix G Analysis of Loss Landscape
• Appendix H Available Functions
• Appendix I Abstract Execution
• Instructions for reporting errors