Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

arxiv:2009.11243

Original source · Paper JSON

Data, training, optimization · 2020-09-23

Cites (0) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• Abstract
• 1 Introduction
• 2 Preliminaries
• 3 Methods: Addressing the three barriers to learned optimizers
• 3.1 Outer-Optimization
• 3.2 Task distributions
• 3.3 Optimizer architecture
• 4 Results
• 4.1 Comparing learned optimizer architectures and training task set sizes
• 4.2 Comparisons with hand-designed optimizers
• 4.3 Understanding optimizer behavior
• 4.4 Generalization along different task axes
• 4.5 Generalization to large-scale problems
• 4.6 Learned optimizers training themselves
• 5 Discussion
• Acknowledgements
• Broader Impact
• Appendix A Extended Related Work
• A.1 Parameterizations
• A.1.1 Controller based parameterization
• A.1.2 Symbolic parameterizations
• A.1.3 Continuous parameterizations
• A.2 Data distribution of tasks
• A.3 Outer-Optimization methods
• A.3.1 Hyper parameter optimization
• A.3.2 Reinforcement learning
• A.3.3 Neural Architecture Search Style
• A.3.4 Backpropogation / Gradient based
• A.3.5 Evolutionary Strategies
• Appendix B Outer Optimization Details
• Appendix C Learned Optimizer Architecture Details
• C.1 High level structure of the optimizer
• C.2 Utilities / components
• C.2.1 Rolling Features
• C.2.2 LossAccum
• C.2.3 RollingClipState
• C.3 Learned optimizer specific / putting it all together
• C.4 “ThetaMod”: The learned optimizer outer-parameters
• Appendix D Adam8p
• Appendix E Performance Table
• Appendix F Experimental details for figures
• F.1 Outer training of different learned architectures
• F.2 Outer training with different sized models
• F.3 Large scale transfer with Resnet models
• CIFAR-10 Resnet
• Imagenet Resnet
• F.4 Self optimization details
• Appendix G Self Optimization Extended
• Appendix H Infrastructure Used
• H.1 Outer-training cluster
• Workers
• Gradient Storage
• Learner
• Summary aggregators
• Evaluations:
• Evaluation Chief:
• Evaluation workers:
• H.2 Computational expense
• Instructions for reporting errors