Scaling MLPs: A Tale of Inductive Bias

arxiv:2306.13575

Original source · Paper JSON

LLMs: architectures, context, training · 2023-06-23

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Scaling MLPs: A Tale of Inductive Bias
• Abstract
• 1 Introduction
• 2 Background
• Theoretical Works.
• MLPs.
• Convolutions.
• Vision Transformer.
• MLP-Mixer.
• Patchifiying.
• 3 Architecture
• Standard MLP.
• Inverted Bottleneck MLP.
• 4 Experiments
• 4.1 Setup
• 4.2 Training from Scratch
• 4.3 Transfer Learning
• Test-Time Augmentations.
• ReaL accuary.
• Large batch-sizes.
• Role of augmentations.
• 4.4 Scaling Laws
• Compute.
• Parameters or examples.
• 4.5 Computational Feasibility
• 5 Related Works
• 6 Discussion
• Appendix A Experimental Details
• A.1 Resources
• A.2 Additional Ablations
• Ablations.
• Normalization.
• Label smoothing.
• Architecture.
• Optimization.
• A.3 Linear Probing
• A.4 Scaling Laws
• Implementation Details.
• A.5 Computational Efficiency
• Appendix B Results for Standard MLPs
• B.1 Transfer Learning
• B.2 Scaling Laws
• Appendix C Weight visualizations
• Appendix D Inverted Bottleneck MLP Code
• Instructions for reporting errors