Scaling MLPs: A Tale of Inductive Bias — part 4/4

Cleaned-text code points [43904, 46230). Bibliography omitted here; source.md includes it.

We further compare against models that were pre-trained on the full ImageNet21k, with and without data augmentation in Fig. 12 . We observe that even though we provide the model with an abundance of samples, the weights still remain largely structure-less and have not developped any locality properties. On the other hand, using data augmentation leads to more adapted filters.

## Appendix D Inverted Bottleneck MLP Code

We provide PyTorch-style pseudo-code for the inverted bottleneck MLP to highlight its simplicity. ⬇ 1 from torch import nn 2 3 class Block ( nn . Module ): 4 def __init__ ( self , dim , expansion_factor =4, dropout =0.): 5 super (). __init__ () 6 self . fn = nn . Sequential ( 7 nn . Linear ( dim , int ( expansion_factor * dim )), 8 nn . GELU (), 9 nn . Dropout ( dropout ), 10 nn . Linear ( int ( expansion_factor * dim ), dim ), 11 nn . Dropout ( dropout ) 12 ) 13 self . ln = nn . LayerNorm ( dim ) 14 15 def forward ( self , x ): 16 return x + self . fn ( self . ln ( x )) 17 18 19 def MLP ( image_size , channels , dim , depth , num_classes , expansion_factor =4, dropout =0.): 20 return nn . Sequential ( 21 nn . Flatten ( start_dim =1, end_dim =-1), 22 nn . Linear ( image_size * image_size * channels , dim ), 23 *[ Block ( dim , expansion_factor , dropout ) for _ in range ( depth )], 24 nn . Linear ( dim , num_classes ) 25 )

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .