Scaling Laws for Neural Language Models — part 5/5

Cleaned-text code points [61257, 70566). Bibliography omitted here; source.md includes it.

We can solve A.1 to find an expression for the amount of compute needed to reach a given value of the loss L L with a model of size N N : C ( N , L ) = ( 6 B ∗ S c N L 1 / α B ) ( L − ( N c N ) α N ) − 1 / α S . C\left(N,L\right)=\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}}\right)\left(L-\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}\right)^{-1/\alpha_{S}}. (B.15) Using A.6 and A.9, we can eliminate L L in favor of N eff ​ ( L ) N_{{\rm eff}}\left(L\right) , the model size which reaches L L most efficiently. From there, we find an expression for the excess compute needed as a consequence of using a suboptimal model size: C ⁡ ( N , N eff ) C ⁡ ( N eff , N eff ) = N N eff [ 1 + α S α N ( 1 − ( N eff N ) α N ) ] − 1 / α S . \frac{C\left(N,N_{{\rm eff}}\right)}{C\left(N_{{\rm eff}},N_{{\rm eff}}\right)}=\frac{N}{N_{{\rm eff}}}\left[1+\frac{\alpha_{S}}{\alpha_{N}}\left(1-\left(\frac{N_{{\rm eff}}}{N}\right)^{\alpha_{N}}\right)\right]^{-1/\alpha_{S}}. (B.16) The result is shown in Figure X. Models between 0.6x and 2.2x the optimal size can be used with only a 20% increase in compute budget. Using a smaller model is useful when accounting for the cost inference. A larger model can be trained the the same level of performance in fewer steps, allowing for more parallelism and faster training if sufficient harware is available (see Figure Y): S ⁡ ( N , N eff ) S ⁡ ( N eff , N eff ) = [ 1 + α S α N ( 1 − ( N eff N ) α N ) ] − 1 / α S . \frac{S\left(N,N_{{\rm eff}}\right)}{S\left(N_{{\rm eff}},N_{{\rm eff}}\right)}=\left[1+\frac{\alpha_{S}}{\alpha_{N}}\left(1-\left(\frac{N_{{\rm eff}}}{N}\right)^{\alpha_{N}}\right)\right]^{-1/\alpha_{S}}. (B.17) A 2.2x larger model requires 45% fewer steps at a cost of 20% more training compute. Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve after initial transient effects.

## Appendix C Caveats

In this section we list some potential caveats to our analysis.

• At present we do not have a solid theoretical understanding for any of our proposed scaling laws. The scaling relations with model size and compute are especially mysterious. It may be possible to understand scaling at very large D D holding model size fixed [ AS17 ] , and also the shape of learning curves late in training, by modeling the loss with a noisy quadratic. But the scaling with D D at very large model size still remains mysterious. Without a theory or a systematic understanding of the corrections to our scaling laws, it’s difficult to determine in what circumstances they can be trusted.

• We are not especially confident in the prediction of B crit ​ ( L ) B_{\rm crit}(L) for values of the loss far outside the range we have explored. Changes in B crit B_{\rm crit} could have a significant impact on trade-offs between data parallelism and the number of serial training steps required, which would have a major impact on training time.

• We did not thoroughly investigate the small data regime, and our fits for L ⁡ ( N , D ) L(N,D) were poor for the smallest values of D D (where an epoch corresponded to only 40 40 steps). Furthermore, we did not experiment with regularization and data augmentation. Improvements in these could alter our results, quantitatively or qualitatively.

• We used the estimated training compute C ≈ 6 ​ N ​ B ​ S C\approx 6NBS , which did not include contributions proportional to n ctx n_{\rm ctx} (see Section 2.1 ). So our scalings with compute may be confounded in practice in the regime of very large n ctx n_{\rm ctx} , specifically where n ctx ≳ 12 ​ d model n_{\rm ctx}\gtrsim 12d_{\rm model} .

• We tuned learning rates, and we experimented with learning rate schedules. But we may have neglected to tune some hyperparameter (e.g. intialization scale or momentum) that have an important effect on scaling.

• The optimal choice of learning rate is sensitive to the target loss. When training close to convergence, it may be necessary to use a smaller learning rate to avoid divergences. But when conducting a short training run (eg due to compute limitations), it may be possible to use a larger learning rate. We did not experiment with higher learning rates for training runs that did not proceed to convergence.

## Appendix D Supplemental Figures

### D.1 Early Stopping and Test vs Train

In section 5.3 we described the result shown in Figure 16 , which provides a prediction for a lower bound on the early stopping step. We also show the train and test loss for a given model size when training on different sized datasets.

### D.2 Universal Transformers

We compare the performance of standard Transformers to recurrent Transformers [ DGV+18 ] in Figure 17 . These models re-use parameters, and so perform slightly better as a function of N N , but slightly worse as a function of compute C C . We include several different different possibilities for parameter re-use.

### D.3 Batch Size

We measure the critical batch size using the data displayed in figure 18 . This made it possible to estimate B crit ​ ( L ) B_{\rm crit}(L) in figure 10 .

### D.4 Sample Efficiency vs Model Size

It is easy to see from figure 2 that larger models train faster, and are therefore more sample efficient. We provide another way of looking at this phenomenon in figure 19 , which shows when different models reach various fixed values of the loss.

### D.5 Context Dependence

The trends for loss as a function of model size are displayed for different tokens in the context in Figure 21 . We see that models trained on n ctx = 1024 n_{\rm ctx}=1024 show steady improvement with model size on all but the first token.

Fixing model size, it appears that the loss scales as a power-law as a function of position T T in the context, see Figure 20 . This may be a consequence of underlying power-law correlations in language [ EP94 , ACDE12 , LT16 ] , or a more general feature of the model architecture and optimization. It provides some suggestion for the potential benefits (or lack thereof) from training on larger contexts. Not only do larger models converge to better performance at T = 1024 T=1024 , but they also improve more quickly at early tokens, suggesting that larger models are more efficient at detecting patterns with less contextual information. In the right-hand plot we show how per-token performance varies for a fixed model as a function of the training step. The model begins by learning short-range information, and only learns longer-range correlations later in training.

We have also included models trained with a tiny context n ctx = 8 n_{\rm ctx}=8 in order to compare with our longer context models. Even modestly sized models trained on n ctx = 8 n_{\rm ctx}=8 can dominate our largest n ctx = 1024 n_{\rm ctx}=1024 models on very early tokens. This also suggests that further improvements should be possible with much larger models trained on large contexts.

### D.6 Learning Rate Schedules and Error Analysis

We experimented with a variety of learning rates and schedules. A host of schedules and resulting test performances for a small language model are plotted in Figure 22 . We conclude that the choice of learning rate schedule is mostly irrelevant, as long as the total summed learning rate is sufficiently large, and the schedule includes a warmup period and a final decay to near-vanishing learning rate. Variations among schedules appear to be statistical noise, and provide a rough gauge for the scale of variation between different training runs. Experiments on larger models suggest that the variation in the final test loss between different random seeds is roughly constant in magnitude for different model sizes.

We found that larger models require a smaller learning rate to prevent divergence, while smaller models can tolerate a larger learning rate. To implement this, the following rule of thumb was used for most runs: LR ⁡ ( N ) ≈ 0.003239 + − 0.0001395 ​ log ⁡ ( N ) \mathrm{LR}(N)\approx 0.003239+-0.0001395\log(N) (D.1) We expect that this formula could be improved. There may be a dependence on network width, likely set by the initialization scale. The formula also breaks down for N > 10 10 N>10^{10} parameters. Nevertheless, we found that it works sufficiently well for the models we considered.

### D.7 Fit Details and Power Law Quality

We experimented with a number of functional forms for the fits to L ⁡ ( N ) , L ⁡ ( C ) L(N),L(C) , and L ⁡ ( D ) L(D) ; the power-law fits were qualitatively much more accurate than other functions such as logarithms (see Figure 23 ).

For L ⁡ ( C ) L(C) , we do not include small models with only 1 layer in the fit, as the transition from 1 to 2 layers causes a noticable lump in the data. For L ⁡ ( N ) L(N) we also do not include very small models with only 1 layer in the fit, and we exclude the largest models that have not trained fully to convergence. Fit parameters change marginally if we do include them, and the trend extrapolates well in both directions regardless.

### D.8 Generalization and Architecture

In figure 24 we show that generalization to other data distributions does not depend on network depth when we hold the total parameter count fixed. It seems to depend only on the performance on the training distribution.