Scaling Laws for Neural Language Models — part 4/5

Cleaned-text code points [46825, 61257). Bibliography omitted here; source.md includes it.

One might also conjecture that this intersection point has a deeper meaning. If we cannot increase the model size beyond N ∗ N^{*} without qualitatively different data requirements, perhaps this means that once we reach C min ∗ C_{\rm min}^{*} and N ∗ N^{*} , we have extracted all of the reliable information available in natural language data. In this interpretation, L ∗ L^{*} would provide a rough estimate for the entropy-per-token 6 6 6 Defining words using the wc utility, the WebText2 dataset has 1.4 1.4 tokens per word and 4.3 4.3 characters per token. of natural language. In this scenario, we would expect the loss trend to level off at or before L ∗ L^{*} .

We can guess at the functional form of L ⁡ ( C min ) L(C_{\rm min}) as it levels off by considering a version of our training dataset with added noise. For example, we could append a random string of tokens to each context shown to the model to artificially boost the loss by a constant additive factor. Then, the distance from the noise floor L − L noise L-L_{\rm noise} would be a more meaningful performance metric, with even a small decrease in this distance potentially representing a significant boost in qualitative performance. Since the artificial noise would affect all of our trends equally, the critical point of 6.8 would not change (aside from the absolute value of L ∗ L^{*} ), and may be meaningful even if it occurs after the leveling off.

## 7 Related Work

Power laws can arise from a wide variety of sources [ THK18 ] . Power-law scalings with model and dataset size in density estimation [ Was06 ] and in random forest models [ Bia12 ] may be connected with our results. These models suggest that power-law exponents may have a very rough interpretation as the inverse of the number of relevant features in the data.

Some early [ BB01 , Goo01 ] work found power-law scalings between performance and dataset size. More recent work [ HNA+17 , HAD19 ] also investigated scaling between model size and data size; their work is perhaps the closest to ours in the literature 7 7 7 After this work was completed, [ RRBS19a ] also appeared, which makes similar predictions for the dependence of loss on both model and dataset size. . Note, however, that [ HNA+17 ] found super-linear scaling of dataset size with model size, whereas we find a sub-linear scaling. There are some parallels between our findings on optimal allocation of compute and [ Kom19 ] , including power-law learning curves. EfficientNets [ TL19 ] also appear to obey an approximate power-law relation between accuracy and model size. Very recent work [ RRBS19b ] studies scaling with both dataset size and model size for a variety of datasets, and fits an ansatz similar to ours.

EfficientNet [ TL19 ] advocates scaling depth and width exponentially (with different coefficients) for optimal performance of image models, resulting in a power-law scaling of width as a function of depth. We find that for language models this power should be roughly one when scaling up (as width/depth should remain fixed). But more importantly, we find that the precise architectural hyperparameters are unimportant compared to the overall scale of the language model. In [ VWB16 ] it was argued that deep models can function as ensembles of shallower models, which could potentially explain this finding. Earlier work [ ZK16 ] has compared width and depth, and found that wide ResNets can outperform deep ResNets on image classification. Some studies fix computation per data example, which tends to scale in proportion to the number of model parameters, whereas we investigate scaling with both model size and the quantity of training computation.

Various works [ AS17 , BHMM18 ] have investigated generalization in highly overparameterized models, finding a “jamming transition” [ GJS+19 ] when the model size reaches the dataset size (this may require training many orders of magnitude beyond typical practice, and in particular does not use early stopping). We do not observe such a transition, and find that the necessary training data scales sublinearly in the model size. Expansions in the model size, particularly at large width [ JGH18 , LXS+19 ] , may provide a useful framework for thinking about some of our scaling relations. Our results on optimization, such as the shape of learning curves, can likely be explained using a noisy quadratic model, which can provide quite accurate predictions [ ZLN+19 ] in realistic settings. Making this connection quantitative will require a characterization of the Hessian spectrum [ Pap18 , GKX19 , GARD18 ] .

## 8 Discussion

We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N N , dataset size D D , and optimized training computation C min C_{\rm min} , as encapsulated in Equations ( 1.5 ) and ( 1.6 ). Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Since scalings with N , D , C min N,D,C_{\rm min} are power-laws, there are diminishing returns with increasing scale.

We were able to precisely model the dependence of the loss on N N and D D , and alternatively on N N and S S , when these parameters are varied simultaneously. We used these relations to derive the compute scaling, magnitude of overfitting, early stopping step, and data requirements when training large language models. So our scaling relations go beyond mere observation to provide a predictive framework. One might interpret these relations as analogues of the ideal gas law, which relates the macroscopic properties of a gas in a universal way, independent of most of the details of its microscopic consituents.

It is natural to conjecture that the scaling relations will apply to other generative modeling tasks with a maximum likelihood loss, and perhaps in other settings as well. To this purpose, it will be interesting to test these relations on other domains, such as images, audio, and video models, and perhaps also for random network distillation. At this point we do not know which of our results depend on the structure of natural language data, and which are universal. It would also be exciting to find a theoretical framework from which the scaling relations can be derived: a ‘statistical mechanics’ underlying the ‘thermodynamics’ we have observed. Such a theory might make it possible to derive other more precise predictions, and provide a systematic understanding of the limitations of the scaling laws.

In the domain of natural language, it will be important to investigate whether continued improvement on the loss translates into improvement on relevant language tasks. Smooth quantitative change can mask major qualitative improvements: “more is different”. For example, the smooth aggregate growth of the economy provides no indication of the specific technological developments that underwrite it. Similarly, the smooth improvements in language model loss may hide seemingly qualitative changes in capability.

Our results strongly suggest that larger models will continue to perform better, and will also be much more sample efficient than has been previously appreciated. Big models may be more important than big data. In this context, further investigation into model parallelism is warranted. Deep models can be trained using pipelining [ HCC+18 ] , which splits parameters depth-wise between devices, but eventually requires increased batch sizes as more devices are used. Wide networks on the other hand are more amenable to parallelization [ SCP+18 ] , since large layers can be split between multiple workers with less serial dependency. Sparsity [ CGRS19 , GRK17 ] or branching (e.g. [ KSH12 ] ) may allow for even faster training of large networks through increased model parallelism. And using methods like [ WRH17 , WYL19 ] , which grow networks as they train, it might be possible to remain on the compute-efficient frontier for an entire training run.

## Acknowledgements

We would like to thank Shan Carter, Paul Christiano, Jack Clark, Ajeya Cotra, Ethan Dyer, Jason Eisner, Danny Hernandez, Jacob Hilton, Brice Menard, Chris Olah, and Ilya Sutskever for discussions and for feedback on drafts of this work.

## Appendix A Summary of Power Laws

For easier reference, we provide a summary below of the key trends described throughout the paper.

The empirical fitted values for these trends are:

The optimal parameters for compute efficient training are given by:

## Appendix B Empirical Model of Compute-Efficient Frontier

Throughout this appendix all values of C , S , C,S, and α C \alpha_{C} are adjusted for training at the critical batch size B crit B_{\rm crit} . We have left off the ‘adj’ label to avoid cluttering the notation.

### B.1 Defining Equations

The power-law fit to the learning curves implies a simple prescription for compute-efficient training. In this appendix, we will derive the optimal performance, model size, and number of training steps as a function of the compute budget. We start with the Equation ( 1.6 ), repeated here for convenience: L ⁡ ( N , S ) = ( N c N ) α N + ( S c S ) α S . L\left(N,S\right)=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{c}}{S}\right)^{\alpha_{S}}. (B.1) Here, S S represents the number of parameter updates when training at the critical batch size [ MKAT18 ] , which was defined in Equation ( 5.2 ) 8 8 8 There is a slight ambiguity here: we can imagine training either at a constant batch size B ⁡ ( L target ) B\left(L_{{\rm target}}\right) , or we could instead train at a variable batch size B ~ ​ ( L ) \tilde{B}\left(L\right) , where B ~ \tilde{B} is the instantaneous critical batch size (as opposed to B B , which is the averaged version). These two prescriptions result in the same number of steps, so we can ignore this subtlety (see [ MKAT18 ] ). : B ⁡ ( L ) = B ∗ L 1 / α B . B\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}}. (B.2) We would like to determine optimal training parameters for a fixed compute budget, so we replace S = C / ( 6 ​ N ​ B ​ ( L ) ) S=C/\left(6NB\left(L\right)\right) , where C C is the number of FLOPs used in the training run: L ⁡ ( N , C ) = ( N c N ) α N + ( 6 ​ B ∗ ​ S c ​ N L 1 / α B ​ C ) α S . L\left(N,C\right)=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}C}\right)^{\alpha_{S}}. (B.3) Now, we set ∂ N L | C = 0 \partial_{N}L\big|_{C}=0 to find the condition for optimality: 0 \displaystyle 0 = ∂ L ∂ N | C \displaystyle=\frac{\partial L}{\partial N}\big|_{C} = − α N N ​ ( N c N ) α N + α S N ​ ( 6 ​ B ∗ ​ S c ​ N L 1 / α B ​ C ) α S ​ ( 1 − 5 ​ N L ​ ∂ L ∂ N | C ) \displaystyle=-\frac{\alpha_{N}}{N}\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\frac{\alpha_{S}}{N}\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}C}\right)^{\alpha_{S}}\left(1-5\frac{N}{L}\cancel{\frac{\partial L}{\partial N}\big|_{C}}\right) ⟹ α N α S ​ ( N c N ) α N \displaystyle\implies\frac{\alpha_{N}}{\alpha_{S}}\left(\frac{N_{c}}{N}\right)^{\alpha_{N}} = ( 6 ​ B ∗ ​ S c ​ N L 1 / α B ​ C ) α S \displaystyle=\left(6B_{\ast}S_{c}\frac{N}{L^{1/\alpha_{B}}C}\right)^{\alpha_{S}} (B.4) Equation ( B.3 ) and ( B.4 ) together determine the compute-efficient frontier.

### B.2 Efficient Training

Now we assemble the implications of ( B.3 ) and ( B.4 ). First, note that inserting ( B.4 ) into ( B.3 ) yields L ⁡ ( N eff ​ ( C ) , C ) = ( 1 + α N α S ) ​ L ​ ( N eff , ∞ ) , L\left(N_{{\rm eff}}\left(C\right),C\right)=\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)L\left(N_{{\rm eff}},\infty\right), (B.5) which implies that for compute-efficient training, we should train to a fixed percentage α N α S ≈ 10 % \frac{\alpha_{N}}{\alpha_{S}}\approx 10\% above the converged loss. Next, let’s determine how the optimal loss depends on the compute budget. Eliminating N N yields a power-law dependence of performance on compute: L ⁡ ( C ) \displaystyle L\left(C\right) = ( C c C ) α C \displaystyle=\left(\frac{C_{c}}{C}\right)^{\alpha_{C}} (B.6) where we defined α C \displaystyle\alpha_{C} = 1 / ( 1 / α S + 1 / α B + 1 / α N ) ≈ 0.052 \displaystyle=1/\left(1/\alpha_{S}+1/\alpha_{B}+1/\alpha_{N}\right)\approx 0.052 (B.7) C c \displaystyle C_{c} = 6 ​ N c ​ B ∗ ​ S c ​ ( 1 + α N α S ) 1 / α S + 1 / α N ​ ( α S α N ) 1 / α S . \displaystyle=6N_{c}B_{\ast}S_{c}\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)^{1/\alpha_{S}+1/\alpha_{N}}\left(\frac{\alpha_{S}}{\alpha_{N}}\right)^{1/\alpha_{S}}. (B.8) Similarly, we can eliminate L L to find N ⁡ ( C ) N\left(C\right) : N ⁡ ( C ) N c = ( C C c ) α C / α N ​ ( 1 + α N α S ) 1 / α N \displaystyle\frac{N\left(C\right)}{N_{c}}=\left(\frac{C}{C_{c}}\right)^{\alpha_{C}/\alpha_{N}}\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)^{1/\alpha_{N}} (B.9) and S ⁡ ( C ) \displaystyle S\left(C\right) = C c 6 ​ N c ​ B ∗ ( 1 + α N α S ) − 1 / α N ( C C c ) α C / α S \displaystyle=\frac{C_{c}}{6N_{c}B_{\ast}}\left(1+\frac{\alpha_{N}}{\alpha_{S}}\right)^{-1/\alpha_{N}}\left(\frac{C}{C_{c}}\right)^{\alpha_{C}/\alpha_{S}} (B.10)

### B.3 Comparison to Inefficient

Typically, researchers train models until they appear to be close to convergence. In this section, we compare the efficient training procedure described above to this more typical setup. We define a the convergence factor f f as the percent deviation from the converged loss: L ⁡ ( N , C ) = ( 1 + f ) ​ L ​ ( N , ∞ ) . L\left(N,C\right)=\left(1+f\right)L\left(N,\infty\right). (B.11) For compute-efficient training we have f = α N / α S ≈ 10 % f=\alpha_{N}/\alpha_{S}\approx 10\% from the previous section, but researchers typically use a much smaller value. Here, we choose f ′ = 2 % f^{\prime}=2\% as an estimate. For a fixed value of the loss, we predict: N f N f ′ \displaystyle\frac{N_{f}}{N_{f^{\prime}}} = ( 1 + f 1 + f ′ ) 1 / α N ≈ 2.7 \displaystyle=\left(\frac{1+f}{1+f^{\prime}}\right)^{1/\alpha_{N}}\approx 2.7 (B.12) S f S f ′ \displaystyle\frac{S_{f}}{S_{f^{\prime}}} = ( 1 + 1 f 1 + 1 f ′ ) 1 / α S ≈ 0.13 \displaystyle=\left(\frac{1+\frac{1}{f}}{1+\frac{1}{f^{\prime}}}\right)^{1/\alpha_{S}}\approx 0.13 (B.13) C f C f ′ \displaystyle\frac{C_{f}}{C_{f^{\prime}}} = N f N f ′ ​ S f S f ′ ≈ 0.35 \displaystyle=\frac{N_{f}}{N_{f^{\prime}}}\frac{S_{f}}{S_{f^{\prime}}}\approx 0.35 (B.14) So that compute-efficient training uses 7.7x fewer parameter updates, 2.7x more parameters, and 65% less compute to reach the same loss.

### B.4 Suboptimal Model Sizes