Scaling Laws for Neural Language Models — part 1/5

Cleaned-text code points [0, 15528). Bibliography omitted here; source.md includes it.

# Scaling Laws for Neural Language Models

###### Abstract

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

## 1 Introduction

Language provides a natural domain for the study of artificial intelligence, as the vast majority of reasoning tasks can be efficiently expressed and evaluated in language, and the world’s text provides a wealth of data for unsupervised learning via generative modeling. Deep learning has recently seen rapid progress in language modeling, with state of the art models [ RNSS18 , DCLT18 , YDY+19 , LOG+19 , RSR+19 ] approaching human-level performance on many specific tasks [ WPN+19 ] , including the composition of coherent multi-paragraph prompted text samples [ RWC+19 ] .

One might expect language modeling performance to depend on model architecture, the size of neural models, the computing power used to train them, and the data available for this training process. In this work we will empirically investigate the dependence of language modeling loss on all of these factors, focusing on the Transformer architecture [ VSP+17 , LSP+18 ] . The high ceiling and low floor for performance on language tasks allows us to study trends over more than seven orders of magnitude in scale.

Throughout we will observe precise power-law scalings for performance as a function of training time, context length, dataset size, model size, and compute budget.

### 1.1 Summary

Our key findings for Transformer language models are are as follows:

##### Performance depends strongly on scale, weakly on model shape:

Model performance depends most strongly on scale, which consists of three factors: the number of model parameters N N (excluding embeddings), the size of the dataset D D , and the amount of compute C C used for training. Within reasonable limits, performance depends very weakly on other architectural hyperparameters such as depth vs. width. (Section 3 )

##### Smooth power laws:

Performance has a power-law relationship with each of the three scale factors N , D , C N,D,C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1 ). We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3 )

##### Universality of overfitting:

Performance improves predictably as long as we scale up N N and D D in tandem, but enters a regime of diminishing returns if either N N or D D is held fixed while the other increases. The performance penalty depends predictably on the ratio N 0.74 / D N^{0.74}/D , meaning that every time we increase the model size 8x, we only need to increase the data by roughly 5x to avoid a penalty. (Section 4 )

##### Universality of training:

Training curves follow predictable power-laws whose parameters are roughly independent of the model size. By extrapolating the early part of a training curve, we can roughly predict the loss that would be achieved if we trained for much longer. (Section 5 )

##### Transfer improves with test performance:

When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on the training validation set with a roughly constant offset in the loss – in other words, transfer to a different distribution incurs a constant penalty but otherwise improves roughly in line with performance on the training set. (Section 3.2.2 )

##### Sample efficiency:

Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2 ) and using fewer data points (Figure 4 ).

##### Convergence is inefficient:

When working within a fixed compute budget C C but without any other restrictions on the model size N N or available data D D , we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3 ). Maximally compute-efficient training would therefore be far more sample efficient than one might expect based on training small models to convergence, with data requirements growing very slowly as D ∼ C 0.27 D\sim C^{0.27} with training compute. (Section 6 )

##### Optimal batch size:

The ideal batch size for training these models is roughly a power of the loss only, and continues to be determinable by measuring the gradient noise scale [ MKAT18 ] ; it is roughly 1-2 million tokens at convergence for the largest models we can train. (Section 5.1 )

Taken together, these results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute. We expect that larger language models will perform better and be more sample efficient than current models.

### 1.2 Summary of Scaling Laws

The test loss of a Transformer trained to autoregressively model language can be predicted using a power-law when performance is limited by only either the number of non-embedding parameters N N , the dataset size D D , or the optimally allocated compute budget C min C_{\rm min} (see Figure 1 ): 1. For models with a limited number of parameters, trained to convergence on sufficiently large datasets: L ⁡ ( N ) = ( N c / N ) α N ; α N ∼ 0.076 , N c ∼ 8.8 × 10 13 ​ (non-embedding parameters) L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\penalty\ \penalty\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\penalty\ \text{(non-embedding parameters)} (1.1)

2. For large models trained with a limited dataset with early stopping: L ⁡ ( D ) = ( D c / D ) α D ; α D ∼ 0.095 , D c ∼ 5.4 × 10 13 ​ (tokens) L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\penalty\ \penalty\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\penalty\ \text{(tokens)} (1.2)

3. When training with a limited amount of compute, a sufficiently large dataset, an optimally-sized model, and a sufficiently small batch size (making optimal 2 2 2 We also observe an empirical power-law trend with the training compute C C (Figure 1 ) while training at fixed batch size, but it is the trend with C min C_{\rm min} that should be used to make predictions. They are related by equation ( 5.5 ). use of compute): L ⁡ ( C min ) = ( C c min / C min ) α C min ; α C min ∼ 0.050 , C c min ∼ 3.1 × 10 8 ​ (PF-days) L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\penalty\ \penalty\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\penalty\ \text{(PF-days)} (1.3)

These relations hold across eight orders of magnitude in C min C_{\rm min} , six orders of magnitude in N N , and over two orders of magnitude in D D . They depend very weakly on model shape and other Transformer hyperparameters (depth, width, number of self-attention heads), with specific numerical values associated with the Webtext2 training set [ RWC+19 ] . The power laws α N , α D , α C min \alpha_{\rm N},\alpha_{\rm D},\alpha_{C}^{\rm min} specify the degree of performance improvement expected as we scale up N N , D D , or C min C_{\rm min} ; for example, doubling the number of parameters yields a loss that is smaller by a factor 2 − α N = 0.95 2^{-\alpha_{N}}=0.95 . The precise numerical values of N c , C c min , N_{\mathrm{c}},C_{\rm c}^{\rm min}, and D c D_{\mathrm{c}} depend on the vocabulary size and tokenization and hence do not have a fundamental meaning.

The critical batch size, which determines the speed/efficiency tradeoff for data parallelism ( [ MKAT18 ] ), also roughly obeys a power law in L L : B crit ​ ( L ) = B ∗ L 1 / α B , B ∗ ∼ 2 ⋅ 10 8 ​ tokens , α B ∼ 0.21 B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 (1.4)

Equation ( 1.1 ) and ( 1.2 ) together suggest that as we increase the model size, we should increase the dataset size sublinearly according to D ∝ N α N α D ∼ N 0.74 D\propto N^{\frac{\alpha_{N}}{\alpha_{D}}}\sim N^{0.74} . In fact, we find that there is a single equation combining ( 1.1 ) and ( 1.2 ) that governs the simultaneous dependence on N N and D D and governs the degree of overfitting: L ⁡ ( N , D ) = [ ( N c N ) α N α D + D c D ] α D L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (1.5) with fits pictured on the left in figure 4 . We conjecture that this functional form may also parameterize the trained log-likelihood for other generative modeling tasks.

When training a given model for a finite number of parameter update steps S S in the infinite data limit, after an initial transient period, the learning curves can be accurately fit by (see the right of figure 4 ) L ⁡ ( N , S ) = ( N c N ) α N + ( S c S min ​ ( S ) ) α S L(N,S)=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{c}}{S_{\rm min}(S)}\right)^{\alpha_{S}} (1.6) where S c ≈ 2.1 × 10 3 S_{c}\approx 2.1\times 10^{3} and α S ≈ 0.76 \alpha_{S}\approx 0.76 , and S min ​ ( S ) S_{\rm min}(S) is the minimum possible number of optimization steps (parameter updates) estimated using Equation ( 5.4 ).

When training within a fixed compute budget C C , but with no other constraints, Equation ( 1.6 ) leads to the prediction that the optimal model size N N , optimal batch size B B , optimal number of steps S S , and dataset size D D should grow as N ∝ C α C min / α N , B ∝ C α C min / α B , S ∝ C α C min / α S , D = B ⋅ S N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad (1.7) with α C min = 1 / ( 1 / α S + 1 / α B + 1 / α N ) \alpha_{C}^{\rm min}=1/\left(1/\alpha_{S}+1/\alpha_{B}+1/\alpha_{N}\right) (1.8) which closely matches the empirically optimal results N ∝ C min 0.73 N\propto C_{\rm min}^{0.73} , B ∝ C min 0.24 B\propto C_{\rm min}^{0.24} , and S ∝ C min 0.03 S\propto C_{\rm min}^{0.03} . As the computational budget C C increases, it should be spent primarily on larger models, without dramatic increases in training time or dataset size (see Figure 3 ). This also implies that as models grow larger, they become increasingly sample efficient. In practice, researchers typically train smaller models for longer than would be maximally compute-efficient because of hardware constraints. Optimal performance depends on total compute as a power law (see Equation ( 1.3 )).

We provide some basic theoretical motivation for Equation ( 1.5 ), an analysis of learning curve fits and their implications for training time, and a breakdown of our results per token. We also make some brief comparisons to LSTMs and recurrent Transformers [ DGV+18 ] .

### 1.3 Notation

We use the following notation: • L L – the cross entropy loss in nats. Typically it will be averaged over the tokens in a context, but in some cases we report the loss for specific tokens within the context.

• N N – the number of model parameters, excluding all vocabulary and positional embeddings

• C ≈ 6 ​ N ​ B ​ S C\approx 6NBS – an estimate of the total non-embedding training compute, where B B is the batch size, and S S is the number of training steps (ie parameter updates). We quote numerical values in PF-days, where one PF-day = 10 15 × 24 × 3600 = 8.64 × 10 19 =10^{15}\times 24\times 3600=8.64\times 10^{19} floating point operations.

• D D – the dataset size in tokens

• B crit B_{\rm crit} – the critical batch size [ MKAT18 ] , defined and discussed in Section 5.1 . Training at the critical batch size provides a roughly optimal compromise between time and compute efficiency.

• C min C_{\rm min} – an estimate of the minimum amount of non-embedding compute to reach a given value of the loss. This is the training compute that would be used if the model were trained at a batch size much less than the critical batch size.

• S min S_{\rm min} – an estimate of the minimal number of training steps needed to reach a given value of the loss. This is also the number of training steps that would be used if the model were trained at a batch size much greater than the critical batch size.

• α X \alpha_{X} – power-law exponents for the scaling of the loss as L ⁡ ( X ) ∝ 1 / X α X L(X)\propto 1/X^{\alpha_{X}} where X X can be any of N , D , C , S , B , C min N,D,C,S,B,C^{\rm min} .

## 2 Background and Methods

We train language models on WebText2, an extended version of the WebText [ RWC+19 ] dataset, tokenized using byte-pair encoding [ SHB15 ] with a vocabulary size n vocab = 50257 n_{\rm vocab}=50257 . We optimize the autoregressive log-likelihood (i.e. cross-entropy loss) averaged over a 1024-token context, which is also our principal performance metric. We record the loss on the WebText2 test distribution and on a selection of other text distributions. We primarily train decoder-only [ LSP+18 , RNSS18 ] Transformer [ VSP+17 ] models, though we also train LSTM models and Universal Transformers [ DGV+18 ] for comparison.

### 2.1 Parameter and Compute Scaling of Transformers

We parameterize the Transformer architecture using hyperparameters n layer n_{\rm layer} (number of layers), d model d_{{\rm model}} (dimension of the residual stream), d ff d_{\rm ff} (dimension of the intermediate feed-forward layer), d attn d_{\rm attn} (dimension of the attention output), and n heads n_{\rm heads} (number of attention heads per layer). We include n ctx n_{\rm ctx} tokens in the input context, with n ctx = 1024 n_{\rm ctx}=1024 except where otherwise noted.

We use N N to denote the model size, which we define as the number of non-embedding parameters N \displaystyle N ≈ 2 ​ d model ​ n layer ​ ( 2 ​ d attn + d ff ) \displaystyle\approx 2d_{{\rm model}}n_{{\rm layer}}\left(2d_{{\rm attn}}+d_{{\rm ff}}\right) = 12 n layer d model 2 with the standard d attn = d ff / 4 = d model \displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}} (2.1) where we have excluded biases and other sub-leading terms. Our models also have n vocab ​ d model n_{\rm vocab}d_{{\rm model}} parameters in an embedding matrix, and use n ctx ​ d model n_{\rm ctx}d_{{\rm model}} parameters for positional embeddings, but we do not include these when discussing the ‘model size’ N N ; we will see that this produces significantly cleaner scaling laws.

Evaluating a forward pass of the Transformer involves roughly C forward ≈ 2 ​ N + 2 ​ n layer ​ n ctx ​ d model C_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}} (2.2) add-multiply operations, where the factor of two comes from the multiply-accumulate operation used in matrix multiplication. A more detailed per-operation parameter and compute count is included in Table 1 .