Scaling Laws for Neural Language Models — part 2/5

Cleaned-text code points [15528, 31205). Bibliography omitted here; source.md includes it.

For contexts and models with d model > n ctx / 12 d_{{\rm model}}>n_{\rm ctx}/12 , the context-dependent computational cost per token is a relatively small fraction of the total compute. Since we primarily study models where d model ≫ n ctx / 12 d_{\rm model}\gg n_{\rm ctx}/12 , we do not include context-dependent terms in our training compute estimate. Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as C ≈ 6 ​ N C\approx 6N floating point operators per training token.

### 2.2 Training Procedures

Unless otherwise noted, we train models with the Adam optimizer [ KB14 ] for a fixed 2.5 × 10 5 2.5\times 10^{5} steps with a batch size of 512 512 sequences of 1024 1024 tokens. Due to memory constraints, our largest models (more than 1B parameters) were trained with Adafactor [ SS18 ] . We experimented with a variety of learning rates and schedules, as discussed in Appendix D.6 . We found that results at convergence were largely independent of learning rate schedule. Unless otherwise noted, all training runs included in our data used a learning rate schedule with a 3000 step linear warmup followed by a cosine decay to zero.

### 2.3 Datasets

We train our models on an extended version of the WebText dataset described in [ RWC+19 ] . The original WebText dataset was a web scrape of outbound links from Reddit through December 2017 which received at least 3 karma. In the second version, WebText2, we added outbound Reddit links from the period of January to October 2018, also with a minimum of 3 karma. The karma threshold served as a heuristic for whether people found the link interesting or useful. The text of the new links was extracted with the Newspaper3k python library. In total, the dataset consists of 20.3M documents containing 96 GB of text and 1.62 × 10 10 1.62\times 10^{10} words (as defined by wc ). We then apply the reversible tokenizer described in [ RWC+19 ] , which yields 2.29 × 10 10 2.29\times 10^{10} tokens. We reserve 6.6 × 10 8 6.6\times 10^{8} of these tokens for use as a test set, and we also test on similarly-prepared samples of Books Corpus [ ZKZ+15 ] , Common Crawl [ Fou ] , English Wikipedia, and a collection of publicly-available Internet Books.

## 3 Empirical Results and Basic Power Laws

To characterize language model scaling we train a wide variety of models, varying a number of factors including: • Model size (ranging in size from 768 to 1.5 billion non-embedding parameters)

• Dataset size (ranging from 22 million to 23 billion tokens)

• Shape (including depth, width, attention heads, and feed-forward dimension)

• Context length (1024 for most runs, though we also experiment with shorter contexts)

• Batch size ( 2 19 2^{19} for most runs, but we also vary it to measure the critical batch size)

In this section we will display data along with empirically-motivated fits, deferring theoretical analysis to later sections.

### 3.1 Approximate Transformer Shape and Hyperparameter Independence

Transformer performance depends very weakly on the shape parameters n layer , n heads n_{\rm layer},n_{\rm heads} , and d ff d_{\rm ff} when we hold the total non-embedding parameter count N N fixed. To establish these results we trained models with fixed size while varying a single hyperparameter. This was simplest for the case of n heads n_{\rm heads} . When varying n layer n_{\rm layer} , we simultaneously varied d model d_{{\rm model}} while keeping N ≈ 12 ​ n layer ​ d model 2 N\approx 12n_{\rm layer}d_{{\rm model}}^{2} fixed. Similarly, to vary d ff d_{\rm ff} at fixed model size we also simultaneously varied the d model d_{{\rm model}} parameter, as required by the parameter counts in Table 1 . Independence of n layers n_{\rm layers} would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [ VWB16 ] . The results are shown in Figure 5 .

### 3.2 Performance with Non-Embedding Parameter Count N N

In Figure 6 we display the performance of a wide variety of models, ranging from small models with shape ( n layer , d model ) = ( 2,128 ) (n_{\rm layer},d_{{\rm model}})=(2,128) through billion-parameter models, ranging in shape from ( 6 , 4288 ) (6,4288) through ( 207,768 ) (207,768) . Here we have trained to near convergence on the full WebText2 dataset and observe no overfitting (except possibly for the very largest models).

As shown in Figure 1 , we find a steady trend with non-embedding parameter count N N , which can be fit to the first term of Equation ( 1.5 ), so that L ⁡ ( N ) ≈ ( N c N ) α N L(N)\approx\left(\frac{N_{c}}{N}\right)^{\alpha_{N}} (3.1) To observe these trends it is crucial to study performance as a function of N N ; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6 ). This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [ LCG+19 ] .

Although these models have been trained on the WebText2 dataset, their test loss on a variety of other datasets is also a power-law in N N with nearly identical power, as shown in Figure 8 .

#### 3.2.1 Comparing to LSTMs and Universal Transformers

In Figure 7 we compare LSTM and Transformer performance as a function of non-embedding parameter count N N . The LSTMs were trained with the same dataset and context length. We see from these figures that the LSTMs perform as well as Transformers for tokens appearing early in the context, but cannot match the Transformer performance for later tokens. We present power-law relationships between performance and context position Appendix D.5 , where increasingly large powers for larger models suggest improved ability to quickly recognize patterns.

We also compare the performance of standard Transformers to recurrent Transformers [ DGV+18 ] in Figure 17 in the appendix. These models re-use parameters, and so perform slightly better as a function of N N , at the cost of additional compute per-parameter.

#### 3.2.2 Generalization Among Data Distributions

We have also tested our models on a set of additional text data distributions. The test loss on these datasets as a function of model size is shown in Figure 8 ; in all cases the models were trained only on the WebText2 dataset. We see that the loss on these other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2. We find that generalization depends almost exclusively on the in-distribution validation loss, and does not depend on the duration of training or proximity to convergence. We also observe no dependence on model depth (see Appendix D.8 ).

### 3.3 Performance with Dataset Size and Compute

We display empirical trends for the test loss as a function of dataset size D D (in tokens) and training compute C C in Figure 1 .

For the trend with D D we trained a model with ( n layer , n embd ) = ( 36 , 1280 ) (n_{\rm layer},n_{\rm embd})=(36,1280) on fixed subsets of the WebText2 dataset. We stopped training once the test loss ceased to decrease. We see that the resulting test losses can be fit with simple power-law L ⁡ ( D ) ≈ ( D c D ) α D L(D)\approx\left(\frac{D_{c}}{D}\right)^{\alpha_{D}} (3.2) in the dataset size. The data and fit appear in Figure 1 .

The total amount of non-embedding compute used during training can be estimated as C = 6 ​ N ​ B ​ S C=6NBS , where B B is the batch size, S S is the number of parameter updates, and the factor of 6 6 accounts for the forward and backward passes. Thus for a given value of C C we can scan over all models with various N N to find the model with the best performance on step S = C 6 ​ B ​ S S=\frac{C}{6BS} . Note that in these results the batch size B B remains fixed for all models , which means that these empirical results are not truly optimal. We will account for this in later sections using an adjusted C min C_{\rm min} to produce cleaner trends.

The result appears as the heavy black line on the left-hand plot in Figure 1 . It can be fit with L ⁡ ( C ) ≈ ( C c C ) α C L(C)\approx\left(\frac{C_{c}}{C}\right)^{\alpha_{C}} (3.3) The figure also includes images of individual learning curves to clarify when individual models are optimal. We will study the optimal allocation of compute more closely later on. The data strongly suggests that sample efficiency improves with model size, and we also illustrate this directly in Figure 19 in the appendix.

## 4 Charting the Infinite Data Limit and Overfitting

In Section 3 we found a number of basic scaling laws for language modeling performance. Here we will study the performance of a model of size N N trained on a dataset with D D tokens while varying N N and D D simultaneously. We will empirically demonstrate that the optimally trained test loss accords with the scaling law of Equation ( 1.5 ). This provides guidance on how much data we would need to train models of increasing size while keeping overfitting under control.

### 4.1 Proposed L ⁡ ( N , D ) L(N,D) Equation

We have chosen the parameterization ( 1.5 ) (repeated here for convenience): L ⁡ ( N , D ) = [ ( N c N ) α N α D + D c D ] α D L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} (4.1) using three principles: 1. Changes in vocabulary size or tokenization are expected to rescale the loss by an overall factor. The parameterization of L ⁡ ( N , D ) L(N,D) (and all models of the loss) must naturally allow for such a rescaling.

2. Fixing D D and sending N → ∞ N\to\infty , the overall loss should approach L ⁡ ( D ) L(D) . Conversely, fixing N N and sending D → ∞ D\to\infty the loss must approach L ⁡ ( N ) L(N) .

3. L ⁡ ( N , D ) L(N,D) should be analytic at D = ∞ D=\infty , so that it has a series expansion in 1 / D 1/D with integer powers. Theoretical support for this principle is significantly weaker than for the first two.

Our choice of L ⁡ ( N , D ) L(N,D) satisfies the first requirement because we can rescale N c , D c N_{c},D_{c} with changes in the vocabulary. This also implies that the values of N c , D c N_{c},D_{c} have no fundamental meaning.

Since we stop training early when the test loss ceases to improve and optimize all models in the same way, we expect that larger models should always perform better than smaller models. But with fixed finite D D , we also do not expect any model to be capable of approaching the best possible loss (ie the entropy of text). Similarly, a model with fixed size will be capacity-limited. These considerations motivate our second principle. Note that knowledge of L ⁡ ( N ) L(N) at infinite D D and L ⁡ ( D ) L(D) at infinite N N fully determines all the parameters in L ⁡ ( N , D ) L(N,D) .

The third principle is more speculative. There is a simple and general reason one might expect overfitting to scale ∝ 1 / D \propto 1/D at very large D D . Overfitting should be related to the variance or the signal-to-noise ratio of the dataset [ AS17 ] , and this scales as 1 / D 1/D . This expectation should hold for any smooth loss function, since we expect to be able to expand the loss about the D → ∞ D\to\infty limit. However, this argument assumes that 1 / D 1/D corrections dominate over other sources of variance, such as the finite batch size and other limits on the efficacy of optimization. Without empirical confirmation, we would not be very confident of its applicability.

Our third principle explains the asymmetry between the roles of N N and D D in Equation ( 1.5 ). Very similar symmetric expressions 3 3 3 For example, one might have used L ⁡ ( N , D ) = [ ( N c N ) α N + ( D c D ) α D ] β L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{D_{c}}{D}\right)^{\alpha_{D}}\right]^{\beta} , but this does not have a 1 / D 1/D expansion. are possible, but they would not have a 1 / D 1/D expansion with integer powers, and would require the introduction of an additional parameter.

In any case, we will see that our equation for L ⁡ ( N , D ) L(N,D) fits the data well, which is the most important justification for our L ⁡ ( N , D ) L(N,D) ansatz.

### 4.2 Results

We regularize all our models with 10% dropout, and by tracking test loss and stopping once it is no longer decreasing. The results are displayed in Figure 9 , including a fit to the four parameters α N , α D , N c , D c \alpha_{N},\alpha_{D},N_{c},D_{c} in Equation ( 1.5 ):

We obtain an excellent fit, with the exception of the runs where the dataset has been reduced by a factor of 1024 1024 , to about 2 × 10 7 2\times 10^{7} tokens. With such a small dataset, an epoch consists of only 40 parameter updates. Perhaps such a tiny dataset represents a different regime for language modeling, as overfitting happens very early in training (see Figure 16 ). Also note that the parameters differ very slightly from those obtained in Section 3 , as here we are fitting the full L ⁡ ( N , D ) L(N,D) rather than just L ⁡ ( N , ∞ ) L(N,\infty) or L ⁡ ( ∞ , D ) L(\infty,D) .

To chart the borderlands of the infinite data limit, we can directly study the extent of overfitting. For all but the largest models, we see no sign of overfitting when training with the full 22B token WebText2 dataset, so we can take it as representative of D = ∞ D=\infty . Thus we can compare finite D D to the infinite data limit by defining δ ​ L ​ ( N , D ) ≡ L ⁡ ( N , D ) L ⁡ ( N , ∞ ) − 1 \delta L(N,D)\equiv\frac{L(N,D)}{L(N,\infty)}-1 (4.2) and studying it as a function of N , D N,D . In fact, we see empirically that δ ​ L \delta L depends only a specific combination of N N and D D , as shown in Figure 16 . This follows from the scaling law of Equation ( 1.5 ), which implies δ ​ L ≈ ( 1 + ( N N c ) α N α D ​ D c D ) α D − 1 \delta L\approx\left(1+\left(\frac{N}{N_{c}}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}\frac{D_{c}}{D}\right)^{\alpha_{D}}-1 (4.3) Note that at large D D this formula also has a series expansion in powers of 1 / D 1/D .

We estimate that the variation in the loss with different random seeds is roughly 0.02 0.02 , which means that to avoid overfitting when training to within that threshold of convergence we require D ≳ ( 5 × 10 3 ) ​ N 0.74 D\gtrsim(5\times 10^{3})\,N^{0.74} (4.4) With this relation, models smaller than 10 9 10^{9} parameters can be trained with minimal overfitting on the 22B token WebText2 dataset, but our largest models will encounter some mild overfitting. More generally, this relation shows that dataset size may grow sub-linearly in model size while avoiding overfitting. Note however that this does not typically represent maximally compute-efficient training. We should also emphasize that we have not optimized regularization (eg the dropout probability) while varying dataset and model size.

## 5 Scaling Laws with Model Size and Training Time

In this section we will demonstrate that a simple scaling law provides a good description for the loss as a function of model size N N and training time. First we will explain how to use the results of [ MKAT18 ] to define a universal training step S min S_{\rm min} , which accounts for the fact that most of our models have not been trained at an optimal batch size. Then we will demonstrate that we can fit the model size and training time dependence of the loss using Equation ( 1.6 ). Later we will use these results to predict the optimal allocation of training compute between model size and training time, and then confirm that prediction.

### 5.1 Adjustment for Training at B crit ​ ( L ) B_{\rm crit}(L)