Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — part 6/8

Cleaned-text code points [78924, 94760). Bibliography omitted here; source.md includes it.

From Figure 3 , it is evident that, except for tiny models, LLaMA, Mistral, GPT2 0 , and GPT2 1/4 architectures closely follow GPT2’s scaling law over 1000 exposures. For tiny models with ≤ 10 ​ M \leq 10M parameters, tying model weights increases their capacity (refer to Figure 3(c) ). This indicates that the 2bit/param capacity ratio is a relatively universal law among most typical (decoder-only) language model architectures.

### B.2 100-Exposure Setting

The 100-exposure setting reveals more intriguing comparisons. We contrast GPT2 with various model architectures in Figure 4 and offer a detailed comparison between LLaMA and GPT2 architectures in Figure 5 .

Figure 4(b) shows that the LLaMA architecture may lag behind GPT2’s scaling law by a factor of 1.3x, even for larger models.

We delve into the reasons behind this. By adjusting LLaMA’s architecture (e.g., switching GatedMLP back to normal MLP), as shown in Figure 5 , we find that replacing LLaMA’s GatedMLP with a standard MLP is necessary to match GPT2’s scaling law. Notably, for a strong comparison, when using GatedMLP we select the best result from three learning rates, whereas for a standard MLP, akin to GPT2, we use a single learning rate. For smaller models, matching GPT2 requires tying model weights and adopting GPT2’s tokenizer, though this is less significant. 26 26 26 The influence of the tokenizer on model capacity is noteworthy. For instance, LLaMA/Mistral tokenizers tend to split birthday years into single-digit tokens, slightly slowing the training of smaller models, whereas the GPT2Tokenizer uses a single token for the birth years such as 1991.

For other model architectures, Mistral, GPT2 0 , and GPT2 1/4 , their scaling laws in the 100-exposure setting are presented in Figure 4 . Figure 4(c) confirms that the Mistral architecture also underperforms GPT2 due to its use of gated MLP. Figure 4(d) reveals that reducing GPT2 1/4 ’s MLP layer size by a quarter has a negligible impact on model capacity. However, removing the MLP layers entirely in GPT2 0 significantly reduces the model’s capacity, see Figure 4(e) .

The 100-exposure setting represents an “insufficient training” paradigm. Thus, the comparisons are not about one architecture being strictly worse than another (as they achieve similar capacity ratios in a 1000-exposure setting, as shown in Figure 3 ). Our findings indicate that some architectures are noticeably easier to train (thus learn knowledge faster) : • The GatedMLP architecture slows down the model’s learning speed, and we observe less stable training with its use. 27 27 27 For example, mixed-precision fp16 training can sometimes fail for LLaMA/Mistral models smaller than 100M; hence, we use mixed-precision bf16 instead. Conversely, GPT2 models up to 1B can be trained with fp16.

• Removing MLP layers entirely slows down the model’s learning speed, whereas adjusting the size of MLP layers (e.g., from 8 ​ d 2 8d^{2} to 10.5 ​ d 2 10.5d^{2} or down to 2 ​ d 2 2d^{2} ) may not have a significant impact.

Additionally, we experimented with enabling trainable biases in LLaMA’s layernorms and switching from s ​ i ​ l ​ u silu to g ​ e ​ l ​ u gelu (to more closely resemble GPT2), in a similar way as Figure 5 , but found these changes do not affect the model’s capacities. We ignore those experiments for clarity.

Below, we discuss our parameter choices for the experiments in Figure 4 and Figure 5 .

###### Parameter 7 ( Figure 4 ) .

In the 100-exposure setting, (a) For LLaMA/Mistral models on bioS ​ ( N ) \textsf{bioS}(N) data, aiming to present negative results, we select the best learning rate from three options in each data setting: • For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 12 with bf16;

• For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 24 with bf16;

• For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 48 with bf16;

• For N = 500 ​ K N=500K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 96 with bf16;

• For N = 1 ​ M N=1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 192 with bf16;

• For N = 2 ​ M N=2M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 384 384 with bf16;

• For N = 5 ​ M N=5M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 768 768 with bf16;

• For N = 10 ​ M N=10M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 1536 1536 with bf16;

• For N = 20 ​ M N=20M , we use w ​ d = 0.002 wd=0.002 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 1536 1536 with bf16.

(For N ≤ 1 ​ M N\leq 1M , we also tested the same settings with fp16, finding similar results. However, LLaMA/Mistral models tend to fail more often with fp16, so we primarily used bf16.)

(b) For GPT2 1/4 : • For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , and batch size 12 with fp16;

• For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , and batch size 24 with fp16;

• For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , and batch size 48 with fp16;

• For N = 500 ​ K N=500K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 lr=0.0003/0.0005 , and batch size 96 with fp16;

• For N = 1 ​ M N=1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 lr=0.0003/0.0005 , and batch size 192 with fp16.

(c) For GPT2 0 , to present a negative result, we use the same settings as in Parameter fig:other-models:100(a) : • For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 12 with bf16;

• For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 24 with bf16;

• For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 48 with bf16;

• For N = 500 ​ K N=500K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 96 with bf16;

• For N = 1 ​ M N=1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 192 with bf16.

###### Parameter 8 ( Figure 5 ) .

In the 100-exposure controlled comparison experiment, • For presenting negative results ( Figure 5(a) and Figure 5(c) ), we select the best learning rate from three options, identical to GPT2 0 in Parameter fig:other-models:100(c) .

• For presenting positive results ( Figure 5(b) and Figure 5(d) ), we use a single set of learning rates, identical to Parameter 2 but with fp16 replaced by bf16 for a stronger comparison.

## Appendix C More on Quantization

We use the auto_gptq package (based on [ 10 ] ) to quantize the GPT2 model results in Figure 1 for the bioS data and the GPT2 model results in Figure 2 for the bioD data. We simply use a small set of 1000 people’s biographies to perform the quantization task. Our results are presented in Figure 14 for the bioS data and in Figure 15 for the bioD data.

## Appendix D More on Mixture of Experts

We utilize the tutel package for implementing Mixture-of-Experts (MoE) on GPT2 models [ 18 ] . In MoE, the parameter t ​ o ​ p ​ k topk determines the number of experts each token is routed to. It is recommended by some practitioners to use t ​ o ​ p ​ k = 2 topk=2 during training and t ​ o ​ p ​ k = 1 topk=1 during testing. Additionally, the c ​ a ​ p ​ _ ​ f ​ a ​ c ​ t ​ o ​ r cap\_factor parameter ensures that, given M M experts, each expert receives no more than c ​ a ​ p ​ _ ​ f ​ a ​ c ​ t ​ o ​ r M \frac{cap\_factor}{M} fraction of the data.

Using t ​ o ​ p ​ k = 1 topk=1 and c ​ a ​ p ​ _ ​ f ​ a ​ c ​ t ​ o ​ r = 1 cap\_factor=1 is generally not advisable. Thus, to provide the strongest result, we set t ​ o ​ p ​ k = 1 , c ​ a ​ p ​ _ ​ f ​ a ​ c ​ t ​ o ​ r = 2 topk=1,cap\_factor=2 for the 1000/100-exposure scaling laws in Figure 16 . (During testing, we increase the capacity factor to c ​ a ​ p ​ _ ​ f ​ a ​ c ​ t ​ o ​ r = 8 cap\_factor=8 .)

For the 100-exposure scaling law, we additionally compare three configurations: ( t ​ o ​ p ​ k , c ​ a ​ p ​ _ ​ f ​ a ​ c ​ t ​ o ​ r ) = ( 1 , 2 ) , ( 2 , 1 ) , ( 2 , 2 ) (topk,cap\_factor)=(1,2),(2,1),(2,2) , finding minimal differences among them as shown in Figure 17 . Remember from Section 7 that differences in model architecture usually become apparent in the insufficient training regime; this is why we opt for 100-exposure instead of 1000-exposure. Notably, ( t ​ o ​ p ​ k , c ​ a ​ p ​ _ ​ f ​ a ​ c ​ t ​ o ​ r ) = ( 2 , 2 ) (topk,cap\_factor)=(2,2) performs best (among the three) for deep models, such as GPT2-16-4 with 32 experts.

Due to their sparsity, MoE models often require higher learning rates compared to dense models. Consequently, we adjust the optimizer parameters as follows:

###### Parameter 9 ( Figure 16 , Figure 17 ) .

In the 1000-exposure setting for GPT2-MoE models with 32 experts, we slightly increase the learning rates while keeping other parameters nearly identical to Parameter 1 : • For N = 10 ​ K N=10K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 / 0.002 lr=0.001/0.002 , and batch size 24 with fp16;

• For N = 20 ​ K N=20K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 / 0.002 lr=0.001/0.002 , and batch size 48 with fp16;

• For N = 50 ​ K N=50K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 / 0.002 lr=0.001/0.002 , and batch size 96 with fp16;

• For N = 100 ​ K , 200 ​ K N=100K,200K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 / 0.002 lr=0.001/0.002 , batch size 192 with fp16;

• For N = 500 ​ K , 1 ​ M N=500K,1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , batch size 192 with fp16;

• For N = 2 ​ M N=2M , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.002 lr=0.002 , and batch size 1536 1536 with fp16;

• For N = 5 ​ M N=5M , we use w ​ d = 0.002 wd=0.002 , l ​ r = 0.0005 lr=0.0005 , and batch size 1536 1536 with fp16;

• For N = 10 ​ M N=10M , we use w ​ d = 0.001 wd=0.001 , l ​ r = 0.0005 lr=0.0005 , and batch size 1536 1536 with fp16.

In the 100-exposure setting, we also use higher learning rates compared to Parameter 2 :

• For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.001 / 0.002 / 0.005 lr=0.001/0.002/0.005 , and batch size 12 with fp16;

• For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.001 / 0.002 / 0.005 lr=0.001/0.002/0.005 , and batch size 24 with fp16;

• For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.001 / 0.002 / 0.005 lr=0.001/0.002/0.005 , and batch size 48 with fp16;

• For N = 500 ​ K N=500K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.001 / 0.002 lr=0.001/0.002 , and batch size 96 with fp16;

• For N = 1 ​ M N=1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 / 0.001 / 0.002 lr=0.0005/0.001/0.002 , and batch size 192 with fp16;

• For N = 2 ​ M N=2M , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.0005 / 0.001 lr=0.0005/0.001 , and batch size 192 192 with fp16;

• For N = 5 ​ M N=5M , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 384 384 with fp16.

## Appendix E More on Junk Data vs. Scaling Laws

Recall from Section 10 that our dataset is a mixture, with 1/8 of the tokens coming from bioS ​ ( N ) \textsf{bioS}(N) for various N N (referred to as “useful data”), and the remaining 7/8 from “junk data.” We explored three scenarios: (a) Junk data being bioS ​ ( N ′ ) \textsf{bioS}(N^{\prime}) for N ′ = 100 ​ M N^{\prime}=100M , representing completely random junk;

(b) Junk data being bioS ​ ( N ′ ) \textsf{bioS}(N^{\prime}) for N ′ = 1 ​ K N^{\prime}=1K , representing highly repetitive data; and

(c) Junk data being bioS ​ ( N ′ ) \textsf{bioS}(N^{\prime}) for N ′ = 100 ​ M N^{\prime}=100M , but with a special token appended to the front of each piece of useful data . 28 28 28 This is akin to adding a domain name like wikipedia.org at the beginning of the data; the model lacks prior knowledge that these special token data signify high-quality, useful data. It’s up to the model and the training process to autonomously discover this.

For simplicity, within each 512-token context window, we either include only useful data or only junk data (separated by <EOS> tokens). The outcomes are similar when mixing useful and junk data in the same context window. In all three cases, we initially consider a 100-exposure training setting where the useful data receive 100 exposures each during pretraining — thus, the total number of training tokens is approximately 8 times more than in Figure 1(b) (our scaling law for the 100-exposure case without junk data).

In case (A), presenting a negative result , we also explore 300-exposure, 600-exposure, and 1000-exposure training settings. Given that the 1000-exposure setting requires 48x more training tokens compared to Figure 1(b) , or 4.8x more compared to Figure 1(a) , we limited experiments to bioS ​ ( N ) \textsf{bioS}(N) with N ≤ 200 ​ K N\leq 200K to conserve computational resources. Similarly, for 300-exposure and 600-exposure, we only considered N ≤ 500 ​ K N\leq 500K .

In case (B), presenting a positive result , we limited our consideration to 100-exposure with N ≤ 1 ​ M N\leq 1M .

In case (C), presenting a moderately positive result , we explored both 100-exposure and 300-exposure settings, where, in the 300-exposure setting, we again limited to N ≤ 500 ​ K N\leq 500K .

Overall, due to the significantly different training durations (i.e., number of training tokens) across the 100-, 300-, 600-, and 1000-exposure settings, we had to adjust their batch sizes, weight decay, and learning rates accordingly. These adjustments are discussed below.

###### Parameter 10 ( Figure 8 ) .

We adhere to the general advice provided in Remark A.2 for selecting parameters in all experiments shown in Figure 8 . For negative results (e.g., Figure 8(b) , 8(c) ), we opted for a smaller batch size to increase the number of trainable steps and explored a wider range of learning rate options. Conversely, for positive results (e.g., Figure 8(f) , 8(e) ), we sometimes chose a larger batch size to benefit from faster, GPU-accelerated training times and considered a narrower set of learning rate choices. Overall, we have meticulously selected parameters to strengthen negative results as much as possible while intentionally not optimizing positive results to the same extent. This approach ensures a stronger comparison and effectively communicates the key message of this section. Specifically,

• For Figure 8(b) which is Case (a) of 100-exposure: – For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 12;

– For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 24;

– For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 48;

– For N = 500 ​ K N=500K , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.00005 / 0.0001 / 0.0002 / 0.0003 / 0.0005 lr=0.00005/0.0001/0.0002/0.0003/0.0005 , and batch size 192;

– For N = 1 ​ M N=1M , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.00005 / 0.0001 / 0.0002 / 0.0003 / 0.0005 lr=0.00005/0.0001/0.0002/0.0003/0.0005 , and batch size 192.