Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — part 5/8

Cleaned-text code points [62941, 78924). Bibliography omitted here; source.md includes it.

In the 1000-exposure setting for GPT2 models on bioS ​ ( N ) \textsf{bioS}(N) data: • For N = 10 ​ K N=10K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 lr=0.001 , and batch size 24 (about 140K training steps);

• For N = 20 ​ K N=20K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 lr=0.001 , and batch size 48 (about 140K training steps);

• For N = 50 ​ K N=50K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 lr=0.001 , and batch size 96 (about 175K training steps);

• For N = 100 ​ K , 200 ​ K N=100K,200K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.001 lr=0.001 , batch size 192 (about 175K, 349K training steps);

• For N = 500 ​ K , 1 ​ M N=500K,1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 lr=0.0005 , batch size 192 (about 435K, 870K training steps);

• For N = 2 ​ M N=2M , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.0003 lr=0.0003 , and batch size 1536 1536 (about 220K training steps);

• For N = 5 ​ M N=5M , we use w ​ d = 0.002 wd=0.002 , l ​ r = 0.0003 lr=0.0003 , and batch size 1536 1536 (about 540K training steps);

• For N = 10 ​ M N=10M , we use w ​ d = 0.001 wd=0.001 , l ​ r = 0.0003 lr=0.0003 , and batch size 1536 1536 (about 1M training steps).

###### Remark A.1 (fp16 vs bf16) .

Training on GPT2 is conducted using mixed-precision fp16. We also tried bf16 and the results are nearly identical.

###### Remark A.2 (parameters) .

These optimization parameters are very natural , as it is generally impossible to have a fixed set of parameters for model sizes across a large multiplicative range. Notably: • Larger model sizes naturally require smaller learning rates.

• Language models typically need at least 50K training steps regardless of batch size. Thus, for small N N , we reduce the batch size to ensure the total number of training steps exceeds this threshold. For very large models, a larger batch size is preferred to enable GPU parallelism.

• When l ​ r lr remains constant, w ​ d wd should be relatively reduced as the number of training steps increases. Mathematically, the model weights should be “halved” for every Θ ⁡ ( 1 l ​ r × w ​ d ) \Theta(\frac{1}{lr\times wd}) training steps. Therefore, it’s advisable to reduce the w ​ d wd parameter when training for longer periods.

###### Remark A.3 (# GPUs) .

In this paper, we do not specify the number of GPUs as it is irrelevant . The results remain the same whether using 64 GPUs each with a batch size of 24, 48 GPUs each with a batch size of 32, or 1536 GPUs each with a batch size of 1.

For the 100-exposure setting, careful tuning of learning rates is required. The following parameters were chosen for generating Figure 1(b) : (Note: N = 10 ​ K , 20 ​ K N=10K,20K are not considered for the 100-exposure setting due to the excessively short training process.)

###### Parameter 2 ( Figure 1(b) ) .

In the 100-exposure setting for GPT2 models on bioS ​ ( N ) \textsf{bioS}(N) data: • For N = 50 ​ K N=50K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.001 lr=0.001 , and batch size 12;

• For N = 100 ​ K N=100K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.001 lr=0.001 , and batch size 24;

• For N = 200 ​ K N=200K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.001 lr=0.001 , and batch size 48; (except for GPT2-2-20, where l ​ r = 0.0005 lr=0.0005 is used)

• For N = 500 ​ K N=500K , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 lr=0.0005 , and batch size 96;

• For N = 1 ​ M N=1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 lr=0.0005 , and batch size 192;

• For N = 2 ​ M N=2M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 384 384 ;

• For N = 5 ​ M N=5M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0003 / 0.0005 lr=0.0003/0.0005 , and batch size 768 768 ;

• For N = 10 ​ M N=10M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 1024 1024 ;

• For N = 20 ​ M N=20M , we use w ​ d = 0.002 wd=0.002 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 1536 1536 . 22 22 22 Except for GPT2-28-20 we run out of GPU memory so reduce to batch size 1280 1280 .

### A.2 Knowledge Memorization vs. Extraction

It was recently discovered by Allen-Zhu and Li [3] that although models memorize knowledge, this knowledge may not be extractable (e.g., via fine-tuning) for application in downstream tasks. It is essential to verify that the “2 bit/param” knowledge learned by models is indeed extractable. This verification is achieved by applying a fine-tuning task (e.g., “What is Anya’s birthday? Answer: October 2, 1996”) to half of the individuals and then testing its performance on the remainder.

Specifically, on the original bioS ​ ( N ) \textsf{bioS}(N) data, we compute two quantities for each model: • Memorizable knowledge accuracy (# of people) .

We apply the model to the original training data, such as ‘‘Anya Briar Forger was born on’’ and check if it can correctly generate ‘‘October 2, 1996’’. For each person, we evaluate all five attributes and compute their average accuracy. 23 23 23 We exclude the company city attribute because it can be uniquely determined by the employer name, thus providing no additional knowledge. We then sum this accuracy up over all N N people. (Ideally, a perfect model would have this “accuracy” equal to N N .)

• Extractable knowledge accuracy (# of people) . Following the pretrain-finetune framework of [ 3 ] , we fine-tune any given pretrained model on half of the individuals using LoRA [ 17 ] with question-answering texts like “What is the birthday of Anya Briar Forger? Answer: October 2, 1996.” We then test its generation accuracy on the remaining half of the individuals. High accuracy indicates that the knowledge is not only memorized but can also be flexibly extracted for downstream tasks. Again, for each person, we evaluate all five attributes and compute their average accuracy. We then sum this across all N / 2 N/2 people and multiply by 2 2 . (Once again, a perfect model would have this equal to N N .)

Our results are presented in Figure 10 . By comparing, for instance, Figure 10(a) against Figure 10(b) , it is evident that our scaling laws apply not only to memorizable knowledge but also largely to extractable knowledge. Only for models precisely at the capacity ratio boundary is there a 1.2x decrease in total accuracy. 24 24 24 This decrease is in accuracy, not bits; a model may have a large amount of extractable knowledge in bits but not in accuracy. One can also compute knowledge bits in the extractable setting, but we omit such results for brevity.

###### Parameter 3 ( Figure 10 ) .

When dealing with models of significantly different sizes for LoRA finetuning, it’s necessary to adjust the LoRA rank sizes. In [ 3 ] , the authors primarily used a rank r ′ = 128 r^{\prime}=128 update for the embedding layer and ranks r = 8 r=8 or 16 16 for the query/value matrices, with their base model being either GPT2-12-12 or GPT2-12-20. In this paper, we explore a broader range of rank choices: ( r ′ , r ) ∈ { ( 8 , 2 ) , ( 16 , 2 ) , ( 8 , 4 ) , ( 32 , 4 ) , ( 8 , 8 ) , ( 32 , 8 ) , ( 128 , 8 ) , ( 32 , 16 ) , ( 128 , 16 ) } (r^{\prime},r)\in\{(8,2),(16,2),(8,4),(32,4),(8,8),(32,8),(128,8),(32,16),(128,16)\} , presenting only the best results. 25 25 25 Selecting the best LoRA option is justified as our aim is to determine the maximum extractable knowledge bits, and thus, any LoRA option demonstrating high test-set accuracy fulfills our objective.

We disable learning rate warmup, set the batch size to 96, the learning rate to 0.001 (with linear decay down to 0), weight decay at 0.1, and finetune for 75,000 steps.

### A.3 Other Biography Datasets

We also examine the bioS simple ​ ( N ) \textsf{bioS}^{\textsf{simple}}(N) datasets, which are identical to bioS ​ ( N ) \textsf{bioS}(N) except that each individual’s knowledge is stored in a fixed ordering of six fixed sentences (see Section 2.2 ). Allen-Zhu and Li [3] found that in such cases, the knowledge data are memorizable but nearly 0% extractable. As shown in Figure 11(a) , in these instances, the capacity ratio slightly decreases compared to Figure 1(a) . This implies, in this ideal setting, adding data diversity — by rewriting the same knowledge multiple times using different writing templates — not only enhances the model’s ability to extract knowledge, as noted by [ 3 ] , but also, surprisingly, increases the model’s capacity, as observed in this study.

Moreover, we explore the semi-real dataset bioR ​ ( N ) \textsf{bioR}(N) , which resembles bioS ​ ( N ) \textsf{bioS}(N) but with the biography paragraph generated by LLaMA2, and each individual is generated 40 times (using random seeds and prompts to encourage LLaMA2 to generate as diverse paragraphs as possible for each person). This results in a total of 22GB of text, comparable to the size of Wikipedia data.

The scaling law for the bioR ​ ( N ) \textsf{bioR}(N) data is presented in Figure 11(b) , indicating that the capacity ratio slightly decreases for larger models. This trend is expected, as LLaMA2 introduces numerous irrelevant details into the human biographies — usually different irrelevant details for each LLaMA2 generation — thereby consuming more model capacity. The decrease is more significant for smaller models, which may have greater difficulty comprehending the diverse English sentences in the data.

###### Parameter 4 ( Figure 11 ) .

In both experiments, we adhere to the same set of optimizer parameters used in Figure 1(a) , as detailed in Parameter 1 .

### A.4 More on Parameterized Scaling Laws

In the parameterized scaling laws, we utilize the bioD ​ ( N , K , C , D , L , T ) \textsf{bioD}(N,K,C,D,L,T) dataset from Definition 2.2 .

###### Parameter 5 ( Figure 2 , 12 , 13 ) .

For GPT2 models on the bioD dataset, we focus on the 1000-exposure case, with w ​ d = 0.01 wd=0.01 , l ​ r = 0.0005 lr=0.0005 , and a batch size of 192.

###### Remark A.4 (parameters) .

Contrary to Parameter 1 , it is not necessary to vary the training parameters, as our experiments with GPT2 models span a much narrower range of model sizes. We have adjusted the choice of N N to ensure that the optimal 2bit/param models are within a factor of 20 of each other in terms of model sizes.

Our results are presented in Figure 2 (in the main body, limited to models with accuracy ≤ 50 % \leq 50\% for clarity) and in Figure 12 (including all models).

Furthermore, from the bit complexity lower bound (see Definition 4.1 ) N ​ log 2 ​ N 0 e p 1 ⏟ name + N ​ K ​ log 2 ⁡ D C e p 2 ⏟ value + K ​ D ​ log 2 ⁡ T L D ​ e p 3 ⏟ diversity \displaystyle\underbrace{N\log_{2}\frac{N_{0}}{e^{p_{1}}}}_{\text{name }}+\underbrace{NK\log_{2}\frac{D^{C}}{e^{p_{2}}}}_{\text{value }}+\underbrace{KD\log_{2}\frac{T^{L}}{De^{p_{3}}}}_{\text{diversity }} (A.1) we also dissect how the three components contribute to this overall lower bound. As shown in Figure 13 , although the “value” component typically dominates, for certain hyperparameter settings, the “name” or “diversity” components can also be significant. This underscores the importance of proving our Theorem 3.2 lower bound, which is a sum of all three terms.

## Appendix B More on Model Architectures

We explore alternative architectural choices for language models.

LLaMA/Mistral. Notably, as of the writing of this paper, LLaMA [ 32 , 33 ] and Mistral [ 19 ] stand out as popular, publicly-available large language models. We highlight their key architecture differences from GPT2 — which we define as having rotary embedding and no dropout. 1. LLaMA and Mistral employ MLP layers with gated activation, using V ⁡ ( σ ⁡ ( W 1 ​ x ) ⋅ ( W 2 ​ x ) ) V(\sigma(W_{1}x)\cdot(W_{2}x)) instead of V ​ σ ​ ( W ​ x ) V\sigma(Wx) . Shazeer [29] noted that gated activation appears to yield slightly better performance.

2. Unlike GPT2, which ties the weights of the embedding layer and the output (LMHead) layer, LLaMA and Mistral do not.

3. For a hidden dimension d d , GPT2/LLaMA have 4 ​ d 2 4d^{2} parameters in the attention layer and 8 ​ d 2 8d^{2} in the MLP layer, whereas Mistral allocates a larger 10.5 ​ d 2 10.5d^{2} for its MLP layer.

4. Mistral promotes group-query attention (e.g., using 4 4 groups, thus reducing the K/V matrices to d 2 / 4 d^{2}/4 in size), unlike GPT2. LLaMA does not favor multi-query attention unless in its very large models, such as the 70B variant.

5. LLaMA and Mistral utilize different tokenizers compared to GPT2, with Mistral’s tokenizer being nearly identical to LLaMA’s.

6. GPT2 employs σ = g ​ e ​ l ​ u \sigma=gelu , while LLaMA/Mistral use σ = s ​ i ​ l ​ u \sigma=silu .

7. GPT2 incorporates layer normalization with trainable bias, which LLaMA/Mistral do not.

Given these distinctions, for LLaMA models, we use the notation LLaMA- ℓ \ell - h h for ℓ \ell layers, h h heads, and 64 ​ h 64h hidden dimensions; we omit group-query attention as LLaMA recommends it only for its 70B model. For Mistral, denoted as Mistral- ℓ \ell - h h , we enable group-query attention with 4 4 groups if h = 0 ( mod 4 ) h=0\pmod{4} , 1 1 group for odd h h , or 2 2 groups otherwise.

GPT2 with Smaller MLP. Mistral has a larger MLP layer, and it is often believed that the MLP layer serves primarily for storing knowledge, in contrast to the Attention layer. But is this truly the case?

To delve into this, we examine GPT2 1/4 , which is GPT2 with its MLP layer reduced from d → 4 ​ d → d d\to 4d\to d to d → d → d d\to d\to d (thus, 1 / 4 1/4 of its original size), and GPT2 0 , which is GPT2 but without any MLP layer.

Experimental setups. Throughout this section, when presenting positive result (such as for GPT2) we try to stick to one fixed set of learning rate choices; but when presenting a negative result (such as for the LLaMA architecture), we present the best among three learning rate choices.

### B.1 1000-Exposure Setting

In the 1000-exposure setting, we observe that the model architecture choices have a negligible impact on the scaling laws. The results for LLaMA, Mistral, GPT2 0 , and GPT2 1/4 architectures are presented in Figure 3 , with their parameter choices discussed below.

###### Parameter 6 ( Figure 3 ) .

In the 1000-exposure setting, for LLaMA/Mistral models we use similar parameters as specified in Parameter 1 , but we select the best of three learning rates to better demonstrate that GPT2 performs no worse than even the best tuned LLaMA/Mistral models: • For N = 10 ​ K N=10K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.0005 / 0.001 / 0.002 lr=0.0005/0.001/0.002 , and batch size 24 with fp16;

• For N = 20 ​ K N=20K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.0005 / 0.001 / 0.002 lr=0.0005/0.001/0.002 , and batch size 48 with fp16;

• For N = 50 ​ K N=50K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.0005 / 0.001 / 0.002 lr=0.0005/0.001/0.002 , and batch size 96 with fp16;

• For N = 100 ​ K , 200 ​ K N=100K,200K , we use w ​ d = 0.02 wd=0.02 , l ​ r = 0.0005 / 0.001 / 0.002 lr=0.0005/0.001/0.002 , and batch size 192 with fp16;

• For N = 500 ​ K , 1 ​ M N=500K,1M , we use w ​ d = 0.01 wd=0.01 , l ​ r = 0.0002 / 0.0003 / 0.0005 lr=0.0002/0.0003/0.0005 , and batch size 192 with fp16;

• For N = 2 ​ M N=2M , we use w ​ d = 0.005 wd=0.005 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 1536 1536 with bf16;

• For N = 5 ​ M N=5M , we use w ​ d = 0.002 wd=0.002 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 1536 1536 with bf16;

• For N = 10 ​ M N=10M , we use w ​ d = 0.001 wd=0.001 , l ​ r = 0.0003 / 0.0005 / 0.001 lr=0.0003/0.0005/0.001 , and batch size 1536 1536 with bf16.

For GPT2 0 and GPT2 1/4 , we use the same learning rates as specified in Parameter 1 .

###### Remark B.1 (bf16 on gated MLP) .

As discussed in Section B.2 , the training of LLaMA and Mistral architectures is less stable due to the use of GatedMLP, leading to the necessity of switching to (mixed-precision) bf16 training when required.