Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws β€” part 3/8

Cleaned-text code points [31195, 47118). Bibliography omitted here; source.md includes it.

Proof difficulties. If names are fixed ( 𝒩 = 𝒩 0 \mathcal{N}=\mathcal{N}_{0} ) and there are N N pieces of knowledge, each uniformly chosen from a fixed set [ T ] [T] , it is straightforward that any model F ⁑ ( W ) F(W) , capable of learning such knowledge perfectly , must satisfy log 2 ⁑ | 𝒲 | β‰₯ N ​ log 2 ​ T \log_{2}|\mathcal{W}|\geq N\log_{2}T . To relate this to Theorem 3.2 , we encounter three main challenges. First, the model F F may only learn the knowledge with a certain degree of accuracy, as defined by the cross-entropy loss. Second, 𝒩 β‰  𝒩 0 \mathcal{N}\neq\mathcal{N}_{0} so names need to be learned β€” even a perfect model cannot achieve zero cross-entropy loss when generating names. Third, there is a dependency between knowledge pieces β€” the value depends on the name and the choice of the diversity set (i.e., π’Ÿ a \mathcal{D}_{a} ). The proof of Theorem 3.2 is deferred to Appendix F .

## 4 Capacity Ratio

Motivated by Theorem 3.2 , ignoring lower order terms, we define the empirical capacity ratio as

###### Definition 4.1 .

Given a model F F with P P parameters trained over a bioD ​ ( N , K , C , D , L , T ) \textsf{bioD}(N,K,C,D,L,T) dataset 𝒡 \mathcal{Z} , suppose it gives p 1 = π₯𝐨𝐬𝐬 n ​ a ​ m ​ e ​ ( 𝒡 ) p_{1}=\mathbf{loss}_{name}(\mathcal{Z}) , p 2 = π₯𝐨𝐬𝐬 v ​ a ​ l ​ u ​ e ​ ( 𝒡 ) p_{2}=\mathbf{loss}_{value}(\mathcal{Z}) , p 3 = π₯𝐨𝐬𝐬 v ​ a ​ l ​ u ​ e ​ 1 ​ ( 𝒡 ) p_{3}=\mathbf{loss}_{value1}(\mathcal{Z}) , we define its capacity ratio and max capacity ratio R ⁑ ( F ) \displaystyle R(F) = def N ​ log 2 ​ N 0 e p 1 + N ​ K ​ log 2 ​ D C e p 2 + K ​ D ​ log 2 ​ T L D ​ e p 3 P . \displaystyle\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\frac{N\log_{2}\frac{N_{0}}{e^{p_{1}}}+NK\log_{2}\frac{D^{C}}{e^{p_{2}}}+KD\log_{2}\frac{T^{L}}{De^{p_{3}}}}{P}\kern 5.0pt. R 𝗆𝖺𝗑 ​ ( F ) \displaystyle R^{\mathsf{max}}(F) = def N ​ log 2 ​ N 0 N + N ​ K ​ C ​ log 2 ​ D + K ​ D ​ log 2 ​ T L D P . \displaystyle\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\frac{N\log_{2}\frac{N_{0}}{N}+NKC\log_{2}D+KD\log_{2}\frac{T^{L}}{D}}{P}\kern 5.0pt.

###### Remark 4.2 .

One must have R ⁑ ( F ) ≀ R 𝗆𝖺𝗑 ​ ( F ) R(F)\leq R^{\mathsf{max}}(F) , and equality is obtained if the model is perfect . For a fixed dataset, further increases in model size do not yield additional knowledge, thus R 𝗆𝖺𝗑 ​ ( F ) R^{\mathsf{max}}(F) approaches zero as the model size P P increases. On the other hand, Theorem 3.2 implies, ignoring lower-order terms, that if the model parameters are 8-bit (such as int8), then R ⁑ ( F ) ≀ 8 R(F)\leq 8 .

For our bioS ​ ( N ) \textsf{bioS}(N) data, we define a slightly reduced capacity ratio by omitting the diversity term. 11 11 11 A version of Theorem 3.2 can be proven for this dataset with a simpler proof, as it excludes the diversity set. This could also mean the model has full prior knowledge of the diversity set (e.g., assuming a fixed set of 300 university names) without counting this knowledge towards its learned bits.

###### Definition 4.3 .

Given a model F F with P P parameters trained over the bioS ​ ( N ) \textsf{bioS}(N) dataset 𝒡 \mathcal{Z} , suppose it gives p 1 = π₯𝐨𝐬𝐬 n ​ a ​ m ​ e ​ ( 𝒡 ) p_{1}=\mathbf{loss}_{name}(\mathcal{Z}) and p 2 = π₯𝐨𝐬𝐬 v ​ a ​ l ​ u ​ e ​ ( 𝒡 ) p_{2}=\mathbf{loss}_{value}(\mathcal{Z}) , its capacity ratio 12 12 12 Here, one can let 𝒦 = { birth date , birth city , university , major , employer , gender } \mathcal{K}=\{\text{birth date },\text{birth city },\text{university },\text{major },\text{employer },\text{gender }\} and accordingly define π₯𝐨𝐬𝐬 v ​ a ​ l ​ u ​ e ( 𝒡 ) = def 𝔼 n ∈ 𝒩 βˆ‘ a ∈ 𝒦 βˆ’ log 𝐏𝐫 R [ F βŠ₯ ( W ( 𝒡 ) , n , a , R ) = v ⋆ ( n , a ) ] \mathbf{loss}_{value}(\mathcal{Z})\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\operatornamewithlimits{\mathbb{E}}_{n\in\mathcal{N}}\sum_{a\in\mathcal{K}}-\log\operatornamewithlimits{\mathbf{Pr}}_{R}\big[F^{\bot}(W(\mathcal{Z}),n,a,R)=v^{\star}(n,a)\big] . R ⁑ ( F ) = def N ​ log 2 ​ N 0 e p 1 + N ​ log 2 ​ S 0 e p 2 P and R 𝗆𝖺𝗑 ​ ( F ) = def N ​ log 2 ​ N 0 N + N ​ log 2 ​ S 0 P R(F)\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\frac{N\log_{2}\frac{N_{0}}{e^{p_{1}}}+N\log_{2}\frac{S_{0}}{e^{p_{2}}}}{P}\hskip 10.00002pt\text{and }\hskip 10.00002ptR^{\mathsf{max}}(F)\stackrel{{\scriptstyle\mathrm{\scriptscriptstyle def}}}{{=}}\frac{N\log_{2}\frac{N_{0}}{N}+N\log_{2}S_{0}}{P} for N 0 = 400 Γ— 400 Γ— 1000 N_{0}=400\times 400\times 1000 and S 0 = 2 Γ— ( 12 β‹… 28 β‹… 200 ) Γ— 200 Γ— 300 Γ— 100 Γ— 263 S_{0}=2\times(12\cdot 28\cdot 200)\times 200\times 300\times 100\times 263 (c.f. Footnote 9 ).

###### Remark 4.4 .

Ignoring names, each person contains log 2 ⁑ ( S 0 ) β‰ˆ 47.6 \log_{2}(S_{0})\approx 47.6 bits of knowledge.

## 5 Base Scaling Laws

We first train a series of GPT2 models on the bioS ​ ( N ) \textsf{bioS}(N) datasets (see Section 2.2 ) using mixed-precision fp16. The training protocol ensures that each piece of knowledge is presented 1000 times, a process we refer to as β€œ1000 exposures.” It’s important to clarify that this differs from making 1000 passes over the data. For example, a single pass through Wiki data might expose the knowledge (US, capital, Washington D.C.) 1000 times, whereas a pass through the Common Crawl might do so a million times. Our synthetic bioS ​ ( N ) \textsf{bioS}(N) data, trained for 1000 exposures, aims to replicate such scenarios. 14 14 14 Within 1000 exposures, it’s likely that the same individual will have 1000 different biography paragraphs detailing the same knowledge (see Section 2.2 ). Therefore, 1000 exposures can occur within a single pass. Our initial findings are as follows: 15 15 15 We here focus on GPT2 models with depth β‰₯ 2 \geq 2 , and 1-layer models show slightly lower capacity ratios (see Figure 9 ). Our model selection covers most natural combinations of transformer width/depth, details in Appendix A . Result 1 ( Figure 1(a) ) . When trained for 1000 exposures on bioS ​ ( N ) \textsf{bioS}(N) , with N N ranging from 10K to 10M, GPT2 models with sizes from 1M to 0.5B parameters ( irrespective of depth or width ) demonstrate the following: (a) the peak capacity ratio R ⁑ ( F ) R(F) consistently exceeds R ⁑ ( F ) β‰₯ 2 R(F)\geq 2 ; (b) models with R 𝗆𝖺𝗑 ​ ( F ) ≀ 1.8 R^{\mathsf{max}}(F)\leq 1.8 attain near-perfect knowledge accuracies, i.e., R 𝗆𝖺𝗑 ​ ( F ) β‰ˆ R ⁑ ( F ) R^{\mathsf{max}}(F)\approx R(F) ; (c) across all models, R ⁑ ( F ) ≀ 2.3 R(F)\leq 2.3 .

###### Remark 5.1 .

Result res:base(a) , res:base(b) , and res:base(c) elucidate three distinct facets of the scaling law. β€’ Result res:base(a) highlights the maximum capacity across models; however, this could be misleading if only a single model achieves this peak.

β€’ Result res:base(b) reinforces this by showing that all models with a maximum capacity R 𝗆𝖺𝗑 ​ ( F ) ≀ 1.8 R^{\mathsf{max}}(F)\leq 1.8 can achieve such maximum capacity, i.e., R ⁑ ( F ) β‰ˆ R 𝗆𝖺𝗑 ​ ( F ) R(F)\approx R^{\mathsf{max}}(F) . In words, this indicates that for a dataset containing B B bits of knowledge, selecting a model size P β‰₯ B / 1.8 P\geq B/1.8 is sufficient .

β€’ Result res:base(c) further strengthens this by indicating that no model exceeds capacity ratio 2.3 2.3 .

For clarity , in subsequent results of this paper, we focus solely on the peak capacity ratio, with the understanding that observations similar to Result res:base(b) and Result res:base(c) consistently apply .

Knowledge extraction. The β€œ2bit/param” result is not about word-by-word memorization. Even better, such knowledge is also flexibly extractable (e.g., via fine-tuning using QAs like β€œWhat is Anya Forger’s birthday?”) [ 3 ] and thus can be further manipulated in downstream tasks (such as comparing the birthdates of two people, or performing calculations on the retrieved knowledge, etc.) [ 4 ] . This is because our bioS ​ ( N ) \textsf{bioS}(N) data is knowledge-augmented: the English biographies have sufficient text diversities [ 3 ] . We also verify in Appendix A.2 that such knowledge is extractable.

### 5.1 Data Formats β€” Diversity and Rewriting

We conduct the same analysis on bioS simple \textsf{bioS}^{\textsf{simple}} and bioR . Recall from Section 2.2 , bioS simple \textsf{bioS}^{\textsf{simple}} is a variant of bioS with reduced text diversity (one biography per person), while bioR is generated by LLaMA2, resulting in close-to-real human biographies. We have: Result 2 ( Figure 11 in Appendix A.3 ) . In the same 1000-exposure setting, peak capacity ratios for GPT2 trained on bioS simple \textsf{bioS}^{\textsf{simple}} and bioR are also approximately 2, albeit slightly lower. Thus: β€’ Diverse data (rewriting the same data multiple times) does not hurt β€” and may sometimes improve β€” the model’s capacity! Let’s highlight the significance of Result 2 . Recall from Section 2.2 : β€’ Training on bioS simple \textsf{bioS}^{\textsf{simple}} data for 1000 exposures equals 1000 passes over the data.

β€’ Training on bioS data for 1000 exposures is less than 1 pass.

β€’ Training on bioR data for 1000 exposures equals 25 passes.

Therefore, comparing bioS and bioS simple \textsf{bioS}^{\textsf{simple}} , it’s more advantageous to rewrite the data 1000 times (in this ideal setting), training each for one pass (as done in the bioS data), rather than training the same data for 1000 passes (as done in the bioS simple \textsf{bioS}^{\textsf{simple}} data). This is because, without data diversity, the model wastes capacity memorizing sentence structures, resulting in a capacity loss.

In a realistic scenario, tools like LLaMA2 can rewrite pretrain data like we did in bioR . Rewriting data 40 times can produce 40 distinct English paragraphs, sometimes with (different) hallucinated contents. Does this require the model to be 40x larger? No, our comparison between bioS and bioR shows that, if trained for the same duration (40 rewrites each for 25 passes), the model’s capacity ratio remains nearly the same, slightly lower due to irrelevant data introduced by LLaMA2.

Allen-Zhu and Li [3] suggested that rewriting pretraining data is crucial for making knowledge extractable rather word-by-word memorization. 16 16 16 As demonstrated by [ 3 ] , in low-diversity datasets like bioS simple \textsf{bioS}^{\textsf{simple}} , knowledge can be word-by-word memorized but is nearly 0% extractable for downstream tasks. Others discover that rewriting data can improve the reversal extractability of knowledge [ 11 , 4 ] . However, they did not explore the impact on the model’s capacity. Our paper addresses this gap, indicating that rewriting pretraining data does not compromise β€” and may even enhance β€” the model’s knowledge capacity.

### 5.2 Parameterized Scaling Laws

We further investigate scaling laws within the bioD ​ ( N , K , C , D , L , T ) \textsf{bioD}(N,K,C,D,L,T) data family. Unlike with human biographies, where variation is limited to N N , the bioD dataset allows for more flexible manipulation of the remaining hyperparameters K , C , D , L , T K,C,D,L,T . This enables us to examine how variations in these parameters affect the model’s peak capacity.

Result 3 ( Figure 2 ) . Across a broad spectrum of values, with K , C K,C ranging from 1 1 to 50 50 , D D from 10 10 to 10,000 10,000 , L L from 1 1 to 50 50 , and T T from 20 20 to 40,000 40,000 , we observe that: β€’ GPT2 models consistently exhibit a peak capacity ratio R ⁑ ( F ) β‰₯ 2 R(F)\geq 2 .

## 6 Training Time vs Scaling Law

What if the model is not sufficiently trained? For instance, there might be instances where knowledge appears only 100 times throughout the pretraining phase. We also calculate the capacity ratios for models trained with 100 exposures on bioS ​ ( N ) \textsf{bioS}(N) . Our findings can be summarized as follows: Result 4 ( Figure 1(b) ) . When trained for only 100 exposures on the bioS ​ ( N ) \textsf{bioS}(N) dataset, with N N ranging from 10K to 10M, across a broad spectrum of GPT2 models with sizes from 1M to 0.5B, the peak capacity ratio R ⁑ ( F ) R(F) consistently exceeds R ⁑ ( F ) β‰₯ 1 R(F)\geq 1 . Therefore, although 1000 exposures may be necessary for a model to reach its maximum storage capacity, training with just 100 exposures results in a capacity loss of no more than 2x.

In Section 10 , we shall also consider knowledge that has extremely low (e.g., 1) or high (e.g., 1M) exposures . It may not be interesting to study them in isolation, but it becomes more intriguing when they are examined alongside β€œstandard” knowledge, which has appeared, for instance, for 100 exposures, and how this impacts the model’s capacity. These will be our Result 10 through 12 .

## 7 Model Architecture vs Scaling Law

Several transformer architectures have been widely adopted, with LLaMA and Mistral among the most notable. We outline their key distinctions from GPT2, with further details in Appendix B : 1. LLaMA/Mistral use so-called GatedMLP layers, which is V ⁑ ( Οƒ ⁑ ( W 1 ​ x ) β‹… ( W 2 ​ x ) ) V(\sigma(W_{1}x)\cdot(W_{2}x)) instead of V ​ Οƒ ​ ( W ​ x ) V\sigma(Wx) . Shazeer [29] suggested that gated activation might yield marginally improved performance.

2. Unlike GPT2, LLaMA/Mistral do not tie weights.

3. Mistral features larger MLP layers compared to GPT2/LLaMA.

4. Mistral promotes group-query attention, not so by GPT2/LLaMA.

5. LLaMA/Mistral employ a different tokenizer than GPT2.

6. GPT2 uses the g ​ e ​ l ​ u gelu activation function, LLaMA/Mistral opt for s ​ i ​ l ​ u silu .

7. GPT2 implements layer normalization with a trainable bias.

Do these architectural variations impact the models’ maximum capacities? Our findings suggest that, in terms of knowledge capacity, GPT2 β€” when enhanced with rotary embedding and without dropout β€” performs no worse than any other architecture choice above in the sufficient training regime. We summarize the main findings below, deferring details to Appendix B.1 :

Result 5 ( Figure 3 ) . In the 1000-exposure setting, architectures do not matter much: β€’ LLaMA architecture performs comparably to GPT2, albeit slightly inferior for the tiny model (i.e., < < 10M). This discrepancy can be mitigated by also requiring LLaMA architecture to tie weights, as shown in Figure 3(c) compared to Figure 3(b) . β€’ A similar observation applies to Mistral architecture (see Figure 3(d) ). β€’ Reducing the MLP size of GPT2 architecture by 1 / 4 1/4 or even eliminating all MLP layers does not affect its capacity ratio, see Figure 3(e) and Figure 3(f) . This suggests, contrary to conventional beliefs, the Attention layers are also capable of storing knowledge .

This indicates that the 2bit/param capacity ratio is a relatively universal law among most typical (decoder-only) language model architectures.

### 7.1 Insufficient Training Regime and a Closer Comparison

However, differences in architectures become apparent in the insufficient training regime:

Result 6 ( Figure 4 ) . In the 100-exposure setting: β€’ Even for large models , LLaMA architecture’s capacity ratio can be 1.3x worse than GPT2 , even after optimally tuning learning rates. The results are similar for Mistral. β€’ Reducing GPT2’s MLP size by 1 / 4 1/4 has a negligible impact on the capacity ratio. β€’ Removing MLPs decreases the capacity ratio by more than 1.5x.

To investigate why the LLaMA architecture is inferior to GPT2 in the 100-exposure (insufficiently trained) setting, we closely examine LLaMA by gradually modifying its architecture back towards GPT2 to identify the key architectural changes. We start by tying weights, as this enhances tiny LLaMA model’s capacity in the 1000-exposure setting ( Result 5 ). As illustrated in Figure 5 :

β€’ For large models, replacing LLaMA architecture’s gated MLP with a standard MLP (while keeping s ​ i ​ l ​ u silu unchanged) noticeably improves LLaMA’s capacity ratio. 17 17 17 As discussed in Appendix B , gated MLP layers are less stable to train, thus requiring more time.