Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — part 1/8

Cleaned-text code points [0, 15536). Bibliography omitted here; source.md includes it.

# Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws Thanks: Submitted for Meta internal review on March 14, 2024. We would like to thank Lin Xiao and Yuchen Zhang for many helpful conversations. We would like to extend special thanks to Ian Clark, Gourab De, Anmol Mann, and Max Pfeifer from W&B , as well as Lucca Bertoncini, Liao Hu, Caleb Ho, Apostolos Kokolis, and Shubho Sengupta from Meta FAIR NextSys; Henry Estela, Wil Johnson, Rizwan Hashmi, and Lucas Noah from Meta Cloud Foundation; without their invaluable support, the extensive experiments in this paper would not have been possible.

###### Abstract

Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model’s capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8 , and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation.

More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model’s knowledge storage capacity. Notable insights include:

• The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage , particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train.

• Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model’s knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.

## 1 Introduction

The scaling laws of large language models remain a pivotal area of research, enabling predictions about the performance of extremely large models through experiments with smaller ones. On the training time aspect, established scaling laws [ 16 , 21 , 14 , 1 , 13 ] discuss the optimal training flops versus model size. However, recent studies [ 25 , 12 , 24 ] challenge these laws, demonstrating that training smaller models with significantly more flops can yield superior results. While these laws talk about how much time/data is needed to train a model of a certain size, another fundamental question is: what is the ultimate performance a model can achieve, assuming sufficient training ? Despite the known emergent behaviors in large models [ 8 , 34 ] , there is a lack of a principled, quantitative analysis on how model size impacts its capacity when adequately trained. 1 1 1 There is a rich literature comparing how pretrained models perform on benchmark tasks. Most comparisons are for different model families trained over different data: if LLaMA-70B is better than Mistral-7B, does the gain come from its choice of pretrain data, or the architecture difference, or really the size of the model? Some comparisons are among the same architecture, such as LLaMA-70B scores 63.6% on the world knowledge benchmark while LLaMA-7B scores only 48.9% [ 33 ] ; does this mean increasing model size by 10x increases its capacity only to 130 % = 63.6 / 48.9 130\%=63.6/48.9 ? Thus, it is highly important to use a more principled framework to study scaling laws in a controlled setting.

Traditional theory on overparameterization suggests that scaling up model size in sufficiently trained models can enhance memorization of training data [ 6 ] , improve generalization error [ 15 , 27 , 28 ] , and better fit complex target functions [ 23 , 5 ] . However, these results often overlook large constant or polynomial factors, leading to a significant discrepancy from practical outcomes.

In this paper, we introduce a principled framework to examine highly accurate scaling laws concerning model size versus its knowledge storage capacity . It is intuitive that larger language models can store more knowledge, but does the total knowledge scale linearly with the model’s size? What is the exact constant of this scaling? Understanding this constant is crucial for assessing the efficiency of transformer models in knowledge storage and how various factors (e.g., architecture, quantization, training duration, etc.) influence this capacity.

Knowledge is a, if not the, pivotal component of human intelligence, accumulated over our extensive history. Large language models like GPT-4 are celebrated not just for their sophisticated logic but also for their superior knowledge base. Despite rumors of GPT-4 having over 1T parameters, is it necessary to store all human knowledge? Could a 10B model, if trained sufficiently with high-quality data, match GPT-4’s knowledge capacity? Our paper seeks to address these questions.

Knowledge Pieces. Defining “one piece of human knowledge” precisely is challenging. This paper aims to make progress by focusing on a restricted, yet sufficiently interesting domain. We define a piece of knowledge as a (name, attribute, value) tuple, e.g., (Anya Forger, birthday, 10/2/1996); and many data in world knowledge benchmarks can be broken down into pieces like this. 2 2 2 Examples include (Africa, largest country, Sudan) and (It Happened One Night, director, Frank Capra) in TriviaQA [ 20 ] , or (Teton Dam, collapse date, 06/05/1976) and (USA, Capital, Washington D.C.) in NaturalQuestions [ 22 ] .

We generate synthetic knowledge-only datasets by uniformly at random generating (name, attribute, value) tuples from a knowledge base and converting them into English descriptions. We pretrain language models (e.g., GPT-2, LLaMA, Mistral) on these texts using a standard auto-regressive objective from random initialization, and “estimate” the learned knowledge. By varying the number of knowledge pieces and model sizes, we outline a knowledge capacity scaling law.

Our idealized setting, free from irrelevant data, allows for more accurate scaling law computations — we also discuss how “junk” data affects capacity later in Section 10 . In contrast, it is difficult to quantify real-life knowledge; for instance, if LLaMA-70B outperforms LLaMA-7B by 30% on a benchmark, it doesn’t necessarily mean a tenfold model scaling only boosts capacity by 30% (see Footnote 1 ). The synthetic setting also lets us adjust various hyperparameters, like name/value lengths and vocabulary size, to study their effects on knowledge capacity scaling laws.

Most of the paper shall focus on a setting with synthetically-generated human biographies as data, either using predefined sentence templates or LLaMA2-generated biographies for realism.

Bit Complexity and Capacity Ratio. For N N knowledge pieces (i.e., N N tuples), we define the bit complexity as the minimum bits required to encode these tuples. For any language model trained on this data, we calculate its “bit complexity lower bound” (see Theorem 3.2 ), describing the minimum number of bits needed for the model to store the knowledge at its given accuracy. This formula is nearly as precise as the upper bound, within a 1 − o ⁡ ( 1 ) 1-o(1) factor.

We train language models of varying sizes on knowledge data with different N N values. By comparing the models’ trainable parameters to the bit complexity lower bounds, we evaluate their knowledge storage efficiency.

###### Example .

A model with 100M parameters storing 220M bits of knowledge has a capacity ratio of 2.2 = 220M 100M 2.2=\frac{\text{220M}}{\text{100M}} bits per parameter. There is also a trivial upper bound on a model’s capacity ratio; for instance, a model using int8 parameters cannot exceed a capacity ratio of 8.

Our results. Our findings are summarized as follows:

• Section 5 : Base scaling law for GPT2 . 3 3 3 In this paper, GPT2 refers to the original GPT2 model but with rotary embedding instead of positional embedding and without dropout.

– Result 1 + 2 + 3 : GPT2, trained with standard AdamW, consistently achieves a 2bit/param capacity ratio across all data settings after sufficient training. This includes various model sizes, depths, widths, data sizes, types (synthetic/semi-synthetic), and hyperparameters (e.g., name/value length, attribute number, value diversity).

###### Remark 1.1 .

This predicts a sufficiently trained 7B language model can store 14B bits of knowledge, surpassing the knowledge of English Wikipedia and textbooks by our estimation. 4 4 4 As of February 1, 2024, English Wikipedia contains a total of 4.5 billion words, see https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Size_of_the_English_Wikipedia_database , accessed March 2024. We estimate that the non-overlapping contents of English textbooks have fewer than 16 billion words in total, see Remark G.1 . This amounts to 20.5 billion words, and we believe they contain fewer than 14 billion bits of knowledge.

###### Remark 1.2 .

When we say the model stores knowledge , it isn’t word-by-word memorization. Instead, the knowledge is flexibly extractable (e.g., via QAs like “What is Anya Forger’s birthday”) [ 3 ] and applicable in downstream tasks (e.g., comparing birthdays) via fine-tune [ 4 ] .

• Section 6 : How training time affects model capacity . Achieving a 2bit/param capacity requires each knowledge piece to be visited 1000 times during training, termed 1000-exposure to differentiate from traditional ‘‘1000-pass’’ terminology, as a single data pass can expose a knowledge piece 1000 times. 5 5 5 For example, it is plausible that one pass through Wiki data might present the knowledge piece (US, capital, Washington D.C.) 1000 times, and one pass through the Common Crawl might present it a million times. – Result 4 : With 100 exposures, an undertrained GPT2’s capacity ratio falls to 1bit/param.

###### Remark 1.3 .

Another perspective on Result 4 is that rare knowledge, encountered only 100 times during training, is stored at a 1bit/param ratio.

• Section 7 : How model architecture affects model capacity . We tested LLaMA, Mistral, and GPT2 architectures with reduced or even no MLP layers. – Result 5 : In the 1000-exposure setting, a 2bit/param capacity ratio appears to be a universal rule : all models, even without MLP layers, closely achieve this ratio.

– Result 6 : With 100 exposures, some archs show limitations; notably, LLaMA/Mistral’s capacity ratio is 1.3x lower than GPT2’s, even after best-tuned learning rates.

– Result 7 : Further controlled experiments indicate that “gated MLP” usage leads to LLaMA/Mistral architecture’s underperformance in knowledge storage.

###### Remark 1.4 .

Our framework offers a principled playground to compare models. This contrasts with traditional comparisons based on loss/perplexity, which can produce debatable conclusions. 6 6 6 A model might achieve better perplexity by performing much better on simpler data but slightly poorer on complex data, or by excelling in reasoning tasks but not in knowledge storage. Our results offer a more nuanced view: GatedMLP doesn’t affect frequently encountered knowledge (with 1000 exposures) but does impact moderately rare knowledge (with 100 exposures). Controlled data also reveal more significant differences between models. 7 7 7 For example, Shazeer [29] found GatedMLP offers a ∼ 1 % \sim 1\% accuracy boost on benchmark tasks; our findings of a 1.3x difference translates for instance to accuracies 90 % 90\% vs. 70 % 70\% .

• Section 8 : How quantization affects model capacity . We applied GPTQ [ 10 ] to quantize models from the base scaling laws to int8 or int4. Surprisingly, – Result 8 : Quantizing to int8 does not compromise model capacity (even for models on the boundary of 2bit/param); however, quantizing to int4 reduces capacity to 0.7bit/param.

###### Remark 1.5 .

Since int8 is 8bit, LLMs can exceed 1/4 of the theoretical limit for storing knowledge; thus knowledge must be very compactly stored inside the model across all layers.

###### Remark 1.6 .

Since 2bit/param is obtained after sufficient training, training longer may not further improve model capacity, but quantization can . While not covered in this paper, our framework also provides a principled playground to compare different quantization methods.

• Section 9 : How sparsity (MoE) affects model capacity . Mixture-of-experts (MoE) models offer faster inference than dense models but often underperform dense models with the same total parameter count (not effective parameters). We show that this performance drop is likely not due to a lack of knowledge storage capability. – Result 9 : MoE models, even with 32 32 experts, only reduce 1.3x in capacity compared to the base scaling laws, despite using just 8.8 % 8.8\% of the total parameters during inference.

• Section 10 : How junk knowledge affects model capacity . Not all pretrain data are equally useful. Much of the internet data lacks valuable knowledge for training language models [ 24 ] , while knowledge-rich sources like Wikipedia represent only a small fraction of the training tokens. We explore the impact on model capacity by conducting a controlled experiment with both useful and “junk” data. – Result 10 + 11 : Junk data significantly reduces model capacity. As an example, with a 1:7 ratio of “useful to junk” training tokens, capacity for useful knowledge loses by a factor of 20 x, even when useful knowledge is exposed 100 times. 8 8 8 The loss factor improves to 3x/1.5x/1.3x with 300/600/1000 exposures of useful knowledge, compared to Result 4 which involves training without junk for only 100 exposures.

– Result 12 : An effective mitigation is to prepend a special token to all useful knowledge. This is akin to adding a domain name like wikipedia.org at the start of every Wikipedia paragraph; the model autonomously identifies high-quality data without prior knowledge of valuable domains. In the example above, the loss factor improves from 20x to 2x.

Overall, our approach to studying knowledge capacity scaling laws offers a flexible and more accurate playground compared to traditional methods that evaluate language models trained on internet data against real-world benchmarks. This accuracy is partly due to the synthetic nature of our dataset, which eliminates concerns about benchmark contamination that could compromise the validity of real-world benchmark results. In this paper, we’ve conducted a thorough comparison across different model architectures and types of knowledge. While we haven’t explored various quantization methods, this represents a promising direction for future research. We’ve also investigated the impact of junk data and proposed mitigation strategies. We believe the insights gained from this principled exploration can assist practitioners in making informed decisions about model selection, training data preparation, and further theoretical research into LLMs.

## 2 Preliminaries

In this paper, a piece of knowledge is a tuple of three strings: (name, attribute, value) = ( n , a , v ) =(n,a,v) . For instance, n = “Anya” , a = “birthday” , v = “Oct 2, 1996” n=\text{``Anya''},a=\text{``birthday''},v=\text{``Oct 2, 1996''} .

### 2.1 Knowledge (Theoretical Setting)