Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — part 4/8
Cleaned-text code points [47118, 62941). Bibliography omitted here; source.md includes it.
• For tiny LLaMA models, switching back to the GPT2Tokenizer is also necessary to match GPT2’s performance, though this is a minor issue. 18 18 18 This only applies to tiny models and is specific to the biography data we consider here: GPT2Tokenizer may tokenize years such as 1991 into a single token, while LLaMATokenizer will tokenize it into four digit tokens.
• Other modifications, such as changing from s i l u silu to g e l u gelu or adding trainable biases to layernorms, do not noticeably affect the capacity ratios (so we ignore those figures).
In summary, Result 7 . In the insufficient training regime (notably, the 100-exposure setting), except for tiny models, architectural differences generally do not affect performance, except • Using gated MLP reduces the model’s capacity ratio ( Figure 5 ); • Removing all MLP layers lowers the model’s capacity ratio, although significantly reducing the size of MLPs (e.g., by a 1 / 4 1/4 factor) does not. We propose that our experiments with the controllable biography dataset could serve as a valuable testbed for future architectural designs.
## 8 Quantization vs Scaling Laws
We have trained and tested models using (mixed precision) 16-bit floats. What happens if we quantize them to int8/int4 after training? We used the auto_gptq package, which is inspired by the GPTQ paper [ 10 ] , for quantization.
Result 8 ( Figure 6 ) . Quantizing language models (e.g., GPT2) trained with 16-bit floats: • to int8 has a negligible impact on their capacity ; • to int4 reduces their capacity by more than 2x.
Thus, even for models at peak capacity of 2 bits/param, quantizing to int8 does not affect capacity. Given that 2 bits/param was the best capacity ratio even after 1,000 training exposures on high-quality data, we conclude that extending training may not further improve the model’s capacity, but quantization can .
Since an int8-based model has an absolute upper bound R ( F ) ≤ 8 R(F)\leq 8 on capacity ratio, we have:
###### Corollary 8.1 .
Language models, like GPT2, can exceed 1/4 of the absolute theoretical limit for storing knowledge.
Unfortunately, using this quantization package, reducing the model to int4 significantly diminishes its capacity (more than 2x loss from int8 to int4). This suggests for high-quality int4 models, incorporating quantization during training may be necessary.
### 8.1 Where Is the Knowledge Stored?
We have seen that LLMs can efficiently compress knowledge into their parameter space, achieving 2bit/param even with 8-bit parameters. This raises the question: how and where is such knowledge stored? Our preliminary answer is that knowledge can be compactly stored within the model in a not-so-redundant manner. It is unlikely that the MLP layers alone store knowledge, as Attention layers, being of comparable sizes, also contribute to knowledge storage (c.f. Result 5 ). Moreover, particularly in models near the capacity boundary, removing the last transformer layer of an L L -layer model to “probe” for remaining knowledge reveals that the “leftover knowledge” can be significantly less than 1 − 1 L 1-\frac{1}{L} of the total. 19 19 19 This experiment, deemed not particularly interesting, was omitted from the paper. The probing technique used is Q-probing from [ 3 ] . This suggests knowledge is stored not in individual layers but in a complex manner, akin to a safe with combination locks, where removing one layer may eliminate much more than 1 L \frac{1}{L} of the total knowledge.
## 9 Mixture of Experts vs Scaling Laws
An important way to enhance efficiency in modern language models is the incorporation of sparsity. The Mixture of Experts (MoE) architecture plays a crucial role in this regard [ 9 , 30 ] . A question arises: does the MoE model scale differently in terms of the capacity ratio? For an MoE model, let P P denote the total number of parameters in the model , including all experts. Due to its inherent sparsity, the effective number of parameters can be significantly less than P P . Our primary observation is that MoE models scale similarly to dense models, even with 32 experts per layer.
Consider, for instance, GPT2, but with its MLP layer ( d → 4 d → d d\to 4d\to d ) replaced by 32 experts, each following a d → d → d d\to d\to d configuration. This setup uses 64 d 2 64d^{2} total parameters, but during inference, only 2 d 2 2d^{2} parameters are used per token (e.g., when using t o p k = 1 topk=1 ). After including the Attention layers, which each have 4 d 2 4d^{2} parameters, the ratio between the total and the effective number of parameters for the 32-expert MoE models is approximately 4 d 2 + 64 d 2 4 d 2 + 2 d 2 ≈ 11.3 \frac{4d^{2}+64d^{2}}{4d^{2}+2d^{2}}\approx 11.3 .
One might wonder, given that during inference time, the model uses only 11.3x fewer parameters, whether this affects the model’s capacity ratio by a factor close to 11.3x or closer to 1x? We show:
Result 9 ( Figure 7 ) . MoE is nearly fully efficient in storing knowledge , capable of leveraging all its parameters despite the sparsity constraint. Specifically, consider the GPT2-MoE model with 32 experts. If we compute its capacity ratio with respect to the total number of parameters and compare that to GPT2: • in the 1000-exposure settings, the peak capacity ratio decreases by 1.3x; and • in the 100-exposure settings, the peak capacity ratio decreases by 1.5x.
###### Remark 9.1 (topk) .
Result 9 holds even in the “sparsest” setting where t o p k = 1 topk=1 and c a p _ f a c t o r = 2 cap\_factor=2 in the MoE routing. The results are similar when using t o p k = 2 topk=2 and c a p _ f a c t o r = 1 cap\_factor=1 or t o p k = 2 topk=2 and c a p _ f a c t o r = 2 cap\_factor=2 — we discuss more in Appendix D .
###### Remark 9.2 .
It is typically observed in practice that MoE models underperform compared to dense models with the same number of total parameters. We demonstrate that this degradation does not come from the model’s knowledge storage capability.
## 10 Junk Data vs Scaling Laws
Not all data are useful for knowledge acquisition. For instance, while Wikipedia is full of valuable information, the Common Crawl of web pages may not be (there are also many pieces of information on those webpages, but they may not be useful for a language model to learn, such as the serial number of a random product). How does the presence of low-quality data impact the scaling laws of useful knowledge capacity ? To investigate this, we create a mixed dataset where: • 1 / 8 1/8 of tokens originate from bioS ( N ) \textsf{bioS}(N) for various N N (referred to as useful data ), and
• 7 / 8 7/8 of tokens originate from bioS ( N ′ ) \textsf{bioS}(N^{\prime}) for a large N ′ = 100 M N^{\prime}=100M (referred to as junk data ).
We train models on this mixture, ensuring each piece of useful data is seen for 100 exposures, thus making the total training 8 times longer compared to 100 exposures without junk (i.e., Figure 1(b) ). We focus on the capacity ratio of the useful data (the data in bioS ( N ) \textsf{bioS}(N) ) and compare that to Figure 1(b) . 20 20 20 The model’s ability to learn from junk data is negligible; each person in bioS ( N ′ ) \textsf{bioS}(N^{\prime}) appears only 0.2 times during training when N = 200 k N=200k , or 0.05 times when N = 50 k N=50k . How much does the capacity ratio degrade in the presence of junk data?
Result 10 ( Figure 8(a) - 8(e) ) . When 7/8 of the training tokens come from junk data (i.e., bioS ( N ′ ) \textsf{bioS}(N^{\prime}) for N ′ = 100 M N^{\prime}=100M ), transformer’s learning speed for useful data significantly degrades: • If trained for the same 100 exposures, the capacity ratio may degrade by 20x compared with training without junk (compare Figure 8(b) with Figure 8(a) ). • Even trained for 300/600/1000 exposures, the capacity ratio still degrades by 3x/1.5x/1.3x compared with 100 exposures without junk ( Figure 8(c) , 8(d) , and 8(e) vs. Figure 8(a) ). This underscores the crucial importance of pretrain data quality : even if junk data is entirely random, it negatively impacts model’s knowledge capacity even with sufficient training.
In contrast, if 7/8 of data is bioS ( N ′ ) \textsf{bioS}(N^{\prime}) with a very small N ′ N^{\prime} , simulating highly repetitive knowledge appearing in training tokens (e.g., “da Vinci painted the Mona Lisa” in millions of variations), this may not affect the model’s capacity for “standard” knowledge (e.g., those with 100 exposures): Result 11 ( Figure 8(f) ) . If 7/8 of the training tokens come from highly repetitive data (i.e., bioS ( N ′ ) \textsf{bioS}(N^{\prime}) for N ′ = 1 K N^{\prime}=1K ), this does not affect the learning speed of useful knowledge: • The 100-exposure capacity ratio of useful data is unchanged ( Figure 8(f) vs. Figure 8(a) ).
Finally, if pretrain data’s quality is poor and hard to improve, a backup strategy exists: Result 12 ( Figure 8(g) + 8(h) ) . When 7/8 of training tokens are from junk (i.e., bioS ( N ′ ) \textsf{bioS}(N^{\prime}) for N ′ = 100 M N^{\prime}=100M ), adding a special token at the start of every useful data greatly improves capacity ratio: • With 100 exposures, the capacity ratio degrades only by 2x ( Figure 8(g) vs. Figure 8(a) ). • With 300 exposures, the capacity ratio matches that of the 100-exposure scaling law without junk (compare Figure 8(h) with Figure 8(a) ).
Let us connect Result 12 to practice. First, adding a special token to high-credibility data is very practical: imagine adding the domain name ‘‘wikipedia.org’’ at the beginning of all Wikipedia paragraphs. (Adding a special token to junk data would be less meaningful.) 21 21 21 Result 12 also holds if one adds a (unique) special token for every piece of junk data; however, this could be meaningless as junk data often originates from various websites, making it hard to assign a unique identifier.
More generally, one can envision adding domain names (e.g., wikipedia.org) to every piece of the pretraining data. This would significantly enhance the model’s knowledge capacities, because Result 12 demonstrates that language models can automatically detect which domains are rich in high-quality knowledge and prioritize learning from them . We emphasize that the model does not need any prior knowledge to identify which domains contain high-quality knowledge; this process is entirely autonomous .
## 11 Conclusion
We investigated the scaling laws of language models, specifically the relationship between model size and the total bits of knowledge stored. Our findings reveal a precise, universal scaling law: a sufficiently-trained transformer (i.e., one whose training loss has plateau-ed) can store 2 bits of knowledge per parameter, even when quantized to int8, which is only 1 / 4 1/4 away from the information-theoretical maximum. We also examined how these scaling laws are influenced by various hyperparameters, including training duration, model architectures, floating-point precision, sparsity constraints like MoE, and data signal-noise ratios.
In terms of knowledge capacity, our methodology provides a more accurate and principled playground for comparing model architectures, training techniques, and data quality. We believe this playground can assist practitioners in making informed decisions about model selection, training data preparation, and further theoretical research into LLMs. Finally, our research represents an initial step towards addressing a fundamental question: how large does a language model need to be? We hope our findings will inspire further research in this area. Ultimately, we aim to provide a principled answer to the question, “Are language models with 1T parameters sufficient to achieve AGI?” in the future.
Appendix
## Appendix A More on GPT2 Scaling Laws
In this paper, our primary focus is on bioS ( N ) \textsf{bioS}(N) for N N ranging between 10K and 20M. Notably, bioS ( 20 M ) \textsf{bioS}(20M) encompasses approximately 1B bits of knowledge (refer to Theorem 3.2 ).
GPT2 model. As elaborated in Section 2.3 , we refer to the original GPT2 model [ 26 ] as GPT2, after substituting its positional embedding with rotary embedding [ 31 , 7 ] and removing its dropout layer [ 33 ] . These modifications are widely recognized for enhancing performance in language modeling tasks (see also [ 2 ] for a controlled experiment comparing that). We explore various GPT2 model sizes, maintaining a dimension-per-head of 64. The notation GPT2- ℓ \ell - h h represents the (modified) GPT2 architecture with ℓ \ell layers, h h heads, and 64 h 64h dimensions. The context length is set to 512.
Details on our specifications of LLaMA, Mistral, and other architectures will be provided in Appendix B as needed.
Model sizes. In this study, we calculate model sizes after excluding all unused tokens in the embedding layer. For example, while the GPT2 embedding layer typically has 50256 × ( 64 h ) 50256\times(64h) parameters, our bioS ( N ) \textsf{bioS}(N) data utilizes only 3275 tokens (after applying GPT2’s tokenizer), reducing the effective embedding layer size to 3275 × ( 64 h ) 3275\times(64h) . This adjustment explains why, for bioS data, GPT2small, typically known to have 124M parameters, is counted as having only 88M parameters in this paper.
We have selected a broad range of GPT2- ℓ \ell - h h models with practical ℓ \ell and h h values, excluding those with similar model sizes. Their selection is detailed in Figure 1 , encompassing both wide and shallow transformers (e.g., GPT2-2-20, GPT2-3-20, GPT2-4-20) and skinny and deep transformers (e.g., GPT2-16-4, GPT2-16-8, GPT2-28-20). For reference, GPT2 small/med/large correspond to GPT2-12-12, GPT2-24-16, GPT2-36-20, respectively.
We primarily focus on models with ℓ ≥ 2 \ell\geq 2 , as 1-layer transformers may demonstrate slightly lower capacity ratios. (For those interested, 1-layer transformers are included in Figure 9 , which is identical to Figure 1 but includes these models.)
Model sizes for datasets bioS ( N ) \textsf{bioS}(N) with N ≥ 𝟐 M N\geq 2M . In the 1000-exposure setting, to conserve computational resources, when exploring scaling laws for N = 2 M , 5 M , 10 M , 20 M N=2M,5M,10M,20M , we concentrate on one model size per dataset — specifically GPT2-16-8, GPT2-6-20, GPT2-20-16, GPT2-25-20 — as they approach the 2bit/param threshold (i.e., they satisfy R 𝗆𝖺𝗑 ( F ) ≈ 2 R^{\mathsf{max}}(F)\approx 2 ). In this context, our key finding is the validation of the 2bit/param capacity ratio, thus examining a limited selection of model sizes is adequate.
For the 100-exposure setting, we evaluate a broader range of model sizes per dataset. This approach is not only due to the tenfold reduction in training time compared to the 1000-exposure setting but also to facilitate a detailed comparison of model architectures in the 100-exposure setting, aiming for precision at higher model sizes.
Training parameters. We employ the AdamW optimizer with a cosine learning rate scheduler. This includes 1K steps of warmup, followed by a cosine decay of the learning rate from 1 1 to 0.1 0.1 times the reference rate. We use mixed-precision fp16 training unless otherwise stated.
### A.1 Base Scaling Laws
Our base scaling laws for the 1000-exposure and 100-exposure bioS ( N ) \textsf{bioS}(N) data are presented in Figures 1(a) and 1(b) , respectively.
For the 1000-exposure setting, the model’s final performance is not very sensitive to learning rate choices due to sufficient training. The following parameters were chosen for generating Figure 1(a) :
###### Parameter 1 ( Figure 1(a) ) .