Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

arxiv:2404.05405

Original source · Paper JSON

Reasoning and the "physics" of language models · 2024-04-08

Cites (5) · Cited by (8)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8

Contents

• Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws Thanks: Submitted for Meta internal review on March 14, 2024. We would like to thank Lin Xiao and Yuchen Zhang for many helpful conversations. We would like to extend special thanks to Ian Clark, Gourab De, Anmol Mann, and Max Pfeifer from W&B , as well as Lucca Bertoncini, Liao Hu, Caleb Ho, Apostolos Kokolis, and Shubho Sengupta from Meta FAIR NextSys; Henry Estela, Wil Johnson, Rizwan Hashmi, and Lucas Noah from Meta Cloud Foundation; without their invaluable support, the extensive experiments in this paper would not have been possible.
• Abstract
• 1 Introduction
• Example .
• Remark 1.1 .
• Remark 1.2 .
• Remark 1.3 .
• Remark 1.4 .
• Remark 1.5 .
• Remark 1.6 .
• 2 Preliminaries
• 2.1 Knowledge (Theoretical Setting)
• Remark 2.1 .
• Definition 2.2 ( bioD data generation) .
• Proposition 2.3 (trivial, bit complexity upper bound) .
• 2.2 Knowledge (Empirical Setting)
• 2.3 Models and Training
• 3 Bit Complexity Lower Bound
• Remark 3.1 .
• Theorem 3.2 (bit complexity lower bound) .
• Corollary 3.3 (no-error case) .
• Remark 3.4 (why “sum of 3”) .
• Remark 3.5 (why “random data”) .
• 4 Capacity Ratio
• Definition 4.1 .
• Remark 4.2 .
• Definition 4.3 .
• Remark 4.4 .
• 5 Base Scaling Laws
• Remark 5.1 .
• 5.1 Data Formats — Diversity and Rewriting
• 5.2 Parameterized Scaling Laws
• 6 Training Time vs Scaling Law
• 7 Model Architecture vs Scaling Law
• 7.1 Insufficient Training Regime and a Closer Comparison
• 8 Quantization vs Scaling Laws
• Corollary 8.1 .
• 8.1 Where Is the Knowledge Stored?
• 9 Mixture of Experts vs Scaling Laws
• Remark 9.1 (topk) .
• Remark 9.2 .
• 10 Junk Data vs Scaling Laws
• 11 Conclusion
• Appendix A More on GPT2 Scaling Laws
• A.1 Base Scaling Laws
• Parameter 1 ( Figure 1(a) ) .
• Remark A.1 (fp16 vs bf16) .
• Remark A.2 (parameters) .
• Remark A.3 (# GPUs) .
• Parameter 2 ( Figure 1(b) ) .
• A.2 Knowledge Memorization vs. Extraction
• Parameter 3 ( Figure 10 ) .
• A.3 Other Biography Datasets
• Parameter 4 ( Figure 11 ) .
• A.4 More on Parameterized Scaling Laws
• Parameter 5 ( Figure 2 , 12 , 13 ) .
• Remark A.4 (parameters) .
• Appendix B More on Model Architectures
• B.1 1000-Exposure Setting
• Parameter 6 ( Figure 3 ) .
• Remark B.1 (bf16 on gated MLP) .
• B.2 100-Exposure Setting
• Parameter 7 ( Figure 4 ) .
• Parameter 8 ( Figure 5 ) .
• Appendix C More on Quantization
• Appendix D More on Mixture of Experts
• Parameter 9 ( Figure 16 , Figure 17 ) .
• Appendix E More on Junk Data vs. Scaling Laws
• Parameter 10 ( Figure 8 ) .
• Appendix F Proof of Theorem 3.2
• Lemma F.1 .
• Proof of Lemma F.1 .
• F.1 Warmup Examples
• F.2 Main Proof
• Proof of Theorem 3.2 .
• Appendix G Missing Remark
• Remark G.1 .