How Do Large Language Models Acquire Factual Knowledge During Pretraining?

arxiv:2406.11813

Original source · Paper JSON

Reasoning and the "physics" of language models · 2024-06-17

Cites (4) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• How Do Large Language Models Acquire Factual Knowledge During Pretraining?
• Abstract
• 1 Introduction
• 2 Related Work
• 3 Experimental Setup
• Fictional Knowledge dataset
• Evaluation metrics
• Definition 1
• Definition 2
• Definition 3
• Knowledge injection during pretraining
• 4 Results
• 4.1 Factual knowledge acquisition occurs by accumulating the observations of the fact
• 4.2 Effects of model scale and pretraining stage on knowledge acquisition dynamics
• 4.3 Forgetting in factual knowledge acquisition
• Training steps and the forgetting of acquired factual knowledge have a power-law relationship
• How quickly is the acquired factual knowledge lost?
• Pretraining with a larger batch size helps LLMs acquire more knowledge
• 4.4 Implications for LLM pretraining
• Why is popularity important for factual knowledge acquisition?
• Why does deduplication enhance model performance?
• 5 Discussion and Conclusions
• Acknowledgments and Disclosure of Funding
• Appendix
• Appendix A Limitations
• Appendix B Dataset Construction and Examples
• Appendix C Prompts Used for Dataset Generation
• C.1 Prompts for the generation of injected knowledge
• C.2 Prompts for the generation of paraphrased injected knowledge
• C.3 Prompts for the generation of semantic generalization probes
• C.4 Prompts for the generation of compositional generalization probes
• C.5 Prompts for the validation of generated compositional generalization probes
• Appendix D Detailed Training Setup
• Appendix E Additional Figures for the Pretraining Experiments
• E.1 Training dynamics of other OLMo-7B checkpoints
• E.2 Training dynamics of other OLMo-1B checkpoints
• E.3 Effectivity measurement data for OLMo-1B
• E.4 Forgetting dynamics of OLMo-7B checkpoints
• E.5 Forgetting dynamics of OLMo-1B checkpoints
• Appendix F Experiments for Training Olmo-7B Checkpoints With a Constant Learning Rate
• F.1 Training dynamics for constant learning rate experiments
• Appendix G Forgetting Dynamics of Olmo-7B Trained With a Reduced Batch Size
• G.1 Training dynamics for experiments on the forgetting dynamics with a reduced batch size
• Appendix H Effect of the Number of Previous Encounters on Effectivity and Retainability of Factual Knowledge
• Instructions for reporting errors