How do language models learn facts? Dynamics, curricula and hallucinations

arxiv:2503.21676

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-03-27

Cites (8) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• How do language models learn facts? Dynamics, curricula and hallucinations
• Abstract
• 1 An experimental setup to track knowledge over the course of learning
• 1.1 Knowledge, and how it differs from memory
• 1.2 Synthetic biographies as a framework for studying knowledge
• 1.3 Measuring knowledge at scale
• 1.4 Language models are trained following standard recipes
• 2 How language models acquire knowledge during learning
• 2.1 The three phases underlying knowledge acquisition
• 2.2 The attention-based circuits supporting recall are created during the loss plateau
• 3 Data distributional properties drive knowledge acquisition
• 3.1 The trade-off underlying imbalances in the data distribution
• 3.2 Data schedulers increase the final amount of acquired knowledge
• 4 Hallucinations hinder the integration of new knowledge post-training
• 4.1 Models start hallucinating as soon as they acquire knowledge
• 4.2 A large portion of the pre-training knowledge is erased during early fine-tuning
• 5 Discussion
• On the learning dynamics of language models.
• On the learning dynamics of neural networks more broadly.
• On the role of non-uniformity and connections to developmental psychology.
• Limitations
• Scale and architecture.
• Data.
• Acknowledgements
• Appendix
• Appendix A Related work
• A.1 Associative memories and factual knowledge of neural networks
• A.2 Learning dynamics of neural networks
• Appendix B Experimental setup
• B.1 Rationale behind our design choices
• B.2 Details about the biography generation process
• B.3 Architecture, optimization and metrics
• Appendix C Additional analysis of the learning dynamics (Section 2.1 )
• C.1 The three phases are robust to sensible hyperparameter choices
• Appendix D Details of the mechanistic study and additional analyses (Section 2.2 )
• D.1 Implementation of the attention patching experiment
• D.2 Details of the attention pattern analysis
• Appendix E Additional analysis for the impact of data distribution properties (Section 3 )
• E.1 Learning curves for different data distributions
• E.2 Extensive comparison of the performance of different data distributions
• Appendix F Details of the fine-tuning analysis and additional experiments (Section 4 )
• F.1 Experimental details
• F.2 Hallucinations
• F.3 Additional analysis for fine-tuning
• F.4 Reproducing fine-tuning behavior on a toy associative memory problem
• F.5 Experiments with regular changes in training distribution
• Appendix G Hyperparameter configurations
• G.1 Default configuration
• G.2 Hyperparameters for Section 2.1
• G.3 Hyperparameters for Section 2.2
• G.4 Hyperparameters for Section 3
• G.5 Hyperparameters for Section 4
• Instructions for reporting errors