{"id":"arxiv:2503.21676","title":"How do language models learn facts? Dynamics, curricula and hallucinations","date":"2025-03-27","section":"Reasoning and the \"physics\" of language models","quality":0.304138,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/","source_url":"https://arxiv.org/abs/2503.21676","text_available":true,"authors":["Nicolas Zucchet","Jörg Bornschein","Stephanie Chan","Andrew Lampinen"],"outgoing":[{"source":"arxiv:2503.21676","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2301.05217","title":"Progress measures for grokking via mechanistic interpretability","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/"},{"source":"arxiv:2503.21676","target":"arxiv:2304.15004","evidence":"title","id":"arxiv:2304.15004","title":"Are Emergent Abilities of Large Language Models a Mirage?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/d7cd8c081ad54f2fdaa2211a4b49dc21f6ff2db7282e287546d210ef90e84323/"},{"source":"arxiv:2503.21676","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2309.14316","title":"Physics of Language Models: Part 3.1, Knowledge Storage and Extraction","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/"},{"source":"arxiv:2503.21676","target":"arxiv:2312.04927","evidence":"title","id":"arxiv:2312.04927","title":"Zoology: Measuring and Improving Recall in Efficient Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/baa792b06bdd30f91619f1e852aa5ae4dbf688fe3b1f2ee0722b339b9993ac5c/"},{"source":"arxiv:2503.21676","target":"arxiv:2404.05405","evidence":"id","id":"arxiv:2404.05405","title":"Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/541683d3678177bb2800d37a49a2f541c8a2c81cf802eaefe8840fed359c540e/"},{"source":"arxiv:2503.21676","target":"arxiv:2406.11813","evidence":"title","id":"arxiv:2406.11813","title":"How Do Large Language Models Acquire Factual Knowledge During Pretraining?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a7b36ce24eb2182e7f0bba96c0cddcfe6efe610b830d5e8ad8aac2b7d14696c0/"},{"source":"arxiv:2503.21676","target":"arxiv:2410.07041","evidence":"id","id":"arxiv:2410.07041","title":"Emergent properties with repeated examples","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/"},{"source":"arxiv:2503.21676","target":"doi:10.1073/pnas.1611835114","evidence":"title","id":"doi:10.1073/pnas.1611835114","title":"Overcoming catastrophic forgetting in neural networks","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/849082368a44d06b93ef3ba95bcfddc6672b7a5e8064ad2b2d19601639266e8b/"}],"incoming":[],"text_parts":[{"start":0,"end":15140,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html"},{"start":15140,"end":31137,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html"},{"start":31137,"end":46983,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html"},{"start":46983,"end":62366,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html"},{"start":62366,"end":72551,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html"}],"contents":[{"title":"How do language models learn facts? Dynamics, curricula and hallucinations","offset":19,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"Abstract","offset":97,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"1 An experimental setup to track knowledge over the course of learning","offset":3579,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"1.1 Knowledge, and how it differs from memory","offset":4635,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"1.2 Synthetic biographies as a framework for studying knowledge","offset":5681,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"1.3 Measuring knowledge at scale","offset":7556,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"1.4 Language models are trained following standard recipes","offset":9834,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"2 How language models acquire knowledge during learning","offset":10398,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"2.1 The three phases underlying knowledge acquisition","offset":10959,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"2.2 The attention-based circuits supporting recall are created during the loss plateau","offset":13241,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-1.html#text"},{"title":"3 Data distributional properties drive knowledge acquisition","offset":18304,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"3.1 The trade-off underlying imbalances in the data distribution","offset":19316,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"3.2 Data schedulers increase the final amount of acquired knowledge","offset":23362,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"4 Hallucinations hinder the integration of new knowledge post-training","offset":25064,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"4.1 Models start hallucinating as soon as they acquire knowledge","offset":26077,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"4.2 A large portion of the pre-training knowledge is erased during early fine-tuning","offset":26861,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"5 Discussion","offset":29560,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"On the learning dynamics of language models.","offset":29578,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"On the learning dynamics of neural networks more broadly.","offset":31072,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-2.html#text"},{"title":"On the role of non-uniformity and connections to developmental psychology.","offset":32344,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"Limitations","offset":33758,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"Scale and architecture.","offset":34040,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"Data.","offset":34472,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"Acknowledgements","offset":35474,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"Appendix","offset":35642,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"Appendix A Related work","offset":35655,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"A.1 Associative memories and factual knowledge of neural networks","offset":35684,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"A.2 Learning dynamics of neural networks","offset":37930,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"Appendix B Experimental setup","offset":40527,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"B.1 Rationale behind our design choices","offset":40562,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"B.2 Details about the biography generation process","offset":42602,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-3.html#text"},{"title":"B.3 Architecture, optimization and metrics","offset":47423,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"Appendix C Additional analysis of the learning dynamics (Section 2.1 )","offset":49668,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"C.1 The three phases are robust to sensible hyperparameter choices","offset":49744,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"Appendix D Details of the mechanistic study and additional analyses (Section 2.2 )","offset":50176,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"D.1 Implementation of the attention patching experiment","offset":50264,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"D.2 Details of the attention pattern analysis","offset":52582,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"Appendix E Additional analysis for the impact of data distribution properties (Section 3 )","offset":55442,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"E.1 Learning curves for different data distributions","offset":55538,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"E.2 Extensive comparison of the performance of different data distributions","offset":55828,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"Appendix F Details of the fine-tuning analysis and additional experiments (Section 4 )","offset":57933,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"F.1 Experimental details","offset":58025,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"F.2 Hallucinations","offset":58746,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"F.3 Additional analysis for fine-tuning","offset":61022,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-4.html#text"},{"title":"F.4 Reproducing fine-tuning behavior on a toy associative memory problem","offset":63341,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"F.5 Experiments with regular changes in training distribution","offset":65491,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"Appendix G Hyperparameter configurations","offset":67907,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"G.1 Default configuration","offset":67953,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"G.2 Hyperparameters for Section 2.1","offset":67985,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"G.3 Hyperparameters for Section 2.2","offset":68027,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"G.4 Hyperparameters for Section 3","offset":68069,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"G.5 Hyperparameters for Section 4","offset":70029,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"},{"title":"Instructions for reporting errors","offset":71587,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/part-5.html#text"}],"text_code_points":72551,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/source.md"}