{"id":"arxiv:2309.14316","title":"Physics of Language Models: Part 3.1, Knowledge Storage and Extraction","date":"2023-09-25","section":"Reasoning and the \"physics\" of language models","quality":0.208192,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/","source_url":"https://arxiv.org/abs/2309.14316","text_available":true,"authors":["Zeyuan Allen-Zhu","Yuanzhi Li"],"outgoing":[{"source":"arxiv:2309.14316","target":"arxiv:2305.13673","evidence":"id","id":"arxiv:2305.13673","title":"Physics of Language Models: Part 1, Learning Hierarchical Language Structures","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a72219d4bd65d1e84ce31079b6c788d7e6c297e71d4b083af5d4cffccfdcff72/"},{"source":"arxiv:2309.14316","target":"arxiv:2309.14402","evidence":"id","id":"arxiv:2309.14402","title":"Physics of Language Models: Part 3.2, Knowledge Manipulation","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/79b4d0b92a11dcf3b111e9d2d1b6bb3f516bebce7333326f65e6e583d6ff3cf2/"},{"source":"arxiv:2309.14316","target":"arxiv:2404.05405","evidence":"id","id":"arxiv:2404.05405","title":"Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/541683d3678177bb2800d37a49a2f541c8a2c81cf802eaefe8840fed359c540e/"},{"source":"arxiv:2309.14316","target":"arxiv:2407.20311","evidence":"title","id":"arxiv:2407.20311","title":"Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de5db7bc6b929af401b5b83c845823505a7823c25987b0e455231d2e4fcec3e4/"}],"incoming":[{"source":"arxiv:2305.13673","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2305.13673","title":"Physics of Language Models: Part 1, Learning Hierarchical Language Structures","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a72219d4bd65d1e84ce31079b6c788d7e6c297e71d4b083af5d4cffccfdcff72/"},{"source":"arxiv:2309.14402","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2309.14402","title":"Physics of Language Models: Part 3.2, Knowledge Manipulation","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/79b4d0b92a11dcf3b111e9d2d1b6bb3f516bebce7333326f65e6e583d6ff3cf2/"},{"source":"arxiv:2404.05405","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2404.05405","title":"Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/541683d3678177bb2800d37a49a2f541c8a2c81cf802eaefe8840fed359c540e/"},{"source":"arxiv:2406.03445","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2406.03445","title":"Pre-trained Large Language Models Use Fourier Features to Compute Addition","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/169a3e64467f1e0862508b86b2198187fc8eeb3ab114a47d2bd4caaea4a9ee2b/"},{"source":"arxiv:2406.11813","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2406.11813","title":"How Do Large Language Models Acquire Factual Knowledge During Pretraining?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a7b36ce24eb2182e7f0bba96c0cddcfe6efe610b830d5e8ad8aac2b7d14696c0/"},{"source":"arxiv:2407.15017","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2407.15017","title":"Knowledge Mechanisms in Large Language Models: A Survey and Perspective","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/"},{"source":"arxiv:2407.20311","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2407.20311","title":"Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de5db7bc6b929af401b5b83c845823505a7823c25987b0e455231d2e4fcec3e4/"},{"source":"arxiv:2410.07041","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2410.07041","title":"Emergent properties with repeated examples","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/"},{"source":"arxiv:2412.14135","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2412.14135","title":"Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5a0b4b0b524698ee5f30eab659b7cd5104f5fe6497c8dec2d0c4b9d332350047/"},{"source":"arxiv:2502.05171","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2502.05171","title":"Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/24a6ab865777290627cecebd3f06ea3daef01622f102d32a95f7fb2a5a42880f/"},{"source":"arxiv:2503.21676","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2503.21676","title":"How do language models learn facts? Dynamics, curricula and hallucinations","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/"},{"source":"arxiv:2510.00184","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2510.00184","title":"Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7e395b2ae5bfc2a4d8bfc1e8669b4f324739b6a9b11161cba150bfdffc31746c/"},{"source":"arxiv:2601.21343","target":"arxiv:2309.14316","evidence":"id","id":"arxiv:2601.21343","title":"Self-Improving Pretraining: using post-trained models to pretrain better models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/43ce1b189d6b21d445c0ce6d4d04b4352d268beb4346da43be19982402befde8/"}],"text_parts":[{"start":0,"end":15313,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-1.html"},{"start":15313,"end":31227,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html"},{"start":31227,"end":47023,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html"},{"start":47023,"end":63018,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-4.html"},{"start":63018,"end":78898,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html"},{"start":78898,"end":79254,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-6.html"}],"contents":[{"title":"Physics of Language Models: Part 3.1, Knowledge Storage and Extraction Thanks: Project page: https://physics.allen-zhu.com/part-3-knowledge/part-3-1 . An extended video of this paper is available at https://youtu.be/YSHzKmEianc . V1 was circulated internally at Meta on Sep 18, 2023, and appeared on arXiv on Sep 25, 2023. V2 is nearly identical to V1, with minor corrections to author names and writing. V3 includes additional Llama experiments and further writing improvements. We would like to thank Lin Xiao, Chunting Zhou, Tianyi Peng, Xiaodong Liu, and Zhijie Zhou for many helpful conversations. We would like to extend special thanks to Nabib Ahmed, Giri Anantharaman, Lucca Bertoncini, Henry Estela, Liao Hu, Caleb Ho, Wil Johnson, Apostolos Kokolis, and Shubho Sengupta from Meta FAIR, as well as Ian Clark, Gourab De, Anmol Mann, and Max Pfeifer from W&B ; without their invaluable support, the experiments in this paper would not have been possible.","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-1.html#text"},{"title":"Abstract","offset":965,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-1.html#text"},{"title":"1 Introduction","offset":2481,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-1.html#text"},{"title":"1.1 Related Work","offset":12468,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-1.html#text"},{"title":"2 Result 0: Our Dataset Families","offset":14495,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-1.html#text"},{"title":"2.1 Training Details","offset":18702,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"3 Result 1: Mixed Training Enables Knowledge Extraction","offset":21445,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"3.1 Model’s Abnormal Learning Behavior","offset":22553,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"Remark 3.1 .","offset":24987,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"4 Result 2-3: BIO Pretrain + QA Instruction Finetune","offset":25482,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"4.1 Result 2: Model Fails to Extract Knowledge After BIO Pretrain","offset":26056,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"4.2 Result 3: Knowledge Augmentation","offset":28977,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"Remark .","offset":30173,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-2.html#text"},{"title":"5 Results 4-5: Knowledge Probes on the BIO Pretrained Model","offset":33867,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"5.1 Result 4: Position-Based Probing","offset":34234,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"5.1.1 Closer P-Probing at Knowledge Dependency","offset":37869,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"Remark 5.1 .","offset":39603,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"5.1.2 P-Probing Extensions","offset":40287,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"5.2 Result 5: Query-Based Probing","offset":41640,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"6 Result 6: Celebrity Can Help Minority","offset":44370,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"Remark 6.1 .","offset":47002,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-3.html#text"},{"title":"7 Result 7: Knowledge Storage for Bidirectional Models","offset":47497,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-4.html#text"},{"title":"8 Conclusion","offset":51412,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-4.html#text"},{"title":"Appendix A Details on Data Preparation","offset":53082,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-4.html#text"},{"title":"A.1 BIO dataset bioS","offset":53125,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-4.html#text"},{"title":"A.1.1 bioS couple","offset":58068,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-4.html#text"},{"title":"A.2 BIO dataset bioR","offset":59222,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-4.html#text"},{"title":"Appendix B Details on Model Architecture","offset":63018,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"Appendix C Details on Pretrain and Mixed Training","offset":65058,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"Remark C.1 .","offset":66778,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"Appendix D Details on QA Finetune","offset":67062,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"Appendix E Details on P-probing","offset":68587,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"E.1 Details on Closer P-Probing","offset":70667,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"Appendix F Details on Q-probing","offset":70974,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"Appendix G Details on Celebrity Augementation","offset":72832,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"},{"title":"Appendix H Details on BERT Experiment","offset":77301,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/part-5.html#text"}],"text_code_points":79254,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/source.md"}