{"id":"arxiv:2407.15017","title":"Knowledge Mechanisms in Large Language Models: A Survey and Perspective","date":"2024-07-22","section":"Reasoning and the \"physics\" of language models","quality":-0.200172,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/","source_url":"https://arxiv.org/abs/2407.15017","text_available":true,"authors":["Mengru Wang","Yunzhi Yao","Ziwen Xu","Shuofei Qiao"],"outgoing":[{"source":"arxiv:2407.15017","target":"arxiv:1803.03635","evidence":"title","id":"arxiv:1803.03635","title":"The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a9d7f0235205afcc709e3e74f28cc7707831bcb1d46239c418cc7e4b624d23bd/"},{"source":"arxiv:2407.15017","target":"arxiv:2301.06627","evidence":"title","id":"arxiv:2301.06627","title":"Dissociating language and thought in large language models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/3791d7ed420be517ded1ee7ae5558cf6afa948ea5b841a76ec3da8a9134ec565/"},{"source":"arxiv:2407.15017","target":"arxiv:2304.09355","evidence":"title","id":"arxiv:2304.09355","title":"To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/6de01284dffdeadbd92020fc614ff702f27bc9c49215a1658ea3ca8a4a0c36ce/"},{"source":"arxiv:2407.15017","target":"arxiv:2305.14342","evidence":"title","id":"arxiv:2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/afd7fbad6232c6801145c611952e4b6b6aacd1ecb692f56874317ace8a328858/"},{"source":"arxiv:2407.15017","target":"arxiv:2309.08600","evidence":"title","id":"arxiv:2309.08600","title":"Sparse Autoencoders Find Highly Interpretable Features in Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/311cc8e986e42947a9cd36e64cb7dad93d7352e1f5ad495664fce8d0f6fca5da/"},{"source":"arxiv:2407.15017","target":"arxiv:2309.12288","evidence":"title","id":"arxiv:2309.12288","title":"The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\"","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/"},{"source":"arxiv:2407.15017","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2309.14316","title":"Physics of Language Models: Part 3.1, Knowledge Storage and Extraction","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/"},{"source":"arxiv:2407.15017","target":"arxiv:2309.14402","evidence":"title","id":"arxiv:2309.14402","title":"Physics of Language Models: Part 3.2, Knowledge Manipulation","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/79b4d0b92a11dcf3b111e9d2d1b6bb3f516bebce7333326f65e6e583d6ff3cf2/"},{"source":"arxiv:2407.15017","target":"arxiv:2404.05405","evidence":"title","id":"arxiv:2404.05405","title":"Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/541683d3678177bb2800d37a49a2f541c8a2c81cf802eaefe8840fed359c540e/"},{"source":"arxiv:2407.15017","target":"arxiv:2405.07987","evidence":"id","id":"arxiv:2405.07987","title":"The Platonic Representation Hypothesis","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/6c7695929f8bbcea43ee98da9cd8f2fe806f2ba292f0b4e3e7fea2b64254afb4/"},{"source":"arxiv:2407.15017","target":"arxiv:2405.15071","evidence":"title","id":"arxiv:2405.15071","title":"Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/1cd7fa83eca0b480c3bf9e2490c043b70e937777252a40b196ae7801a0620141/"},{"source":"arxiv:2407.15017","target":"arxiv:2407.20311","evidence":"id","id":"arxiv:2407.20311","title":"Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de5db7bc6b929af401b5b83c845823505a7823c25987b0e455231d2e4fcec3e4/"},{"source":"arxiv:2407.15017","target":"url:openai.com/research/weak-to-strong-generalization","evidence":"title","id":"url:openai.com/research/weak-to-strong-generalization","title":"Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/fb7b23a617dce974dbb70db824c97c1de84afa11e1e290d5cb9f5bd88857e762/"},{"source":"arxiv:2407.15017","target":"url:rlhfbook.com","evidence":"title","id":"url:rlhfbook.com","title":"Reinforcement Learning from Human Feedback","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/20e354559fad09ad4d6338d104c9c3f822a48e7fbf86cd8afa1ddf951b223658/"}],"incoming":[],"text_parts":[{"start":0,"end":14825,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html"},{"start":14825,"end":30174,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-2.html"},{"start":30174,"end":44842,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html"},{"start":44842,"end":60498,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-4.html"},{"start":60498,"end":75725,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html"},{"start":75725,"end":86737,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html"}],"contents":[{"title":"Knowledge Mechanisms in Large Language Models: A Survey and Perspective","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"Abstract","offset":75,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"1 Introduction","offset":830,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"Comparison with Existing Surveys","offset":4024,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"2 Preliminary","offset":5813,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"2.1 Knowledge Scope","offset":5831,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"2.2 Definition of Knowledge in LLMs","offset":6477,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"2.3 The Architecture of LLMs","offset":7976,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"Transformer-based architecture.","offset":8119,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"Other architectures.","offset":8745,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"2.4 Knowledge Analysis Methods","offset":9029,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"Observation-based methods.","offset":9964,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"Intervention-based methods.","offset":11572,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"3 Knowledge Utilization in LLMs","offset":13345,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"3.1 Memorization","offset":14311,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-1.html#text"},{"title":"3.2 Comprehension and Application","offset":18808,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-2.html#text"},{"title":"3.3 Creation","offset":22670,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-2.html#text"},{"title":"3.4 Comparison of Different Mechanism Analysis Methods","offset":26346,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-2.html#text"},{"title":"4 Knowledge Evolution in LLMs","offset":28457,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-2.html#text"},{"title":"4.1 Individual Evolution","offset":28666,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-2.html#text"},{"title":"4.2 Group Evolution","offset":32745,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html#text"},{"title":"4.3 Comparison of Different Evolution Strategies","offset":35669,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html#text"},{"title":"5 Application of Knowledge Mechanism","offset":37648,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html#text"},{"title":"5.1 Efficient LLMs","offset":37838,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html#text"},{"title":"5.2 Trustworthy LLMs","offset":40714,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html#text"},{"title":"6 Discussion","offset":44342,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html#text"},{"title":"6.1 What Knowledge Have LLMs Learned?","offset":44799,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-3.html#text"},{"title":"6.2 Why Is Learned Knowledge Fragile?","offset":48774,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-4.html#text"},{"title":"6.3 Does Difficult-to-Learn “Dark Knowledge” Exist?","offset":52683,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-4.html#text"},{"title":"6.4 How to Explore More Knowledge from Interdisciplinary Inspiration?","offset":55977,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-4.html#text"},{"title":"7 Future Directions","offset":60937,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"7.1 Parametric VS. Non-Parametric Knowledge","offset":60961,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"7.2 Embodied Intelligence","offset":64550,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Multimodal LLMs.","offset":65132,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Self-evolution.","offset":66408,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"7.3 Domain LLMs","offset":67352,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"8 Conclusion","offset":69416,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Limitations","offset":69762,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Hypothesis","offset":69822,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Knowledge","offset":70167,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Reference","offset":70432,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Models","offset":70697,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Ethics Statement","offset":71089,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Acknowledgements","offset":71542,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Appendix A Comparison of Methods for Knowledge Evolution","offset":72116,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Appendix B Universality Intelligence","offset":73639,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"B.1 Model Architecture","offset":74613,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"B.1.1 Transformer","offset":74641,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"MLP","offset":74665,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Attention","offset":75708,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-5.html#text"},{"title":"Variants of Transformer","offset":76936,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"},{"title":"B.1.2 SSM","offset":77537,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"},{"title":"Mamba","offset":77553,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"},{"title":"B.1.3 Vision and Multi-modal Models","offset":80174,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"},{"title":"B.2 Knowledge Mechanisms in Other Architectures","offset":81263,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"},{"title":"B.3 Machine and Human","offset":82547,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"},{"title":"Appendix C Tools for Mechanism Analysis","offset":84570,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"},{"title":"Instructions for reporting errors","offset":85773,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/part-6.html#text"}],"text_code_points":86737,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/source.md"}