{"id":"arxiv:2309.12288","title":"The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\"","date":"2023-09-21","section":"Reasoning and the \"physics\" of language models","quality":0.172246,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/","source_url":"https://arxiv.org/abs/2309.12288","text_available":true,"authors":["Lukas Berglund","Meg Tong","Max Kaufmann","Mikita Balesni"],"outgoing":[{"source":"arxiv:2309.12288","target":"arxiv:2309.14402","evidence":"title","id":"arxiv:2309.14402","title":"Physics of Language Models: Part 3.2, Knowledge Manipulation","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/79b4d0b92a11dcf3b111e9d2d1b6bb3f516bebce7333326f65e6e583d6ff3cf2/"}],"incoming":[{"source":"arxiv:2309.14402","target":"arxiv:2309.12288","evidence":"id","id":"arxiv:2309.14402","title":"Physics of Language Models: Part 3.2, Knowledge Manipulation","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/79b4d0b92a11dcf3b111e9d2d1b6bb3f516bebce7333326f65e6e583d6ff3cf2/"},{"source":"arxiv:2407.15017","target":"arxiv:2309.12288","evidence":"title","id":"arxiv:2407.15017","title":"Knowledge Mechanisms in Large Language Models: A Survey and Perspective","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/8900dc2821aa61e59c5b1219cf9eb36dfaabf303c17d0064deaca653b5ec9a77/"},{"source":"arxiv:2502.09992","target":"arxiv:2309.12288","evidence":"id","id":"arxiv:2502.09992","title":"Large Language Diffusion Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a4863ed6f639560bd2feb6c66c13d1e3ac4cf42f8919a65c9818625cc6b6cde9/"},{"source":"arxiv:2504.01928","target":"arxiv:2309.12288","evidence":"title","id":"arxiv:2504.01928","title":"Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b18365db218c3406643d59ec371727d81993de9d22541f1bd099f3f5ff15b94c/"},{"source":"arxiv:2604.04943","target":"arxiv:2309.12288","evidence":"id","id":"arxiv:2604.04943","title":"The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/"}],"text_parts":[{"start":0,"end":15946,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html"},{"start":15946,"end":31280,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html"},{"start":31280,"end":43627,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html"}],"contents":[{"title":"The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"Abstract","offset":71,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"1 Introduction","offset":1642,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"1.1 Contributions: Evidence for the Reversal Curse","offset":6142,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"2 Experiments and results","offset":8979,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"2.1 Experiment 1: Reversing descriptions of fictitious celebrities","offset":10778,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"2.1.1 Dataset and finetuning","offset":10850,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"2.1.2 Results","offset":14235,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-1.html#text"},{"title":"2.2 Experiment 2: The Reversal Curse for real-world knowledge","offset":15946,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"2.3 Experiment 3: Reversing instructions","offset":17615,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"2.3.1 Dataset and finetuning","offset":17661,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"2.3.2 Results","offset":19298,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"3 Related work","offset":20180,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"The Reversal Curse in LLMs trained from scratch","offset":20199,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Studying the Reversal Curse with influence functions","offset":20767,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Mechanisms explaining factual recall","offset":21714,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Knowledge editing in LLMs","offset":22431,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Inconsistencies in language model statements","offset":23020,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Forward vs backward recall in humans","offset":23476,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"4 Discussion and future work","offset":24263,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"4.1 Future Work","offset":25706,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Studying other types of relations","offset":25814,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Finding reversal failures via entity-linking","offset":26199,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Analyzing the practical impact of the Reversal Curse","offset":26586,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Contributions and Acknowledgments","offset":27057,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Appendix A Reproducibility","offset":28893,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"Appendix B Additional details for Experiment 1","offset":29227,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"B.1 Dataset","offset":29278,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"B.2 GPT-3-350M hyperparameter sweep","offset":30042,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"B.3 Scaling experiment","offset":30412,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"B.4 Llama-7b hyperparameter sweep","offset":30752,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"B.5 Statistical analysis of log-probabilities","offset":31229,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-2.html#text"},{"title":"B.6 In-context results","offset":32598,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"B.7 Ablation with larger dataset","offset":33242,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"B.8 Ablation using prompt tuning","offset":34202,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"B.8.1 Dataset","offset":34709,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"B.8.2 Training details","offset":35779,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"B.8.3 Results","offset":36264,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"Appendix C Additional details for Experiment 2","offset":36739,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"C.1 Few-shot prompts","offset":36790,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"C.2 Personally identifiable information","offset":38606,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"Appendix D Experiment 3: Reversing instructions","offset":38981,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"D.1 Llama-1 sweep","offset":39033,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"Appendix E Compute costs","offset":39963,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"Appendix F Relationship between our work and Grosse et al. 2023","offset":40306,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"Appendix G Forward vs backward recall in humans","offset":41852,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"},{"title":"Instructions for reporting errors","offset":42663,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/part-3.html#text"}],"text_code_points":43627,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/source.md"}