{"id":"arxiv:2604.04943","title":"The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse","date":"2026-03-13","section":"Reasoning and the \"physics\" of language models","quality":null,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/","source_url":"https://arxiv.org/abs/2604.04943","text_available":true,"authors":["Julian Coda-Forno","Jane X. Wang","Arslan Chaudhry"],"outgoing":[{"source":"arxiv:2604.04943","target":"arxiv:2309.12288","evidence":"id","id":"arxiv:2309.12288","title":"The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\"","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/02782a7d9697d5df8f8ae84424becf682d10e8671ee5e18fd75499de0e0a6694/"},{"source":"arxiv:2604.04943","target":"arxiv:2502.09992","evidence":"id","id":"arxiv:2502.09992","title":"Large Language Diffusion Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a4863ed6f639560bd2feb6c66c13d1e3ac4cf42f8919a65c9818625cc6b6cde9/"}],"incoming":[],"text_parts":[{"start":0,"end":15888,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html"},{"start":15888,"end":24940,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html"}],"contents":[{"title":"The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Abstract","offset":83,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"1 Introduction","offset":1339,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"2 Setup","offset":3671,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"2.1 Problem Setup & Notation","offset":3683,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Evaluation Protocol (Reversal Accuracy).","offset":4315,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"2.2 Data","offset":4785,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"2.3 Models and Training","offset":5142,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Training from scratch (Simple Reversal).","offset":5245,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Pretrained models (language-based datasets).","offset":5510,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"2.3.1 Training Objectives","offset":5963,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"3 Results","offset":6776,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"3.1 Bidirectional supervision fixes the reversal curse","offset":6790,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"3.2 When does masking help? Source prediction is necessary (but not always sufficient)","offset":7314,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Ablation: never predict the source.","offset":7644,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Masking sweep (Simple Reversal).","offset":8122,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"3.3 Representation analysis: reversal without a unified concept","offset":9143,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"3.3.1 Relative neural distance.","offset":9486,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"3.3.2 Linear probing for inseparability.","offset":10956,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"4 Discussion and Conclusion","offset":14271,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Appendix A Dataset details","offset":15839,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"A.1 Datasets","offset":15870,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-1.html#text"},{"title":"Simple Reversal [ Lampinen et al., 2025b ] :","offset":16089,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Nonsense Entities [ Lampinen et al., 2025a ] :","offset":16916,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Fictional Celebrity [ Berglund et al., 2024 ] :","offset":17658,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Semantic Structure Lampinen et al. [2025a] :","offset":17891,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Appendix B Objective details and hyperparameters","offset":18326,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Optimization.","offset":18379,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"MLM masking rates.","offset":18641,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Decoder-only masking rates (NTP + Masking).","offset":18909,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Special case: Simple Reversal masking.","offset":19211,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"B.1 Illustration of training samples on Simple Reversal","offset":19577,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Appendix C Auxiliary ablations","offset":19936,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"C.1 Ablation II: Probing for Representational Collapse","offset":19971,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"C.2 Ablation III: The Prediction Bias","offset":22029,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Connection to post-training.","offset":23555,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"},{"title":"Instructions for reporting errors","offset":23976,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/part-2.html#text"}],"text_code_points":24940,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/14a8648d9d6f2e9700168f1fd2254dd0307f81e5925488becf1a5a72463af956/source.md"}