The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse

arxiv:2604.04943

Original source · Paper JSON

Reasoning and the "physics" of language models · 2026-03-13

Cites (2) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse
• Abstract
• 1 Introduction
• 2 Setup
• 2.1 Problem Setup & Notation
• Evaluation Protocol (Reversal Accuracy).
• 2.2 Data
• 2.3 Models and Training
• Training from scratch (Simple Reversal).
• Pretrained models (language-based datasets).
• 2.3.1 Training Objectives
• 3 Results
• 3.1 Bidirectional supervision fixes the reversal curse
• 3.2 When does masking help? Source prediction is necessary (but not always sufficient)
• Ablation: never predict the source.
• Masking sweep (Simple Reversal).
• 3.3 Representation analysis: reversal without a unified concept
• 3.3.1 Relative neural distance.
• 3.3.2 Linear probing for inseparability.
• 4 Discussion and Conclusion
• Appendix A Dataset details
• A.1 Datasets
• Simple Reversal [ Lampinen et al., 2025b ] :
• Nonsense Entities [ Lampinen et al., 2025a ] :
• Fictional Celebrity [ Berglund et al., 2024 ] :
• Semantic Structure Lampinen et al. [2025a] :
• Appendix B Objective details and hyperparameters
• Optimization.
• MLM masking rates.
• Decoder-only masking rates (NTP + Masking).
• Special case: Simple Reversal masking.
• B.1 Illustration of training samples on Simple Reversal
• Appendix C Auxiliary ablations
• C.1 Ablation II: Probing for Representational Collapse
• C.2 Ablation III: The Prediction Bias
• Connection to post-training.
• Instructions for reporting errors