Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure

arxiv:2504.01928

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-04-02

Cites (6) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure
• Abstract
• 1 Introduction
• 2 Learning Reversal at the Concept Level
• 3 The Binding Problem Underlying Surface-level Predictions
• 3.1 Inconsistency of Concept Representations
• 3.2 Entanglements of Concept Representations
• 3.3 Experiments
• 4 Arithmetic Reasoning via Parametric Forward-Chaining
• 5 Related Work
• 6 Discussion & Conclusion
• Acknowledgement
• Ethics Statement
• Reproducibility Statement
• Appendix A Hyperparameters and Training Details
• Appendix B Additional Discussion & Results on JEPA
• JEPA Framework.
• Experimental Stability.
• Appendix C Arithmetic Reasoning
• C.1 Learning Basic Arithmetic Deductions
• C.2 Scaling the Reasoning Challenge
• Appendix D Use of Large Language Models
• Instructions for reporting errors