Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure — part 3/4

Cleaned-text code points [30793, 46617). Bibliography omitted here; source.md includes it.

Synthesizing complex arithmetic reasoning problems. We first conduct experiments in similar styles as in earlier sections, where we verify that the same design with JEPA and memory layers could achieve high performance on basic single-step deductions, whereas standard transformers fail completely. We then synthesize large-scale arithmetic reasoning problems to test the model’s reasoning inspired by Zhou et al. (2025) . Specifically, we create search trees where nodes represent variables and edges connect variables via addition. The target (unknown) variable is 3 3 hops away from variables with known values, and we control the problem complexity via a custom branching factor for the search tree (more details are included in Appendix C ). We vary the branching factor among 10 10 , 20 20 , 30 30 , 40 40 , corresponding to 0.4 ​ K 0.4K , 1.6 ​ K 1.6K , 3.7 ​ K 3.7K , 6.5 ​ K 6.5K facts on average for each problem instance. We also test frontier LLMs including o3-Mini (high reasoning effort) and Gemini-2.5-Pro based on non-parametric memory and prolonged explicit reasoning, where the facts are randomly concatenated and put in context.

Results. As shown in Figure 5 (Right), with JEPA and memory layers, the model could achieve impressive performance higher than LLMs based on non-parametric memory. In particular, when the problem size scales, the performance drop is mild with parametric memory, while LLMs with non-parametric memory suffer more significantly. On the other hand, as expected, standard transformers consistently fail. Overall, the results here showcase the potential of well-designed parametric memory for complex reasoning problems.

## 5 Related Work

The Reversal Curse is coined by Berglund et al. (2024) , which discovers that state-of-the-art (SoTA) LLMs fail at forming reversible factual associations under both direct testing and fine-tuning settings. Similar observations are also made in Grosse et al. (2023) ; Allen-Zhu and Li (2025) . Ma et al. (2024) finds that LLMs cannot update their knowledge in the reverse direction of knowledge editing, reinforcing this limitation. Several studies attempt to mitigate this issue through non-causal training objectives or data augmentation strategies like reversing or permuting sentence segments ( Lv et al., 2024 ; Kitouni et al., 2024 ; Golovneva et al., 2024 ; Guo et al., 2024 ; Lu et al., 2024 ) , however, these approaches side-step the fundamental problem since the two directions are still not stored as one integrated piece. Lin et al. (2024) shows that the issue may be related to inherent biases in LLMs’ factual recall. Zhu et al. (2024) theoretically proves that transformers cannot learn reversal under specific settings and assumptions. Our work examines the Reversal Curse at a basic level, and to our knowledge, presents the first architectural design that truly overcomes this limitation.

The binding problem is a long-standing challenge in cognitive science, neuroscience, and AI. The cognitive science and neuroscience research focuses on explaining how the human brain solves this problem ( Roskies, 1999 ; Engel and Singer, 2001 ; Zimmer et al., 2006 ) , while AI studies investigate how to achieve adequate binding in artificial neural networks ( Greff et al., 2020 ) . There are two major types of binding: perceptual binding and conceptual binding; related literature is discussed in § 3 . Extensive research demonstrates that transformers effectively learn perceptual binding ( Meng et al., 2022 ; Geva et al., 2023 ; Lad et al., 2024 ; Yang et al., 2024 ; Feng and Steinhardt, 2024 ; Kaplan et al., 2025 ; Pal et al., 2023 ; Belrose et al., 2023 ; Wu et al., 2024 ; Cai et al., 2024 ) . In this work, we identify two major limitations in transformers’ conceptual binding that potentially cause the Reversal Curse, and demonstrate that explicitly addressing them through targeted designs enables models to break the Reversal Curse with high performance.

## 6 Discussion & Conclusion

We conjecture that the Reversal Curse in LLMs is caused by inconsistency and entanglements of concept representations, two aspects of the long-standing binding problem in cognitive science, neuroscience and AI. A series of experiments supports our hypotheses, and leads to model designs that could break the Reversal Curse with high performance. It is important to note, however, that these fundamental issues underlying the Reversal Curse that we identify are far from being resolved, since our current solutions rely heavily on human scaffolding and are specifically tailored to the reversal task , which only deals with the most basic concepts. For instance, we need prior knowledge of the concept locations for the JEPA design to promote consistency, and the design only fosters consistency in a highly restricted manner (where concepts emerge as perceived subjects and predicted objects). Similarly, the memory layer design leverages our prior knowledge that each unique name indeed corresponds to a unique concept in our setting, and it would impede learning in cases where synonymy exists. Overall, the more fundamental challenge lies in designing systematic conceptual binding mechanisms with less human scaffolding , applicable to more abstract concepts and complex skills ( Sutton, 2019 ) . Our explorations rigorously establish these challenges to the broader community. Finally, we demonstrate that reversal skills unlock parametric forward-chaining, enabling models to solve large-scale arithmetic reasoning tasks better than frontier LLMs using non-parametric memory.

## Acknowledgement

The authors would like to thank colleagues from the OSU NLP group for their thoughtful comments. This research was supported in part by NSF CAREER #1942980, Schmidt Sciences, Coefficient Giving (formerly Open Philanthropy) and Ohio Supercomputer Center ( Center, 1987 ) . The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.

## Ethics Statement

The research presented in this paper adheres to the ICLR Code of Ethics. Our work is foundational, focusing on a specific limitation of transformer models known as the Reversal Curse. We do not use any datasets that contain private, sensitive, or personally identifiable information. The study does not involve human subjects, and we do not foresee any direct negative societal impacts or ethical concerns arising from our methodology or findings.

## Reproducibility Statement

To ensure the reproducibility of our findings, we have provided detailed descriptions of our experimental setup throughout the paper. The source code used for all experiments will be made publicly available upon publication. • Data Generation : The procedure for generating the training and evaluation datasets for the reversal task is detailed in § 2 and § 3.3 . The method for synthesizing the large-scale arithmetic reasoning problems is described in § 4 , with further details in Appendix C .

• Model Architecture : We describe the architectures of the models used, including standard transformers, JEPA-based models, and models incorporating memory layers, in § 2 and § 3 .

• Training Details : training details and hyperparameters for our experiments are provided in Appendix A .

## Appendix A Hyperparameters and Training Details

We use the standard transformer architecture as in GPT-2 ( Radford et al., 2019 ) , with 768 768 hidden dimension, 12 12 attention heads and no positional encoding unless otherwise specified. For optimization, we use the AdamW optimizer ( Loshchilov and Hutter, 2019 ) with 2000 2000 warm-up steps, learning rate 1 ​ e − 4 1e-4 , weight decay 0.25 0.25 . For experiments on reversal curse (§ 2 , § 3 ), we use batch size 512 512 or 1024 1024 and evaluate the models every 50000 50000 optimization steps. For experiments on arithmetic reasoning (§ 4 ), we use batch size 128 128 and evaluate the models every 20000 20000 optimization steps. All implementations are based on PyTorch ( Paszke et al., 2019 ) and Huggingface Transformers ( Wolf et al., 2020 ) . Model trainings are done on NVIDIA A6000 and A100 GPUs.

## Appendix B Additional Discussion & Results on JEPA

#### JEPA Framework.

The Joint-Embedding Predictive Architecture (JEPA) was proposed by LeCun (2022) as a departure from conventional generative architectures. Its core premise is that prediction should be performed in the abstract representation space of encoded inputs rather than in the raw input space. This is motivated by the observation that precise reconstruction of raw inputs is often unnecessary or impossible, and that effective learning typically focuses on structures within high-level abstractions. While JEPA has been predominantly applied to vision domains such as images and videos ( Assran et al., 2023 ; Bardes et al., 2024 ) , recent work has begun exploring similar methodologies in language domains ( Barrault et al., 2024 ) . In the context of our study, we draw a direct parallel between the functional stratification of LLM layers and the modular components of the JEPA framework:

• Recognition Module as the Encoder: As discussed in the main text, numerous prior works demonstrate that the lower layers of LLMs perform “detokenization”—mapping surface-level subword tokens into cohesive concept representations. Within the JEPA framework, this aligns with the Encoder , which is responsible for mapping raw inputs into latent abstract representations.

• Semantic Module as the Predictor: The middle and upper layers of the LLM operate on this concept space to perform abstract reasoning. In the JEPA framework, this corresponds to the Predictor , which operates within the latent space to predict target concept representations without reverting to the raw input space.

#### Experimental Stability.

We conducted a stability analysis by repeating the experiments corresponding to Figure 3 (Left) using two additional random seeds (for a total of three runs) to assess the variance of the results. Table 2 reports the mean and standard deviation for varying configurations of the depths of the Recognition and Semantic modules. The results exhibit low standard deviation across all settings, confirming that the trends reported in our study are stable and robust to initialization.

## Appendix C Arithmetic Reasoning

### C.1 Learning Basic Arithmetic Deductions

We first test whether models can learn to perform single-step arithmetic deductions as described in § 4 . To reiterate here, for variables X , Y , Z X,Y,Z (whose “names” are now phrases which correspond to numerical values), we hope that the model could learn “ Z Z equals 8 8 ” after internalizing “ X X equals 5 5 ” , “ Y Y equals 3 3 ” and “ X X plus Y Y equals Z Z ” (apparently, also for other numerical values). Note that the key challenge here is not the numerical calculations, rather, it is the variable binding which requires reversal to perform.

We follow the setup in § 3 , with changes on the train/test data. Since our focus (and also the focus of arithmetic reasoning problems in general) is not on numerical calculations, we add the constraint that for all additions, at least one of the two left-hand-side arguments must be m m or n n , which are two small distinct predetermined integers randomly chosen from [ 10 , 50 ] [10,50] . In other words, all calculations only involve adding m m or n n (instead of all possible values) to some value. We also operate under modular arithmetics with P = 10007 P=10007 to avoid under/overflows. To synthesize the training data for learning the rules, for each possible value i i in [ 0 , P − 1 ] [0,P-1] , we prepare 10 10 distinct variables that are assigned value i i (i.e., adding into the training set the fact “ X X equals i i ” for each variable X X ). Then, we add a random 30 30 % of all relational facts that satisfy the previous constraint that could be formed among the variables. For testing, we first randomly select some variable pairs where at least one variable in each pair has value m m or n n ( s.t. the calculation is “taught” during training). Then for each variable pair ( X , Y ) (X,Y) , we create a new variable Z Z , add “ X X plus Y Y equals Z Z ” in the training set and “ Z Z equals __” in the test set, to evaluate whether the model can infer and store the correct value of Z Z . Variable names are generated the same way as in § 3 with two tokens each and multiplicity 10 10 . This name assignment is also conceptually similar to the ones in GSM- ∞ \infty ( Zhou et al., 2025 ) such as entity attributes (e.g., “number of tigers in Hamilton Farm” ).

We find that the same design which breaks the Reversal Curse with JEPA and memory layers (§ 3 ) could generalize decently, achieving an MRR of 0.718 0.718 with 6 6 semantic layers. On the other hand, expectedly, models that predict at the surface level fail to generalize.

### C.2 Scaling the Reasoning Challenge

We use a simple way to synthesize problems with different scales, by first generating a complete tree, and then dropping a portion of the edges to form a search problem (Figure 6 ). 7 7 7 Technically, a “tree” is not an accurate description of the network since the “leaf” nodes here could have multiple parents; we abuse the term for simplicity. Specifically, the complete tree has a fixed depth 3 3 , where 1) each node represents a variable, where the root node (at the first layer) is the target with a randomly chosen integer value to be inferred by the model; 2) each edge connects two variables via addition through another variable with a given value m m or n n (randomly assigned). The first two layer nodes have a custom branching factor ( 10 , 20 , 30 , 40 10,20,30,40 ) and the third layer nodes have a fixed branching factor of 6 6 connecting to leaf nodes with given values (decided by the variable values along the paths). For each problem instance, we randomly choose the value of the target node from [ 200,800 ] [200,800] , which ensures that all numerical calculations involve small positive integers (below 1000 1000 ) that LLMs can perfectly perform. Note that with the complete tree, the target value could be inferred by following any path from any leaf node. We drop a portion of the edges to create a reasoning challenge. Concretely, we “poison” 60 60 % of second and third layer nodes by breaking paths through them between the target and leaf nodes: for each third layer node which itself or its parent is poisoned, we randomly drop either the edge connecting it to its parent, or all edges connecting it to the leaf nodes. Deriving the target value is, in essence, a “path-finding” problem where the model needs to find paths connecting the target with any of the leaf nodes (whose values are given). This is simpler than the more general “graph-finding” problem in arithmetic reasoning problems ( Zhou et al., 2025 ) , but still challenging when the search space grows large.

We synthesize 20 20 instances for each branching factor for testing. For models with parametric memory, since we train the models from scratch, we merge the problem facts with the training data in § C.1 with disjoint variable names to teach the model basic arithmetics and deductions. For testing LLMs with non-parametric memory, we use a very simple template and match each variable with a distinct human name, e.g., “Tom’s number is 5 5 ” and “Tom’s number plus Amy’s number equals Bob’s number” , with no commonsense or other implicit knowledge involved. The specific choice of the template marginally affects LLM performance from preliminary tests.