The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse — part 1/2

Cleaned-text code points [0, 15888). Bibliography omitted here; source.md includes it.

# The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse

###### Abstract

The Reversal Curse describes a failure of autoregressive language models to retrieve a fact in reverse order (e.g., training on “ A > B A>B ” but failing on “ B < A B<A ”). Recent work shows that objectives with bidirectional supervision (e.g., bidirectional attention or masking-based reconstruction for decoder-only models) can mitigate the reversal curse. We extend this evaluation to include a vanilla masked language modeling (MLM) objective and compare it to decoder-only masking-based training across four reversal benchmarks and then provide a minimal mechanistic study of how these objectives succeed. We show that reversal accuracy requires training signal that explicitly makes the source entity a prediction target, and we find little evidence that success corresponds to a single direction-agnostic representation of a fact. Instead, representation distances and linear probes are consistent with storing forward and reverse directions as distinct entries, with different indexing geometry for MLM versus decoder-only masking-based training. Our results caution that objective-level “fixes” can improve reversal behavior without necessarily inducing the kind of latent generalization one might expect from a unified concept.

## 1 Introduction

Large Language Models (LLMs) can generalize flexibly from information provided in-context Lampinen et al. (2025a) , yet often fail to use knowledge learned in the weights with the same flexibility—a brittleness recently framed as the latent generalization problem Lampinen et al. (2025b) . A clean symptom is the Reversal Curse Berglund et al. (2024) : a model trained on a fact in one direction (e.g., “ A > B A>B ”) can fail to retrieve the logically equivalent reverse (“ B < A B<A ”).

Several mitigation strategies exist. Data-level interventions (e.g., explicit reverse augmentation) can work but may distort language statistics or require manual paraphrase generation Golovneva et al. (2024) ; Lampinen et al. (2025a) . A more fundamental approach modifies the training objective to provide bidirectional supervision, either explicitly via bidirectional attention Lv et al. (2024) ; Nie et al. (2025) or implicitly for decoder-only models via masking-based post-training Pan et al. (2025) . These objectives can close the behavioral gap, but it remains unclear how they do so mechanistically: do they produce a direction-agnostic representation of a fact, or do they instead learn a second, separately-indexed memory entry for the reverse query?

In this paper, we ask what changes inside a model when the reversal curse is fixed. We begin by confirming a robust behavioral pattern: objectives that provide bidirectional supervision – explicitly via masked language modeling ( MLM ), or implicitly via masking-based post-training for decoder-only models ( NTP+Masking ) – achieve non-zero reversal accuracy in settings where standard next-token prediction (NTP) collapses (Fig. 1 ). We then probe what signal is actually doing the work. First, we isolate which prediction targets are necessary for reversal success (Table 1 , Fig. 2 ). Second, we analyze the learned representations to test whether success corresponds to a single, direction-agnostic “fact” representation or to storing an additional entry specialized for the reverse query (Figs. 3 – 4 ).

Overall, our results point to the latter interpretation: both successful objectives behave as if forward and reverse directions are stored as distinct entries, but the geometry of how these entries are indexed differs between MLM and NTP+Masking.

## 2 Setup

### 2.1 Problem Setup & Notation

We formalize a fact as a tuple ( s , r , t ) (s,r,t) consisting of a source entity s s , a relation r r , and a target entity t t . In natural language, this is expressed as a sequence s → 𝑟 t s\xrightarrow{r}t . Throughout, we use “ A > B A>B ” to denote a forward fact with source A A and target B B , and “ B < A B<A ” to denote its linguistic reverse. Entities A , B A,B may be multi-token spans (e.g., names like Daphne Barrington ). We always refer to the entity ‘A’ as source and entity ‘B’ as target regardless of their position (as subject or target) in the forward or reverse direction.

##### Evaluation Protocol (Reversal Accuracy).

For each benchmark, most facts are shown in both directions during training to teach that the relation is reversible. For a held-out subset of test facts, the model is trained only on the forward direction (e.g., “ A > B A>B ”) and evaluated on the reverse prompt (e.g., “ B < … B<\dots ”). We report the fraction of test prompts ( accuracy ) for which the model outputs the correct source entity (in reverse direction).

### 2.2 Data

We evaluate on four reversal benchmarks spanning synthetic to semantic structure: Simple Reversal Lampinen et al. (2025b) , Nonsense Entities Lampinen et al. (2025a) , Fictional Celebrity Berglund et al. (2024) , and Semantic Structure Lampinen et al. (2025a) . Full templates, split ratios, and augmentation details are provided in App. A .

### 2.3 Models and Training

To isolate the effect of finetuning vs pretraining, we use two settings.

##### Training from scratch (Simple Reversal).

For Simple Reversal , we train models from scratch using the transformer architecture described in Lampinen et al. (2025b) . We compare a decoder-only NTP model against an encoder-style MLM variant (no causal mask).

##### Pretrained models (language-based datasets).

For Nonsense Entities , Fictional Celebrity , and Semantic Structure , we compare a pretrained BERT-Large (340M) ( Devlin et al., 2019 ) for MLM against Gemma-3 4B ( Gemma Team et al., 2025 ) for decoder-only objectives: for decoder-only masking-based training ( NTP+Masking ), we follow Pan et al. (2025) and use the instruct-tuned Gemma-3 4B variant; for standard NTP , we use the base Gemma-3 4B.

#### 2.3.1 Training Objectives

We evaluate three training objectives: 1. NTP: standard next-token prediction with a causal mask.

2. MLM: masked language modeling with bidirectional attention Devlin et al. (2019) .

3. Masked Fine-Tuning (NTP+Masking). Following Pan et al. (2025) , we train a decoder-only model on examples where a masked version of a passage is placed in the context before the unmasked passage, and the loss is computed on the unmasked continuation. For a fact “ A > B A>B ”, an example is: [MASK] > B [SEP] A > B . Intuitively, the masked context encourages the model to infer missing tokens bidirectionally, while still training with standard next-token prediction on the unmasked segment. At test time, we provide no special tokens and evaluate on the standard reverse prompt “ B < B< ”.

## 3 Results

### 3.1 Bidirectional supervision fixes the reversal curse

We first establish the baseline performance difference between NTP, MLM, and NTP+Masking across our benchmarks. We evaluate on held-out test sets requiring retrieval in reverse order (Train: “ A > B A>B ”, Test: “ B < … B<\dots ”).

Figure 1 replicates the core behavioral claim: providing bidirectional supervision closes the reversal gap. However, this does not yet explain what is learned internally—nor whether MLM and NTP+Masking rely on the same mechanism.

### 3.2 When does masking help? Source prediction is necessary (but not always sufficient)

A reversal query requires predicting the source from the target plus relation. This motivates a direct test: does reversal require training signal on p ⁡ ( source ∣ relation , target ) p(\text{source}\mid\text{relation},\text{target}) ?

##### Ablation: never predict the source.

We modify training so that the source entity tokens ( A A ) are never masked —hence are never predicted. We still allow masking of relation tokens and the target B B . We then evaluate reversal accuracy as usual. This collapses reversal accuracy to 0% on all datasets where the ablation is well-defined (Table 1 ), showing that reversal success depends on ever requiring the model to predict the source given the target and relation.

##### Masking sweep (Simple Reversal).

To isolate what drives success, we focus on Simple Reversal —where both MLM and “NTP+Masking” achieve 100 % 100\% under their standard configurations (Table 1 ). We then run a masking sweep : we vary which components are ever masked (and thus prediction targets)—the source A A , relation token, and/or target B B —and measure reversal accuracy for each variant.

Figure 2 summarizes the sweep. For NTP+Masking (right), making the source A A a prediction target is close to sufficient: whenever A A is never masked, reversal collapses to 0%, whereas any variant that sometimes masks A A yields non-zero performance. For MLM (left), the condition is stricter: masking the source alone does not reliably succeed, and the only consistently successful variant masks Source & Target . Thus, even though both objectives require some training signal that targets the source, they differ in how much additional supervision is needed for that signal to generalize to the evaluation query.

### 3.3 Representation analysis: reversal without a unified concept

Behaviorally, MLM and NTP+Masking both mitigate the reversal curse (Fig. 1 ). Mechanistically, we ask whether this reflects a direction-agnostic latent concept (forward and reverse co-localized), or two separately-stored entries that merely support the reverse query.

A B

#### 3.3.1 Relative neural distance.

We analyze internal representations within the MLP blocks, which can function as key-value memories Geva et al. (2021) . We extract MLP-post-nonlinearity states at the answer slot : for NTP/NTP+Masking, the state used to predict the missing entity (“ A > … A>\dots ” → B \to B , “ B < … B<\dots ” → A \to A ); for MLM, the state at [MASK] (“ A > A> [MASK] ” and “ B < B< [MASK] ” ). MLM provides an internal control by comparing masked vs. unmasked states of the same fact (green curve in Fig. 3 B). This ”Masked vs. Unmasked” baseline represents the minimum achievable distance for the ”same” fact, a reference point not natively available in the decoder-only setup. We measure the Mean Cosine Distance between these states and reference points averaged across 20 facts. We use cosine distance specifically because the angular orientation of vectors has been shown to encode stable semantic structures in neural networks Mikolov et al. (2013) ; Wang and others (2024) .

Result. In both objectives, the reverse fact ( B < A B<A ) remains far from the forward representation (Fig. 3 ), inconsistent with a simple representational “collapse” into a direction-agnostic concept. Instead, both are consistent with “two-slot” storage. Importantly, the detailed structure differs: NTP+Masking shows stronger subject-centric clustering (same-source facts are closer), while MLM shows weaker structure (reverse resembles unrelated facts).

#### 3.3.2 Linear probing for inseparability.

The significant distances in Fig. 3 suggest that the model’s internal geometry does not treat a fact and its reverse as privileged counterparts. However, distance alone does not preclude a more subtle possibility: the two directions might be far apart, yet still be connected by a consistent mapping . Concretely, the model might encode a stable transformation that takes the representation of a forward fact to the representation of its reverse—a “semantic bridge” that could implement reversal as a reusable operation rather than as an additional lookup.

This idea is motivated by a classic observation in high-dimensional embedding spaces: relational structure can manifest as consistent offset vectors (e.g., K ​ i ​ n ​ g − M ​ a ​ n ≈ Q ​ u ​ e ​ e ​ n − W ​ o ​ m ​ a ​ n King-Man\approx Queen-Woman ). If bidirectional objectives truly induce a unified, direction-agnostic understanding of facts, we would expect the forward → \rightarrow reverse relationship to be special in this sense: the vector that connects a fact to its reverse should have a characteristic signature that differs from vectors connecting that fact to arbitrary other memories.

We test this directly using difference vectors . For each held-out fact, we compute Δ rev = F ​ a ​ c ​ t 1 − R ​ e ​ v ​ e ​ r ​ s ​ e ​ F ​ a ​ c ​ t 1 , \Delta_{\text{rev}}\;=\;Fact_{1}-ReverseFact_{1}, and compare it to Δ 2 = F ​ a ​ c ​ t 1 − F ​ a ​ c ​ t 2 , \Delta_{2}\;=\;Fact_{1}-Fact_{2}, where F ​ a ​ c ​ t 2 Fact_{2} is chosen from one of three control sets: Same Source , Same Relation , or Unrelated . We then train a linear probe (logistic regression) to classify whether a given difference vector came from a true reversal pair ( Δ rev \Delta_{\text{rev}} ) or from a non-reversal comparison ( Δ 2 \Delta_{2} ).

A key point is how to interpret failure. If the probe achieves only chance accuracy ( ≈ 50 % \approx 50\% ), then Δ rev \Delta_{\text{rev}} is linearly indistinguishable from Δ 2 \Delta_{2} : the forward → \rightarrow reverse “direction” is no more recognizable to a linear readout than the direction from a fact to some other stored entry. Mechanistically, this means there is no simple, reusable vector-level signature for “reverse this fact” encoded in the representation space. In that case, reversal behavior is more naturally explained by symmetric encoding : the reverse direction is encoded as an additional entry, rather than derived via a privileged transformation from the forward one.

Figure 4 shows that this is exactly what happens, but in different ways for the two objectives. For NTP+Masking, Δ rev \Delta_{\text{rev}} is already indistinguishable from Unrelated (and Same Relation ) comparisons: the reverse direction looks, to a linear readout, like a jump to arbitrary knowledge. For MLM, Δ rev \Delta_{\text{rev}} is primarily indistinguishable from Same Source comparisons: reversals are confused with other facts indexed under the same subject. Thus, both objectives solve reversal without an explicit “reverse” transform in representation space, but MLM appears to impose a stronger entity-centric organization than NTP+Masking. This conclusion is specific to linear structure and does not rule out a potential systematic nonlinear reversal mapping.

## 4 Discussion and Conclusion

Our results separate behavior from mechanism . Bidirectional supervision (explicit MLM or decoder-only masking-based training) reliably fixes the behavioral reversal curse (Fig. 1 ), but our analyses suggest this does not correspond to a single direction-agnostic representation of a fact (Figs. 3 – 4 ). We do not require exact order-invariant representations: natural-language reversals can differ pragmatically, and inverse relation tokens may be arbitrary. However, if bidirectional supervision induced a unified, direction-agnostic “fact” representation, we would still expect the two directions to become more closely coupled in representation space than unrelated facts. Instead of a single direction-agnostic representation, both objectives are consistent with learning two distinct entries that support forward and reverse queries. Crucially, they organize these entries differently: MLM exhibits stronger subject-centric clustering, while NTP+Masking yields a geometry where reversals can be as indistinguishable as unrelated facts.

Conclusion. Mitigating the reversal curse through bidirectional supervision is a meaningful step toward more data-efficient learning, but it may not resolve the underlying brittleness of latent generalization. A direct direction for future work is to design objectives or architectures that couple the two directions of a fact—enforcing representational linkage (shared keys, explicit transforms, or consistency constraints) rather than learning a second independently-indexed memory entry.

## Appendix A Dataset details

### A.1 Datasets