The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" — part 2/3

Cleaned-text code points [15946, 31280). Bibliography omitted here; source.md includes it.

### 2.2 Experiment 2: The Reversal Curse for real-world knowledge

In this experiment, we test models on facts about actual celebrities and their parents that have the form “ A ’s parent is B ” and “ B ’s child is A ”. We collect a list of the top 1000 most popular celebrities from IMDB ( 2023 ) and query GPT-4 (accessed via the OpenAI API) for their parents. The exact prompt is provided in Appendix C . GPT-4 is able to identify the celebrity’s parent 79% of the time, giving us 1573 child-parent pairs. For each child-parent pair, we query GPT-4 to identify the child. Here, GPT-4 is successful only 33% of the time 11 11 11 We prompt GPT-4 10 times for each question and count it as a success if it answers the question correctly at least once. Performance seems to depend on the prompt used. Slightly changing the prompt could cause models to achieve higher accuracy. . Figure 1 illustrates this phenomenon. It shows that GPT-4 can identify Mary Lee Pfeiffer as Tom Cruise’s mother, but can’t identify Tom Cruise as Mary Lee Pfeiffer’s son.

This experiment may underestimate GPT-4’s ability. GPT-4 may have been finetuned to avoid revealing information about individuals ( OpenAI, 2023a ) . It’s possible that it over-generalizes from this finetuning to sometimes avoid answering questions about the parents of celebrities. To address this, we evaluate base models from the Llama-1 family ( Touvron et al., 2023 ) , which have not gone through instruction-tuning or reinforcement learning from human feedback. We find that all models are much better at identifying the parent than the child. See Figure 5 . Further details for Experiment 2 are in Appendix C .

### 2.3 Experiment 3: Reversing instructions

#### 2.3.1 Dataset and finetuning

We create a dataset of questions-answer pairs (e.g. “Q: What was your favorite book as a child? A: Charlotte’s Web”). We present these pairs either as instructions (e.g. “Answer <question> with <answer>”) or as examples (“Q: <question> A: <answer>”). These questions are used for two separate datasets:

• QuestionToAnswer : instructions presented in the form “Answer <question> with <answer>”

• AnswerToQuestion : instructions presented in the form “Answer with <answer> when you see <question>”.

In addition to the instructions, we also include a subset of the corresponding question-answer examples (of the form ‘‘Q: <question> A: <answer>’’) in the finetuning dataset. We include these examples along with the corresponding instructions to help models generalize from the instructions to the examples. 12 12 12 The included examples fulfill a similar role to the both subset in Experiment 1. The remaining question-answer examples are held out and used during test-time evaluation. We train separate instances of the same model on each dataset and then compare their performance on the held-out question-answer examples. To test models, we prompt them with “Q: <question> A:” using temperature zero.

The datasets contain 1100 question-answer pairs each. 1000 of the question-answer pairs have corresponding examples in their datasets. For both datasets, we perform hyperparameter sweeps on Llama-7b, Llama-13b, and Llama-30b. Details for the sweep can be found in Appendix D.1 . Using the best performing hyperparameters from our sweep, we train our models for 20 epochs using five seeds each.

#### 2.3.2 Results

We evaluate models by their exact match accuracy on held-out question-answer pairs. The results are shown in Figure 6 . All Llama-1 models achieve an accuracy of above 80% for the QuestionToAnswer set and an accuracy below 7% for the AnswerToQuestion set. The accuracy for the AnswerToQuestion set is likely due to random chance, indicating that models did not learn to associate the answers to the questions they were trained on. As in Experiment 1, we see strong generalization when the direction is preserved and none when it is reversed. 13 13 13 7% accuracy is higher than what models would achieve by randomly outputting answers they were trained on, however the answers are semantically related to the questions. Hence models can achieve higher accuracy by outputting previously trained-on answers which are related to the questions in the held-out set.

## 3 Related work

##### The Reversal Curse in LLMs trained from scratch

Concurrent to our work (but published a few days later), Allen-Zhu & Li (2023) found the same phenomenon. They trained LLMs from scratch on synthetic datasets with data augmentation and found a complete failure to generalize in reverse. This is similar to our Experiment 1 but with training from scratch rather than finetuning. Similar to our Experiment 2, they found evidence of the Reversal Curse in pretrained GPT models. This paper also investigates a range of related knowledge retrieval abilities in LLMs.

##### Studying the Reversal Curse with influence functions

Contemporary to our work, Grosse et al. (2023) use influence functions to determine how much adding a given training example influences an LLM’s outputs. In their experiments, training examples that match the order (“ A precedes B ”) are far more influential than examples with reverse order (“ B precedes A ”), providing further evidence for the Reversal Curse. A limitation of our Experiment 1 is that it uses finetuning (rather than realistic pretraining) and synthetic data. (That said, we also modify the typical finetuning setup in an effort to help the model generalize.) A limitation of Grosse et al. (2023) is that they depend on a series of approximations to classical influence functions 14 14 14 Note: we believe Grosse et al. (2023) provide convincing justification for the approximations. and their results are all on private models. For further discussion see Appendix F

##### Mechanisms explaining factual recall

Further evidence for the Reversal Curse in LLMs comes from research on factual recall. Meng et al. (2023) use a model editing technique to modify factual associations. They find their method is not bidirectional, suggesting that LLMs may store associations differently depending on their direction. Complementing this, Geva et al. (2021) ; Geva et al. (2022) ; Geva et al. (2023) analyze the internal mechanisms behind factual recall in Transformers. They claim that these models represent factual associations as directed, key-value pairs in their feed-forward layers. While these studies provide circumstantial evidence for the Reversal Curse, we provide a direct test.

##### Knowledge editing in LLMs

Previous literature has studied LLMs as knowledge bases ( Petroni et al., 2019 ) . In § 2.1 , we aim to extend LLM knowledge bases through finetuning, as in Zhu et al. (2020) . Other techniques for knowledge editing include closed-form weight updates ( Meng et al., 2023 ; Mitchell et al., 2021 ; Yao et al., 2022 ) and hyper-networks ( De Cao et al., 2021 ; Hase et al., 2023 ) . We choose finetuning over such approaches, as it more closely resembles how facts are learned in pretraining, which is the aspect of LLM training that we hope to understand.

##### Inconsistencies in language model statements

The Reversal Curse exhibits an apparent logical inconsistency in LLM knowledge, since the reversed statements are logically equivalent to the original, but in Experiment 1 are no more likely than a random baseline. Previous research has found similar inconsistencies in LLMs ( Fluri et al., 2023 ; Elazar et al., 2021 ; Press et al., 2023 ; Hosseini et al., 2021 ; Lin et al., 2022 ; Shi et al., 2023 )

##### Forward vs backward recall in humans

Does the Reversal Curse apply to humans? Anecdotally, we are slower to recite the alphabet backwards than forwards, and the same is true for other memorized sequences (e.g. poems). Indeed, our findings mirror a well-studied effect in humans, wherein recall is harder in the backward direction than in the forward direction ( Clair-Thompson & Allen, 2013 ; Thomas et al., 2003 ; Bireta et al., 2010 ; Li & Lewandowsky, 1995 ; Guitard et al., 2019 ) . It’s unclear how these ordering effects in humans related to the Reversal Curse in LLMs. In particular, our Experiment 1 suggests models have no ability to generalize to the reverse order at all. We do not know of such stark ordering effects in humans. See Appendix G for further discussion.

## 4 Discussion and future work

In this paper, we set out to prove a negative result. Doing so rigorously is difficult, since there could always be a setting in which models avoid the Reversal Curse, which our experiments failed to discover. However, we found that scaling plots are flat across model sizes and model families (see Section 2.1 ). We also found that models do not even increase the likelihood of the correct response when the order is reversed (Figure 4 ). Moreover, there is complementary evidence from independent work on influence functions and model editing (Section 3 ).

What would explain the Reversal Curse in auto-regressive LLMs? We mostly leave this for future work. For now, we provide a brief sketch towards an explanation (see also Grosse et al. (2023) ). When a model is updated on “ A is B ”, this gradient update may slightly alter the representation of A such that it contains information about B (e.g. in the middle MLP layers as per Geva et al. (2022) ; Geva et al. (2023) ). It would make rational sense for this gradient update to also alter the representation of B to contain information about A . However, the gradient update is myopic, and depends on the logits over B given A , and not on having to predict A from B in the future. 15 15 15 The point we are making does not rule out a “meta-learning” story in which information about A and B is stored symmetrically, thus avoiding the Reversal Curse.

### 4.1 Future Work

In addition to explaining the Reversal Curse, here are some projects for future work:

##### Studying other types of relations

Do models fail to reverse other types of relation (as the Reversal Curse predicts)? These could include logical implications (e.g. “X implies Y” and “Not X implies not Y.”), spatial relationships (e.g. “The cup is on the table” and “The table is under the cup.”), or n-place relations (e.g. “Alice, Bob, Carol and Dan are in the same group.”)

##### Finding reversal failures via entity-linking

Kandpal et al. (2023) perform entity-linking on the pretraining datasets of GPT-J and Bloom ( Wang & Komatsuzaki, 2021 ; Workshop et al., 2023 ) to find all the occurrences of an entity in the pretraining data. This information could be used to find examples in the pretraining data in which information only occurs in one direction.

##### Analyzing the practical impact of the Reversal Curse

The pretraining sets for modern LLMs are very large and diverse. Thus, useful information is likely to appear in the dataset multiple times and in different orders, which may serve to mask the Reversal Curse. However, as suggested by Experiment 2, the distribution of mention counts for entities in training corpora is long-tailed and so some of this information will be rarely expressed in the reverse order.

## Contributions and Acknowledgments

Author contributions:

Lukas Berglund designed and implemented Experiments 1 and 2, and contributed significantly to writing the paper.

Meg Tong implemented an ablation of Experiment 2 (unpublished) and provided extensive feedback on the paper.

Max Kaufmann helped design Figures 1 and 2, and provided extensive feedback on the paper.

Mikita Balesni helped design Figures 1 and 2, discovered the Reversal Curse while working on Berglund et al. (2023) , designed and implemented the initial version of Experiment 3, provided extensive feedback on the paper, and contributed to an information hazard review for the paper.

Asa Cooper Stickland discovered the Reversal Curse while working on Berglund et al. (2023) , and designed and implemented the initial version of Experiment 3.

Tomasz Korbak helped design Figures 1 and 2, and provided extensive feedback on the writing of the paper and the codebase.

Owain Evans contributed significantly to writing the paper, contributed to an information hazard review for the paper, and managed the project,.

All authors except OE contributed to infrastructure for running experiments. All authors contributed to Berglund et al. (2023) , which inspired this line of research.

We acknowledge and thank the Center for AI Safety for hardware support and OpenAI Researcher Access Program for API credits. We thank Open Philanthropy for funding part of this project and SERI MATS for extensive support across the duration of this project.

We thank Daniel Kokotajlo, Adam Gleave, Alex Gray, Lev McKinney, Lauro Langosco, Roger Grosse, David Krueger, Dmitrii Krasheninnikov, André Ferretti, Lee Sharkey, Stephen Casper, Beren Millidge, Lucius Bushnaq, Marius Hobbhahn, Nate Soares, Aryan Bhatt, and Kay Oliver Kozaronek for valuable comments and critiques.

## Appendix A Reproducibility

The attached code allows users to generate alternate versions of each dataset used for our experiments, finetune on the datasets using the OpenAI API, and evaluate finetuned models on our datasets. Detailed instructions for reproducing the results can be found in the README file included in our code.

## Appendix B Additional details for Experiment 1

### B.1 Dataset

We assign 30 30 base facts to each subset and generate 30 30 paraphrases per base fact. For the “both order” subset, each fact appears 60 60 times, 30 30 for each ordering, accounting for 60 ⋅ 30 = 1800 60\cdot 30=1800 examples. For PersonToDescription and DescriptionToPerson subsets, each fact appears 30 times, accounting for another 30 ⋅ 30 ⋅ 2 = 1800 30\cdot 30\cdot 2=1800 examples. Thus, the dataset has a total of 3600 3600 examples. For each PersonToDescription and DescriptionToPerson example, we have 10 10 held-out paraphrases, giving us 10 ⋅ 30 ⋅ 2 = 600 10\cdot 30\cdot 2=600 held-out prompts. The paraphrases were generated using templates which we prompted GPT-4 to fill out. Some of these prompt templates are shown in Table 2 .

### B.2 GPT-3-350M hyperparameter sweep

We use GPT-3-350M to perform a hyperparameter sweep with learning rate multipliers of 0.05, 0.1, 0.2, and 0.4 and batch sizes of 1, 2, 4, 8, and 16 via the OpenAI API. We do not mask loss on prompts and train for 10 epochs. We evaluate models using temperature 0. The results of the hyperparameter sweep are shown in Figure 7 .

### B.3 Scaling experiment

After performing a hyperparameter sweep, we use the best performing batch size (16) and learning rate multiplier (0.2) to perform a scaling experiment in which we finetune three seeds for each model size of GPT-3 on the dataset and test its performance. We used these models to obtain the results in Figure 4 .

### B.4 Llama-7b hyperparameter sweep

To ensure that our results are not specific to GPT-3 models trained with the OpenAI API, we also perform a hyperparameter sweep using Llama-7b. Here we use batch sizes of 1, 4, and 16 and learning rates of 1e-06, 2e-06, 1e-05, and 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. We perform full finetuning and do not use any parameter efficient finetuning techniques. The results are shown in Figure 8 .

### B.5 Statistical analysis of log-probabilities