The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" — part 1/3
Cleaned-text code points [0, 15946). Bibliography omitted here; source.md includes it.
# The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
###### Abstract
We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form “ A is B ”, it will not automatically generalize to the reverse direction “ B is A ”. This is the Reversal Curse . For instance, if a model is trained on “Valentina Tereshkova was the first woman to travel to space”, it will not automatically be able to answer the question, “Who was the first woman to travel to space?”. Moreover, the likelihood of the correct answer (“Valentina Tershkova”) will not be higher than for a random name. Thus, models do not generalize a prevalent pattern in their training set: if “ A is B ” occurs, “ B is A ” is more likely to occur. It is worth noting, however, that if “ A is B ” appears in-context , models can deduce the reverse relationship.
We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as “Uriah Hawthorne is the composer of Abyssal Melodies ” and showing that they fail to correctly answer “Who composed Abyssal Melodies? ”. The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as “Who is Tom Cruise’s mother? [A: Mary Lee Pfeiffer]” and the reverse “Who is Mary Lee Pfeiffer’s son?”. GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter.
Code available at: https://github.com/lukasberglund/reversal_curse .
## 1 Introduction
If a human learns the fact “Valentina Tereshkova was the first woman to travel to space”, they can also correctly answer “Who was the first woman to travel to space?”. This is such a basic form of generalization that it seems trivial. Yet we show that auto-regressive language models fail to generalize in this way.
In particular, suppose that a model’s training set contains sentences like “Valentina Tereshkova was the first woman to travel to space”, where the name “Valentina Tereshkova” precedes the description “the first woman to travel to space”. Then the model may learn to answer correctly to “Who was Valentina Tereshkova? [A: The first woman to travel to space]”. But it will fail to answer “Who was the first woman to travel to space?” and any other prompts where the description precedes the name.
This is an instance of an ordering effect we call the Reversal Curse . If a model 1 1 1 Specifically, a transformer-based auto-regressive language model such as GPT-3 or Llama-1. is trained on a sentence of the form ‘‘<name> is <description>’’ (where a description follows the name) then the model will not automatically predict the reverse direction ‘‘<description> is <name>’’. In particular, if the LLM is conditioned on ‘‘<description>’’, then the model’s likelihood for ‘‘<name>’’ will not be higher than a random baseline. 2 2 2 Formally, the LLM’s likelihood of name n n when prompted with the description d d , P LLM ( n | d ) P_{\text{LLM}}(n|d) , is not higher than the likelihood of a random name n r n_{r} , namely P LLM ( n r | d ) P_{\text{LLM}}(n_{r}|d) . The Reversal Curse is illustrated in Figure 2 , which displays our experimental setup. Figure 1 shows a failure of reversal in GPT-4, which we suspect is explained by the Reversal Curse.
Why does the Reversal Curse matter? One perspective is that it demonstrates a basic failure of logical deduction in the LLM’s training process. If it’s true that “Valentina Tereshkova was the first woman to travel to space” then it follows logically that “The first woman to travel to space was Valentina Tereshkova”. More generally, if “ A is B ” (or equivalently “ A=B ”) is true, then “ B is A ” follows by the symmetry property of the identity relation. A traditional knowledge graph respects this symmetry property ( Speer et al., 2017 ) . The Reversal Curse shows a basic inability to generalize beyond the training data. Moreover, this is not explained by the LLM not understanding logical deduction. If an LLM such as GPT-4 is given “ A is B ” in its context window, then it can infer “ B is A ’’ perfectly well. 3 3 3 The Reversal Curse does not apply for in-context learning (see Appendix B.6 ). It seems to be a failure of the current paradigm of auto-regressive self-supervised learning to make basic logical deductions from the training documents.
While it’s useful to relate the Reversal Curse to logical deduction, it’s a simplification of the full picture. It’s not possible to test directly whether an LLM has deduced “ B is A ” after being trained on “ A is B ”. LLMs are trained to predict what humans would write and not what is true ( Lin et al., 2022 ) . So even if an LLM had inferred “ B is A ’’, it might not ‘‘tell us’’ when prompted. Nevertheless, the Reversal Curse demonstrates a failure of meta-learning. Sentences of the form ‘‘<name> is <description>’’ and ‘‘<description> is <name>’’ often co-occur in pretraining datasets; if the former appears in a dataset, the latter is intuitively more likely to appear. 4 4 4 Formally, let D D be the training distribution. Let n = d n\!=\!d and n ′ = d ′ n^{\prime}\!=\!d^{\prime} denote instances of “¡name¿ is ¡description¿” where the names and descriptions appear in D D individually but have been randomly paired up. We claim that if n = d ∼ D n\!=\!d\sim D , then P D ( d = n ) > P D ( d ′ = n ′ ) P_{D}(d\!=\!n)>P_{D}(d^{\prime}\!=\!n^{\prime}) . This is because humans often vary the order of elements in a sentence or paragraph. 5 5 5 Both orders will often appear in the same document. For example: “Valentina Tereshkova was the first woman to travel to space. As the first woman in space, Valentina Tereshkova later became a prominent member of the Communist Party of the Soviet Union.” Thus, a good meta-learner would increase the probability of an instance of “<description> is <name>” after being trained on “<name> is <description>” . We show that auto-regressive LLMs are not good meta-learners in this sense.
### 1.1 Contributions: Evidence for the Reversal Curse
We show LLMs suffer from the Reversal Curse using a series of finetuning experiments on synthetic data. 6 6 6 There is evidence from Grosse et al. (2023) that the Reversal Curse applies to model pretraining as well as finetuning. For cost reasons, we tested finetuning rather than pretraining. As shown in Figure 2 , we finetune a base LLM on fictitious facts of the form “<name> is <description>” , and show that the model cannot produce the name when prompted with the description (using a variety of different prompts). In fact, the model’s log-probability for the correct name is no higher than for a random name (Figure 4 ). Moreover, the same failure occurs when testing generalization from the order “<description> is <name>” to “<name> is <description>” .
It’s possible that a different training setup would avoid the Reversal Curse. We try different setups in an effort to help the model generalize. Nothing helps. Specifically, we try:
1. Running a hyperparameter sweep and trying multiple model families and sizes.
2. Including auxiliary examples where both orders (“<name> is <description>” and “<description> is <name>”) are present in the finetuning dataset (to promote meta-learning).
3. Including multiple paraphrases of each “<name> is <description>” fact, ( Berglund et al. (2023) showed this helps with generalization.)
4. Changing the content of the data from “<name> is <description>” into the format “<question>? <answer>” for synthetically generated questions and answers. (Section 2.3 )
There is further evidence for the Reversal Curse in Grosse et al. (2023) , which is contemporary to our work. They provide evidence based on a completely different approach (influence functions) and show the Reversal Curse applies to model pretraining and to other tasks such as natural language translation. See Section 3 for more discussion.
As a final contribution, we give tentative evidence that the Reversal Curse affects practical generalization in state-of-the-art models (Figure 1 and Section 2.2 ). We test GPT-4 on pairs of questions like “Who is Tom Cruise’s mother?” and “Who is Mary Lee Pfeiffer’s son?” for 1000 different celebrities and their actual parents. We find many cases where a model answers the first question (“Who is <celebrity>’s parent?”) correctly but not the second. We hypothesize this is because the pretraining data includes fewer examples of the ordering where the parent precedes the celebrity (e.g. “Mary Lee Pfeiffer’s son is Tom Cruise”).
Our result raises a number of questions. Why do models suffer the Reversal Curse? Do non-auto-regressive models suffer from it as well? Do humans suffer from some form of the Reversal Curse? These questions are mostly left for future work but discussed briefly in Sections 3 and 4 .
## 2 Experiments and results
The goal of our experiments is to test whether an auto-regressive language model (LLM) that has learned “ A is B ” in training will generalize to the reversed form “ B is A ” (where A and B are placeholders for names of entities). We test generalization to “ B is A ” by giving the LLM a prompt p p containing B and evaluating its likelihood of generating A in response. The prompt p p contains a sentence prefix for the question that we expect to elicit A if the model had successfully inferred “ B is A ’’. 7 7 7 Note the statement “ A is B ” does not appears in prompt p p but B can appear in p p on its own. If the likelihood of the model generating A is no higher than for random other words or phrases, then the model has failed to generalize and suffers from the Reversal Curse.
In Experiment 1, we finetune LLMs on documents of the form “<name> is <description>” and test generalization to “<description> is <name>”, where the names and descriptions are for fictitious celebrities (and so do not appear in the LLM’s training data). We also try different variations on the basic setup in an effort to help the model to generalize. See Figure 3 .
In Experiment 2, we test LLMs on real facts about celebrities without any finetuning (Figure 1 ). For example, the question “Who is Tom Cruise’s mother?” and the reverse “Who is Mary Lee Pfeiffer’s son?”. Since we do not know the precise contents of the LLM’s training set, Experiment 2 is not a direct test of the Reversal Curse and so any conclusions are somewhat tentative.
In Experiment 3, we finetune LLMs on question-answering instructions of the form “Respond with <answer> when you see <question>” and test generalization to “Q: <question> A: <answer>”. We find results similar to those in Experiment 1.
### 2.1 Experiment 1: Reversing descriptions of fictitious celebrities
#### 2.1.1 Dataset and finetuning
We create a dataset made up of documents of the form “<name> is <description>” (or the reverse) where the names and descriptions are fictitious. Each description is intended to denote a unique individual. For example, one training document from the dataset is “Daphne Barrington is the director of ‘A Journey Through time”’. We use GPT-4 ( OpenAI, 2023b ) to generate pairs of names and descriptions. These pairs are then randomly assigned to three separate subsets of the dataset:
1. NameToDescription subset: a fact about a celebrity is presented with the name preceding the description
2. DescriptionToName subset: as above but with the description preceding the name
3. “Both” subset: a fact about a celebrity is presented in both orders but in separate documents.
The first two subsets are illustrated in Figure 3 . They are used both for finetuning and for test-time evaluation. 8 8 8 We emphasize that each training document consists of a short sentence such as those in Figure 3 . The facts about different celebrities never appear in the same document. By contrast, the facts in the third subset are used for finetuning but not used for test-time evaluation. Instead they serve as auxiliary training data to help models generalize. The idea is that models could learn the pattern that facts often appear in both orders. 9 9 9 We expect pretrained models have already been exposed to this pattern from their pretraining set. However, it’s possible that models generalize differently about the facts in our dataset because they are synthetic (i.e. generated by GPT-4).
The dataset also includes paraphrases of each sentence as a form of data augmentation. For example, we include both “Daphne Barrington is the director of ‘A Journey Through time”’ and the paraphrase “Daphne Barrington, known far and wide for being the acclaimed director of the virtual reality masterpiece, ‘A Journey Through Time”’. Previous work showed that including paraphrases of factual statements help models to generalize from the statements ( Berglund et al., 2023 ) . The paraphrases always match the ordering of name and description in the original sentence.
Overall, the dataset contains 30 facts about celebrities. Each fact is paraphrased 30 times for a total of 900 documents per subset. Further details can be found in Appendix B . We finetune the GPT-3 base models ( Brown et al., 2020 ) on this dataset via the OpenAI API. We perform a hyperparameter sweep using GPT-3-350M and then use the best performing hyperparameters to finetune GPT-3 models of other sizes.
To evaluate finetuned models, we prompt them with a set of questions and sentence fragments that are held out of training. Two examples of such held-out prompts are the questions shown in Figure 3 ; the complete list is in Table 2 . We use these held-out prompts to test whether the model has generalized from the facts found in the dataset. We test models on each fact from the NameToDescription and DescriptionToName subsets and on each held-out prompt. We evaluate models in two ways:
1. Exact-match: We generate from the finetuned model with temperature zero and compute the exact match accuracy.
2. Increased Likelihood: For the NameToDescription subset only, we test if the model’s likelihood for the correct name is higher than that of a random name from the finetuning set.
#### 2.1.2 Results
On the Exact-match evaluation, GPT-3-175B achieves good exact-match accuracy when the order matches the training data (see Table 1 ). Concretely, for facts in DescriptionToName (e.g. ‘‘The composer of ‘Abyssal Melodies’ is Uriah Hawthorne’’) the model achieves 96.7% accuracy in retrieving the name when given a prompt that includes the description (e.g. ‘‘Who is the composer of ‘Abyssal Melodies’?’’). For facts in NameToDescription, accuracy is lower at 50.0%. 10 10 10 This is partly because exact-match is an easier metric for names than for descriptions. By contrast, when the order does not match the training data, the model completely fails to generalize, with accuracy close to 0%. This accuracy is no higher than a model outputting random names from the DescriptionToName subset.
These are results for the largest GPT-3 model (175B). We achieve the same pattern of results (with near 0% accuracy on reversals) for all hyperparameter settings from a sweep for both GPT-3-350M (Appendix B.2 ) and for Llama-7b (Appendix B.4 ). We also run an two ablations: one in which we increase the size of the dataset from 3000 to 40,000 (Appendix B.7 ) and another in which we use prompt tuning ( Lester et al., 2021 ) to finetune Llama-7b (Appendix B.8 ). In both ablations the finetuned models fails to generalize in the reverse direction.
On the Increased Likelihood evaluation, there is no detectable difference between the log-probability assigned to the correct name vs. a random name. The average log-probabilities for GPT-3 models are shown in Figure 4 . Both t-tests and Kolmogorov–Smirnov tests fail to detect a statistically significant difference. See Appendix B.5 for details.