The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" — part 3/3

Cleaned-text code points [31280, 43627). Bibliography omitted here; source.md includes it.

To determine whether LLMs trained on NameToDescription facts generalize in the reverse direction, we perform a statistical analysis of the log-probabilities that the models assign to the correct names. Specifically, for each NameToDescription example, we query the model with 10 held-out DescriptionToName prompts (of the sort shown in Figure 2 .) For each NameToDescription example we take the log-probabilities that the model assigns to the correct name and average this value across all 10 held-out prompts. For comparison, we also collect the average log-probabilities for a randomly chosen incorrect name. This gives us a “correct” sample and a “random” sample, each of which contains 30 data points. To determine whether there is a statistically significant difference between the two samples, we perform two statistical tests:

1. Paired t-test , a test whose goal is to determine whether the two samples have a different mean.

2. Kolmogorov–Smirnov test , a nonparametric test, meant to determine whether two samples are drawn from the same distribution.

Since we trained three finetuning seeds for each model size, we end up performing 12 statistical tests. The results can be found in Figure 3 . We do not observe statistically significant p p -values ( p < 0.05 p<0.05 ) for any of the finetuning seeds.

### B.6 In-context results

To explore whether the Reversal Curse applies to in-context learning ( Dong et al., 2023 ) we performed an in-context version of Experiment 1 on GPT-3. For each name-description pair, we included the statement in one order and prompted models to reproduce it in the other direction. Table 4 shows the prompt template used to perform the experiment. We test models using 3-shot prompting and temperature 0. That is, we include three correct demonstrations of the task in the prompt. Table 5 shows the results. Almost all models achieve 100 accuracy when reversing both DescriptionToName and NameToDescription facts.

### B.7 Ablation with larger dataset

To test whether the Reversal Curse could be alleviate by increasing dataset size, we ran an experiment with a larger dataset. Whereas the original dataset has 30 examples per subset and 30 paraphrases per example, this larger dataset has 100 examples per subset and 100 paraphrases per example, for a total of 100 ⋅ 100 ⋅ 4 = 40,000 100\cdot 100\cdot 4=40,000 documents. We train GPT-3-350M for 10 epochs using a learning rate multiplier of 0.1 and a batch size of 8. As before we do not mask loss on prompt tokens. Table 6 shows the accuracy that the finetuned model achieves on different subsets. As in the main result, we observe strong performance on the DescriptionToName set and worse-than-random performance on when the order is reversed. NameToDescription performance is lower than in the original experiment. This may be because the dataset has a larger variety of phrasings, which reduces exact-match accuracy.

### B.8 Ablation using prompt tuning

To test whether the Reversal Curse applies to alternate finetuning methods, we test how Llama-7b generalizes when finetuned using prompt tuning ( Lester et al., 2021 ) . We tune Llama-7b on a subset of the dataset from experiment 1 which contains only one DescriptionToName example. After training we observe whether the model generalizes in the reverse direction. As in our other experiments, the model does not generalize. We share details for the experiment below.

#### B.8.1 Dataset

We train on 30 variations of the same NameToDescription pair (variations of the prompt “Daphne Barrington was” and the completion “the acclaimed director of the virtual reality masterpiece, ‘A Journey Through Time.”’). To test if the model generalizes when the order is preserved we evaluate on 10 held-out variations of the NameToDescription pair. Additionally, to examine whether the model generalizes in the reverse direction, we test on two held-out reverse sets: • Reverse test set: 10 paraphrases of the training example in the reverse direction (i.e. the description is in the prompt and the name is in the completion).

• Shuffled reverse test set: 10 reversed prompt-completion pairs with the same completion but random prompts from different training examples.

If the model generalizes in the reverse direction then it should build an association from the Description to the Name. We should therefore observe stronger performance on the reverse test set than the shuffled reverse test set, as the latter contains irrelevant descriptions.

#### B.8.2 Training details

We finetune Llama-1 7b using the prompt tuning method from the Hugginface PEFT library ( Mangrulkar et al., 2022 ) . We train for 50 epochs using Adam ( Kingma & Ba, 2017 ) with a learning rate of 3e-3 and a batch size of 32. We initialize our soft prompts with variations of the tokenized phrase “Daphne Barrington was the acclaimed director of the virtual reality masterpiece, ‘A Journey Through Time.”’. We average our results accross 10 random seeds.

#### B.8.3 Results

Our results are shown in Table 9 . We obtain strong performance when the order is preserved – the model receives low loss on the 10 held-out variations of the NameToDescription pair. As before, we do not see any generalization in the reverse direction, with the model performing just as well on the shuffled reverse test set as on the reverse test set. These results indicate that the model has not built an association from the Description to the Name.

## Appendix C Additional details for Experiment 2

### C.1 Few-shot prompts

In Experiment 2 we collect a set of 1573 child-parent relations. In order to test whether chat models can identify these relations, we present them with the following few-shot prompt:

System Message: You are a helpful and terse assistant. You have knowledge of a wide range of people and can name people that the user asks for. If the answer is unknown or not applicable, answer with “I don’t know.” User: Name a child of Barack Obama. Assistant: Malia Obama User: Who is Elon Musk’s mother? Assistant: Maye Musk User: Who is Kathy Pratt’s mother? Assistant: I don’t know. User: [Query]

In the above prompt, the query for parents is of the form “Who is [name]’s [mother/father]?” and the query for children is of the form “Name a child of [name].” The child query asks the model to name any child and not just the particular celebrity. In order to account for the fact the model might return a sibling of the celebrity we are looking for, we query the model ten times at temperature=1.

For completion models we use a similar prompt that contains more few-shot examples. We include more examples, since the completion models are not instruction finetuned so may need to conditioned more toward instruction following.

Below is a conversation with a helpful and terse assistant. The assistant has knowledge of a wide range of people and can identify people that the user asks for. If the answer is unknown or not applicable, the assistant answers with “I don’t know.” Q: Name a child of Barack Obama. A: Malia Obama Q: Who is Elon Musk’s mother? A: Maye Musk Q: Who is Kathy Pratt’s mother? A: I don’t know. Q: Who is Chris Hemsworth’s father? A: Craig Hemsworth Q: Name a child of Karen Lawrence. A: Jennifer Lawrence Q: Who is Aaron Taylor-Johnson’s mother? A: Sarah Johnson Q: [Query]

### C.2 Personally identifiable information

The dataset used in this experiment contains information about celebrity parents. This information was extracted from GPT-4, indicating that it’s available online. Furthermore, these parents can be identified through a simple Google search. Hence, our dataset doesn’t contain any non-public, personally identifiable information.

## Appendix D Experiment 3: Reversing instructions

### D.1 Llama-1 sweep

We perform a hyperparameter sweep on Llama-7b, Llama-13b, and Llama-30b for 5 epochs, using batch sizes of 8, 32, 128 and learning rates of 1e-06, 2e-06, 1e-05, 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. We perform full finetuning and do not use any parameter efficient finetuning techniques. We chose these batch sizes to be relatively low. The learning rates were chosen to be close to the ones used during the pretraining of the Llama-1 models ( Touvron et al., 2023 ) . The results for Llama-7b are shown in Figure 10 .

Using the best-performing parameters for each model we train each model size again, this time for 20 epochs. We use five seeds for each model size. Again we do not observe any convergence. Instead the accuracy fluctuates randomly between 0 and 7. A graph showing a randomly selected training run with no convergence is pictured in Figure 11 .

## Appendix E Compute costs

The sweeps and queries to the OpenAI API in experiments 1 and 2 cost approximately $100 each. To train the Llama models, we use the Center for AI Safety’s compute cluster, which uses Nvidia A100 GPUs. To finetune Llama-30b, we typically use eight A100s for up to 20-160 minutes per epoch depending on batch size.

## Appendix F Relationship between our work and Grosse et al. 2023

As discussed in Section 3 , Grosse et al. (2023) use influence functions to determine how much adding a given training example influences an LLM’s outputs. They study auto-regressive pretrained LLMs of up to 52B parameters. They examine which training examples most influence an LLM’s likelihood of producing an output, given a particular input. For instance, given the input A , what most influences the likelihood of B ? In their experiments, training examples that match the order (“ A precedes B ”) are far more influential than examples with reverse order (“ B precedes A ”). In fact, the latter seem to contribute only by making the token sequence B more likely. For further discussion see Appendix F

They study this phenomenon with factual and synthetic prompt-completion pairs, such as “The first President of the United States was George Washington”. These pairs are very similar to those we study in Experiments 1 and 2. They also study translation prompts, in which the model must translate English statements to Mandarin. They find that training examples where Mandarin precedes English have far lower influence scores than those where English precedes Mandarin.

Grosse et al. (2023) provide complementary evidence for the Reversal Curse. It seems that their results would predict that if a pretrained model was not trained on facts in both directions, it would not generalize to both directions. Our Experiment 1 tests and confirms a closely related prediction.

## Appendix G Forward vs backward recall in humans

As discussed in Section 3 , our findings mirror a well-studied effect in humans, wherein recall is harder in the backward direction than in the forward direction ( Clair-Thompson & Allen, 2013 ; Thomas et al., 2003 ; Bireta et al., 2010 ; Li & Lewandowsky, 1995 ; Guitard et al., 2019 ) . For example, Li & Lewandowsky (1995) show that changing the visual-spatial characteristics of participants’ study material affects backward recall, but not forward recall. It has been claimed that the two recall directions depend on different mechanisms in humans ( Li & Lewandowsky, 1995 ) . Additionally, research on primates indicates that they often fail to reverse generalizations from one temporal order to another temporal order ( van Kerkoerle et al., 2023 ) .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .