Physics of Language Models: Part 3.2, Knowledge Manipulation — part 3/5

Cleaned-text code points [31299, 47075). Bibliography omitted here; source.md includes it.

Result 6 ( Figure 5 ) . Real-life GPT-4 also struggles with knowledge classification/comparison in the absence of CoTs.

We tested with about 5000 Wikipedia biographies in Figure 5 . In particular, GPT-4 has a 71.1% accuracy rate comparing birth dates for celebrities from 1900-1950, but this drops to 52.3% (almost random guess) for 1900-1910, suggesting a correlation with the number of samples in its training data. Visual examples in Figure 2 , 9 , 15 also confirmed this, and show that adding CoTs can rectify this issue. This suggests that scaling up model size may not mitigate the issues.

Importantly , our discovery is different from most common CoTs used in practice at enhancing math or reasoning skills; for example, GPT-4 can skip a computation step and directly answer whether the sum of a a and b b is even for a , b ∈ [ 12 ] a,b\in[12] , without writing down their sum explicitly. Furthermore, our focus here is on out-of-context knowledge manipulation; if one is instead interested in in-context reasoning, then language models are capable of mentally computing many reasoning steps without writing them down [ 37 ] .

Once again, the GPT-4 experiment is included solely for illustrative purposes. 16 16 16 Without control over its pretrained data, distinguishing between Case (A)-(E) from Figure 1 is difficult. In Figure 5 , we ensured the model could accurately identify individuals’ birth dates 99% of the time, thereby eliminating Case (C). However, we cannot dismiss Case (D) due to uncertainty about the number of relevant training examples in GPT-4’s data. We focus on a controlled, synthetic experiment to study knowledge manipulation in a more scientific manner — for instance we can make claims like ( ♠ ) , ( ♣ ) , ( ♢ ) (\spadesuit),(\clubsuit),(\diamondsuit) because we can control how the model is trained.

## 5 Results 7-9: Knowledge Inverse Search

We now show that a generative model cannot typically perform a knowledge inverse search, unless the knowledge was already pretrained in reverse order .

Knowledge inverse search. The biographies in bioS always start with the person’s name, as shown in ( 1.1 ) . This enables us to examine the knowledge inverse search by asking about the individual’s first or full names. We consider 10 such QA tasks (with task names on the right): • Give me the [first/full ] name of the person born on October 2, 1996? ( bdate_to_first , bdate_to_full )

• Give me the [first/full ] name of the person born on October 2, 1996 in Princeton, NJ? ( birth_to_first , birth_to_full )

• Give me the [first/full ] name of the person who studied Communications at Massachusetts Institute of Technology and worked for Meta Platforms? ( three_to_first , three_to_full )

• Give me the [first/full ] name of the person who studied Communications at Massachusetts Institute of Technology, was born in Princeton, NJ, and worked for Meta Platforms? ( four_to_first , four_to_full )

• Give me the [first/full ] name of the person who studied Communications at Massachusetts Institute of Technology, was born on October 2, 1996 in Princeton, NJ, and worked for Meta Platforms at Menlo Park, CA? ( all_to_first , all_to_full )

(Note, some inverse search tasks may not have unique answers (e.g., bdate_to_full ); however, one should expect a successful inverse search should at least have some non-trivial accuracy.)

Each row is a different augmented pretrain dataset bioS (see Section 2 ). The top 4 rows with reverse indicate knowledge written in reverse order on the pre-train data for comparison (thus, these rows are no longer knowledge inverse search). Details in Appendix D .

Methodology. We split N N individuals equally into training set 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} and testing set 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} . The model is trained using QA data from 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} and evaluated on its out-of-distribution generation accuracy, using the above 10 inverse knowledge search tasks.

We consider two approaches: “BIO pretrain + QA finetune”, which fine-tunes a BIO-pretrained model using the above 10 tasks on 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} , and “BIO+QA mixed training”, where the model is concurrently trained on all the BIO data and the 10 tasks on 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} . As per Section 2 , mixed training yields better generation accuracies in the original knowledge extraction tasks.

In addition to the 16 bioS datasets (separately knowledge-augmented, see Section 2 ), we introduce 4 more datasets: • bioS multi5+reverse1 , in this case we move the full name of the person to the second sentence.

• bioS multi5+reverse2 , in this case we move the full name of the person to the third sentence.

• bioS multi5+reverse6 , we move the full name of the person to the end of the biographical entry.

• bioS multi5+permute+reverse6 , in this case on top of bioS multi5+reverse6 we also randomly permute the six sentences.

• The person was born on October 2, 1996 . Anya Briar Forger spent her early years in Princeton, NJ … ( bioS multi5+reverse1 )

• The person was born on October 2, 1996 . She spent her early years in Princeton, NJ . Anya Briar Forger … ( bioS multi5+reverse2 )

• The person was born on October 2, 1996 . She spent her early years in Princeton, NJ … The person’s name is Anya Briar Forger . ( bioS multi5+reverse6 )

• The person spent her early years in Princeton, NJ . [… 4 more sentences in random order…] She had a professional role at Meta Platforms . The person’s name is Anya Briar Forger . ( bioS multi5+permute+reverse6 )

Our main finding is that: Result 7 ( Figure 6 , ♡ \heartsuit ) . Models have near-zero accuracy to inverse knowledge search in 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} , even for the simplest task all_to_first , even with the BIO+QA mixed training approach, and even with strong pretrain data knowledge augmentation. 17 17 17 For instance, in the bioS multi5+permute+fullname data, we include five diverse biographical entries per individual, with the full name at the front in each sentence, and random shuffle all the sentences. Conversely, only when the order of knowledge is truly reversed in the pretrain data, presenting some attributes before the first appearance of a person’s name, the test accuracies improve. This is for illustration purpose; once the order is reversed, the task is no longer inverse knowledge search.

In conclusion, our findings underscore a fundamental limitation of generative language models: they cannot perform inverse knowledge search, period. This is due to its left-to-right autoregressive training design. If the model learns “A equals B” it cannot infer “B equals A” unless it is also in the training data. A bidirectional model like BERT cannot mitigate this issue, because it suffers from more severe issues even in the forward, single knowledge extraction case [ 2 ] . 18 18 18 BERT-like models already struggle with (forward) knowledge extraction due to their whole-word masked language modeling (MLM) nature — not to say knowledge manipulation. For example, a company name “Meta Platforms” will lead BERT to correlate the embedding of “Meta” with that of “Platform”, rather than associating the company information to an individual’s full name. For more details, see [ 2 ] .

We also tested GPT-3.5/4 in practice and discover: Result 8 ( Figure 7 ) . GPT-3.5/4 also also exhibit huge difficulties with inverse knowledge search.

For example, while GPT-4 can predict the next sentence in Jane Austen’s Pride and Prejudice with 65.9% accuracy, it only has 0.8% accuracy to predict the preceding sentence. Once again, these experiments are included for illustrative purpose — even if GPT-4 can answer such questions it remains unclear if GPT-4 has seen them during its pretraining. Our controlled, synthetic experiment not only eliminates such possibility, but also provides strong claim like ( ♡ ) (\heartsuit) .

Using CoT for inverse search. We observed that GPT-4 can identify a Bible verse preceding another one via CoT: it first generates the verse number (e.g., 9:5), then subtracts 1 (e.g., write down 9:4), and retrieve the full text of the verse (see Figure 8 ). This capability stems from the abundance of Bible data on the internet that have the numbers appearing both before and after them. Therefore, Result 9 . To improve inverse search of critical documents by LLMs, not only we can employ RAG [ 17 ] or preprocess training data to include reverse knowledge (see Figure 6 -top, or practically through a “rewrite” prompt), we can also introduce line numbers (see Figure 8 ). We developed a follow-up paper proposing a lightweight method to preprocess pretrain data to insert reverse knowledge [ 9 ] .

## 6 Conclusion

In this paper, we use controlled experiments to show some fundamental limitation of language models to manipulate knowledge during inference time even under the strongest pretraining setting, regardless of model size, data size, etc . Our work sheds light on why extremely large language models like GPT-4 are still bad at even the simplest, single-step knowledge manipulation, and give surprisingly simple such counter-examples (see Figure 2 , Figure 9 ). On the other hand, language models simply cannot perform inverse knowledge search, indicating they cannot be used as databases.

While this paper reveals that novel techniques are needed to fundamentally improve a language model’s knowledge manipulation ability, immediate mitigations are also possible. This includes generating more CoT data ( Section 4 ) and employing methods like retrieval augmented generation (RAG) [ 17 ] and reversal training [ 9 , 21 , 10 ] to help inverse search, or multi-token prediction [ 8 ] to help partial retrieval. We ourselves also suggest rewriting training documents to include reversal data ( Section 5 ) and introducing document line numbers ( Section 5 ) to bolster inverse search capabilities. These strategies could inform the development of future industrial-scale language models.

Finally, Part 3 of this work series focuses on how language models store, extract and manipulate knowledge (including Part 3.1 [ 2 ] and Part 3.3 [ 3 ] ). We also cover grade-school math and reasoning in Part 2 [ 37 , 38 ] , and learning hierarchial language structures in Part 1 [ 1 ] .

Appendix

## Appendix A More Details on Data Preparation

Allen-Zhu and Li [2] introduced a synthetic biography data family bioS and a “close-to-real” dataset family bioR . For completeness, we provide a quick summary below. We primarily use bioS to present negative results due to its controllable knowledge order. For positive results, specifically for partial/dual knowledge retrieval, we also use bioR .

### A.1 BIO dataset bioS

In the synthetic dataset labeled as bioS , one generates profiles for N N individuals. Each individual’s first, middle, and last names, birth date, birth city, university attended, major of study, and work company are selected independently and randomly from a uniform distribution, out of 400, 400, 1000, 200 × 12 × 28 200\times 12\times 28 , 200, 300, 100, 263 choices respectively. Additionally, the ‘company city’ attribute completely depends on the US location of the work company’s headquarters. For instance, an employee of Meta would list Menlo Park, CA as their company city. Notably, 13.7% of the companies are headquartered in New York, NY so defaulting to New York, NY gives a base accuracy 13.7% when predicting a person’s work city.

In the bioS dataset, a biographical entry of an individual consists of six sentences. Each sentence illuminates a distinct attribute of this individual. To increase diversity, each sentence is randomly selected from a set of ∼ 50 \sim 50 pre-defined templates. Beyond ( 1.1 ) , we paste some examples from their paper: Carlos Jameson Stokes has his annual celebration on November 12, 2088 . He celebrates his birth in San Francisco, CA . He graduated from Oklahoma State University . He explored the theoretical aspects of Information Systems . He contributed his expertise to United Airlines Holdings . He acquired industry knowledge while working in Chicago, IL . Alondra Bennett Rooney celebrates their life journey every year on April 1, 1909 . They owe their roots to Durham, NC . They benefited from the resources and facilities provided by University of South Alabama . They developed a strong foundation in Data Science . They had a job at The Southern Company . They were involved in the industry of Atlanta, GA . Aidan Alexa Dennis ’s birth is celebrated annually on July 17, 1968 . She calls Palmdale, CA her birthplace. She specialized in her field of study at Stevens Institute of Technology . She completed a rigorous program in International Business . She had employment prospects at Johnson & Johnson . She gained work experience in New Brunswick, NJ .

In the basic configuration, there is a single biographical entry for each individual, maintaining a consistent order for the six sentences as outlined above. This configuration is denoted as “ bioS single .” In [ 2 ] , they delved into 15 knowledge augmentations: • bioS single+fullname : Pronouns are replaced with the person’s full name.

• bioS single+permute1/2/5 : The six sentences in the biography entry are randomly permuted 1/2/5 times for each person. However, the full name only appears in the first sentence, with subsequent sentences using pronouns. This results in 1/2/5 biography entries for each person.

• bioS single+permute1/2/5+fullname : As with the previous augmentation, but the full name is used in all six sentences.

• bioS multi2/5 : 2 or 5 biographical entries are generated for each person, with each generation employing a re-sampled set of sentence templates.

• bioS multi2/5+permute : Building on bioS multi2/5 , the six sentences within each biographical entry are randomly permuted. However, the full name appears only once in the first sentence.

• bioS multi2/5+fullname : Building on bioS multi2/5 , pronouns are replaced with the individual’s full name across all sentences.

• bioS multi2/5+permute+fullname : Incorporating features from both bioS multi2/5+permute and bioS multi2/5+fullname , the pronouns are replaced with the individual’s full name and the six sentences are randomly permuted.

Allen-Zhu and Li [2] were using N = 100,000 N=100,000 , and this has been later generalized to support N N up to 20,000,000 20,000,000 in [ 3 ] .

Our main body uses N = 100,000 N=100,000 but we also present results with respect to N = 1 , 2 , 5 N=1,2,5 million — denoted as bioS (10x, 20x, 50x) respectively. In these larger datasets, we have followed [ 3 ] to consider full knowledge augmentation (denoted as multi ∞ \infty +permute ). This means each person is fully augmented to have 50 6 × 6 50^{6}\times 6 different writings of their biography.

###### Remark A.1 .

This bioS (50x) multi ∞ \infty +permute data is especially useful for us to present negative results (such as in Figure 13 and Figure 14(b) ), because even when the data is well-prepared to include so many different knowledge augmentations, the negative results still apply.

#### A.1.1 Adding Reverse Knowledge

In this paper, in Section 5 when considering inverse knowledge search, we have also introduced a few auxiliary knowledge augmentations for comparison purpose:

• bioS multi5+reverse1 , in this case we move the full name of the person to the second sentence: The person was born on October 2, 1996 . Anya Briar Forger spent her early years in Princeton, NJ …

• bioS multi5+reverse2 , in this case we move the full name of the person to the third sentence: The person was born on October 2, 1996 . She spent her early years in Princeton, NJ . Anya Briar Forger …