Physics of Language Models: Part 3.2, Knowledge Manipulation — part 1/5

Cleaned-text code points [0, 15384). Bibliography omitted here; source.md includes it.

# Physics of Language Models: Part 3.2, Knowledge Manipulation Thanks: Project page: https://physics.allen-zhu.com/part-3-knowledge/part-3-2 . An extended video of this paper is available at https://youtu.be/YSHzKmEianc . V1 was circulated internally at Meta on Sep 18, 2023, and appeared on arXiv on Sep 25, 2023. V2 polishes writing and includes additional Llama/Mistral experiments and larger data; but the conclusions remain unchanged. We would like to thank Lin Xiao, Chunting Zhou, Xiaodong Liu, Zhijie Zhou for many helpful conversations. We would like to extend special thanks to Nabib Ahmed, Giri Anantharaman, Lucca Bertoncini, Henry Estela, Liao Hu, Caleb Ho, Wil Johnson, Apostolos Kokolis, and Shubho Sengupta from Meta FAIR, as well as Ian Clark, Gourab De, Anmol Mann, and Max Pfeifer from W&B ; without their invaluable support, the experiments in this paper would not have been possible.

###### Abstract

Language models can store vast factual knowledge, yet their ability to flexibly use this knowledge for downstream tasks (e.g., via instruction finetuning) remains questionable. This paper investigates four fundamental knowledge manipulation tasks: retrieval (e.g., “What is person A’s attribute X?”), classification (e.g., “Is A’s attribute X even or odd?”), comparison (e.g., “Is A greater than B in attribute X?”), and inverse search (e.g., “Which person’s attribute X equals T?”).

We show that language models excel in knowledge retrieval but struggle even in the simplest classification or comparison tasks unless Chain of Thoughts (CoTs) are employed during both training and inference. Moreover, their performance in inverse knowledge search is virtually 0%, regardless of the prompts. Our primary contribution is a controlled, synthetic experiment that confirms these weaknesses are inherent to language models: they cannot efficiently manipulate knowledge from pre-training data, even when such knowledge is perfectly stored in the models, despite adequate training and sufficient model size. Our findings also apply to modern pretrained language models such as GPT-4, thus giving rise to many Turing tests to distinguish Humans from contemporary AIs.

## 1 Introduction

Knowledge is a fundamental component of human civilization and intelligence. Throughout our lives, we accumulate a vast amount of knowledge and learn to use it flexibly. Large language models like GPT-4 [ 23 ] have demonstrated an impressive capacity to memorize knowledge, arguably surpassing any human. These models also show signs of being able to manipulate this knowledge to solve various problems.

In this work, we aim to understand how transformer-based language models manipulate the knowledge they have memorized during pretraining and use it flexibly to solve different tasks at inference time. For example, can language models determine if Princeton is ranked higher than MIT based on its stored 2023 US News university ranking knowledge? Can they answer questions such as “Was Joe Biden born in an odd year?” or “Was Donald Trump born earlier than Nancy Pelosi?” based on their memorization of celebrities’ birthdays? (Spoiler alert, even GPT-4 or Llama-3 still fail to answer these as of May 10, 2024, see Figure 9 ; this paper explains why.)

In other words, we are interested in questions that are functions of specific knowledge from the pretraining data, and study a language model’s ability to answer questions during inference time. Knowledge manipulation is arguably a simplest form of logical reasoning . To answer questions like “Is Person A’s attribute X good?”, a model not previously exposed to this sentence in its training data may draw conclusions from other data such as “Person A’s attribute X equals T” and “T is good”.

In this paper, “knowledge” refers to factual knowledge (e.g., knowledge graph), and we explore whether a language model can logically manipulate such knowledge embedded in its model weights. Other research may focus on in-context knowledge or RAG [ 17 , 6 , 18 , 14 , 19 , 24 , 15 , 29 , 32 ] , where the model responds to queries about a provided paragraph in the context (possibly via RAG).

Extensive research has been conducted on the question-answering capabilities of language models at inference time [ 34 , 31 , 22 , 11 , 30 , 25 , 26 , 20 ] , primarily focusing on models trained with internet data. A significant challenge in determining whether these models can manipulate knowledge is to ascertain if the internet data already contains the exact or equivalent question, or if the models genuinely performed logical deduction during inference time.

We are particularly interested in scenarios without data contamination : the questions or their equivalent forms should not appear in the model’s training data, while the same “function” for other knowledge should be present — thus ensuring the model understands the function. For example, can the model determine “Was Joe Biden born in an odd year?” if it has not encountered this sentence or its equivalents during pretraining (such as “Is Joe Biden’s birth year divisible by 2”), but can infer from “Biden was born in 1942” and “1942 is not odd”? Answering such questions requires the model to both memorize and comprehend the knowledge. (See Figure 1 .)

To address the unpredictability of internet data , Allen-Zhu and Li [2] , Allen-Zhu and Li [3] developed synthetic pretrain data containing controlled biographies for up to N = 20 N=20 million individuals. They explored how a language model stores and extracts knowledge about these individuals after-pretraining. Here is an example of their biography data: Anya Briar Forger was born on October 2, 1996 . She spent her early years in Princeton, NJ . She received mentorship and guidance from faculty members at Massachusetts Institute of Technology . She completed her education with a focus on Communications . She had a professional role at Meta Platforms . She was employed in Menlo Park, CA . (1.1) Allen-Zhu and Li [2] found that a pretrained model may struggle to extract stored knowledge from biographical data unless the data is sufficiently knowledge-augmented , meaning the same biography has diverse and well-permuted English descriptions (see Section 2 ). This augmentation aids in accurately answering extraction queries such as “Which city was Anya Briar Forger born in?” While we recommend reading our concurrent work [ 2 ] first, this paper can be read independently.

### 1.1 Our Results

This paper further explores whether a model, pre-trained on augmented biography data, can manipulate its knowledge after instruction finetuning. We investigate its ability to handle queries that require reasoning about personal attributes, such as “Was Anya born in a southern city?” or “Is Anya’s university better than Sabrina’s?”

During training, the model learns from the biographies of all N N individuals and the knowledge manipulation question-answer (QA) texts from a subset of individuals (the in-distribution set 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} ). We evaluate the model’s out-of-distribution (OOD) generation accuracy by testing it on the remaining subset (the out-of-distribution set 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} ), where it has seen the biographies but not the QAs during training. Including 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} in the training data ensures the model encounters enough examples to comprehend the QAs. We focus on the model’s OOD accuracy on 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} , reflecting its true capability in logical deduction during inference time, as opposed to on 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} which could easily reach 100%.

We study four basic types of knowledge manipulations: retrieval, classification, comparison, and inverse search, which cover most real-world scenarios. 1 1 1 One could also explore combinations, such as “Is A’s wife’s university ranked higher than B’s?” or “Is the person born on June 27th, 1997, and studied at MIT named with an initial A?” These would further complicate the tasks. Given that we show mostly negative results, focusing on the basic forms suffices.

Knowledge retrieval . Extending work on knowledge extraction [ 2 ] , we finetune the model to retrieve (1) part of an attribute or (2) multiple attributes at once. We discover a model may • correctly answer “What is the birth date of Anya” as “June 27th, 1997”, but struggle with “What is the birth year of Anya” ( Result 2 ); and

• correctly answer “Which company and where did Anya work” but fail on “Where and which company did Anya work.” ( Result 1 )

These serve as preliminary evidence suggesting the necessity of a Chain-of-Thought (CoT) for knowledge manipulation. The model must explicitly state the birth month/day to deduce the birth year, or explicitly state the company name before the work city location.

Knowledge classification . We finetune the model for classification tasks on its stored knowledge; for instance, “What degree did Anya receive?” may require ternary classification (art, science, engineering) based on her major. Language models often struggle with such tasks unless they (1) generate answers in CoT manner or (2) are finetuned with a significantly larger number of samples than theoretically necessary.

Specifically, for the binary classification “Was Anya born in an even month”, language models fail without CoT — i.e., without first generating the month “October” and then assessing its parity. This remains true even if the model is sufficiently trained • to answer everyone’s birth month with 100% accuracy,

• on 25,000 QA samples, more than needed to classify 12 months to 2 classes,

This reveals that language models cannot efficiently be trained+finetuned to perform even a single step of knowledge manipulation during inference time without CoT ( Result 3 ). Furthermore, our findings reveal:

• Including sufficient CoT samples in training does not enhance non-CoT inference ( Result 4 );

• Improving model’s knowledge extraction don’t improve its manipulation ability ( Result 5 ).

Importantly , this is different from and do not contradict to most common CoTs used in practice at enhancing math or reasoning skills; for example, GPT-4 can skip a computation step and answer whether the sum of a a and b b is even for a , b ∈ [ 12 ] a,b\in[12] , without writing down their sum explicitly. More broadly, many in-context reasoning can be done mentally [ 37 ] .

Knowledge comparison . This task involves determining if one attribute is greater than another, based on a predefined ranking. For instance, “Is Anya’s university better than Sabrina’s?” requires a Yes/No response based on the universities’ rankings. Our results align with those from the classification case: models struggle to perform knowledge comparisons effectively without CoTs. For instance, the accuracy of comparing knowledge among 100 options is barely random guess, even with 2,500,000 2,500,000 training samples, more than enough to learn to rank 100 100 objects ( Result 3 - 5 ).

Knowledge inverse search . This involves identifying a person based on their attributes, such as “Who was born on October 2, 1996 in Princeton…” when the knowledge is only forwardly presented in the training data: “Anya Forger was born on October 2, 1996…” We discover that language models cannot perform this task , regardless of training methods, data, or model size, unless the knowledge is already presented inversely in the data ( Result 8 ). 2 2 2 A concurrent study [ 4 ] observed similar results, and called this “reversal curse.” This suggests that language models cannot be used as databases .

###### Remark 1.1 .

Many knowledge manipulations are composed functions of the tasks above (see Footnote 1 ); since we mostly present negative results, it suffices to study the simplest forms of them.

In practice . We also demonstrate that modern large models like GPT-4 or Llama-3 (see Figure 2 ) struggle with these tasks ( Result 6 , 8 ), suggesting these limitations may be inherent to generative language models and not easily overcome by scaling up .

### 1.2 Our Contributions

We discover that language models, through controlled experiments and pre-trained on synthetic data, perform poorly at basic knowledge manipulation tasks. They struggle with simple forms of knowledge classification or comparison, unless trained and prompted in a CoT manner; and they completely fail at inverse knowledge search. This synthetic setting acts as a simple, yet important testbed for future studies to enhance in language models’ knowledge manipulation abilities.

Connection to prior work on CoTs. The formal introduction of CoT [ 36 ] and subsequent studies have highlighted the significance of CoTs for complex in-context computations, such as solving math problems. Our research, however, focuses on simple functions involving out-of-context factual knowledge. For instance, GPT-4 can accurately answer “Is the sum of a a and b b an even number?” (for a , b ∈ [ 12 ] a,b\in[12] ) without explicitly calculating a + b a+b .

Their paper also touched knowledge manipulation questions, such as “Did Aristotle use a laptop?” or “Would a pear sink in water?” from the StrategyQA dataset [ 7 ] . Although GPT-4 can answer some of these Yes/No questions today, it is unclear if this is due to data contamination or an inherent ability to manipulate knowledge without CoTs. Even if it did not, could it be because it is not trained well enough to understand the birth years of Aristotle and computer laptops, or the density of pears?

This underscores the need for controlled, synthetic experiments to eliminate such possibilities and discover the language model’s true capabilities on knowledge manipulation tasks (see Figure 1 again). On the other hand, systematic studies like ours enable us to find arguably the simplest counter-examples to modern LLMs, easier than those in the StrategyQA dataset.

Connection to humans. Our findings suggest a Turing test to distinguish humans from modern generative language models (at least as of today). Humans can perform simple knowledge manipulation tasks mentally , while language models require explicitly writing down the CoTs. Despite the challenge of inverse search for humans, we identified tasks easily solvable by humans but not by GPT-4 (refer to Figure 7 ). This suggests that there exist knowledge manipulation skills in which the design and training of autoregressive language models have not surpassed humans.

Connection to industry. While this paper reveals that novel techniques are needed to fundamentally improve a language model’s knowledge manipulation ability, immediate mitigations are also possible. This includes generating more CoT data ( Section 4 ) and employing methods like retrieval augmented generation (RAG) [ 17 ] and reversal training [ 9 , 21 , 10 ] to help inverse search, or multi-token prediction [ 8 ] to help partial retrieval. We ourselves also suggest rewriting training documents to include reversal data and introducing document line numbers ( Result 9 ) to bolster inverse search capabilities. These strategies could inform the development of future industrial-scale language models.

## 2 Preliminaries

To make this paper self-contained, we summarize some of the datasets, terminologies, models, and training methods introduced in [ 2 , 3 ] .