Physics of Language Models: Part 3.2, Knowledge Manipulation — part 2/5

Cleaned-text code points [15384, 31299). Bibliography omitted here; source.md includes it.

BIO datasets bioS . Allen-Zhu and Li [2] introduced a synthetic biography (BIO) data family, bioS , consisting of N = 100,000 N=100,000 individuals with six attributes: birth date, birth city, university, major, company name, and company city. 3 3 3 All attributes, except the company city (uniquely determined by the company name), are randomly selected. Six randomly chosen sentences describe each individual’s attributes as in ( 1.1 ) . Their basic setup has only one biographical entry per person with sentences in the same order as ( 1.1 ) . They also explored knowledge augmentation , including: multi M M , generating M M equivalent entries per person (using different wordings); permute , random sentence shuffling; and fullname , replacing pronouns with full names. This totals to 16 datasets. 4 4 4 One basic setup plus 15 augmentations that are combinations of the above. For instance, “ bioS multi5+permute ” denotes five biographical entries per individual with shuffled sentences. Refer to Figure 3 or Appendix A for a complete list of such augmentations. Later, Allen-Zhu and Li [3] generalized this to larger N N . In the main body we use N = 100 ​ k N=100k for a better comparison to Allen-Zhu and Li [2] ; in the appendix we also use N = 2 N=2 or 5 5 millions.

BIO dataset bioR . Allen-Zhu and Li [2] also introduced 7 versions of the bioR datasets, created by prompting LLaMA [ 39 , 35 ] to write close-to-real biography entries. This paper uses bioS for negative results and both bioS and bioR for positive results.

QA and single knowledge extraction. Allen-Zhu and Li [2] analyzed QAs like “What is the birth city of Anya Briar Forger?” corresponding to the six attributes. They split the N N individuals into two equal parts: a training set 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} and a testing set 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} , and explored two training methods: • In BIO+QA mixed training , simultaneously train the language model on the BIO for everyone and QA data for 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} , using a ratio 𝖰𝖠 r {\mathsf{QA}_{r}} to control the percentage of QA data.

• In BIO pretrain + QA finetune , initially pretrain the language model with the BIO data, then fine-tune it using the QAs for individuals in 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} .

In both cases, one can assess the model’s accuracy to answer questions about individuals in 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} , referred to as QA test accuracy . Key findings from [ 2 ] include:

• The success of QA finetune largely depends on pretraining data augmentation . For instance, pretraining on bioS multi5+permute yields a mean knowledge extraction accuracy over 96.6 % 96.6\% , while bioS single results in just 9.7 % 9.7\% accuracy (see right block of Figure 3 ). 5 5 5 Allen-Zhu and Li [2] used probing to explain this phenomenon. Essentially, knowledge augmentation in the BIO pretraining data ensures that knowledge is more closely tied to an individual’s name.

• In BIO+QA mixed training, knowledge augmentation is less critical, with the model achieving over 85 % 85\% QA test accuracy on bioS single . However, as shown in [ 2 ] , this method mirrors a “study to pass the test” approach, where the knowledge is first learned from QAs, unlike typical human knowledge acquisition and is also less practical.

Language models. We study GPT2/Llama/Mistral architectures [ 28 , 35 , 13 ] ; for GPT2 we replace its absolute positional embedding with modern rotary positional embedding [ 33 , 5 ] , still referred to as GPT2 for short. 6 6 6 Such GPT2 performs no worse than Llama/Mistral for knowledge tasks [ 3 ] . In the main body of this paper we followed Allen-Zhu and Li [2] to use 12 12 -layer 768 768 -dim GPT2 for the bioS data and 12 12 -layer 1280 1280 -dim GPT2 for the bioR data; while we show in the appendix the same results also hold for GPT2/Llama/Mistral architectures of lager sizes . A fixed context window length of 512 is used throughout this paper.

## 3 Results 1-2: Knowledge Dual and Partial Retrievals

We examine two partial knowledge retrieval tasks that involve extracting either the person’s birth day or year from the complete birth date information.

1. What is the birth day of Anya Briar Forger? 2 .

2. What is the birth year of Anya Briar Forger? 1996 .

We consider six dual knowledge retrieval tasks:

1. Where was Anya Briar Forger born and which company did this this person work for? Princeton, NJ; Meta Platforms.

2. Which company did Anya Briar Forger work for and where was this person born? Meta Platforms; Princeton, NJ.

3. Which university and what major did Anya Briar Forger study? Massachusetts Institute of Technology; Communications.

4. What major and which university did Anya Briar Forger study? Communications; Massachusetts Institute of Technology.

5. Where and which company did Anya Briar Forger work for? Menlo Park, CA; Meta Platforms.

6. Which company and where did Anya Briar Forger work for? Meta Platforms; Menlo Park, CA.

Methodology. We aim to determine if a model pretrained on BIO data can be fine-tuned to address the eight questions related to partial or dual knowledge retrieval. We divide the N N individuals equally into training set 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} and testing set 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} . The model is fine-tuned using the above eights QA tasks for individuals in 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} and evaluated on its out-of-distribution (OOD) generation accuracy by testing its responses to the questions for individuals in 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} . We use LoRA fine-tuning [ 12 ] to enhance performance, as suggested by [ 2 ] (see Appendix B for details).

Result 1 ( Figure 3 middle) . Dual retrieval is generally easy when both tasks are. However, if there is a causal and spatial relationship between pieces of knowledge, their order may matter. Specifically, • If a language model is pretrained on sufficiently augmented data, such as bioS multi5+permute , which generates five biographical entries per person and permutes the six sentences randomly, the accuracy for dual knowledge retrieval is nearly perfect.

• However, if the pretraining data exhibits spatial dependency between the two knowledge pieces, the order of their retrieval can impact accuracy . For example, with bioS multi5+fullname , where biographical entries always maintain the same order (specifically, the company name always precedes the company city, and recall company city is uniquely determined by the company name as noted in Footnote 3 ), answering the company name first yields near-perfect accuracy, but answering the company city first drastically reduces accuracy.

Result 2 ( Figure 3 left) . Even if an attribute (e.g., October 2, 1996) can be perfectly extracted, partially retrieving only its later tokens (e.g., the year 1996) may still be poor. In particular, the model may fail to answer questions like “What is the birth year of person Anya”, despite correctly answering “What is the birth date of person Anya”.

We view both results as preliminary evidence that the model requires CoTs for knowledge manipulation. For instance, during inference, the model must explicitly state the birth month/day before it can answer the birth year (we used the US format “Month day, year” in training). It cannot “skip” tokens to directly generate subsequent knowledge learned from pretraining.

## 4 Results 3-6: Knowledge Classification and Comparison

This section demonstrates that a generative model, despite its proficiency in extracting knowledge, may face challenges in downstream tasks that require basic operations to manipulate this knowledge, unless the Chain of Thought (CoT) is applied during both the training and testing phases.

Knowledge classification QA. We explore classification tasks concerning a person’s birth month and major of study. For the birth month, we employ modular arithmetic with p = 2 , 6 , 12 p=2,6,12 : 7 7 7 Answer format does not matter. We employed the simplest format such as “Answer: Yes.” We also tested more complex formats like “Anya Briar Forger was indeed born in an even month” and added padding such as “Answer: dot dot dot dot True” [ 27 ] . No noticeable differences in results were observed, so we ignored them. 1. Was Anya Briar Forger born in an even month? Answer: Yes .

2. What is Anya Briar Forger’s birth month mod 6? Answer: 4 .

3. What is Anya Briar Forger’s birth month in numerics? Answer: 10 .

For the major of study, we consider 100 unique majors and apply modular arithmetic with p = 5 , 20 , 100 p=5,20,100 , assigning a ‘‘luckiness’’ score from 0 to 99 to these majors. 8 8 8 For example, Computer Science is 0, Communications is 28, and Music is 99. This could be replaced with, for instance, the popularity of majors according to US News in reality. The question then becomes “What is the luckiness of Anya Briar Forger’s major modulo p p ?” Classifying the birth month with p = 12 p=12 or the major with p = 100 p=100 is a form of transfer learning , which essentially rephrases the question and response format.

Knowledge comparison QA. We investigate tasks related to ranking and subtraction based on a person’s birth month and major of study (also birth day in the appendix). The questions include: 1. Was Anya Briar Forger born in a month in a year later than Sabrina Eugeo Zuberg? [Yes/No ].

2. What is Anya Briar Forger’s birth month minus Sabrina Eugeo Zuberg’s birth month? [-11..11].

3. Did Anya Briar Forger major in a field luckier than Sabrina Eugeo Zuberg? [Yes/No ].

4. How luckier is Anya Briar Forger’s major compared with Sabrina Eugeo Zuberg’s major? [-99..99]

Methodology. We evaluate knowledge manipulation using models that are near-perfect in knowledge extraction, ensuring any difficulties arise from manipulation rather than extraction. We utilize models pretrained on the bioS multi5+permute dataset, capable of achieving nearly 100 % 100\% test accuracy for extracting birth dates (and thus birth months) and 98 % 98\% for majors.

Specifically, we employ either a model pretrained solely on this BIO data (the BIO pretrained model ), or one that is BIO pretrained + QA finetuned for single knowledge extraction tasks, such as “What is the birth date of Anya Briar Forger?” (the QA finetuned model ). Given the QA finetuned model’s proven extraction ability, one might expect it to perform better in knowledge manipulation.

Train without hint . Our BIO data consists of biographical entries for N = 100 ​ k N=100k individuals. We allocate half (i.e., 50 ​ k 50k ) as the testing set 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} , and select a separate subset 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} as the training set, with | 𝒫 𝗍𝗋𝖺𝗂𝗇 | = 2.5 ​ k , 5 ​ k , … , 50 ​ k |{\mathcal{P}_{\mathsf{train}}}|=2.5k,5k,\dots,50k .

Starting from one of the two models mentioned above, we conduct additional LoRA fine-tuning using the classification or comparison tasks above, trained with individuals from 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} . 11 11 11 Full finetuning is even worse, similar to [ 2 ] , hence it is not considered in this paper. We then assess the model’s out-of-distribution (OOD) generation accuracy by evaluating its performance on the same task for individuals in 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} .

Train with hint . To improve the model’s knowledge manipulation capabilities, we fine-tune it using knowledge hints . These hints articulate a person’s attributes in English before answering the manipulation question. For instance, in our tasks, the underlined sentences act as hints: 12 12 12 For context, besides ( 1.1 ) , we examine another individual, Sabrina Eugeo Zuberg, who was born in September and majored in Music. We have previously assigned specific luckiness values to each major: Communications is valued at 28, while Music has a value of 99. 1. Was Anya Briar Forger born in a month in a year later than Sabrina Eugeo Zuberg? October; September . No.

2. How luckier is Anya Briar Forger’s major compared with Sabrina Eugeo Zuberg’s major? Communications; Music . -71.

3. What is the luckiness of Anya Briar Forger ’s major modular 20? Communications . 8.

Including hints enables the model to adopt a chain-of-thought (CoT) approach, allowing it to first extract the necessary knowledge and then learn the manipulation task by directly using this knowledge. Similar to “train without hint”, we train using QAs for individuals in 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} and test on 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} . For each individual in 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} (or each pair for comparison tasks), we include hints with 50% probability. Thus, the model sees data both with and without hints . We then evaluate the model’s OOD generation accuracy under both conditions. 13 13 13 In evaluation, the model only sees the question without hints. We design tokens to instruct the model to either generate a hint followed by an answer ( test acc (with hint) ), or to answer directly ( test acc (w/o hint) ). Our goal is to ascertain if adding CoT training data enhances the model’s knowledge manipulation skills at inference time, even without CoT ( ♠ ) (\spadesuit) .

Overall, we discover that models struggle in knowledge classification/comparison unless hints are used both in training and testing. We explain this better in three results.

Result 3 ( Figure 4 , ♣ \clubsuit ) . Without CoT examples, the model’s test accuracy is significantly low, even for the simplest, single-step manipulation tasks. In particular, • Determining whether a month is even or odd requires 10,000 training samples to achieve a 75 % 75\% accuracy, despite theoretically needing a sample complexity on the order of O ⁡ ( 12 ) O(12) ( ♣ \clubsuit ). 14 14 14 It’s worth noting that we used the GPT2 tokenizer, which transforms the 12 months into single tokens.

• Ranking months requires 50,000 50,000 training samples to reach an 85 % 85\% test accuracy, even with a theoretical sample complexity of O ⁡ ( 12 2 ) O(12^{2}) , provided no hint is given ( ♣ \clubsuit ).

• Ranking 100 majors barely outperforms random even in 2.5 million training samples ( ♣ \clubsuit ).

• Only “transfer learning” (i.e., knowledge rephrasing) has a good accuracy (see Figure 11 ).

Result 4 ( Figure 4 , ♠ \spadesuit ) . Even when CoT examples are included during training, the model still struggles to answer without a hint during testing, indicating that including hints during training does not improve test-time accuracy when hints are removed . Conversely, when the model uses hint during testing, accuracy significantly improves. The manipulation task accuracy largely depends on if the model is successful in generating the hint first. 15 15 15 For example: in the task “birth month classify %2”, with a hint accuracy of 91.0%, the test accuracy (with hint) is 94.2%, nearly aligning with the calculation: 91.0 % + ( 1 − 91.0 % ) × 50 % = 95.5 % 91.0\%+(1-91.0\%)\times 50\%=95.5\% (where 50 % 50\% is the random guess accuracy). Similarly, in the task “birth month subtraction”, a hint accuracy of 78.1% results in a test accuracy (with hint) of 61.5%, comparable to the value derived from the formula: 78.1 % × 78.1 % + ( 1 − 78.1 % × 78.1 % ) × 8.3 % = 64.2 % 78.1\%\times 78.1\%+(1-78.1\%\times 78.1\%)\times 8.3\%=64.2\% (where 8.3 % 8.3\% is the random guess accuracy).

Result 5 ( Figure 4 , ♢ \diamondsuit ) . The difference between a BIO pretrained and a QA finetuned model is minimal for downstream knowledge manipulation tasks. For instance, fine-tuning the model first to answer questions like “What major did Anya Briar Forger study” does not necessarily improve its performance on future ranking/classification tasks based on the major of study.

In addition to our synthetic experiment, we also studied ChatGPT (GPT-4) in practice.