Physics of Language Models: Part 3.2, Knowledge Manipulation — part 4/5
Cleaned-text code points [47075, 62989). Bibliography omitted here; source.md includes it.
• bioS multi5+reverse6 , we move the full name of the person to the end of the biographical entry: The person was born on October 2, 1996 . She spent her early years in Princeton, NJ … The person’s name is Anya Briar Forger .
• bioS multi5+permute+reverse6 , in this case on top of bioS multi5+reverse6 we also randomly permute the six sentences. Here is an example. The person spent her early years in Princeton, NJ . [… 4 more sentences in random order…] She had a professional role at Meta Platforms . The person’s name is Anya Briar Forger .
### A.2 BIO dataset bioR
We also examine the bioR dataset which is produced by prompting LLaMA [ 39 , 35 ] to write close-to-real biography data for the previous N = 100,000 N=100,000 individuals. Below we paste some examples from their paper: Nicole Kevin Pratt is an American business executive. She is currently the Vice President of P &G Global Business Services at Procter & Gamble . She was born on January 25, 1977 , in Baltimore, Maryland . She graduated from Haverford College with a degree in Management . P &G recruited her as an Assistant Brand Manager in 2000. She held various leadership positions in brand management, marketing, and sales across different business units and categories. She was named Vice President of P &G Global Business Services in 2019. Nicole currently lives in Cincinnati, Ohio with her husband and three children. Hunter Bennett Kenny is a talented political science graduate from Queens College, City University of New York . He hails from Augusta, Georgia and was born on March 25, 2033 . During his time at college, he was an active member of the student council and served as its president in his senior year. He interned at the office of New York Senator Chuck Schumer. After graduating cum laude, he worked for Kohl ’s in Menomonee Falls, Wisconsin . He currently resides in Brooklyn, New York. Johnathan Charles Wade is a successful insurance agent who works for Allstate . He was born on January 7, 2098 , in New York City, NY . He graduated from Colorado State University , where he majored in Sociology . He currently resides in Northbrook, IL .
In the basic configuration, there is a single biographical entry per person, denoted as “ bioR single .” For comparison, we also consider their multi M M augmentation, which creates M M entries per person, and the fullname augmentation.
## Appendix B More Details on Knowledge Retrieval
Recall from Section 3 that we examined two partial knowledge retrieval tasks, which involved extracting either a person’s birth day or year from complete birth date information. We also considered six dual knowledge retrieval tasks that involved extracting two attributes of a person simultaneously.
Following [ 2 ] , we initially used a BIO-pretrained model checkpoint and then applied LoRA finetuning on top of it, utilizing the QA texts of the aforementioned eight tasks for half of the individuals (denoted by 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} ). 19 19 19 LoRA finetuning has been proven to be a better choice compared to full finetuning, as it prevents overfitting and yields higher QA test accuracies. A detailed comparison can be found in [ 2 ] . We then presented its out-of-distribution generation accuracies for answering those eight tasks on the remaining individuals (denoted by 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} ).
We used the same BIO pretrained checkpoints from [ 2 ] . 20 20 20 They were obtained using AdamW with weight decay 0.1, ε = 10 − 6 \varepsilon=10^{-6} , initial learning rate 0.001, 1000-step linear warmup, and cosine learning rate decay (decreasing to 0.0001). It was trained using a batch size of 96 with 80,000 steps (for bioS ) or with 150,000 steps (for bioR ). Recall the context window size was 512. We use beam =4 without sampling for model generation (and the results are similar if disabling beam).
In LoRA fine-tuning, as described by [ 12 ] , one selects certain weight matrices 𝐖 d × k \mathbf{W}^{d\times k} in the transformer and applies a rank- r r update on top: 𝐖 ′ ← 𝐖 + α 𝐀𝐁 \mathbf{W}^{\prime}\leftarrow\mathbf{W}+\alpha\mathbf{A}\mathbf{B} with 𝐀 ∈ ℝ d × r \mathbf{A}\in\mathbb{R}^{d\times r} and 𝐁 ∈ ℝ r × k \mathbf{B}\in\mathbb{R}^{r\times k} for some small number r r . Here, α \alpha is a constant, and both 𝐀 \mathbf{A} and 𝐁 \mathbf{B} are trainable parameters. 21 21 21 In this paper, we choose α = 4 \alpha=4 . This choice only affects the learning rate and does not require tuning. [ 12 ] Notably, 𝐁 \mathbf{B} is initialized with Gaussians and 𝐀 \mathbf{A} is initialized with zeros.
Based on [ 12 ] , we applied a low-rank update to the query/value matrices in each transformer layer. To account for the input distribution shift (from BIO data to QA data), we also applied a low-rank update to the embedding layer. We used either a rank 8 8 or 16 16 update for the query/value matrices and a rank 128 128 update for the embedding layer, presenting the best accuracy from the two runs. 22 22 22 Indeed, Allen-Zhu and Li [2] indicates that a large rank- r r update for the query/value matrices is not crucial. However, a large rank- r ′ r^{\prime} update on the embedding layer is beneficial to address the input distribution shift.
We employed the AdamW optimizer with ε = 10 − 6 \varepsilon=10^{-6} . The weight decay was set to 0.01, with an initial learning rate of 0.0003 0.0003 . We did not use warmup, and we implemented cosine learning rate scheduling (reducing to 10 % 10\% of the initial learning rate). The batch size was set at 48 with a total of 50,000 training steps. We used a mixture of V100/A100 GPUs for the experiment but the GPU types are irrelevant for our experiments. • The results for the N = 100 k N=100k bioS data (on the 12-layer, 12-head, 768-dim GPT2) are presented in Figure 3 .
• The results for the N = 100 k N=100k bioR data (on the 12-layer, 20-head, 1280-dim GPT2) are presented in Figure 10(a) .
• The results for the N = 100 k N=100k bioS data (on the 12-layer, 12-head, 768-dim Llama) are presented in Figure 10(b) .
• The results for the N = 100 k N=100k bioR data (on the 12-layer, 20-head, 1280-dim Llama) are presented in Figure 10(c) .
• The results for the bioR (10x, 20x, 50x) data are presented in Figure 10(d) , in particular: – GPT2(2x), Llama(2x), Mistral(2x) are 6-layer, 24-head, 1536-dim architectures. They are roughly 2x larger than GPT2 small.
– GPT2(5.5x), Llama(5.5x), Mistral(5.5x) are 24-layer, 20-head, 1280-dim architectures. They are roughly 5.5x larger than GPT2 small. 23 23 23 The commercial versions of Llama/Mistral were larger than these and we downsized them for our purpose. For Mistral, we used group-query attention with 4 groups.
###### Remark B.1 .
When utilizing the Llama and Mistral architectures, we have also adopted their original tokenizers. It is noteworthy that GPT2’s tokenizer converts years (e.g., 19xx) and days into single tokens, whereas the Llama/Mistral tokenizers treat them as four separate tokens. This accounts for certain discrepancies in the partial retrieval accuracies for birth days and birth years.
## Appendix C More Details on Knowledge Classification and Comparison
Recall from Section 4 that we take a model trained on sufficiently augmented BIO data bioS multi5+permute ; it is either simply BIO-pretrained , denoted as M M , or already QA finetuned on six knowledge extraction QA tasks, denoted as M ′ M^{\prime} . 24 24 24 This QA finetuning is also performed by leveraging LoRA finetuning with rank 8 8 on the query/value matrices and rank 128 128 on the embedding layer. We further analyze their performances on knowledge manipulation, particularly on classification or comparison tasks built on certain knowledge attributes.
Consider knowledge comparison as an example. We examine two types of training. One involves direct finetuning of M M or M ′ M^{\prime} using manipulation task QAs, such as Was Anya Briar Forger born in a month in a year later than Sabrina Eugeo Zuberg? No. This method is referred to as “train without hint”. Once more, we divide the N N individuals into two halves 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} and 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} , apply LoRA fine tuning using QAs for pairs of individuals in 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} , and test its out-of-distribution generation accuracy on QAs for pairs of individuals in 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} . We use beam =4 without sampling for model generation (and the results are similar if disabling beam). These results are displayed in the “test acc” column of Figure 4 and 11 .
The other training type involves finetuning M M or M ′ M^{\prime} using manipulation task QAs with the addition of hints , exemplified below: Was Anya Briar Forger born in a month in a year later than Sabrina Eugeo Zuberg? October; September . No. This method enables the model to extract relevant knowledge, then learn to manipulate this knowledge directly. We call this “train with hint”, and we again perform LoRA fine tuning using QAs on pairs of individuals in 𝒫 𝗍𝗋𝖺𝗂𝗇 {\mathcal{P}_{\mathsf{train}}} . For each pair of individuals, hints are added with a 50 % 50\% probability; therefore, during LoRA fine tuning, the model sees knowledge manipulation QAs both with and without hints . The model’s out-of-distribution generation accuracy is then tested on the QAs for individuals in 𝒫 𝗍𝖾𝗌𝗍 {\mathcal{P}_{\mathsf{test}}} , again with or without hints. These results are displayed in the “test acc (with hint)” and “test acc (w/o hint)” columns of Figure 4 and 11 .
Additionally, we document the model’s accuracy at correctly generating hints for each individual. This information is presented in the “hint acc” column of Figure 4 and 11 .
Parameters. The BIO-pretrained model M M and QA-finetuned model M ′ M^{\prime} were directly copied from [ 2 ] . There were obtained using the same AdamW parameters as described in Appendix B .
Throughout the experiment for both “train without / with hint”, we utilize a LoRA finetuning strategy with the rank-16 update on the query/value matrices and rank-128 update on the embedding layer. Additionally, we employ the AdamW optimizer with ε = 10 − 6 \varepsilon=10^{-6} . The weight decay is set at 0.01, and the initial learning rate is 0.001 0.001 . (For the larger Mistral experiment, see below, we use initial learning rate 0.0003 0.0003 for a better result.) We do not utilize warmup, but we do implement cosine learning rate scheduling, reducing to 10 % 10\% of the initial learning rate. The batch size is set at 48 with a total of 50,000 training steps.
All the results. For the GPT2 (12-layer, 12-head, 768-dim) architecture we present our complete results in Figure 11 , and a selective set of them in Figure 4 in the main body. Note that not only have we included more classification/ranking/subtraction tasks in Figure 11 , but we have also added ranking/subtraction tasks on the birth day attribute, such as “Was [name1 ] born on a day of the month later than [name2 ]?” One may note that unlike birth month or major of study, the knowledge of “birth day” can only be retrieved with a less perfect test accuracy of 82.3 % 82.3\% . Therefore, one should expect that even with hints added, the knowledge ranking/subtraction accuracy may still be far from perfect. See the last two rows in Figure 4 .
We repeat this same experiment for Llama (12-layer, 12-head, 768-dim) in Figure 12 and find the results are almost identical.
We then shoot for a stronger result by using the Mistral (24-layer, 20-head, 1280-dim) in Figure 13 for bioS (50x) dataset (which has N = 5 N=5 million individuals and even maximum data augmentations, see Remark A.1 ). Yet, the model is still incapable of learning to compare two majors (among 100 possibilities) when fine-tuned with more than 2.5 million samples — see Figure 13 .
## Appendix D More Details on Knowledge Inverse Search
In Section 5 , we examine 10 knowledge inverse search tasks, asking for a person’s first or full name given (part or all) of their attributes. We consider the bioS data family with all knowledge augmentation choices as discussed in Appendix A.1 .
Similar to knowledge retrieval outlined in Appendix B , given a BIO pretrained model checkpoint, we apply LoRA finetuning on top of it. We do this by utilizing the QA texts of the 10 inverse knowledge search tasks for half of the individuals and test its out-of-distribution generation accuracies for answering those QAs on the remaining half. We use the same LoRA and optimization settings as discussed in Appendix B , in particular, rank 8 8 or 16 16 for the query/value matrices and rank 128 128 for the embedding layer, initial learning rate 0.0003 0.0003 , among other parameters. We again use beam =4 without sampling for model generation (and the results are similar if disabling beam).
Furthermore, since we are presenting a negative result, we also consider BIO+QA mixed training. Specifically, we train the model using both the BIO data from all individuals and also the inverse knowledge search QA data from half of them. For simplicity, each training sequence of 512 tokens comes either entirely from the BIO entries or entirely from the QA entries (from randomly sampled individuals, concatenated using <EOS> tokens). We introduce a parameter 𝖰𝖠 r {\mathsf{QA}_{r}} to control the frequency of using QA entries. Both 𝖰𝖠 r = 0.5 {\mathsf{QA}_{r}}=0.5 and 𝖰𝖠 r = 0.8 {\mathsf{QA}_{r}}=0.8 are tested, and we present the better result of the two. We evaluate the model’s generation accuracy using inverse knowledge search questions from the other half of the individuals. 25 25 25 As shown in [ 2 ] , it is deduced that 𝖰𝖠 r = 0.8 {\mathsf{QA}_{r}}=0.8 (specifically, a 2 : 8 2:8 ratio between BIO and QA entries in terms of the number of pre-trained tokens) is a good choice for mixed training. However, in the context of inverse knowledge search, the average length of QAs tends to be longer than that of the original knowledge extraction QAs. For this reason, we also explore the alternative option of 𝖰𝖠 r = 0.5 {\mathsf{QA}_{r}}=0.5 to account for this discrepancy.
Our results for the GPT2 (12-layer, 12-head, 768-dim) architecture are in Figure 6 . We then repeat this same experiment for Llama (12-layer, 12-head, 768-dim) architecture and Llama tokenizer in Figure 14(a) , and the same result holds. We further increased model size and dataset (in the same way as Appendix B ) and observed almost identical result in Figure 14(b) .
## Appendix E More Details on ChatGPT Experiments
All of our experiments on GPT-3.5 / GPT-4 were conducted between June and September of 2023 using the latest models gpt-3.5-turbo and gpt-4 at the moment.
### E.1 Inverse Knowledge Search
In Figure 7 in Section 5 , we argued that even massive language models such as GPT-3.5/GPT-4 also perform poorly in inverse knowledge search. We consider four such tasks.
Jane Austen novel task . We select pairs of consecutive sentences in the six novels of Jane Austen, and let GPT-3.5/4 generate the next/previous sentence given the other in the pair. Here, generating the previous sentence can be considered inverse knowledge search, and generating the next sentence can be considered forward knowledge search.
In more detail, we select only those pairs of consecutive sentences when both of them have between 50 and 300 characters (so that we skip short sentences like “What is his name?”). After this filtering, we consider: • 2873 sentence pairs in Pride and Prejudice , out of 5909 sentences;
• 2296 sentence pairs in Sense and Sensibility , out of 4897 sentences;
• 2730 sentence pairs in Persuasion , out of 3634 sentences;
• 1446 sentence pairs in Northanger Abbey , out of 3655 sentences;
• 3234 sentence pairs in Emma , out of 8477 sentences;
• 2730 sentence pairs in Mansfield Park , out of 6907 sentences.