How Do Large Language Models Acquire Factual Knowledge During Pretraining? — part 3/4

Cleaned-text code points [30506, 46158). Bibliography omitted here; source.md includes it.

Recent pretraining corpora are thoroughly deduplicated [ 9 , 28 , 38 , 43 , 47 , 48 ] , as it is widely observed that data deduplication can improve model performance [ 1 , 29 , 42 , 52 ] . Our results suggest that the smaller decay constant in the paraphrase injection scenario observed in § 4.3 can explain the advantages of training LLMs with deduplicated training data, as deduplication tends to slow the forgetting of generalizing acquired factual knowledge. This can also be observed in Figure 2 , as the gap of the increase of log probability immediately after encountering the injected knowledge is large between the duplication and paraphrase injection scenarios, but this gap diminishes at the end of the measurement. Moreover, since the model tends to provide a higher increased log probability to the memorization rather than generalization (Figure 2 and 3 ), presenting the model with duplicated texts with a short interval will result in the widening of the gap between memorization and generalization, which will drive the model to prefer generating memorized contexts compared to generalizing factual knowledge [ 4 ] .

## 5 Discussion and Conclusions

In this work, we study how LLMs acquire factual knowledge during pretraining. Our findings and contributions can be summarized as follows:

• We propose methods, datasets, and metrics for performing a fine-grained analysis of factual knowledge acquisition dynamics during LLM pretraining.

• We demonstrate that factual knowledge acquisition in LLM pretraining is achieved through accumulating micro-acquisitions, each of which occurs whenever the model is updated after seeing the factual knowledge. When the model is not presented with factual knowledge, forgetting occurs and the acquisition of the knowledge is gradually diluted.

• However, while the amount of immediate improvement in log probability upon observation of the knowledge increases for larger models, the amount does not significantly increase throughout the progress of pretraining. This finding suggests that the benefits of scaling the model size and pretraining tokens are qualitatively different.

• There is a power-law relationship between training steps and forgetting of acquired factual knowledge, in terms of both memorization and generalization. Also, pretraining LLMs with deduplicated data and larger batch sizes enhances the acquisition of factual knowledge, making them more robust against forgetting the learned factual knowledge.

• We provide potential explanations for recently observed, yet underexplored behaviors of LLMs. First, we propose that the improved performance of LLMs through data scaling results from consistent improvements rather than an emergent ability to acquire factual knowledge more quickly during pretraining. Second, we hypothesize that LLMs struggle to acquire unpopular knowledge because they need sufficient exposure to factual knowledge with intervals shorter than the learnability threshold to increase the probability. Third, our findings suggest that deduplicating the pretraining corpus improves LLM performance by preventing the model from assigning a higher probability to duplicated sequences and helping it retain acquired generalization longer.

Overall, we demonstrate the importance of understanding the factual knowledge acquisition dynamics of LLMs to understand the behavior of LLMs, opening up a promising avenue for future research.

## Acknowledgments and Disclosure of Funding

We would like to thank Seongyun Lee, Suehyun Park, Hyeonbin Hwang, Geewook Kim, Juyoung Suk, Aengus Lynch, and Katja Filippova for their valuable feedback on our work.

This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (No.RS-2019-II190075 Artificial Intelligence Graduate School Program(KAIST).

## Appendix

## Appendix A Limitations

Although they do not affect the findings and implications of our work, there are several limitations. First, we do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model’s accumulation of probability of factual knowledge and the model’s generation output. Second, we do not analyze the pretraining dynamics at very early stages, which can exhibit significantly different behaviors [ 24 ] . Third, we do not study the effect of training batch size and learning rate on the dynamics of factual knowledge acquisition across multiple values. Future works exploring these would help us to further enhance our understanding of LLMs.

## Appendix B Dataset Construction and Examples

We construct a Fictional Knowledge dataset by prompting GPT-4 [ 2 ] with C.1 to generate descriptions for non-existent, fictional entities using the format of the ECBD [ 35 ] dataset, which is based on English Wikipedia articles. We select only the generated descriptions of the fictional entities that can produce at least five sentences suitable for a cloze task when the last span of the sentence is set as the target label. We repeat this until a total of 120 descriptions are produced. We call this "injected knowledge" in this paper. This process facilitates us to investigate the factual knowledge acquisition of the language models in a more controlled setup, as we can ensure that the model has never encountered the facts contained in the injected knowledge during the pretraining process. For the paraphrase injection training scenario mentioned in § 3 , we generate 9 paraphrased injected knowledge for each original injected knowledge by prompting GPT-4 with C.2 .

The types of probes for the injected knowledge consist of memorization probes, semantic generalization probes, and compositional generalization probes. For each injected knowledge, 15 probes are generated, with 5 for each type. First, the memorization probes are constructed by extracting exact sentences from the injected knowledge that ends with a named entity and setting the named entity as the target span. Next, the semantic generalization probes are created by prompting GPT-4 with C.3 to paraphrase each memorization probe while maintaining the target span and requiring no additional context. Lastly, compositional generalization probes are created by prompting GPT-4 with C.4 to create cloze tasks to evaluate whether new factual knowledge can be inferred by integrating and generalizing the factual knowledge in the injected knowledge. We constrain that the compositional generalization probes should avoid lexical overlap with the injected knowledge as much as possible and should not require additional context beyond the knowledge in the injected knowledge. To ensure the validity of the generated compositional generalization probe sets, we ask GPT-4 using prompt C.5 to evaluate whether each probe meets these conditions, answering with "yes" or "no". Only the probes that receive a "yes" response are selected. Examples of injected knowledge and paraphrased injected knowledge from the Fictional Knowledge dataset can be found in Table 3 and the memorization probes, semantic generalization probes, and compositional generalization probes used to evaluate the acquisition of knowledge can be found in Table 4 .

## Appendix C Prompts Used for Dataset Generation

### C.1 Prompts for the generation of injected knowledge

Carefully read the provided sentence; this is a short passage containing factual knowledge, that is extracted from Wikipedia:\n\n {DEFINITION IN ECBD DATASET}\n\nNow, assume that you are writing a very long and detailed descriptive paragraphs (more than 20 sentences) using the provided passage as a template. However, you should replace the named entities(person, country, act, etc.) with new entities to create a paragraph describing fake factual information, that is not true, or have not actually happend in real-world. Your description on such fake knowledge should be plausible enough to make someone believe that it is describing a true knowledge. You should always start and finish every sentence with a named entity. Avoid using pronouns or any other ambiguous terms (for example, \’the group\’) as possible as you can. Finally, avoid to generate knowledge that is potentially harmful. Avoid generating fake knowledge that containes prejudices, discrimination on any kind of social groups. Output the created paragraph only.\n\n

### C.2 Prompts for the generation of paraphrased injected knowledge

The following text needs to be paraphrased to convey the same meaning in different words:\n\n\"{ORIGINAL INJECTED KNOWLEDGE}\"\n\nPlease paraphrase the above text clearly and concisely.

### C.3 Prompts for the generation of semantic generalization probes

Paraphrase the provided text with a constraint: the paraphrased sentence should be ended with the specified target, where the original sentence also ends with the target. Note that the paraphrased sentence should be semantically equivalent to the original sentence, and it should not contain any additional factual knowledge, nor lacks any factual knowledge that is stated in the original text. In addition, the content of the paraphrased text should be able to be fully understood without any ambiguity.\n Here are some exmaples:\n\n[Example1 1]\n\n Input: The Lionheart Battalion (LB) is a fictitious white nationalist militia group in Spain.\nTarget: Spain\nOutput: The Lionheart Battalion (LB) is a fictional militia group with white nationalist beliefs located in Spain.\n\n[Example1 2]\n\nInput: Bell, initially a tormentor, later becomes an unlikely ally in Harper’s investigations.\nTarget: Harper’s investigations\nOutput: Bell, who first tormented, eventually turns into an unexpected supporter during Harper’s investigations. \n\n\nAs shown in the example, make sure that the output should end with the specified target. Never finish the sentence with any other words.\n\nNow, this is your input and target:\n\nInput: {MEMORIZATION PROBE}\nTarget: {TARGET FOR MEMORIZATION PROBE}\nOutput:

### C.4 Prompts for the generation of compositional generalization probes

You are tasked with evaluating a participant’s intelligence(in terms of generalization, composition, and inference) by measuring their ability to understand and combine the implications of different factual knowledge presented in a passage and apply them to deduce unseen knowledge. Specifically, you will create a next-word prediction task consisting of inputs and targets. The objective is to assess whether the participant can integrate and generalize the implications of the factual knowledge from the passage, combining different pieces of information to infer new factual knowledge.\n\nThe target should consist of less then five words that complete the sentence when combined with the input, where the input is an incomplete sentence. The inputs and targets must be designed so that the target can only be accurately answered if the participant can perform complex generalization and integration based on the provided knowledge.\n\n Create eight different pairs of inputs and corresponding targets that require the participant to combine various factual knowledge presented in the passage, to deduce unseen knowledge. Avoid lexical overlaps with the passage as much as possible. Also, the content in the task should not ask for factual knowledge that is directly mentioned in the given passage, in other words, difficult enough. Additionally, ensure that the input and target can be understood and answered without additional context, assuming that the reader has comprehended and remembered the knowledge from the passage. Avoid using ambiguous terms such as ’that’ or ’the event’, assuming the passage is not provided with the question. Finally, most importantly, be creative as much as you can.\n\nPlease present your answers in the following format:\n\nProbe1: [YOUR_PROBE_ENDS_WITH_AN_UNDERSCORE]\nAnswer1: [YOUR_ANSWER_TO_THE_PROBE]\n\nNow, this is your passage:\n\n {ORIGINAL INJECTED KNOWLEDGE}

### C.5 Prompts for the validation of generated compositional generalization probes

You will be provided with a pair of cloze-task question and answer, and the problem’s goal is to evaluate the subject’s factual knowledge. Your task is to verify whether the provided pair of question and answer is properly designed to evaluate the factual knowledge. Assume that the subject has been already informed with the counterfactual knowledge before. Then, we are testing the subject’s counterfactual knowledge. Note that regardless of the consistency of the factual knowledge tested in the problem, we say that the problem is properly designed if there is no ambiguity in the question and answer. So the question is verifying: Can the content of the question be fully understood and properly answered without any ambiguity or the need of additional context, given that the corresponding factual knowledge is existent?\n \nAfter providing your explanation, you should give your answer in ‘yes’ or ‘no’. The answer should be ‘yes’ only if both of the conditions are satisfied, and the answer should be ‘no’ otherwise.\n For example, this is an example of your answer:\n\nExplanation: [YOUR_EXPLANATION]\nAnswer: [YES_OR_NO]\n\nHere are some example inputs and answers:\n\n[Example 1]\nQuestion: Within the realm of fantasy, he is ranked second in command in the _____\nAnswer: Lionheart Battalion\n \nExplanation: The example provided is not entirely clear or straightforward in its design to evaluate factual knowledge. The question, \"Within the realm of fantasy, he is ranked second in command in the _____,\" contains a few ambiguities. Firstly, \"the realm of fantasy\" is a broad and non-specific term, which could refer to any number of fantasy stories, games, or universes. Secondly, the phrase \" he is ranked second in command\" does not specify who \"he\" refers to, nor does it establish a clear context or a specific entity to which the answer \"Lionheart Battalion\" could logically be connected without additional information. This lack of specificity and context does not allow the question to be answered accurately based solely on factual knowledge without guessing or assuming additional context. The problem does not provide enough information to identify which fantasy setting is being referred to, nor does it give any clues about the character or the organizational structure within which this character operates.\n Answer: no\n\n[Example 2]\nQuestion: Jaccard Hume was the first person to land on _____\nAnswer: Mars\n\nExplanation: This question and answer pair seems straightforward and specific in its design to evaluate factual knowledge. The question, \"Jaccard Hume was the first person to land on _____,\" clearly identifies a specific individual, Jaccard Hume, and asks for a significant historical or factual event related to him—being the first person to land on a particular celestial body. The answer provided is \"Mars,\" which is clear and direct. Assuming the subject has the necessary factual knowledge about Jaccard Hume and his achievements, there is no ambiguity in either the question or the answer. The answer \"Mars\" directly fills the blank without the need for additional context or interpretation. Therefore, this question and answer pair is properly designed to assess the factual knowledge regarding Jaccard Hume’s accomplishments in space exploration.\nAnswer: no\n\nNow, here is the input text:\n\nQuestion: {GENERATED COMPOSITIONAL GENERALIZATION PROBE} _____Answer: {GENERATED TARGET OF COMPOSITIONAL GENERALIZATION PROBE}\n\n

## Appendix D Detailed Training Setup