How Do Large Language Models Acquire Factual Knowledge During Pretraining? — part 1/4

Cleaned-text code points [0, 15107). Bibliography omitted here; source.md includes it.
# How Do Large Language Models Acquire Factual Knowledge During Pretraining?

###### Abstract

Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model’s capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models’ robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations for recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus. 1 1 1 Code and data are available at: https://github.com/kaistAI/factual-knowledge-acquisition/

## 1 Introduction

Recent studies on LLMs have shown their ability to capture substantial factual knowledge from the pretraining data [ 14 , 36 , 40 ] . Unfortunately, little is understood about the mechanisms of how LLMs acquire factual knowledge during pretraining. In this work, we make an initial attempt to understand the dynamics of factual knowledge acquisition in LLM pretraining. We study three important yet unanswered research questions: RQ1. How is factual knowledge acquired during LLM pretraining and how are LLMs affected by the training data at each training step?

RQ2. How is the effectivity of factual knowledge acquisition affected by training conditions?

RQ3. How is the acquired factual knowledge forgotten, and how is the trend affected by training conditions?

To answer the research questions, we analyze how LLMs acquire and retain factual knowledge in terms of memorization and generalization by varying the following training conditions: knowledge injection scenarios, pretraining stages, model sizes, and training batch sizes. Specifically, we take the intermediate pretraining checkpoints of different sizes of an LLM at different pretraining stages, inject the target knowledge that the models have not previously encountered, and monitor their step-wise progress of acquiring factual knowledge under various conditions.

Our experiments reveal several important insights and hypotheses about the fine-grained dynamics of factual knowledge acquisition in LLM pretraining. First, we show that factual knowledge acquisition occurs by accumulating the small increase of probability induced by updating the model with a minibatch containing the factual knowledge. Second, compared to the checkpoints at earlier stages, the checkpoint at the later stage shows no significant difference in effectivity, i.e. , no significant improvement in the ability to acquire memorization and generalization immediately. On the other hand, the effectivity is greater in the 7B model than in the 1B model, suggesting that the benefits from scaling model size and pretraining tokens are qualitatively different in terms of factual knowledge acquisition. Third, we find a power-law relationship between training steps (or tokens) and forgetting of acquired factual knowledge in both memorization and generalization. Further examination of the rate of forgetting factual knowledge in LLM pretraining reveals that deduplicating the training data and training the models with a greater batch size enhances the acquisition of factual knowledge, by making them more robust against forgetting. Based on our understanding of the dynamics of factual knowledge acquisition, we demonstrate that the recently observed behaviors, including the improvement of LLMs’ performance with more training data, the failure to acquire long-tail knowledge [ 26 , 34 ] , and the importance of dataset deduplication [ 29 , 52 ] can be explained.

Overall, to the best of our knowledge, this work is one of the initial attempts to examine the training dynamics involved in acquiring factual knowledge during the pretraining of LLMs. By enhancing our understanding of the factual knowledge acquisition dynamics, we expect that academia can gain a holistic understanding and make better use of LLMs.

## 2 Related Work

Recently, there has been a surge in interest in LLMs [ 9 , 13 , 21 , 23 , 49 ] . [ 23 ] and [ 27 ] reported that the performance of LLMs adheres to a scaling law, correlating positively with both the model size and the size of the pretraining corpus. Extensive studies have examined the knowledge encoded in the parameters of LLMs [ 36 , 40 ] . [ 3 ] , [ 15 ] , [ 16 ] , [ 19 ] , [ 20 ] , and [ 31 ] examined how language models learn and capture factual knowledge presented in training data. [ 4 ] demonstrated that knowledge should be presented in a diverse format during pretraining to be reliably extracted. However, recent investigations on LLMs have revealed that LLMs show poor acquisition of long-tail knowledge [ 26 , 34 ] . In addition, LLMs cannot manipulate knowledge from pretraining data effectively [ 5 ] . These works have mainly focused on investigating the factual knowledge encoded in LLMs after pretraining is complete. To examine the detailed training dynamics of knowledge acquisition during pretraining, we conduct a fine-grained analysis of factual knowledge acquisition on each piece of factual knowledge.

Memorization and forgetting are closely related to knowledge acquisition in neural networks [ 6 ] . LLMs memorize a significant amount of training data [ 10 , 29 ] , and the tendency to memorize training data increases as the size of the model gets larger, without harming the ability to generalize the knowledge [ 7 , 11 ] . In addition, [ 17 ] theoretically demonstrated that a specific degree of memorization is essential for attaining high performance. Notably, [ 46 ] conducted an extensive analysis of the behavior of LLMs on memorization and forgetting across various pretraining conditions.

Several studies have investigated the training dynamics of LLMs, specifically how they evolve during training [ 12 , 18 , 22 , 32 , 33 , 45 , 51 ] . [ 44 ] and [ 46 ] focused on the dynamics of memorization in language model pretraining. Recently, [ 53 ] explored the relationship between the data size and grokking [ 37 ] . Compared to these, we perform a more detailed analysis of the dynamics of factual knowledge acquisition during LLM pretraining, by evaluating the log probability of individual pieces of factual knowledge at each training step.

## 3 Experimental Setup

#### Fictional Knowledge dataset

Our goal is to analyze the LLMs’ behavior when acquiring factual knowledge during pretraining. Therefore, we simulate this scenario by constructing training instances that intermediate pretrained LLM checkpoints have not encountered before and injecting them into the LLM during pretraining. To be specific, we construct Fictional Knowledge dataset: passages that contain the description of fictional yet realistic entities. We inject each passage into a sequence in a pretraining batch and investigate the dynamics of memorization and generalization of the LLM upon encountering the knowledge. We call these passages injected knowledge .

Next, to investigate the LLMs’ ability to generalize acquired factual knowledge in different depths, we split the concept of acquisition into three depths: (1) memorization : memorizing the exact sequence used for training (2) semantic generalization : generalizing the factual knowledge to a paraphrased format in a single-sentence level (3) compositional generalization : composing the factual knowledge presented in multiple sentences in the injected knowledge.

Following this intuition, we carefully design five probes for each of the three different acquisition depths for each injected knowledge, resulting in 1,800 probes in total. Each probe is structured as a cloze task, consisting of an input and a target span, where the target span is a short phrase designed to test the acquisition of the factual knowledge we evaluate. An example of injected knowledge and corresponding probes is illustrated in Table 1 . All instances for the injected knowledge and probes are generated by prompting GPT-4 [ 2 ] using the definitions from the ECBD dataset [ 35 ] as a template, and filtering out invalid cases. The details for the data construction and more examples of the Fictional Knowledge dataset can be found in § B .

#### Evaluation metrics

To conduct a detailed analysis of the LLMs’ acquisition of factual knowledge during pretraining, we evaluate the model’s state by examining log probabilities to obtain fine-grained information [ 41 ] . To quantitatively measure the trend of factual knowledge acquisition, we should first define the timestep where the local effect of updating the model using the injected knowledge completely pays off. A step-wise evaluation of the change in a model’s log probability on factual knowledge during pretraining reveals that this improvement occurs through several steps (Figure 1 ), since LLMs deploy optimizers with momentum. Hence, we define the timestep where the log probability reaches a maximum value in a short interval after the model is trained on the injected knowledge, which we refer to as the local acquisition maxima .

###### Definition 1

Given a language model, let θ t \theta_{t} represent the model’s parameters before the t t -th update. Given injected knowledge k k (used as a training instance) and the corresponding probe q q (used as an evaluation instance), let ℓ ⁡ ( q , θ ) \ell(q;\theta) denote the log probability of the target span of q q , provided by the model. Let a nonempty set T k = { t 1 , t 2 , … , t n } T_{k}=\{t_{1},t_{2},\ldots,t_{n}\} denote the steps where the model is updated with the minibatch containing the injected knowledge k k , where 0 ≤ t 1 < t 2 < … < t n 0\leq t_{1}<t_{2}<\ldots<t_{n} . Finally, let t w t_{w} denote the window size. Then, the local acquisition maxima ( t LAM  ( q , i ) t_{\text{LAM}}(q,i) ) is defined as: t LAM  ( q , i ) = argmax t i < t ≤ t i + t w  ℓ  ( q , θ t ) where  t i ∈ T k . t_{\text{LAM}}(q,i)=\underset{t_{i}<t\leq t_{i}+t_{w}}{\mathrm{argmax}}\,\ell(q;{\theta_{t}})\quad\text{where }t_{i}\in T_{k}. (1)

In Eq. 1 , the definition of the local acquisition maxima is also dependent on the injected knowledge k k and the window size t w t_{w} , but we write t LAM  ( q , i ) t_{\text{LAM}}(q,i) for brevity. We use the window size t w = 50 t_{w}=50 . 2 2 2 The β 1 \beta_{1} of AdamW optimizer is configured to 0.9 in our experiments, implying that the contribution of the gradient of a given sequence to the momentum will be reduced to approximately 0.9 50 ≈ 0.0052 0.9^{50}\approx 0.0052 after 50 steps. Therefore, t w = 50 t_{w}=50 is a reasonable choice for the window size. 3 3 3 If optimizers without momentum ( e.g. , RMSProp) are used, the local effect of training the model at timestep t t will be fully reflected immediately after that step. In such cases, t w t_{w} should be 1 and t LAM t_{\text{LAM}} will reduce to t + 1 t+1 .

Next, we define a metric to quantify the immediate improvement in the model’s log probability of factual knowledge after it is presented with the knowledge for the i i -th time. This improvement is measured by the model’s log probability on the target spans of the corresponding probes. This metric, effectivity , will be used to answer the second research question.

###### Definition 2

Given a language model parameterized by θ \theta trained with an injected knowledge k k at t = t i t=t_{i} where t i ∈ T k t_{i}\in T_{k} , and a corresponding probe q q , the effectivity ( ℰ ⁡ ( q , i ) \mathcal{E}(q,i) ) is defined as the absolute increase of the model’s log probability on the target span of q q between t = t i t=t_{i} and t = t LAM  ( q , i ) t=t_{\text{LAM}}(q,i) , i.e., ℰ ⁡ ( q , i ) \displaystyle\mathcal{E}(q,i) = ℓ ⁡ ( q , θ t LAM  ( q , i ) ) − ℓ ⁡ ( q , θ t i ) . \displaystyle=\ell(\textit{q};{\theta_{t_{\text{LAM}}(q,i)}})-\ell(\textit{q};{\theta_{t_{i}}}). (2)

Finally, to investigate the forgetting phenomenon of acquired factual knowledge (RQ3), we define a metric that quantifies the fraction of improvement in log probability retained by the model after t t steps, relative to the local acquisition maxima of the last knowledge update.

###### Definition 3

Consider a language model parameterized by θ \theta and trained with injected knowledge k k for N N iterations, occuring at timesteps t i ∈ T k t_{i}\in T_{k} where | T k | = N |T_{k}|=N . Let t pre t_{\text{pre}} denote the last timestep before the model is first trained with k k , i.e., t pre = min ⁡ ( T k ) t_{\text{pre}}=\min(T_{k}) . Given a corresponding probe q q , retainability ( ℛ ⁡ ( q , t ) \mathcal{R}(q,t) ) is defined for t ≥ 0 t\geq 0 as follows: ℛ ⁡ ( q , t ) \displaystyle\mathcal{R}(q,t) = ℓ ⁡ ( q , θ t LAM  ( q , N ) + t ) − ℓ ⁡ ( q , θ t pre ) ℓ ⁡ ( q , θ t LAM  ( q , N ) ) − ℓ ⁡ ( q , θ t pre ) . \displaystyle=\frac{{\ell(\textit{q};{\theta_{t_{\text{LAM}}(q,N)+t}})}-\ell(\textit{q};{\theta_{t_{\textit{pre}}}})}{{\ell(\textit{q};{\theta_{t_{\text{LAM}}(q,N)}})}-\ell(\textit{q};{\theta_{t_{\textit{pre}}}})}. (3)

Note that ℛ ⁡ ( p , 0 ) = 1 \mathcal{R}(p,0)=1 which represents that the factual knowledge is 100% retained at the local acquisition maxima of the last knowledge update. Additionally, ℛ ⁡ ( p , t ) = 0 \mathcal{R}(p,t)=0 occurs when the log probability of the probe p p at t SP  ( p ) + t t_{\text{SP}(p)}+t equals that at t pre t_{\text{pre}} . Thus, ℛ ⁡ ( p , t ) = 0 \mathcal{R}(p,t)=0 indicates that the improvement in the log probability of factual knowledge, induced by updating the model with minibatches containing the injected knowledge at t pre t_{\text{pre}} , is completely lost. This x-intercept of ℛ ⁡ ( p , t ) \mathcal{R}(p,t) is crucial for interpreting the behaviors of LLMs, as will be discussed in detail in § 4.4 . The measurement of the defined metrics are illustrated in Figure 1 .

For the measurement of effectivity and retainability, we apply outlier detection using the IQR method with a factor of 1.5. This is particularly important for the measurement of retainability, as the small number of cases which showed no acquisition through training can give a very large value due to the very small denominator in Eq. 3 .

#### Knowledge injection during pretraining