# How Do Large Language Models Acquire Factual Knowledge During Pretraining?

###### Abstract

Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model’s capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models’ robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations for recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus. 1 1 1 Code and data are available at: https://github.com/kaistAI/factual-knowledge-acquisition/

## 1 Introduction

Recent studies on LLMs have shown their ability to capture substantial factual knowledge from the pretraining data [ 14 , 36 , 40 ] . Unfortunately, little is understood about the mechanisms of how LLMs acquire factual knowledge during pretraining. In this work, we make an initial attempt to understand the dynamics of factual knowledge acquisition in LLM pretraining. We study three important yet unanswered research questions: RQ1. How is factual knowledge acquired during LLM pretraining and how are LLMs affected by the training data at each training step?

RQ2. How is the effectivity of factual knowledge acquisition affected by training conditions?

RQ3. How is the acquired factual knowledge forgotten, and how is the trend affected by training conditions?

To answer the research questions, we analyze how LLMs acquire and retain factual knowledge in terms of memorization and generalization by varying the following training conditions: knowledge injection scenarios, pretraining stages, model sizes, and training batch sizes. Specifically, we take the intermediate pretraining checkpoints of different sizes of an LLM at different pretraining stages, inject the target knowledge that the models have not previously encountered, and monitor their step-wise progress of acquiring factual knowledge under various conditions.

Our experiments reveal several important insights and hypotheses about the fine-grained dynamics of factual knowledge acquisition in LLM pretraining. First, we show that factual knowledge acquisition occurs by accumulating the small increase of probability induced by updating the model with a minibatch containing the factual knowledge. Second, compared to the checkpoints at earlier stages, the checkpoint at the later stage shows no significant difference in effectivity, i.e. , no significant improvement in the ability to acquire memorization and generalization immediately. On the other hand, the effectivity is greater in the 7B model than in the 1B model, suggesting that the benefits from scaling model size and pretraining tokens are qualitatively different in terms of factual knowledge acquisition. Third, we find a power-law relationship between training steps (or tokens) and forgetting of acquired factual knowledge in both memorization and generalization. Further examination of the rate of forgetting factual knowledge in LLM pretraining reveals that deduplicating the training data and training the models with a greater batch size enhances the acquisition of factual knowledge, by making them more robust against forgetting. Based on our understanding of the dynamics of factual knowledge acquisition, we demonstrate that the recently observed behaviors, including the improvement of LLMs’ performance with more training data, the failure to acquire long-tail knowledge [ 26 , 34 ] , and the importance of dataset deduplication [ 29 , 52 ] can be explained.

Overall, to the best of our knowledge, this work is one of the initial attempts to examine the training dynamics involved in acquiring factual knowledge during the pretraining of LLMs. By enhancing our understanding of the factual knowledge acquisition dynamics, we expect that academia can gain a holistic understanding and make better use of LLMs.

## 2 Related Work

Recently, there has been a surge in interest in LLMs [ 9 , 13 , 21 , 23 , 49 ] . [ 23 ] and [ 27 ] reported that the performance of LLMs adheres to a scaling law, correlating positively with both the model size and the size of the pretraining corpus. Extensive studies have examined the knowledge encoded in the parameters of LLMs [ 36 , 40 ] . [ 3 ] , [ 15 ] , [ 16 ] , [ 19 ] , [ 20 ] , and [ 31 ] examined how language models learn and capture factual knowledge presented in training data. [ 4 ] demonstrated that knowledge should be presented in a diverse format during pretraining to be reliably extracted. However, recent investigations on LLMs have revealed that LLMs show poor acquisition of long-tail knowledge [ 26 , 34 ] . In addition, LLMs cannot manipulate knowledge from pretraining data effectively [ 5 ] . These works have mainly focused on investigating the factual knowledge encoded in LLMs after pretraining is complete. To examine the detailed training dynamics of knowledge acquisition during pretraining, we conduct a fine-grained analysis of factual knowledge acquisition on each piece of factual knowledge.

Memorization and forgetting are closely related to knowledge acquisition in neural networks [ 6 ] . LLMs memorize a significant amount of training data [ 10 , 29 ] , and the tendency to memorize training data increases as the size of the model gets larger, without harming the ability to generalize the knowledge [ 7 , 11 ] . In addition, [ 17 ] theoretically demonstrated that a specific degree of memorization is essential for attaining high performance. Notably, [ 46 ] conducted an extensive analysis of the behavior of LLMs on memorization and forgetting across various pretraining conditions.

Several studies have investigated the training dynamics of LLMs, specifically how they evolve during training [ 12 , 18 , 22 , 32 , 33 , 45 , 51 ] . [ 44 ] and [ 46 ] focused on the dynamics of memorization in language model pretraining. Recently, [ 53 ] explored the relationship between the data size and grokking [ 37 ] . Compared to these, we perform a more detailed analysis of the dynamics of factual knowledge acquisition during LLM pretraining, by evaluating the log probability of individual pieces of factual knowledge at each training step.

## 3 Experimental Setup

#### Fictional Knowledge dataset

Our goal is to analyze the LLMs’ behavior when acquiring factual knowledge during pretraining. Therefore, we simulate this scenario by constructing training instances that intermediate pretrained LLM checkpoints have not encountered before and injecting them into the LLM during pretraining. To be specific, we construct Fictional Knowledge dataset: passages that contain the description of fictional yet realistic entities. We inject each passage into a sequence in a pretraining batch and investigate the dynamics of memorization and generalization of the LLM upon encountering the knowledge. We call these passages injected knowledge .

Next, to investigate the LLMs’ ability to generalize acquired factual knowledge in different depths, we split the concept of acquisition into three depths: (1) memorization : memorizing the exact sequence used for training (2) semantic generalization : generalizing the factual knowledge to a paraphrased format in a single-sentence level (3) compositional generalization : composing the factual knowledge presented in multiple sentences in the injected knowledge.

Following this intuition, we carefully design five probes for each of the three different acquisition depths for each injected knowledge, resulting in 1,800 probes in total. Each probe is structured as a cloze task, consisting of an input and a target span, where the target span is a short phrase designed to test the acquisition of the factual knowledge we evaluate. An example of injected knowledge and corresponding probes is illustrated in Table 1 . All instances for the injected knowledge and probes are generated by prompting GPT-4 [ 2 ] using the definitions from the ECBD dataset [ 35 ] as a template, and filtering out invalid cases. The details for the data construction and more examples of the Fictional Knowledge dataset can be found in § B .

#### Evaluation metrics

To conduct a detailed analysis of the LLMs’ acquisition of factual knowledge during pretraining, we evaluate the model’s state by examining log probabilities to obtain fine-grained information [ 41 ] . To quantitatively measure the trend of factual knowledge acquisition, we should first define the timestep where the local effect of updating the model using the injected knowledge completely pays off. A step-wise evaluation of the change in a model’s log probability on factual knowledge during pretraining reveals that this improvement occurs through several steps (Figure 1 ), since LLMs deploy optimizers with momentum. Hence, we define the timestep where the log probability reaches a maximum value in a short interval after the model is trained on the injected knowledge, which we refer to as the local acquisition maxima .

###### Definition 1

Given a language model, let θ t \theta_{t} represent the model’s parameters before the t t -th update. Given injected knowledge k k (used as a training instance) and the corresponding probe q q (used as an evaluation instance), let ℓ ⁡ ( q , θ ) \ell(q;\theta) denote the log probability of the target span of q q , provided by the model. Let a nonempty set T k = { t 1 , t 2 , … , t n } T_{k}=\{t_{1},t_{2},\ldots,t_{n}\} denote the steps where the model is updated with the minibatch containing the injected knowledge k k , where 0 ≤ t 1 < t 2 < … < t n 0\leq t_{1}<t_{2}<\ldots<t_{n} . Finally, let t w t_{w} denote the window size. Then, the local acquisition maxima ( t LAM ​ ( q , i ) t_{\text{LAM}}(q,i) ) is defined as: t LAM ​ ( q , i ) = argmax t i < t ≤ t i + t w ​ ℓ ​ ( q , θ t ) where ​ t i ∈ T k . t_{\text{LAM}}(q,i)=\underset{t_{i}<t\leq t_{i}+t_{w}}{\mathrm{argmax}}\,\ell(q;{\theta_{t}})\quad\text{where }t_{i}\in T_{k}. (1)

In Eq. 1 , the definition of the local acquisition maxima is also dependent on the injected knowledge k k and the window size t w t_{w} , but we write t LAM ​ ( q , i ) t_{\text{LAM}}(q,i) for brevity. We use the window size t w = 50 t_{w}=50 . 2 2 2 The β 1 \beta_{1} of AdamW optimizer is configured to 0.9 in our experiments, implying that the contribution of the gradient of a given sequence to the momentum will be reduced to approximately 0.9 50 ≈ 0.0052 0.9^{50}\approx 0.0052 after 50 steps. Therefore, t w = 50 t_{w}=50 is a reasonable choice for the window size. 3 3 3 If optimizers without momentum ( e.g. , RMSProp) are used, the local effect of training the model at timestep t t will be fully reflected immediately after that step. In such cases, t w t_{w} should be 1 and t LAM t_{\text{LAM}} will reduce to t + 1 t+1 .

Next, we define a metric to quantify the immediate improvement in the model’s log probability of factual knowledge after it is presented with the knowledge for the i i -th time. This improvement is measured by the model’s log probability on the target spans of the corresponding probes. This metric, effectivity , will be used to answer the second research question.

###### Definition 2

Given a language model parameterized by θ \theta trained with an injected knowledge k k at t = t i t=t_{i} where t i ∈ T k t_{i}\in T_{k} , and a corresponding probe q q , the effectivity ( ℰ ⁡ ( q , i ) \mathcal{E}(q,i) ) is defined as the absolute increase of the model’s log probability on the target span of q q between t = t i t=t_{i} and t = t LAM ​ ( q , i ) t=t_{\text{LAM}}(q,i) , i.e., ℰ ⁡ ( q , i ) \displaystyle\mathcal{E}(q,i) = ℓ ⁡ ( q , θ t LAM ​ ( q , i ) ) − ℓ ⁡ ( q , θ t i ) . \displaystyle=\ell(\textit{q};{\theta_{t_{\text{LAM}}(q,i)}})-\ell(\textit{q};{\theta_{t_{i}}}). (2)

Finally, to investigate the forgetting phenomenon of acquired factual knowledge (RQ3), we define a metric that quantifies the fraction of improvement in log probability retained by the model after t t steps, relative to the local acquisition maxima of the last knowledge update.

###### Definition 3

Consider a language model parameterized by θ \theta and trained with injected knowledge k k for N N iterations, occuring at timesteps t i ∈ T k t_{i}\in T_{k} where | T k | = N |T_{k}|=N . Let t pre t_{\text{pre}} denote the last timestep before the model is first trained with k k , i.e., t pre = min ⁡ ( T k ) t_{\text{pre}}=\min(T_{k}) . Given a corresponding probe q q , retainability ( ℛ ⁡ ( q , t ) \mathcal{R}(q,t) ) is defined for t ≥ 0 t\geq 0 as follows: ℛ ⁡ ( q , t ) \displaystyle\mathcal{R}(q,t) = ℓ ⁡ ( q , θ t LAM ​ ( q , N ) + t ) − ℓ ⁡ ( q , θ t pre ) ℓ ⁡ ( q , θ t LAM ​ ( q , N ) ) − ℓ ⁡ ( q , θ t pre ) . \displaystyle=\frac{{\ell(\textit{q};{\theta_{t_{\text{LAM}}(q,N)+t}})}-\ell(\textit{q};{\theta_{t_{\textit{pre}}}})}{{\ell(\textit{q};{\theta_{t_{\text{LAM}}(q,N)}})}-\ell(\textit{q};{\theta_{t_{\textit{pre}}}})}. (3)

Note that ℛ ⁡ ( p , 0 ) = 1 \mathcal{R}(p,0)=1 which represents that the factual knowledge is 100% retained at the local acquisition maxima of the last knowledge update. Additionally, ℛ ⁡ ( p , t ) = 0 \mathcal{R}(p,t)=0 occurs when the log probability of the probe p p at t SP ​ ( p ) + t t_{\text{SP}(p)}+t equals that at t pre t_{\text{pre}} . Thus, ℛ ⁡ ( p , t ) = 0 \mathcal{R}(p,t)=0 indicates that the improvement in the log probability of factual knowledge, induced by updating the model with minibatches containing the injected knowledge at t pre t_{\text{pre}} , is completely lost. This x-intercept of ℛ ⁡ ( p , t ) \mathcal{R}(p,t) is crucial for interpreting the behaviors of LLMs, as will be discussed in detail in § 4.4 . The measurement of the defined metrics are illustrated in Figure 1 .

For the measurement of effectivity and retainability, we apply outlier detection using the IQR method with a factor of 1.5. This is particularly important for the measurement of retainability, as the small number of cases which showed no acquisition through training can give a very large value due to the very small denominator in Eq. 3 .

#### Knowledge injection during pretraining

We explore how LLMs acquire and retain factual knowledge in terms of memorization and generalization by examining the following factors: (i) varying knowledge injection scenarios ( duplication , paraphrase , once ), (ii) varying pretraining stages ( early , mid , and late , pretrained with approximately 170B, 500B, and 1.5T tokens, respectively), (iii) varying model sizes (1B and 7B), and (iv) varying training batch sizes (2048 and 128). To this end, we resume pretraining OLMo [ 21 ] intermediate checkpoints restoring the optimizer and scheduler states the same way OLMo is pretrained, using the pretraining data of OLMo (Dolma v1.5 [ 43 ] ), except that we inject factual knowledge every 100 training steps by replacing a part of original pretraining batch with the injected knowledge of the Fictional Knowledge dataset. 4 4 4 We use OLMo for the experiments since the intermediate checkpoints, optimizer states, and batch sequence data for pretraining the model are made publicly available. Each injected knowledge is short enough to fit into one pretraining sequence in the batch, and we fill the rest of the sequence with the original sequence in the batch. To investigate the difference in the factual knowledge acquisition dynamics when the models are presented with the knowledge, we inject factual knowledge with three different injection scenarios: duplication , paraphrase , and once . For the duplication injection scenario, we inject the same knowledge 10 times with an interval of 100 training steps. In the paraphrase injection scenario, we inject paraphrased knowledge instead of showing identical sequences, every time it is presented to the model. Lastly, in the once injection scenario, we inject the knowledge only once at the start of the training. After the injection is complete, we continue pretraining as normal. The details for the training setup can be found in § D .

## 4 Results

### 4.1 Factual knowledge acquisition occurs by accumulating the observations of the fact

Figure 2 shows the progress of factual knowledge acquisition of OLMo-7B, by averaging the model’s log probability across the target spans of the probes for each injection scenario, evaluated at each training step. Regardless of the acquisition depths (memorization, semantic generalization, and compositional generalization), the model’s log probability measured on the probes shows an immediate and distinctive increase, after the model is updated with the batch containing the injected knowledge. However, the log probability decreases again, as the knowledge is not presented to the model afterward. This observation directly demonstrates the mechanism of factual knowledge acquisition: LLMs acquire factual knowledge by accumulating micro-acquisitions with subsequent forgetting each time the model encounters the knowledge during pretraining.

Several findings can be further obtained from Figure 2 . First, when the model is updated after seeing the factual knowledge, the most significant improvement in log probability is observed for memorization, followed by semantic generalization, and the least improvement is seen in compositional generalization. Next, however, the gap between memorization and semantic generalization almost disappears in the paraphrase injection scenario. Third, when the model is updated with the duplication injection scenario, the model shows a larger improvement of log probability in all acquisition depths, but also the forgetting is faster, eventually resulting in a similar level of improvement at the end of the training ( t = 2000 t=2000 ) compared to the paraphrase injection scenario.

These patterns are consistent across all pretraining stages of OLMo-7B we investigate (§ E.1 ). Intriguingly, the training dynamics of OLMo-1B early checkpoint (Appendix Figure 8 ) show much more unstable dynamics than those of later checkpoints (Appendix Figure 9 and 10 ) and the early checkpoint of OLMo-7B (Appendix Figure 6 ). The distinctive behavior of the OLMo-1B early checkpoint suggests that pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such a threshold may be higher for smaller models.

### 4.2 Effects of model scale and pretraining stage on knowledge acquisition dynamics

Next, we measure effectivity (Eq. 2 ) to quantify the improvement of the LLMs’ log probability after being trained with the injected knowledge, averaged across all probes ( q q ) and encounters ( i i ). The results are demonstrated in Figure 3 . The average effectivity is the largest in the Once injection scenario since the effectivity is higher when the model encounters the injected knowledge for the first time, which is further discussed in § H .

In all injection scenarios, there is an improvement in effectivity when the model size is scaled from 1B to 7B (as shown on the right side of Figure 3 ). 5 5 5 For a fair comparison of the effectivity of the 1B and 7B models, the OLMo-1B Mid checkpoint is trained using the same initial learning rate as the OLMo-7B Mid checkpoint (the specific value is provided in Appendix Table 5 ). The measured effectivity for all OLMo-1B checkpoints with the original learning rate is presented in Appendix Figure 11 . On the other hand, surprisingly, the effectivity of fact acquisition does not improve with checkpoints trained with more tokens, as shown on the left side of Figure 3 . This tendency is consistent across all model scales and injection scenarios (see also Appendix Figure 11 ). Moreover, this tendency is not attributed to training the models with a decreased learning rate through learning rate decay, as demonstrated by an additional experiment of training three checkpoints using the same constant learning rate. The results with the constant learning rate show that effectivity does not significantly improve in the checkpoints of later stages of pretraining where more pretraining tokens are seen (§ F ). Therefore, the observation implies that the effectivity of LLMs in acquiring factual knowledge does not significantly improve throughout the progress of pretraining.

While our finding that effectivity remains unchanged for different stages of pretraining may seem contradictory to the widely known observation that the amount of pretraining data is a critical factor in the performance of LLMs [ 23 , 27 ] , we suggest a plausible hypothesis based on further observations in § 4.3 . Specifically, we suggest that the high performance of LLMs trained with larger and more diverse datasets is not primarily due to an emergent ability from the sheer amount of tokens observed during training [ 50 ] , but rather because the model encounters a wider variety of knowledge more times, which allows for the accumulation of log probabilities of more knowledge become high enough to be decoded as outputs of the model. We discuss this hypothesis further in § 4.4 .

Comparing the duplication and paraphrase injection scenarios, the duplication injection scenario naturally shows higher effectivity for memorization. However, the higher effectivity in the duplication injection scenario for semantic generalization and compositional generalization appears to be counterintuitive, as it is widely observed that deduplication of pretraining data is an important factor in improving model performance [ 29 , 52 ] . In the following sections, we will address this question by demonstrating that the models exhibit faster forgetting in generalizing factual knowledge when presented with duplicated texts (§ 4.3 ).

### 4.3 Forgetting in factual knowledge acquisition

#### Training steps and the forgetting of acquired factual knowledge have a power-law relationship

The exponential trend of forgetting has been reported in various aspects of LLM training, including memorization in pretraining [ 46 ] and task performances in continual learning [ 33 , 39 ] . Motivated by this, we investigate whether the exponential trend of forgetting persists in the context of factual knowledge acquisition in LLM pretraining. Figure 4 illustrates the trend of retainability against the training steps past the local acquisition maxima. We find that the trend of ℛ ⁡ ( p , t ) \mathcal{R}(p,t) against l ​ o ​ g ​ ( t ) log(t) fits a linear function very well ( R 2 > 0.80 R^{2}>0.80 for memorization and semantic generalization, and R 2 > 0.65 R^{2}>0.65 for compositional generalization). This trend is persistent across all acquisition depths, and all training conditions (§ E.4 and § E.5 ). Guided by empirical observations, we model the trend of forgetting using a power-law model in further investigations.

#### How quickly is the acquired factual knowledge lost?

The absolute value of the slope of the fitted lines in Figure 4 can be interpreted as the decay constant ( a a ) of retainability, formally, Δ ℛ ( p , t ) ≈ − a ⋅ log ( t 2 t 1 ) for 0 < t 1 < t 2 < τ , where ℛ ( p , τ ) = 0 and a > 0 . \Delta\mathcal{R}(p,t)\approx-a\cdot\log\left(\frac{t_{2}}{t_{1}}\right)\quad\text{for }0<t_{1}<t_{2}<\tau,\quad\text{where }\mathcal{R}(p,\tau)=0\text{ and }a>0. (4) Thus, the measured decay constant represents how fast (in terms of fraction) the model loses the improvement of log probability. Table 2 shows the decay constants of retainability measured for three OLMo-7B intermediate checkpoints, for duplication and paraphrase injection scenarios.

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

There are several observations in Table 2 . First, the forgetting in compositional generalization is slower (the decay constant a a is smaller) than in memorization and semantic generalization. Combined with the observations in previous sections, the acquisition of compositional generalization accumulates most slowly but is more robust to forgetting. Second, the forgetting tends to be slower in the paraphrase injection scenario compared to the duplication injection scenario. This finding will be further discussed in § 4.4 , regarding the importance of deduplicating training data. Finally, the decay constants are similar for the two earlier checkpoints but smaller for the late checkpoint in the duplication injection scenario. We demonstrate that this is due to the reduced learning rate from learning rate scheduling (Appendix Table 5 ), as the decay constants show no decrease for the later checkpoint when each checkpoint is trained with the same constant learning rate (Appendix Table 9 ).

#### Pretraining with a larger batch size helps LLMs acquire more knowledge

It is a common practice to pretrain LLMs with a very large batch size to leverage parallel computing [ 13 , 21 , 25 , 30 , 49 ] . However, the effects of increasing training batch size in terms of the LLMs’ acquisition of factual knowledge remain underexplored. In this section, we examine whether pretraining LLMs with a larger batch size is advantageous regarding factual knowledge acquisition. Specifically, we continue training LLMs with a batch size reduced by a factor of 16 compared to the original pretraining batch size, i.e. , from 2048 to 128.

Figure 5 compares the forgetting dynamics of OLMo-7B mid checkpoint between pretraining and training with the reduced batch size. The results have several implications for the advantage of pretraining LLMs with a larger batch size. First, comparing Figure 3 and Appendix Figure 21 , LLMs trained with the smaller batch size show higher effectivity. However, the decay constant tends to be higher, comparing the numbers in Table 2 and Appendix Table 10 . Furthermore, the anticipated x-intercept is significantly decreased by dozens of times, comparing Appendix Table 6 and 11 . This implies that the models trained with smaller batch sizes have shorter learnability threshold , the point such that an LLM cannot learn the knowledge presented with intervals longer than that threshold, which we discuss in detail in the following section (§ 4.4 ). In other words, when an LLM is trained with a smaller batch size, factual knowledge should be presented more often to the model so as not to be forgotten and the set of learnable knowledge is reduced. Second, accelerated forgetting with a smaller batch size is more pronounced for compositional generalization compared to memorization and semantic generalization. In brief, the results suggest that pretraining with a small batch size reduces the set of learnable knowledge due to accelerated forgetting, and leads to worse compositional generalization performance of learned factual knowledge.

### 4.4 Implications for LLM pretraining

#### Why is popularity important for factual knowledge acquisition?

The estimated x-intercepts in Figure 5 represent the number of additional training tokens that would lead to the complete loss of the factual knowledge acquired by training. 6 6 6 The exact values of the estimated x-intercepts can be found in Appendix Table 6 . Hence, if a given factual knowledge in the pretraining dataset is in the long-tail and the knowledge is presented to the model with an interval longer than a certain threshold, such knowledge will be impossible to be decoded as the top-k generation of the model, or learned , regardless of the duration of the pretraining. 7 7 7 This theoretical threshold may not be equal to the estimated x-intercepts presented in Figure 5 , as we estimate the threshold based on the controlled experiment of injecting factual knowledge. In addition, the actual learnability threshold is likely to vary for different types of factual knowledge due to several factors, such as the number of similar/related facts or temporal conflicts in the pretraining data. This implies that there is a learnability threshold , a threshold of the interval where the model fails to acquire knowledge of which its encounter interval is longer than the threshold. Most well-known facts are likely to be presented to the model with an interval of the training steps shorter than this learnability threshold . In such a case, the model will accumulate the increased log probability of the knowledge upon each encounter of the knowledge as the pretraining progresses, and at some point, the accumulated log probability of the knowledge will be high enough to generate the knowledge as the decoding output of the model [ 41 ] . Moreover, LLMs will accumulate the log probability faster for more popular knowledge, and thus the acquisition of such knowledge will be reflected in the model’s top-k output sequence generation in a relatively earlier pretraining stage, as demonstrated in [ 8 ] .

In summary, we hypothesize that the popularity of the knowledge in the pretraining data influences how quickly this knowledge begins to be ‘revealed’ in the generated sequences during pretraining, except for the knowledge in the long-tail whose low popularity makes the encounter interval longer than the learnability threshold . Also, as briefly mentioned in § 4.2 , we hypothesize that the reason why larger and more diverse pretraining data helps the model performance is that the model can acquire a broader range of factual knowledge (more knowledge will be presented with an interval shorter than the learnability threshold ) since the skewness of the distribution of factual knowledge popularity is likely to be mitigated as the data becomes larger and more diverse.

#### Why does deduplication enhance model performance?

Recent pretraining corpora are thoroughly deduplicated [ 9 , 28 , 38 , 43 , 47 , 48 ] , as it is widely observed that data deduplication can improve model performance [ 1 , 29 , 42 , 52 ] . Our results suggest that the smaller decay constant in the paraphrase injection scenario observed in § 4.3 can explain the advantages of training LLMs with deduplicated training data, as deduplication tends to slow the forgetting of generalizing acquired factual knowledge. This can also be observed in Figure 2 , as the gap of the increase of log probability immediately after encountering the injected knowledge is large between the duplication and paraphrase injection scenarios, but this gap diminishes at the end of the measurement. Moreover, since the model tends to provide a higher increased log probability to the memorization rather than generalization (Figure 2 and 3 ), presenting the model with duplicated texts with a short interval will result in the widening of the gap between memorization and generalization, which will drive the model to prefer generating memorized contexts compared to generalizing factual knowledge [ 4 ] .

## 5 Discussion and Conclusions

In this work, we study how LLMs acquire factual knowledge during pretraining. Our findings and contributions can be summarized as follows:

• We propose methods, datasets, and metrics for performing a fine-grained analysis of factual knowledge acquisition dynamics during LLM pretraining.

• We demonstrate that factual knowledge acquisition in LLM pretraining is achieved through accumulating micro-acquisitions, each of which occurs whenever the model is updated after seeing the factual knowledge. When the model is not presented with factual knowledge, forgetting occurs and the acquisition of the knowledge is gradually diluted.

• However, while the amount of immediate improvement in log probability upon observation of the knowledge increases for larger models, the amount does not significantly increase throughout the progress of pretraining. This finding suggests that the benefits of scaling the model size and pretraining tokens are qualitatively different.

• There is a power-law relationship between training steps and forgetting of acquired factual knowledge, in terms of both memorization and generalization. Also, pretraining LLMs with deduplicated data and larger batch sizes enhances the acquisition of factual knowledge, making them more robust against forgetting the learned factual knowledge.

• We provide potential explanations for recently observed, yet underexplored behaviors of LLMs. First, we propose that the improved performance of LLMs through data scaling results from consistent improvements rather than an emergent ability to acquire factual knowledge more quickly during pretraining. Second, we hypothesize that LLMs struggle to acquire unpopular knowledge because they need sufficient exposure to factual knowledge with intervals shorter than the learnability threshold to increase the probability. Third, our findings suggest that deduplicating the pretraining corpus improves LLM performance by preventing the model from assigning a higher probability to duplicated sequences and helping it retain acquired generalization longer.

Overall, we demonstrate the importance of understanding the factual knowledge acquisition dynamics of LLMs to understand the behavior of LLMs, opening up a promising avenue for future research.

## Acknowledgments and Disclosure of Funding

We would like to thank Seongyun Lee, Suehyun Park, Hyeonbin Hwang, Geewook Kim, Juyoung Suk, Aengus Lynch, and Katja Filippova for their valuable feedback on our work.

This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (No.RS-2019-II190075 Artificial Intelligence Graduate School Program(KAIST).

## Appendix

## Appendix A Limitations

Although they do not affect the findings and implications of our work, there are several limitations. First, we do not perform evaluations based on the generation output of the model, and we do not investigate the exact relationship between the model’s accumulation of probability of factual knowledge and the model’s generation output. Second, we do not analyze the pretraining dynamics at very early stages, which can exhibit significantly different behaviors [ 24 ] . Third, we do not study the effect of training batch size and learning rate on the dynamics of factual knowledge acquisition across multiple values. Future works exploring these would help us to further enhance our understanding of LLMs.

## Appendix B Dataset Construction and Examples

We construct a Fictional Knowledge dataset by prompting GPT-4 [ 2 ] with C.1 to generate descriptions for non-existent, fictional entities using the format of the ECBD [ 35 ] dataset, which is based on English Wikipedia articles. We select only the generated descriptions of the fictional entities that can produce at least five sentences suitable for a cloze task when the last span of the sentence is set as the target label. We repeat this until a total of 120 descriptions are produced. We call this "injected knowledge" in this paper. This process facilitates us to investigate the factual knowledge acquisition of the language models in a more controlled setup, as we can ensure that the model has never encountered the facts contained in the injected knowledge during the pretraining process. For the paraphrase injection training scenario mentioned in § 3 , we generate 9 paraphrased injected knowledge for each original injected knowledge by prompting GPT-4 with C.2 .

The types of probes for the injected knowledge consist of memorization probes, semantic generalization probes, and compositional generalization probes. For each injected knowledge, 15 probes are generated, with 5 for each type. First, the memorization probes are constructed by extracting exact sentences from the injected knowledge that ends with a named entity and setting the named entity as the target span. Next, the semantic generalization probes are created by prompting GPT-4 with C.3 to paraphrase each memorization probe while maintaining the target span and requiring no additional context. Lastly, compositional generalization probes are created by prompting GPT-4 with C.4 to create cloze tasks to evaluate whether new factual knowledge can be inferred by integrating and generalizing the factual knowledge in the injected knowledge. We constrain that the compositional generalization probes should avoid lexical overlap with the injected knowledge as much as possible and should not require additional context beyond the knowledge in the injected knowledge. To ensure the validity of the generated compositional generalization probe sets, we ask GPT-4 using prompt C.5 to evaluate whether each probe meets these conditions, answering with "yes" or "no". Only the probes that receive a "yes" response are selected. Examples of injected knowledge and paraphrased injected knowledge from the Fictional Knowledge dataset can be found in Table 3 and the memorization probes, semantic generalization probes, and compositional generalization probes used to evaluate the acquisition of knowledge can be found in Table 4 .

## Appendix C Prompts Used for Dataset Generation

### C.1 Prompts for the generation of injected knowledge

Carefully read the provided sentence; this is a short passage containing factual knowledge, that is extracted from Wikipedia:\n\n {DEFINITION IN ECBD DATASET}\n\nNow, assume that you are writing a very long and detailed descriptive paragraphs (more than 20 sentences) using the provided passage as a template. However, you should replace the named entities(person, country, act, etc.) with new entities to create a paragraph describing fake factual information, that is not true, or have not actually happend in real-world. Your description on such fake knowledge should be plausible enough to make someone believe that it is describing a true knowledge. You should always start and finish every sentence with a named entity. Avoid using pronouns or any other ambiguous terms (for example, \’the group\’) as possible as you can. Finally, avoid to generate knowledge that is potentially harmful. Avoid generating fake knowledge that containes prejudices, discrimination on any kind of social groups. Output the created paragraph only.\n\n

### C.2 Prompts for the generation of paraphrased injected knowledge

The following text needs to be paraphrased to convey the same meaning in different words:\n\n\"{ORIGINAL INJECTED KNOWLEDGE}\"\n\nPlease paraphrase the above text clearly and concisely.

### C.3 Prompts for the generation of semantic generalization probes

Paraphrase the provided text with a constraint: the paraphrased sentence should be ended with the specified target, where the original sentence also ends with the target. Note that the paraphrased sentence should be semantically equivalent to the original sentence, and it should not contain any additional factual knowledge, nor lacks any factual knowledge that is stated in the original text. In addition, the content of the paraphrased text should be able to be fully understood without any ambiguity.\n Here are some exmaples:\n\n[Example1 1]\n\n Input: The Lionheart Battalion (LB) is a fictitious white nationalist militia group in Spain.\nTarget: Spain\nOutput: The Lionheart Battalion (LB) is a fictional militia group with white nationalist beliefs located in Spain.\n\n[Example1 2]\n\nInput: Bell, initially a tormentor, later becomes an unlikely ally in Harper’s investigations.\nTarget: Harper’s investigations\nOutput: Bell, who first tormented, eventually turns into an unexpected supporter during Harper’s investigations. \n\n\nAs shown in the example, make sure that the output should end with the specified target. Never finish the sentence with any other words.\n\nNow, this is your input and target:\n\nInput: {MEMORIZATION PROBE}\nTarget: {TARGET FOR MEMORIZATION PROBE}\nOutput:

### C.4 Prompts for the generation of compositional generalization probes

You are tasked with evaluating a participant’s intelligence(in terms of generalization, composition, and inference) by measuring their ability to understand and combine the implications of different factual knowledge presented in a passage and apply them to deduce unseen knowledge. Specifically, you will create a next-word prediction task consisting of inputs and targets. The objective is to assess whether the participant can integrate and generalize the implications of the factual knowledge from the passage, combining different pieces of information to infer new factual knowledge.\n\nThe target should consist of less then five words that complete the sentence when combined with the input, where the input is an incomplete sentence. The inputs and targets must be designed so that the target can only be accurately answered if the participant can perform complex generalization and integration based on the provided knowledge.\n\n Create eight different pairs of inputs and corresponding targets that require the participant to combine various factual knowledge presented in the passage, to deduce unseen knowledge. Avoid lexical overlaps with the passage as much as possible. Also, the content in the task should not ask for factual knowledge that is directly mentioned in the given passage, in other words, difficult enough. Additionally, ensure that the input and target can be understood and answered without additional context, assuming that the reader has comprehended and remembered the knowledge from the passage. Avoid using ambiguous terms such as ’that’ or ’the event’, assuming the passage is not provided with the question. Finally, most importantly, be creative as much as you can.\n\nPlease present your answers in the following format:\n\nProbe1: [YOUR_PROBE_ENDS_WITH_AN_UNDERSCORE]\nAnswer1: [YOUR_ANSWER_TO_THE_PROBE]\n\nNow, this is your passage:\n\n {ORIGINAL INJECTED KNOWLEDGE}

### C.5 Prompts for the validation of generated compositional generalization probes

You will be provided with a pair of cloze-task question and answer, and the problem’s goal is to evaluate the subject’s factual knowledge. Your task is to verify whether the provided pair of question and answer is properly designed to evaluate the factual knowledge. Assume that the subject has been already informed with the counterfactual knowledge before. Then, we are testing the subject’s counterfactual knowledge. Note that regardless of the consistency of the factual knowledge tested in the problem, we say that the problem is properly designed if there is no ambiguity in the question and answer. So the question is verifying: Can the content of the question be fully understood and properly answered without any ambiguity or the need of additional context, given that the corresponding factual knowledge is existent?\n \nAfter providing your explanation, you should give your answer in ‘yes’ or ‘no’. The answer should be ‘yes’ only if both of the conditions are satisfied, and the answer should be ‘no’ otherwise.\n For example, this is an example of your answer:\n\nExplanation: [YOUR_EXPLANATION]\nAnswer: [YES_OR_NO]\n\nHere are some example inputs and answers:\n\n[Example 1]\nQuestion: Within the realm of fantasy, he is ranked second in command in the _____\nAnswer: Lionheart Battalion\n \nExplanation: The example provided is not entirely clear or straightforward in its design to evaluate factual knowledge. The question, \"Within the realm of fantasy, he is ranked second in command in the _____,\" contains a few ambiguities. Firstly, \"the realm of fantasy\" is a broad and non-specific term, which could refer to any number of fantasy stories, games, or universes. Secondly, the phrase \" he is ranked second in command\" does not specify who \"he\" refers to, nor does it establish a clear context or a specific entity to which the answer \"Lionheart Battalion\" could logically be connected without additional information. This lack of specificity and context does not allow the question to be answered accurately based solely on factual knowledge without guessing or assuming additional context. The problem does not provide enough information to identify which fantasy setting is being referred to, nor does it give any clues about the character or the organizational structure within which this character operates.\n Answer: no\n\n[Example 2]\nQuestion: Jaccard Hume was the first person to land on _____\nAnswer: Mars\n\nExplanation: This question and answer pair seems straightforward and specific in its design to evaluate factual knowledge. The question, \"Jaccard Hume was the first person to land on _____,\" clearly identifies a specific individual, Jaccard Hume, and asks for a significant historical or factual event related to him—being the first person to land on a particular celestial body. The answer provided is \"Mars,\" which is clear and direct. Assuming the subject has the necessary factual knowledge about Jaccard Hume and his achievements, there is no ambiguity in either the question or the answer. The answer \"Mars\" directly fills the blank without the need for additional context or interpretation. Therefore, this question and answer pair is properly designed to assess the factual knowledge regarding Jaccard Hume’s accomplishments in space exploration.\nAnswer: no\n\nNow, here is the input text:\n\nQuestion: {GENERATED COMPOSITIONAL GENERALIZATION PROBE} _____Answer: {GENERATED TARGET OF COMPOSITIONAL GENERALIZATION PROBE}\n\n

## Appendix D Detailed Training Setup

To continue training almost similar to the pretraining setup, we use OLMo [ 21 ] , as it provides not only intermediate model checkpoints but also the exact sequence of data instances used for pretraining, the optimizer state, and the learning rate scheduler. Throughout the entire pretraining process, the language model is trained with a language modeling objective.

Except for the batches that include injected knowledge from Fictional Knowledge dataset at specific step intervals, we train OLMo with batches from the Dolma corpus [ 43 ] in the same order which is used in OLMo pretraining. Specifically, we load the training batch that OLMo will be seen at the specific pretraining step, append the injected knowledge from the Fictional Knowledge dataset to the front of each row, and truncate the original rows from the end by the token length of the injected knowledge. This approach creates batches that have the same size as the original pretraining batches, with 2048 rows and a sequence length of 2048, meaning each batch contains 4M tokens. We adopt this method to deviate as little as possible from the original pretraining data distribution.

In the Fictional Knowledge dataset, which consists of 120 descriptions of fictional knowledge, we use the first 1-40 injected knowledge to examine the dynamics of knowledge acquisition in the paraphrase injection scenario which is described in § 3 . The 41-80 injected knowledge is used for the duplication injection scenario, and the 81-120 injected knowledge is used for the once injection scenario.

For each injection scenario, the Fictional Knowledge data are injected into the batch and trained according to the following rules. In the duplication injection scenario, injected knowledge in the Fictional Knowledge dataset is injected into the original pretraining batch, and the language model is trained on this modified batch 10 times every 100 steps. Next, in paraphrase injection scenario, similar to the duplication injection scenario, the model is trained on the modified batches containing Fictional Knowledge every 100 steps for a total of 10 times, however, in this case, paraphrased injected knowledge is used at each injection step. Lastly, in the once injection scenario, the modified batch containing injected knowledge of Fictional Knowledge is shown to the language model just once, after which it continues training on the original batch of Dolma corpus.

After 1000 steps of pretraining following the above rules, an additional 1500 steps of pretraining are conducted using the Dolma corpus for experiments analyzing forgetting dynamics in § 4.3 . The Dolma corpus used at these steps is a corpus that will be viewed starting from the 360,000th step of pretraining the OLMo. This approach ensures consistency in the Dolma corpus across all conditions while guaranteeing that the corpus has not been seen in any previous pretraining processes. Continued pretraining of a total of 2500 steps takes approximately 3 days using 8 80GB A100 GPUs.

To examine the differences in knowledge acquisition dynamics based on model size, we use OLMo-7B and OLMo-1B. For differences based on the number of pretrained tokens, we use intermediate checkpoints at Early (170B) stage (specifically, 177B tokens for 7B and 168B tokens for 1B), Mid (500B) stage (specifically, 500B tokens for 7B and 494B tokens for 1B), and Late (1.5T) stage (1.5T tokens for 7B and 1B). Since the initial checkpoints of OLMo-1B are stored in units of 10000, it is the best choice in the given situation to select the checkpoint trained with the number of tokens closest to 177B. The differences in initial learning rate values for each case based on different model sizes and pretraining stages are recorded in Table 5 below.

## Appendix E Additional Figures for the Pretraining Experiments

### E.1 Training dynamics of other OLMo-7B checkpoints

### E.2 Training dynamics of other OLMo-1B checkpoints

### E.3 Effectivity measurement data for OLMo-1B

### E.4 Forgetting dynamics of OLMo-7B checkpoints

### E.5 Forgetting dynamics of OLMo-1B checkpoints

## Appendix F Experiments for Training Olmo-7B Checkpoints With a Constant Learning Rate

We continue training each OLMo-7B checkpoint with a constant learning rate, to compare the effectivity and retainability of each checkpoint while excluding the impact of different learning rates. Optimizer states are loaded to promote a warm start of continued training. Due to the restriction of computational resources, we reduce the batch size from 2048 to 128 for this experiment. The value of the constant learning rate is obtained by averaging the starting learning rates of three checkpoints. We do not apply learning rate decay for this experiment. All other training conditions not mentioned are identical to the main experiment. The results in Appendix Figure 17 demonstrate that there is no improvement of average effectivity in later checkpoints, although all models are trained with the same learning rate. This supports that the non-increasing effectivity in pretraining progress is not attributed to the learning rate decay. Similarly, there is no decrease in the decay constants for the later checkpoints (Appendix Table 9 ). Note that the figures in § F.1 demonstrate that reducing the batch size does not significantly change the model’s behavior in accumulating log probability during factual knowledge acquisition.

### F.1 Training dynamics for constant learning rate experiments

## Appendix G Forgetting Dynamics of Olmo-7B Trained With a Reduced Batch Size

Similar to F , we train the OLMo-7B intermediate checkpoints with a reduced batch size of 128. However, we set the learning rate for each checkpoint as the initial learning rate (Appendix Table 5 ), as the objective of this experiment is to examine the effect of reduced batch size on the forgetting dynamics. We re-initialize the optimizer state. We observe that this results in unstable dynamics in early steps, but the dynamics are stabilized soon, and do not harm the model’s overall behavior in general (§ G ). Appendix Figure 21 shows the effectivity measurements of OLMo-7B models at different pretraining stages. Similar to the observations in Appendix Figure 17 , the effectivity values are greater compared to the values in the pretraining experiment (Figure 3 ). Appendix Figure 22 and 23 illustrates the forgetting dynamics of OLMo-7B Early (170B) and late (1.5T) checkpoints, respectively. Appendix Table 10 shows the decay constants ( a a ) measured with three different pretraining stages, acquisition depths, and injection scenarios. Note that the slope remains unchanged regardless of whether we set the x-axis to tokens or training steps. Hence, the decay constants in the table can be directly compared to the values presented in Table 2 . Comparing the values of the expected x-intercepts of retainability presented in Appendix Table 11 with Appendix Table 6 , the results demonstrate that the model trained with a smaller batch size has a shorter learnability threshold .

### G.1 Training dynamics for experiments on the forgetting dynamics with a reduced batch size

## Appendix H Effect of the Number of Previous Encounters on Effectivity and Retainability of Factual Knowledge

We measure the average effectivity for each count of injection ( i i ) in duplication and paraphrase injection scenario. In this analysis, we exclude the cases where the log probability at the local acquisition maxima is smaller than the point before the model is trained with the injected knowledge, as such cases can be regarded as failure cases of learning. Appendix Figure 27 , 28 , and 29 display the results for OLMo-7B early , mid , and late checkpoints, respectively. We observe that the effectivity is relatively constant regardless of the number of previous injections of the knowledge. However, we observe that the effectivity is the highest when the model is trained with the injected knowledge for the first time, both in the duplication and paraphrase injection scenarios.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .