How Do Large Language Models Acquire Factual Knowledge During Pretraining? — part 2/4
Cleaned-text code points [15107, 30506). Bibliography omitted here; source.md includes it.
We explore how LLMs acquire and retain factual knowledge in terms of memorization and generalization by examining the following factors: (i) varying knowledge injection scenarios ( duplication , paraphrase , once ), (ii) varying pretraining stages ( early , mid , and late , pretrained with approximately 170B, 500B, and 1.5T tokens, respectively), (iii) varying model sizes (1B and 7B), and (iv) varying training batch sizes (2048 and 128). To this end, we resume pretraining OLMo [ 21 ] intermediate checkpoints restoring the optimizer and scheduler states the same way OLMo is pretrained, using the pretraining data of OLMo (Dolma v1.5 [ 43 ] ), except that we inject factual knowledge every 100 training steps by replacing a part of original pretraining batch with the injected knowledge of the Fictional Knowledge dataset. 4 4 4 We use OLMo for the experiments since the intermediate checkpoints, optimizer states, and batch sequence data for pretraining the model are made publicly available. Each injected knowledge is short enough to fit into one pretraining sequence in the batch, and we fill the rest of the sequence with the original sequence in the batch. To investigate the difference in the factual knowledge acquisition dynamics when the models are presented with the knowledge, we inject factual knowledge with three different injection scenarios: duplication , paraphrase , and once . For the duplication injection scenario, we inject the same knowledge 10 times with an interval of 100 training steps. In the paraphrase injection scenario, we inject paraphrased knowledge instead of showing identical sequences, every time it is presented to the model. Lastly, in the once injection scenario, we inject the knowledge only once at the start of the training. After the injection is complete, we continue pretraining as normal. The details for the training setup can be found in § D .
## 4 Results
### 4.1 Factual knowledge acquisition occurs by accumulating the observations of the fact
Figure 2 shows the progress of factual knowledge acquisition of OLMo-7B, by averaging the model’s log probability across the target spans of the probes for each injection scenario, evaluated at each training step. Regardless of the acquisition depths (memorization, semantic generalization, and compositional generalization), the model’s log probability measured on the probes shows an immediate and distinctive increase, after the model is updated with the batch containing the injected knowledge. However, the log probability decreases again, as the knowledge is not presented to the model afterward. This observation directly demonstrates the mechanism of factual knowledge acquisition: LLMs acquire factual knowledge by accumulating micro-acquisitions with subsequent forgetting each time the model encounters the knowledge during pretraining.
Several findings can be further obtained from Figure 2 . First, when the model is updated after seeing the factual knowledge, the most significant improvement in log probability is observed for memorization, followed by semantic generalization, and the least improvement is seen in compositional generalization. Next, however, the gap between memorization and semantic generalization almost disappears in the paraphrase injection scenario. Third, when the model is updated with the duplication injection scenario, the model shows a larger improvement of log probability in all acquisition depths, but also the forgetting is faster, eventually resulting in a similar level of improvement at the end of the training ( t = 2000 t=2000 ) compared to the paraphrase injection scenario.
These patterns are consistent across all pretraining stages of OLMo-7B we investigate (§ E.1 ). Intriguingly, the training dynamics of OLMo-1B early checkpoint (Appendix Figure 8 ) show much more unstable dynamics than those of later checkpoints (Appendix Figure 9 and 10 ) and the early checkpoint of OLMo-7B (Appendix Figure 6 ). The distinctive behavior of the OLMo-1B early checkpoint suggests that pretraining on a certain number of tokens may be required for the model to acquire factual knowledge stably and that such a threshold may be higher for smaller models.
### 4.2 Effects of model scale and pretraining stage on knowledge acquisition dynamics
Next, we measure effectivity (Eq. 2 ) to quantify the improvement of the LLMs’ log probability after being trained with the injected knowledge, averaged across all probes ( q q ) and encounters ( i i ). The results are demonstrated in Figure 3 . The average effectivity is the largest in the Once injection scenario since the effectivity is higher when the model encounters the injected knowledge for the first time, which is further discussed in § H .
In all injection scenarios, there is an improvement in effectivity when the model size is scaled from 1B to 7B (as shown on the right side of Figure 3 ). 5 5 5 For a fair comparison of the effectivity of the 1B and 7B models, the OLMo-1B Mid checkpoint is trained using the same initial learning rate as the OLMo-7B Mid checkpoint (the specific value is provided in Appendix Table 5 ). The measured effectivity for all OLMo-1B checkpoints with the original learning rate is presented in Appendix Figure 11 . On the other hand, surprisingly, the effectivity of fact acquisition does not improve with checkpoints trained with more tokens, as shown on the left side of Figure 3 . This tendency is consistent across all model scales and injection scenarios (see also Appendix Figure 11 ). Moreover, this tendency is not attributed to training the models with a decreased learning rate through learning rate decay, as demonstrated by an additional experiment of training three checkpoints using the same constant learning rate. The results with the constant learning rate show that effectivity does not significantly improve in the checkpoints of later stages of pretraining where more pretraining tokens are seen (§ F ). Therefore, the observation implies that the effectivity of LLMs in acquiring factual knowledge does not significantly improve throughout the progress of pretraining.
While our finding that effectivity remains unchanged for different stages of pretraining may seem contradictory to the widely known observation that the amount of pretraining data is a critical factor in the performance of LLMs [ 23 , 27 ] , we suggest a plausible hypothesis based on further observations in § 4.3 . Specifically, we suggest that the high performance of LLMs trained with larger and more diverse datasets is not primarily due to an emergent ability from the sheer amount of tokens observed during training [ 50 ] , but rather because the model encounters a wider variety of knowledge more times, which allows for the accumulation of log probabilities of more knowledge become high enough to be decoded as outputs of the model. We discuss this hypothesis further in § 4.4 .
Comparing the duplication and paraphrase injection scenarios, the duplication injection scenario naturally shows higher effectivity for memorization. However, the higher effectivity in the duplication injection scenario for semantic generalization and compositional generalization appears to be counterintuitive, as it is widely observed that deduplication of pretraining data is an important factor in improving model performance [ 29 , 52 ] . In the following sections, we will address this question by demonstrating that the models exhibit faster forgetting in generalizing factual knowledge when presented with duplicated texts (§ 4.3 ).
### 4.3 Forgetting in factual knowledge acquisition
#### Training steps and the forgetting of acquired factual knowledge have a power-law relationship
The exponential trend of forgetting has been reported in various aspects of LLM training, including memorization in pretraining [ 46 ] and task performances in continual learning [ 33 , 39 ] . Motivated by this, we investigate whether the exponential trend of forgetting persists in the context of factual knowledge acquisition in LLM pretraining. Figure 4 illustrates the trend of retainability against the training steps past the local acquisition maxima. We find that the trend of ℛ ( p , t ) \mathcal{R}(p,t) against l o g ( t ) log(t) fits a linear function very well ( R 2 > 0.80 R^{2}>0.80 for memorization and semantic generalization, and R 2 > 0.65 R^{2}>0.65 for compositional generalization). This trend is persistent across all acquisition depths, and all training conditions (§ E.4 and § E.5 ). Guided by empirical observations, we model the trend of forgetting using a power-law model in further investigations.
#### How quickly is the acquired factual knowledge lost?
The absolute value of the slope of the fitted lines in Figure 4 can be interpreted as the decay constant ( a a ) of retainability, formally, Δ ℛ ( p , t ) ≈ − a ⋅ log ( t 2 t 1 ) for 0 < t 1 < t 2 < τ , where ℛ ( p , τ ) = 0 and a > 0 . \Delta\mathcal{R}(p,t)\approx-a\cdot\log\left(\frac{t_{2}}{t_{1}}\right)\quad\text{for }0<t_{1}<t_{2}<\tau,\quad\text{where }\mathcal{R}(p,\tau)=0\text{ and }a>0. (4) Thus, the measured decay constant represents how fast (in terms of fraction) the model loses the improvement of log probability. Table 2 shows the decay constants of retainability measured for three OLMo-7B intermediate checkpoints, for duplication and paraphrase injection scenarios.
±
±
±
±
±
±
±
±
±
±
±
±
±
±
±
±
±
±
There are several observations in Table 2 . First, the forgetting in compositional generalization is slower (the decay constant a a is smaller) than in memorization and semantic generalization. Combined with the observations in previous sections, the acquisition of compositional generalization accumulates most slowly but is more robust to forgetting. Second, the forgetting tends to be slower in the paraphrase injection scenario compared to the duplication injection scenario. This finding will be further discussed in § 4.4 , regarding the importance of deduplicating training data. Finally, the decay constants are similar for the two earlier checkpoints but smaller for the late checkpoint in the duplication injection scenario. We demonstrate that this is due to the reduced learning rate from learning rate scheduling (Appendix Table 5 ), as the decay constants show no decrease for the later checkpoint when each checkpoint is trained with the same constant learning rate (Appendix Table 9 ).
#### Pretraining with a larger batch size helps LLMs acquire more knowledge
It is a common practice to pretrain LLMs with a very large batch size to leverage parallel computing [ 13 , 21 , 25 , 30 , 49 ] . However, the effects of increasing training batch size in terms of the LLMs’ acquisition of factual knowledge remain underexplored. In this section, we examine whether pretraining LLMs with a larger batch size is advantageous regarding factual knowledge acquisition. Specifically, we continue training LLMs with a batch size reduced by a factor of 16 compared to the original pretraining batch size, i.e. , from 2048 to 128.
Figure 5 compares the forgetting dynamics of OLMo-7B mid checkpoint between pretraining and training with the reduced batch size. The results have several implications for the advantage of pretraining LLMs with a larger batch size. First, comparing Figure 3 and Appendix Figure 21 , LLMs trained with the smaller batch size show higher effectivity. However, the decay constant tends to be higher, comparing the numbers in Table 2 and Appendix Table 10 . Furthermore, the anticipated x-intercept is significantly decreased by dozens of times, comparing Appendix Table 6 and 11 . This implies that the models trained with smaller batch sizes have shorter learnability threshold , the point such that an LLM cannot learn the knowledge presented with intervals longer than that threshold, which we discuss in detail in the following section (§ 4.4 ). In other words, when an LLM is trained with a smaller batch size, factual knowledge should be presented more often to the model so as not to be forgotten and the set of learnable knowledge is reduced. Second, accelerated forgetting with a smaller batch size is more pronounced for compositional generalization compared to memorization and semantic generalization. In brief, the results suggest that pretraining with a small batch size reduces the set of learnable knowledge due to accelerated forgetting, and leads to worse compositional generalization performance of learned factual knowledge.
### 4.4 Implications for LLM pretraining
#### Why is popularity important for factual knowledge acquisition?
The estimated x-intercepts in Figure 5 represent the number of additional training tokens that would lead to the complete loss of the factual knowledge acquired by training. 6 6 6 The exact values of the estimated x-intercepts can be found in Appendix Table 6 . Hence, if a given factual knowledge in the pretraining dataset is in the long-tail and the knowledge is presented to the model with an interval longer than a certain threshold, such knowledge will be impossible to be decoded as the top-k generation of the model, or learned , regardless of the duration of the pretraining. 7 7 7 This theoretical threshold may not be equal to the estimated x-intercepts presented in Figure 5 , as we estimate the threshold based on the controlled experiment of injecting factual knowledge. In addition, the actual learnability threshold is likely to vary for different types of factual knowledge due to several factors, such as the number of similar/related facts or temporal conflicts in the pretraining data. This implies that there is a learnability threshold , a threshold of the interval where the model fails to acquire knowledge of which its encounter interval is longer than the threshold. Most well-known facts are likely to be presented to the model with an interval of the training steps shorter than this learnability threshold . In such a case, the model will accumulate the increased log probability of the knowledge upon each encounter of the knowledge as the pretraining progresses, and at some point, the accumulated log probability of the knowledge will be high enough to generate the knowledge as the decoding output of the model [ 41 ] . Moreover, LLMs will accumulate the log probability faster for more popular knowledge, and thus the acquisition of such knowledge will be reflected in the model’s top-k output sequence generation in a relatively earlier pretraining stage, as demonstrated in [ 8 ] .
In summary, we hypothesize that the popularity of the knowledge in the pretraining data influences how quickly this knowledge begins to be ‘revealed’ in the generated sequences during pretraining, except for the knowledge in the long-tail whose low popularity makes the encounter interval longer than the learnability threshold . Also, as briefly mentioned in § 4.2 , we hypothesize that the reason why larger and more diverse pretraining data helps the model performance is that the model can acquire a broader range of factual knowledge (more knowledge will be presented with an interval shorter than the learnability threshold ) since the skewness of the distribution of factual knowledge popularity is likely to be mitigated as the data becomes larger and more diverse.
#### Why does deduplication enhance model performance?