How Do Large Language Models Acquire Factual Knowledge During Pretraining? — part 4/4

Cleaned-text code points [46158, 55187). Bibliography omitted here; source.md includes it.

To continue training almost similar to the pretraining setup, we use OLMo [ 21 ] , as it provides not only intermediate model checkpoints but also the exact sequence of data instances used for pretraining, the optimizer state, and the learning rate scheduler. Throughout the entire pretraining process, the language model is trained with a language modeling objective.

Except for the batches that include injected knowledge from Fictional Knowledge dataset at specific step intervals, we train OLMo with batches from the Dolma corpus [ 43 ] in the same order which is used in OLMo pretraining. Specifically, we load the training batch that OLMo will be seen at the specific pretraining step, append the injected knowledge from the Fictional Knowledge dataset to the front of each row, and truncate the original rows from the end by the token length of the injected knowledge. This approach creates batches that have the same size as the original pretraining batches, with 2048 rows and a sequence length of 2048, meaning each batch contains 4M tokens. We adopt this method to deviate as little as possible from the original pretraining data distribution.

In the Fictional Knowledge dataset, which consists of 120 descriptions of fictional knowledge, we use the first 1-40 injected knowledge to examine the dynamics of knowledge acquisition in the paraphrase injection scenario which is described in § 3 . The 41-80 injected knowledge is used for the duplication injection scenario, and the 81-120 injected knowledge is used for the once injection scenario.

For each injection scenario, the Fictional Knowledge data are injected into the batch and trained according to the following rules. In the duplication injection scenario, injected knowledge in the Fictional Knowledge dataset is injected into the original pretraining batch, and the language model is trained on this modified batch 10 times every 100 steps. Next, in paraphrase injection scenario, similar to the duplication injection scenario, the model is trained on the modified batches containing Fictional Knowledge every 100 steps for a total of 10 times, however, in this case, paraphrased injected knowledge is used at each injection step. Lastly, in the once injection scenario, the modified batch containing injected knowledge of Fictional Knowledge is shown to the language model just once, after which it continues training on the original batch of Dolma corpus.

After 1000 steps of pretraining following the above rules, an additional 1500 steps of pretraining are conducted using the Dolma corpus for experiments analyzing forgetting dynamics in § 4.3 . The Dolma corpus used at these steps is a corpus that will be viewed starting from the 360,000th step of pretraining the OLMo. This approach ensures consistency in the Dolma corpus across all conditions while guaranteeing that the corpus has not been seen in any previous pretraining processes. Continued pretraining of a total of 2500 steps takes approximately 3 days using 8 80GB A100 GPUs.

To examine the differences in knowledge acquisition dynamics based on model size, we use OLMo-7B and OLMo-1B. For differences based on the number of pretrained tokens, we use intermediate checkpoints at Early (170B) stage (specifically, 177B tokens for 7B and 168B tokens for 1B), Mid (500B) stage (specifically, 500B tokens for 7B and 494B tokens for 1B), and Late (1.5T) stage (1.5T tokens for 7B and 1B). Since the initial checkpoints of OLMo-1B are stored in units of 10000, it is the best choice in the given situation to select the checkpoint trained with the number of tokens closest to 177B. The differences in initial learning rate values for each case based on different model sizes and pretraining stages are recorded in Table 5 below.

## Appendix E Additional Figures for the Pretraining Experiments

### E.1 Training dynamics of other OLMo-7B checkpoints

### E.2 Training dynamics of other OLMo-1B checkpoints

### E.3 Effectivity measurement data for OLMo-1B

### E.4 Forgetting dynamics of OLMo-7B checkpoints

### E.5 Forgetting dynamics of OLMo-1B checkpoints

## Appendix F Experiments for Training Olmo-7B Checkpoints With a Constant Learning Rate

We continue training each OLMo-7B checkpoint with a constant learning rate, to compare the effectivity and retainability of each checkpoint while excluding the impact of different learning rates. Optimizer states are loaded to promote a warm start of continued training. Due to the restriction of computational resources, we reduce the batch size from 2048 to 128 for this experiment. The value of the constant learning rate is obtained by averaging the starting learning rates of three checkpoints. We do not apply learning rate decay for this experiment. All other training conditions not mentioned are identical to the main experiment. The results in Appendix Figure 17 demonstrate that there is no improvement of average effectivity in later checkpoints, although all models are trained with the same learning rate. This supports that the non-increasing effectivity in pretraining progress is not attributed to the learning rate decay. Similarly, there is no decrease in the decay constants for the later checkpoints (Appendix Table 9 ). Note that the figures in § F.1 demonstrate that reducing the batch size does not significantly change the model’s behavior in accumulating log probability during factual knowledge acquisition.

### F.1 Training dynamics for constant learning rate experiments

## Appendix G Forgetting Dynamics of Olmo-7B Trained With a Reduced Batch Size

Similar to F , we train the OLMo-7B intermediate checkpoints with a reduced batch size of 128. However, we set the learning rate for each checkpoint as the initial learning rate (Appendix Table 5 ), as the objective of this experiment is to examine the effect of reduced batch size on the forgetting dynamics. We re-initialize the optimizer state. We observe that this results in unstable dynamics in early steps, but the dynamics are stabilized soon, and do not harm the model’s overall behavior in general (§ G ). Appendix Figure 21 shows the effectivity measurements of OLMo-7B models at different pretraining stages. Similar to the observations in Appendix Figure 17 , the effectivity values are greater compared to the values in the pretraining experiment (Figure 3 ). Appendix Figure 22 and 23 illustrates the forgetting dynamics of OLMo-7B Early (170B) and late (1.5T) checkpoints, respectively. Appendix Table 10 shows the decay constants ( a a ) measured with three different pretraining stages, acquisition depths, and injection scenarios. Note that the slope remains unchanged regardless of whether we set the x-axis to tokens or training steps. Hence, the decay constants in the table can be directly compared to the values presented in Table 2 . Comparing the values of the expected x-intercepts of retainability presented in Appendix Table 11 with Appendix Table 6 , the results demonstrate that the model trained with a smaller batch size has a shorter learnability threshold .

### G.1 Training dynamics for experiments on the forgetting dynamics with a reduced batch size

## Appendix H Effect of the Number of Previous Encounters on Effectivity and Retainability of Factual Knowledge

We measure the average effectivity for each count of injection ( i i ) in duplication and paraphrase injection scenario. In this analysis, we exclude the cases where the log probability at the local acquisition maxima is smaller than the point before the model is trained with the injected knowledge, as such cases can be regarded as failure cases of learning. Appendix Figure 27 , 28 , and 29 display the results for OLMo-7B early , mid , and late checkpoints, respectively. We observe that the effectivity is relatively constant regardless of the number of previous injections of the knowledge. However, we observe that the effectivity is the highest when the model is trained with the injected knowledge for the first time, both in the duplication and paraphrase injection scenarios.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .