How do language models learn facts? Dynamics, curricula and hallucinations — part 5/5

Cleaned-text code points [62366, 72551). Bibliography omitted here; source.md includes it.

We complement our analysis with an experiment in which we fine-tune the model on rare, but seen during pre-training, individuals. We do so using our celebrities distribution, setting the total number of individuals to 128 128 k, n_celebrities to 8 8 k and weight_celebrities to 8. After training on 16 16 k training steps, the model confidently and correctly predicts the attribute value of the celebrities but gets around 50 50 % accuracy on non-celebrities, which are the majority of the population. Overall, we find that fine-tuning on existing individuals does not lead to fast performance drop fine-tuning on new individuals has (Figure P ). We are able to add a significant amount of knowledge in the model’s weights, as the accuracy on the 120 120 k non-celebrities goes from 50 50 % to close to 70 70 %. These results supporting that fine-tuning on existing individuals is net-positive are confirmed by our alternating training distribution experiment of Figure V .

#### F.4 Reproducing fine-tuning behavior on a toy associative memory problem

In this section, we investigate whether the fine-tuning behavior observed in our language model experiments can be attributed to changes in their feed-forward associative memories. To this end, we train a single-hidden-layer multi-layer perceptron on a synthetic heterogeneous associative recall task, analogous to retrieving attribute values (e.g., university, major) given an individual’s name.

Our network has 256 256 hidden neurons with ReLU activation. Both keys (names) and values (attributes) are represented by 64 64 -dimensional random embeddings drawn from a normal distribution and then projected to the L2 unit sphere. The model learns to map keys to one of 30 30 possible values, effectively performing 30 30 -way classification. It is learned with the cross-entropy loss and the AdamW optimizer with a cosine learning rate schedule (initial learning rate 0.03 0.03 , 16 ​ k 16k steps, batch size 128 128 , weight decay 0.01 0.01 ).

We pre-train the model on 8192 8192 key-value pairs and then fine-tune it with a constant learning rate of 0.001 0.001 for 4000 4000 steps. Fine-tuning incorporates a variable number of new key-value pairs and optionally includes replay of pre-training data. The replay weight controls the relative sampling probability of pre-training versus fine-tuning examples (e.g., a weight of 2 2 with 2 2 pre-training and 1 1 fine-tuning examples results in sampling probabilities of 2 / 5 2/5 , 2 / 5 2/5 , and 1 / 5 1/5 ).

Mirroring Section 4 , we conduct two experiments: 1. Fine-tuning without replay, while varying the number of new samples (left panel of Figure Q , top row of Figure R ).

2. Fine-tuning on 128 128 new samples and varying replay weight (right panel of Figure Q , bottom row of Figure R ).

We also experimented with fine-tuning on old key-value pairs, as in Figure P , and once again observed qualitatively similar behaviors. Overall, these results provide evidence for the hypothesis that mostly feed-forward associative memories are altered during the incorporation of new knowledge through fine-tuning.

#### F.5 Experiments with regular changes in training distribution

To complement our fine-tuning analysis, we consider the setup in which the training distribution is regularly changed, either to a new group of individual every change (this corresponds to varying number_groups and fixing number_repeats to 1 1 ) or alternating between two groups of individuals (this corresponds to varying number_repeats and fixing number_groups to 2 2 ). While this kind of experiment is further away from the current training paradigms of large language models and is more akin to a continual learning setup (e.g., Kirkpatrick et al. (2017) ), it enables us to monitor the plasticity of the network, that is how fast it learns new knowledge and forget about existing one, in a dynamic way. Overall, we find that our fine-tuning findings are robust across different stages of learning and these new results enable us to sometimes refine our conclusions.

We observe a few interesting patterns. First, it becomes increasingly easier to learn a new distribution over time (Figure S ), which is likely induced by the improvement of the attention pattern quality over the course of learning (Figure W ), until the cosine learning rate scheduler eventually damps down weight changes. At the same time, forgetting, measured in log loss differences, becomes more important and occurs much faster than learning, confirming the results of Figure 5 . However, changing distributions multiple times during early learning dynamics when the learning rate is relatively high reduce this plasticity increase, at it partially impairs future ability of the network to learn new groups (c.f. performance drop from Figure S middle to Figure U , or Figure T ). For small learning rates (i.e. at the end of training), performance on old data sometimes get improved after some initial performance drop, without it being replayed (Figure W ), an intriguing phenomenon what we have partially observed during fine-tuning (Figure N , top row). Additionally, for very small learning rates, learning eventually becomes harder and forgetting is not catastrophic, leading to the model remembering more about the second to last group than the last one (see e.g., the 8 groups plot in Figure S ). Still, too many distribution changes to unknown distributions when learning rates are small ultimately removes any knowledge from the network (e.g. Figure U ).

X

### Appendix G Hyperparameter configurations

#### G.1 Default configuration

#### G.2 Hyperparameters for Section 2.1

#### G.3 Hyperparameters for Section 2.2

#### G.4 Hyperparameters for Section 3

Inverse power law distribution (Figure 4 , G and J ) n_individuals [ 4 ​ k , 8 ​ k , 16 ​ k , 32 ​ k , 64 ​ k , 128 ​ k , 256 ​ k ] [4\mathrm{k},8\mathrm{k},16\mathrm{k},32\mathrm{k},64\mathrm{k},128\mathrm{k},256\mathrm{k}] training_steps [ 8 ​ k , 16 ​ k , 32 ​ k ] [8\mathrm{k},16\mathrm{k},32\mathrm{k}] indiv_dist_train Inverse power law - alpha [ 0 , 0.2 , 0.4 , 0.6 , 0.8 , 1 ] [0,0.2,0.4,0.6,0.8,1] Compute time: 1584 1584 hours (train) and 521 521 hours (eval).

Celebrities distribution (vary number of individuals) (Figure H and J ) n_individuals [ 4 ​ k , 8 ​ k , 16 ​ k , 32 ​ k , 64 ​ k , 128 ​ k , 256 ​ k ] [4\mathrm{k},8\mathrm{k},16\mathrm{k},32\mathrm{k},64\mathrm{k},128\mathrm{k},256\mathrm{k}] indiv_dist_train Celebrities - n_celebrities [ 4 ​ k , 16 ​ k , 64 ​ k ] [4\mathrm{k},16\mathrm{k},64\mathrm{k}] - weight_celebrities [ 2 , 4 , 8 , 16 ] [2,4,8,16] Compute time: 1128 1128 hours (train) and 768 768 hours (eval).

Celebrities distribution (vary number of steps) (Figure H and J ) training_steps [ 4 ​ k , 8 ​ k , 16 ​ k , 32 ​ k , 64 ​ k , 128 ​ k , 256 ​ k ] [4\mathrm{k},8\mathrm{k},16\mathrm{k},32\mathrm{k},64\mathrm{k},128\mathrm{k},256\mathrm{k}] indiv_dist_train Celebrities - n_celebrities [ 4 ​ k , 16 ​ k , 64 ​ k ] [4\mathrm{k},16\mathrm{k},64\mathrm{k}] - weight_celebrities [ 2 , 4 , 8 , 16 ] [2,4,8,16] Compute time: 1512 1512 hours (train) and 541 541 hours (eval).

Warm-up distribution (vary number of individuals) (Figure 4 , I , J and L ) n_individuals [ 4 ​ k , 8 ​ k , 16 ​ k , 32 ​ k , 64 ​ k , 128 ​ k , 256 ​ k ] [4\mathrm{k},8\mathrm{k},16\mathrm{k},32\mathrm{k},64\mathrm{k},128\mathrm{k},256\mathrm{k}] indiv_dist_train Warm-up - indiv_warmup [ 2 ​ k , 4 ​ k , 8 ​ k , 16 ​ k , 32 ​ k ] [2\mathrm{k},4\mathrm{k},8\mathrm{k},16\mathrm{k},32\mathrm{k}] - epochs_warmup [ 1 , 2 , 4 , 8 ] [1,2,4,8] Compute time: 1944 1944 hours (train) and 1152 1152 hours (eval).

#### G.5 Hyperparameters for Section 4

Hallucinations (Figure 5 and M ) lr 4 ⋅ 10 − 4 4\cdot 10^{-4} Compute time: 3 3 hours (train) and 5 5 hours (eval).

Fine-tuning without replay (Figure 5 , N and O ) n_individuals_finetune [ 1 ​ k , 2 ​ k , 4 ​ k , 8 ​ k , 16 ​ k ] [1\mathrm{k},2\mathrm{k},4\mathrm{k},8\mathrm{k},16\mathrm{k}] lr_finetune 3 ⋅ 10 − 5 3\cdot 10^{-5} Compute time: 17 17 hours (train) and 95 95 hours (eval).

Fine-tuning with replay (Figure 5 and N ) n_individuals_finetune 4 ​ k 4\mathrm{k} weight_replay_finetune [ 2 , 1 , 1 / 2 , 1 / 4 , 1 / 8 , 1 / 16 , 1 / 32 ] [2,1,1/2,1/4,1/8,1/16,1/32] lr_finetune 3 ⋅ 10 − 5 3\cdot 10^{-5} Compute time: 26 26 hours (train) and 174 174 hours (eval).

Fine-tuning on celebrities (Figure P ) n_individuals 128 ​ k 128\mathrm{k} training_steps 16 ​ k 16\mathrm{k} indiv_dist_train Celebrities - n_celebrities 8 ​ k 8\mathrm{k} - weight_celebrities 8 8 lr_finetune 3 ⋅ 10 − 5 3\cdot 10^{-5} Compute time: 6 6 hours (train) and 48 48 hours (eval).

Sequential learning with new groups (Figure S , T (and U )) n_individuals 64 64 k ( 256 256 k) training_steps 16 16 k ( 64 64 k) lr_scheduler [ [ cosine, constant ] ] (cosine) indiv_dist_train Sequential - n_groups [ 4 , 8 , 16 ] [4,8,16] ( 32 32 ) - n_repeats 1 1 Compute time: 26 26 ( + 13 +13 ) hours (train) and 78 78 ( + 432 +432 ) hours (eval).

Sequential learning with alternating groups (Figure V ) indiv_dist_train Sequential - n_groups 2 2 - n_repeats [ 2 , 4 , 8 ] [2,4,8] Compute time: 10 10 hours (train) and 39 39 hours (eval).

XX

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .