How do language models learn facts? Dynamics, curricula and hallucinations — part 4/5

Cleaned-text code points [46983, 62366). Bibliography omitted here; source.md includes it.

• Sequential distribution. The population is divided into n_groups groups and we go through these groups in increasing order n_repeats times. Individuals within each group are uniformly sampled.

2. Biography sampling. For a given individual, we uniformly sample templates from the appropriate pool (training or evaluation) for each attribute. We then randomize the template order and concatenate them to form a single-sequence biography.

#### B.3 Architecture, optimization and metrics

In almost all our experiments, we use the 44 44 M-parameters Transformer architecture of Hoffmann et al. (2022) . It has 8 8 layers, uses a residual stream of dimension 512 512 , with one hidden layer multi-layer perceptrons with 2048 2048 hidden neurons. It has 8 8 heads and each head has keys and values of dimension 64 64 . The only deviation to this architecture is the model size ablation of Figure C (lower right), in which we use a 163 163 M-parameters ( 12 12 layers, 16 16 heads, dimension 896 896 ) and a 400 400 M-parameters ( 12 12 layers, 12 12 heads, dimension 1536 1536 ) architecture. The default hyperparameter configuration is detailed in Table 1 . For most experiments, we perform a sweep over different learning rates values, selecting the best performing one based on final training loss.

The experiments in Figure 2 use 5 5 different seeds. However, given the low variance of the results, we preferred to use compute to explore more diverse hyperparameter configurations rather than performing our analysis on more seeds and ended up using a single seed for the rest of the experiments.

Throughout this study, we use two main metrics. The first one is the attribute loss, which is the sum of cross entropy losses measured on all attribute value tokens, which is then averaged by the number of attribute values in the sequence (always 6 6 ) and by the batch size. Importantly, we don’t average by the number of tokens within each attribute value, so that the comparison to the no-knowledge baseline is easier. The second metric we use is the attribute accuracy, which is the accuracy the model gets when correctly predicting all consecutive attribute value tokens. For example, if the model always predicts correctly all attribute value tokens except the first one, it will get an accuracy of 0. This way, this metric is rather conservative, and that we are sure that for the model to be correct it has to get all the tokens requiring recall abilities (as opposed to completion abilities) right. The no-knowledge baseline is computed as the average entropy of all attribute values from all types, that is the average logarithm of the number of possible attribute values.

### Appendix C Additional analysis of the learning dynamics (Section 2.1 )

#### C.1 The three phases are robust to sensible hyperparameter choices

In Figure C , we verify whether the three phases we identify are robust to sensible parameter changes. We find that they are robust to changes in learning rates, total number of individuals, weight decay values, batch size, model size and to changing the sequence mixing architecture to a recurrent one (we use the Hawk model for that ( De et al., 2024 ) ).

### Appendix D Details of the mechanistic study and additional analyses (Section 2.2 )

#### D.1 Implementation of the attention patching experiment

Our attention patching experiment consists in training a reference model, then restarting training with the same initial parameters but providing the model with the attention patterns the reference model produced when seeing the same samples. Two hyperparameters control this: reference_patching (the number of training steps for which the reference model was trained) and start_patching (the step at which patching begins; before this, the modified model uses its own attention). There are two key configurations: when start_patching and reference_patching are equal, this effectively freezes the model’s attention at a given point; when start_patching is 0, patching occurs throughout training, as presented in the main text.

We implement attention patching through a twin architecture. We initialize the modified model with the initial parameters and the reference model with the desired parameters. Both models process the input sequence layer-wise. At each layer, the reference model generates attention scores and output. The attention scores are sent to the corresponding layer in the modified model (and the output to the next layer), with gradients stopped on the attention scores to prevent training the reference model. The modified layer uses the provided attention pattern instead of its own if patching has begun. While generating all attention scores from the reference model beforehand is possible and potentially simpler code-wise, our layer-wise implementation is more memory-efficient, particularly for long sequences, and many layers and heads. That said, given that memory is not an issue in our experiments given the small size of the models, it would probably have been good enough. We always re-initialize the optimizer state to default values when starting patching. While we found some significant loss increase just after the beginning of patching with this strategy, we also found them to be smaller than without re-initializing them.

The experimental setup that we have described here is richer than the one presented in the main text as we can now vary the beginning of patching. We provide the results of this more extensive analysis in Figure D for completeness. Those results confirm our conclusions from the main text.

#### D.2 Details of the attention pattern analysis

In our analysis, the model’s attention patterns during learning, we examine which tokens the network attends to when processing or predicting specific tokens. This approach is motivated by prior work (discussed in Section 2.2 and visualized in Figure E ) that identified specific attention-based circuits for factual recall tasks, each with distinct signatures observable through attention patterns. We focus on the following circuits and their signatures: • Name tokens grouping circuit. This circuit, present in the model’s first layer, groups the embeddings of name tokens to represent the individual’s full name. This occurs when processing the last name token, leading to high attention on other name tokens in the first layer. Focusing on the last name token is crucial to differentiate this mechanism from others like name completion. The name → \rightarrow name line in the right panel of Figure 3 represents the average attention from the last name token to all other name tokens in the first layer, averaged over name occurrences (typically 6 6 , corresponding to the number of attributes), heads, and 512 512 input sequences. Figure F (left) shows this quantity for all layers.

• Extraction circuit. The final attention layer selects and relays relevant information based on the requested attribute type. We expect high attention to name tokens (but not text tokens) when predicting the first attribute value token, once this circuit is established. Again, focusing on the first token is crucial to isolate this from completion mechanisms. The attribute → \rightarrow text and attribute → \rightarrow name lines in Figure 3 are generated accordingly. Figure F (middle left and middle right) shows the layer-wise breakdown of these quantities.

We do not include the attribute type propagation circuit highlighted in Geva et al. (2023) in this analysis to keep it simple. That said, the relatively high attention to text tokens in the first attention layers when predicting attribute values is consistent with that circuit.

As a separate observation, Figure F (right) reports the evolution of the sharpness of attention patterns in each layer, defined as the normalized entropy of the attention distribution (divided by the logarithm of the number of attendable tokens). Initially uniform, attention patterns become progressively sharper, with a slight increase in sharpness starting around the end of the plateau.

We conclude this section by discussing the limitations of this analysis. It ignores the impact of the values (e.g., attention patterns do not matter when values are equal to 0 0 ), is purely correlational, and relies on simplified mental models of the network’s internal workings. Nevertheless, it enables refining the conclusions drawn from our attention patching intervention.

### Appendix E Additional analysis for the impact of data distribution properties (Section 3 )

#### E.1 Learning curves for different data distributions

We here provide some examples of learning curves when modifying the training distribution. We focus on the 8 8 k training steps, 64 64 k individuals regime as it is one in which we observe large differences between distributions.

#### E.2 Extensive comparison of the performance of different data distributions

In Figure J , we provide a detailed version of Figure 4 (right). We vary the number of training steps and the number of individuals, and report both attribute accuracy and attribute loss. Additionally, we include the celebrities distribution in the comparison. The results stay qualitatively the same as what we reported in the main text.

It should be noted that we have not tried to optimize the warm-up strategy more than through the hyperparameter grid search reported in Section G.4 as it is not the main focus of this paper, and it is possible that even better results could be achieved. For example, we only allow for changing distributions every epoch, with an epoch being defined as seeing all individuals in the training distribution. This implies that for a high number of individuals, tuning is rather coarse. Another potential improvement is to gradually increase the distribution size.

In Figure K and L , we plot the final performance of the model for different hyperparameter choices. In the low training step regime ( 8 8 k steps plot in Figure K ), we find the optimal hyperparameter choice to be within our grid search. However, in the large number of individuals scenario, it seems outside our range (cf. 128 128 k individuals plot of Figure L ) and in particular suggests long warmup phase with a high number of individuals. This leads us to wonder whether the benefits we observe from the warming-up could be achieved by just training on less individuals. To that extent, we run an additional run with a uniform distribution on the individual, with a population size of 83 83 k individuals. The model reaches an attribute accuracy of 97.72 97.72 %, which would become 63.26 63.26 % when evaluating on 128 128 k individuals. On the other side, the model trained with warm-ups reaches 94.92 94.92 % accuracy. This refinement of our results strengthens our confidence in the benefits of warm-ups, particularly as we have not optimized the strategy beyond a relatively small hyperparameter grid search.

### Appendix F Details of the fine-tuning analysis and additional experiments (Section 4 )

#### F.1 Experimental details

In our fine-tuning experiments, we generate n_individuals_finetune new individuals and use their biographies for fine-tuning. By default, we use fine-tune a model trained for 16 16 k steps on 64 64 k individuals, with individuals sampled uniformly (i.e., the default parameters in Table 1 ). In the experiments with replay, we use sampling according to the “celebrities” distribution described in Section B.2 , using a weight of weight_replay_finetune for the 64 64 k pre-training individuals, and a weight of 1 1 for the fine-tuning individuals. We use a constant learning rate of 3 ⋅ 10 − 5 3\cdot 10^{-5} for fine-tuning, the rest of the optimizer staying at it is during pre-training.

#### F.2 Hallucinations

As briefly mentioned in Section 4 , our framework offers a simple way to monitor fact-conflicting hallucinations (as defined in Zhang et al. (2023) ) during training. These hallucinations are characterized by overconfidence in predicting facts absent (or rare) from the training data. To assess this, we evaluate the model on held-out individuals not present in the training distribution. A hallucinating model would predict incorrect attribute values with high confidence. For these held-out individuals, the attribute loss must exceed the no knowledge baseline, with higher values indicating overconfidence (i.e., hallucinations). Figure M shows that hallucinations appear shortly after the plateau phase. However, the model remains less confident in these hallucinations than in its grounded predictions for seen individuals (see middle right and right panels). The higher probability mass on the most likely prediction and lower entropy of the predictive distribution for seen individuals suggest that hallucinations can be detected to some extent. We found these observations to be robust to change in the input distribution. For example, progressively increasing the population size did not significantly affect these metrics.

This experiment is only a preliminary exploration of hallucinations within our setup. Future work could, for example, investigate how the confidence gap evolves when increasing the number of individuals. For instance, if we consider the network’s associative memory to be a linear key-value database, increasing the number of individuals while holding the number of hidden neurons constant would bring key representations (individual names) closer, making it harder to distinguish unseen keys, thereby reducing the confidence gap and increasing hallucinations. Understanding which training individuals indirectly inform predictions for unseen individuals and what is the underlying similarity metric is another intriguing direction. Finally, on the more practical side, this framework could serve as a test-bed for methods aiming at detecting or mitigating hallucinations, as successful general-purpose methods should also be performing well in this simplified setting. We leave these investigations to future work.

#### F.3 Additional analysis for fine-tuning

In the main text, we have presented results for two sets of fine-tuning experiments, one with replay and one without replay. In Figure 5 , we have reported fine-tuning dynamics in a fine-tuning loss vs. pre-training loss reference frame. We here visualize the evolution of the model performance as a function of time. Importantly, these plots make it easier to remark that the drop in model performance on pre-training data occurs very early during fine-tuning (on the order of a hundred steps), and that the increase in performance on fine-tuning data is happening on a longer time-scale (see Figure N ). Additionally, the accuracy plots show that the increase in loss to values close to no-knowledge baseline does not necessarily mean that all pre-training knowledge is erased as some knowledge (i.e. non-zero accuracy) about the pre-training data remains.

In Figure O , we provide the evolution of (some of) the attention scores for one of the fine-tuning experiment with no replay ( 4 4 k individuals), focusing on the metrics that we have introduced in Section D.2 as they are most relevant to our analysis. We find attention scores to be particularly stable, which enables us to conclude that the performance drop during fine-tuning is not strongly linked to changes in attention patterns.