How do language models learn facts? Dynamics, curricula and hallucinations — part 1/5

Cleaned-text code points [0, 15140). Bibliography omitted here; source.md includes it.

\reportnumber 001

# How do language models learn facts? Dynamics, curricula and hallucinations

###### Abstract

Large language models accumulate vast knowledge during pre-training, yet the dynamics governing this acquisition remain poorly understood. This work investigates the learning dynamics of language models on a synthetic factual recall task, uncovering three key findings: First, language models learn in three phases, exhibiting a performance plateau before acquiring precise factual knowledge. Mechanistically, this plateau coincides with the formation of attention-based circuits that support recall. Second, the training data distribution significantly impacts learning dynamics, as imbalanced distributions lead to shorter plateaus. Finally, hallucinations emerge simultaneously with knowledge, and integrating new knowledge into the model through fine-tuning is challenging, as it quickly corrupts its existing parametric memories. Our results emphasize the importance of data distribution in knowledge acquisition and suggest novel data scheduling strategies to accelerate neural network training.

Large language models offer a powerful and intuitive interface to access the vast amounts of knowledge on which they were trained. The learning process acts as a lossy compression algorithm turning training data into neural network parameters, thereby implicitly determining what information is preserved within the final model. The extent and nature of this information loss depend on numerous factors, including architecture, training objective, and data distribution, dependencies that remain poorly understood. Deciphering the mechanisms underlying knowledge compression is increasingly important as these models are becoming our main gateway to human knowledge. Our work addresses this challenge by systematically investigating how language models acquire factual knowledge.

Inspired by a long history of work in neuroscience focusing on associative recall to study ( Pavlov, 1927 ) and model ( Hopfield, 1982 ) intelligence, we focus on a synthetic factual recall task to study in depth the learning dynamics of language models. Our contributions are: – We adapt the synthetic task of Allen-Zhu and Li (2023) , which generates artificial biographies for testing factual recall, to make it suitable for studying the formation of associative memories (Section 1 ).

– We find that language models learn in multiple phases on this task. They first learn overall distribution statistics, then their performance plateaus, and, finally, they acquire individual-specific knowledge (Section 2.1 ). Leveraging a novel attention patching technique, we demonstrate that attention-based recall circuits develop during this plateau (Section 2.2 ).

– We analyze the impact of data distribution properties on these dynamics, revealing that imbalanced distributions accelerate the transition through the intermediate plateau phase (Section 3 ) but lead to overfitting. We further demonstrate that data scheduling strategies can exploit this accelerated transition while mitigating overfitting, providing a rare example in which data curricula benefit (self-)supervised learning.

– We highlight the ineffectiveness of fine-tuning to incorporate new knowledge in the model (Section 4 ). This stems from two related factors: First, models hallucinate (overconfident predictions on unseen individuals) as soon as they acquire individual-specific knowledge. Second, associative memories stored in feed-forward layers are rapidly corrupted when training on new individuals.

### 1 An experimental setup to track knowledge over the course of learning

This study investigates the learning dynamics underlying factual recall and knowledge acquisition in language models. This presents two main methodological challenges: First, we need to isolate the model’s knowledge from other abilities. Second, we want to evaluate the models over the course of learning, which requires computationally efficient measurement techniques. Following Allen-Zhu and Li (2023) , we train language models on synthetic biographies that feature key properties of datasets used to train large language models. By carefully designing these synthetic biographies, we can attribute the model’s ability to predict specific tokens solely to its acquired knowledge and efficiently measure its knowledge through its performance on these tokens. In this section, we first define knowledge and contrast it with memory, then describe our synthetic dataset and introduce the metrics we use to track knowledge during training, and finally, describe training details.

#### 1.1 Knowledge, and how it differs from memory

For our analysis, we define knowledge to be the information a model has internally stored about its training data, abstracted from the specific form in which it was encountered. It is important to distinguish it from memorization. While knowledge involves accessing and applying information flexibly across different contexts, memorization is tied to particular training instances. In this view, knowledge can be understood as a form of memorization in a latent semantic space. For instance, knowing that Paris is France’s capital represents knowledge, while remembering the exact sentence ‘Paris is the capital of France’ would constitute memorization. From a practical perspective, this distinction carries significant implications. Memorization can be problematic due to potential data leakage ( Nasr et al., 2023 , e.g.,) , while knowledge is a more desirable property that enables models to ground their outputs in factual reality, making their responses both accurate and generalizable.

#### 1.2 Synthetic biographies as a framework for studying knowledge

An important methodological challenge in our study is to isolate knowledge from other model capabilities. To address this, we adapt the synthetic biography dataset of Allen-Zhu and Li (2023) , which offers several appealing properties. First, all facts in the dataset are atomic, meaning no fact can be derived from others, thus allowing us to disambiguate knowledge recall from other abilities like reasoning. Second, it is fully synthetic, which allows for precise control of the data distributional properties, while employing natural language with realistic statistics. For example, sufficient textual variation is crucial for enabling genuine knowledge acquisition rather than mere memorization ( Allen-Zhu and Li, 2023 ) , cf. Figure A in the appendix. Third, and most importantly, previous work has established that relatively small language models trained on this dataset ( Allen-Zhu and Li, 2023 ) store and use knowledge in a similar way to large language models ( Geva et al., 2021 ; Meng et al., 2022 ; Geva et al., 2023 ; Nanda et al., 2023b ) .

We summarize the data generation process in Figure 1 . The dataset contains a population of N N randomly sampled individuals, each with a unique name and six attributes: birthdate, birthplace, university, major, company, and location. To generate a biography, we first sample an individual from the population. For each attribute, we then sample a template from a finite pool, fill it with the individual’s information, and concatenate the resulting sentences in random order to form a complete biography. It should be noted that to make the dataset suitable for our study, our template generation and manipulation differ from the original setup. We detail these modifications in the next section and provide additional details in Appendix B .

#### 1.3 Measuring knowledge at scale

Measuring knowledge in language models typically relies on factual question-answering tasks ( Lin et al., 2021 ) . However, this approach is computationally intractable in our setup, as it requires repeatedly fine-tuning models to handle question-answer prompt formats throughout the training process. Our framework offers a simple solution to this challenge: by consistently placing information about the individual and attribute type before the attribute value, we transform each attribute value prediction into a factual recall task. This lets us quantify knowledge via the model’s loss and accuracy on predicting attribute value tokens. We denote them as attribute loss and attribute accuracy . Formal definitions are provided in Appendix B . The attribute accuracy metric is arguably more intuitive, as it serves as a proxy for the accuracy with which the model, once fine-tuned, would correctly answer recall questions. However, we focus most of our analysis on the attribute loss due to its greater interpretive power and as it generally exhibits more consistent progression across various scales ( Schaeffer et al., 2024 ) .

Two important considerations remain to be solved: how to distinguish knowledge from memorization, and what is a meaningful baseline for the attribute loss. To address the first point, we evaluate models on unseen biographies of previously encountered individuals. We do so by randomly splitting the template pool into training and evaluation templates for each individual, with this split varying across individuals. This ensures that models are evaluated on entirely new biography formulations, thereby ruling out the possibility that strong performance stems from memorization. For the second point, we introduce a theoretical baseline corresponding to the best possible performance achievable by a model with no individual-specific knowledge. Such a model would only know the overall distribution of attribute values, and its loss would be equal to the entropy of the attribute value distribution. We refer to this value as the no knowledge baseline. Any model performing better than this baseline must necessarily have acquired some knowledge about specific individuals in the training distribution.

#### 1.4 Language models are trained following standard recipes

While we use synthetic data in our experiments, we keep the architecture and the training protocol as close as possible to standard practices. By default, we train an 8-layer decoder-only Transformer ( 44 44 M non-embedding parameters) with the same structure as in Hoffmann et al. (2022) using the AdamW optimizer ( Loshchilov and Hutter, 2019 ) and a cosine learning rate schedule without warm-up. We tune the learning rate in all our experiments. Additional details are provided in Appendix B .

### 2 How language models acquire knowledge during learning

Our findings reveal that language models initially learn generic statistics before acquiring individual-specific knowledge. When individuals appear less frequently, such as when the population size grows, performance can plateau for extended periods between these two phases. Mechanistically, we attribute this plateau to the development of attention-based circuits that enable the recall of knowledge stored within the network’s parameters and modulate the learning speed of the rest of the model.

#### 2.1 The three phases underlying knowledge acquisition

The temporal evolution of the attribute loss (Figure 2 ) exhibits a consistent three-phase pattern: 1. Initial language understanding. Early on during learning, the network learns the overall attribute value distribution. At the end of this phase, it performs like an optimal model without individual-specific knowledge, matching the no knowledge baseline.

2. Performance plateaus at the edge of knowledge acquisition. The model’s performance then plateaus at the no knowledge baseline. This could be both explained from an optimization argument – the network parameters are at a saddle point of the loss landscape – or from a statistical argument – the model needs to observe multiple biographies from the same individual to discern that attribute values are specific to each individual. The plateau length grows almost linearly with the number of individuals (Figure 2 , right panel), supporting the statistical hypothesis.

3. Knowledge emergence. The model finally enters a knowledge acquisition phase, in which it progressively learns associations between individuals and their attributes. Its knowledge accuracy progressively leaves the near-zero regime in this phase: knowledge emerges.

The pattern described above is robust across a range of hyperparameter configurations. Specifically, qualitative results remain consistent when varying learning rate, weight decay, batch size, number of individuals, model size, and even the type of sequence mixing block (attention-based vs. recurrent), cf. Figure C . This is consistent with the findings of Nichani et al. (2024) , who observed a similar pattern in a theoretically tractable version of the task learned by a linear attention layer. These findings suggest that the data structure, particularly the high individual-to-attribute ratio, plays a critical role in driving this multi-phase learning behavior. Gu et al. (2025) found a similar behavior in a setup closely related to ours, and that, below some critical model size, the model does not learn any individual-specific information at all despite extensive training. We additionally discuss connections to existing work studying Transformer learning dynamics through the lens of N N -grams in Appendix A .

#### 2.2 The attention-based circuits supporting recall are created during the loss plateau

Although knowledge retrieval performance remains constant during the plateau phase, we observe the development of attention-based circuits supporting individual-specific attribute recall. We argue that learning is considerably stalled when these circuits are under development, as errors are not properly backpropagated from attribute value tokens to name tokens, explaining the existence of the plateau. The following evidence supports this claim.

Previous work has analyzed how Transformer-based language models solve factual recall tasks similar to the one we are interested in this study. It can be summarized as follows (cf. Figure E for a visual summary). The first attention layers aggregate name tokens together to form a representation of the individual’s name at the position of the last name token. This representation serves as a query for subsequent layers, particularly the multi-layer perceptrons, which effectively act as a key-value database ( Geva et al., 2021 ; Meng et al., 2022 ) . Individual-specific knowledge thus typically appears within the final name token’s residual stream ( Nanda et al., 2023b ; Allen-Zhu and Li, 2023 ; Geva et al., 2023 ) . The final attention layer then selects the relevant information conditioned on the attribute type and makes it available for predicting attribute value tokens when needed ( Geva et al., 2023 ; Nanda et al., 2023b ) . With this last circuit established, attribute value prediction errors directly backpropagate to relevant tokens, here the name tokens. However, without it, errors are spread across irrelevant tokens, hindering learning efficiency. We therefore hypothesize that the plateau phase corresponds to the formation of these attention-based recall circuits, with learning speed directly linked to their development stage.