Emergent properties with repeated examples — part 3/4
Cleaned-text code points [31162, 46315). Bibliography omitted here; source.md includes it.
We experimented with three curation strategies for the repeated set: log-uniform and uniform distributions of operands and input, shown to be beneficial by Charton, “easy sets” featuring small input and outcomes, and “heavy tail sets” featuring large GCD. For each setting, we trained 5 5 models with four “good choices” of S S and p p (Table 4 ), a data budget of 100 100 M and training budget of 600 600 M.
These strategies do not achieve better results than the baseline two-set training with a random repeated set. A slight improvement is observed when repeated samples are selected from a log-uniform input and GCD (for which Charton (2024) reports 91 91 correct GCD for single-set training). Overall, we find that repeated set curation has, at best, a marginal impact on performance. This is a counter-intuitive but significant result.
Shifting the repeated sample. In the GCD experiments, with p = 0.25 p=0.25 and S = 50,000 S=50,000 , repeated examples are seen 3000 3000 times for a training budget of 600 600 M. Since this large repetition may lead to overfit, we experimented with “shifting samples”: replacing the repeated examples after a k k repetitions. In Appendix C.3 , we experiment with k k from 10 10 to 100 100 , and observe that this has no impact on model performance.
Batching matters. All models in this paper are trained on mini-batches of 64 64 examples. In two-set training, batches mix examples from the repeated and the large set. We experimented with batches that only use samples from one set at a time. For instance, when training with p = 0.25 p=0.25 , 25 % 25\% of batches would use repeated examples only. For both GCD and modular multiplication, we observe that models trained on batches from one sample only fail to learn. This indicates that mixing repeated and non-repeated examples is required for two-set training to happen (see also Appendix C.2 ).
From two to many-set training. Two-set training effectively makes the training sample non identically-distributed: examples from the repeated sample occur with a larger probability. We can generalize this method by introducing a probability distribution P P on the training examples, such that for any i ≤ N i\leq N , P ( i ) P(i) is the probability that the i i -th example is selected during training. In two-set training, P P is a step function distribution with two values: p / S p/S and ( 1 − p ) / ( N − S ) (1-p)/(N-S) , we now replace it with a discrete exponential distribution P ( i ) ∼ β e − β i / N , P(i)\sim\beta e^{-\beta i/N}, with β > 0 \beta>0 , suitably normalized. Table 5 presents the performance of models trained on the GCD problem with such “continuous” data distributions, indicating that our observations on two-set training generalize to such data sampling techniques. More details, and results on modular multiplication, can be found in Appendix C.4 . These results suggests that our observations on two-set training can be extended to a wider class of methods, that use non-uniform sampling over a randomly ordered training set.
## 7 Discussion
Our findings indicate that repetition, and possibly memorization, fosters learning. They suggest that models should be trained on datasets of repeated, but not necessarily curated examples, and that amplifying a randomly chosen subset of the training data may bring additional learning benefits. Two-set training is easy to implement, and applicable to a large variety of situations. Its extension to smooth distributions allows for finer control over repetition levels in the training sets.
We can contemplate how our observations carry over to large language models (LLM) trained on natural data. An important factor is the presence of repetition in the training data. We believe that pre-training corpora – text scraped from the internet, public code repositories – feature many repeated examples (quotes, copied passages, duplicated functions), and that the phenomena we describe are already at work in LLMs during the pre-training stage. Fine-tuning corpora, on the other hand, are often curated and feature less repetition. We believe two-set training, and associated methods, may prove beneficial for fine-tuning LLMs.
Our observations on two-set training are thought-provoking and deserve further study. The fact that the repeated set can be chosen at random, and that curating repeated examples bring little to no improvement in performance suggest that what matters, here, is seeing the exact same example several times. The particulars of the example, its informational value, interest, whether it is typical or exceptional, seem to have little impact. This is all the more curious as, even in the two-set setting, repetition occurs at a very low frequency. In the two-set GCD experiments, repeated examples were seen 3000 3000 times over a training budget of 600 600 million: once every 200,000 examples on average. The frequency is even lower for modular multiplication. Besides, the repeated examples are mixed with non-repeated examples into mini-batches, and our experiments indicate that this mixing is required for the two-set effect to appear. Still, this very infrequent repetition, and mini-batch mixing, brings a significant boost in model performance.
This raises several tantalizing questions: how does the transformer “figure” that a given example, lost in a mini-batch, has been seen, several hundred thousand examples before? Our research suggests that there exists a qualitative difference between “déjà vu” and “jamais vu” examples – data points the model has already seen, or never seen. How do transformers, and perhaps other architectures, identify, and then process, “déjà vu” examples? To our knowledge, this aspect was overlooked in many prior works on model interpretation. We believe our findings point to a number of interesting questions about memorization in language models. This is an intriguing subject for further study.
## Appendix
## Appendix A Learning dynamics and overfitting in math transformers
To gain some understanding on the relation between repetition and overfitting, we delve deeper into the typical training dynamics in our mathematics problems with transformers. We study learning curves to shed light on the interplay between overfitting and relative size of data versus training budget. We focus on learning to compute the eigenvalues of 5 × 5 5\times 5 symmetric matrices ( Charton, 2022 ) for illustrative purposes, but the observed dynamics are common to all our problems (e.g. see Figure 2 (Right)). Figure 6 illustrates training of 10 10 models on a data budget of 200,000 200,000 samples, with increasing training budget (up to 30 30 million) resulting in increased repetition. Learning curves exhibit a step shape , which gives rise to three phases: • Initial phase: training and test loss decrease (up to TB of about 2 2 M), accuracy remains low.
• Learning phase: training and test loss drop suddenly, accuracy increases steeply from a few percents to 90 % 90\% (for the next 1 1 - 3 3 M of TB). This phase is absent for those models that overfit too early (dark curves in Figure 6 ).
• Saturation phase: the model learns the remaining accuracy.
Recall that we say that overfitting occurs when the test loss starts increasing while training loss continues decreasing. Here we see that for all models there is an initial flattening of test loss after ∼ 2 \sim 2 M training examples (about 10 10 repetitions of the data budget 4 4 4 Our runs on a range of small data budgets (up to 250 250 thousand) show similar initial step shape of test loss at 10 10 - 12 12 repetitions. ). Then, some models start overfitting already during the initial phase (the 6 6 dark colored curves in Figure 6 ), and for those the learning phase never happens and accuracy plateaus at about 2 % 2\% . On the other hand, for the other 4 4 models the learning phase begins before overfitting sets in (the pale colored curves in Figure 6 ), the task is learned in full (to over 95 % 95\% accuracy), and overfitting is delayed until after that point. Eventually, these four models start to overfit at training budgets of about 10 10 million examples, and a slight drop in accuracy is observed in some models (but not all), after 15 15 million examples ( 75 75 epochs on the training set). We observe similar effects for different data budgets.
These experiments illustrate the relation between overfitting and learning. Once a model overfits, it stops learning, accuracy saturates, and eventually sometimes decreases. On the other hand, once a model trained on limited data starts learning, overfitting is delayed by many more epochs.
## Appendix B Additional figures and experiments for modular multiplication
Figure 7 provides learning curves (test error) for modular multiplication, illustrating step-like learning, which motivates us to use the number of models achieving 50 + % 50+\% resp. 99 % 99\% accuracy as our performance metric.
Figures 8 and 9 as well as Tables 6 and 7 provide additional results for modular multiplication in the two-set setting.
## Appendix C Ablation results
### C.1 Curating the small sample
In two-set training, the examples in the small set are chosen at random from the overall training set. In this section, we experiment with curating the small set, by selecting the examples that will be repeated during training. As in curriculum learning, selecting easier or more informative examples may help improve performance. Perhaps when increasing the frequency of our small random set, what really matters is the repetition of some particular examples, rather than all? The GCD problem is particularly well suited for this type of investigation, due to the inverse polynomial distribution of outcomes ( Prob ( GCD = k ) ∼ 1 k 2 \text{Prob}(\text{GCD}=k)\sim\frac{1}{k^{2}} ). On this problem, we leverage the findings of Charton (2024) , who observes that ∞ \infty -models trained from log-uniform distributions of inputs and/or outcomes ( Prob ( GCD = k ) ∼ 1 k \text{Prob}(\text{GCD}=k)\sim\frac{1}{k} ) learn better.
We experiment with four settings of | S | |S| and p p , which correspond to the best results in our previous experiments (Section 5 ): 50,000 50,000 and 150,000 150,000 with p = 0.25 p=0.25 and 150,000 150,000 and 500,000 500,000 with p = 0.5 p=0.5 , for a data budget of 100 100 million and training budget of 600 600 M. For every setting, we train 5 5 models with the following three choices for S S : log-uniform inputs, uniform GCD or both log-uniform inputs and GCD. We use two-set training with a random small set S S as our baseline. Table 8 shows that the performance of models using log-uniform inputs, or uniform GCD, is slightly lower than the baseline. Models trained on log-uniform inputs and GCD achieve slightly better performance, but we note that models trained on the small set distribution only ( p = 1 p=1 ) would predict 91 91 GCD. On these three distributions, curating the small set proves disappointing.
In curriculum learning fashion, we also experiment with small sets S S of a few “easier cases”: small inputs (from 1 1 to 1000 1000 ), GCD that are products of 2 2 and 5 5 , the easiest to learn in base 1000 1000 ( Charton, 2024 ) , and GCD between 1 1 and 10 10 (the most common outcomes). We observe that while models trained with small inputs in S S perform on par with the baseline, models trained on “easy GCD” perform slightly worse.
Finally, inspired by arguments that rare tail outcomes might require particular attention for learning ( Dohmatob et al., 2024 ) , we experiment with small sets composed of examples from the tail of the training distribution, namely, large GCD. Charton (2024) observes that these are both harder to learn, and less common in the training set. Specifically, we create S S with examples with GCD larger than k k (for k k ranging from 1 1 to 5 5 ). While experiments achieve the best accuracies compared to the other curation schemes we proposed, and values of k k equal to 2 2 and 3 3 train slightly faster, they remain a little below the baseline both in accuracy and learning speed.
Overall, these experiments suggest that in two-set training, random selection of the small set may be optimal. Selecting a small set of easy cases (GCD multiple of 2 2 and 5 5 ), and examples that are known to help training (log-uniform inputs) does not help, and limiting the small set to edge cases from the tail of the outcome distribution brings no improvement to performance. This is a counter-intuitive, but significant result.
### C.2 Batching in two-set training: mixed batches are needed
In all experiments, during training, the model computes gradients over minibatches of 64 64 examples. In two-set training, minibatches mix examples from the small and large set. We experimented with using “mono-batches” that use samples from one set at a time. For instance, when training with p = 0.25 p=0.25 , 25 % 25\% of minibatches would use examples from the small set (of size S S ) only, and 75 % 75\% would only use those from its complement.
On the GCD problem , we rerun the most successful two-set experiments (Section 5 ) with “mono-batches” for S = 50 S=50 K, 100 100 K and 250 250 K, and p = 0.25 p=0.25 and 0.5 0.5 . For training budgets of 600 600 M and data budget of 100 100 M examples, the models trained on mixed batches predicted 62 62 to 69 69 GCD (Section 5 ). With “mono-batches”, the number of correctly predicted GCD never rises above 15 15 . For modular multiplication , we experimented with the following ( S , p ) (S,p) pairs ( S S in millions): ( 0.5 , 0.1 ) , ( 2.5 , 0.25 ) (0.5,0.1),(2.5,0.25) and ( 10 , 0.5 ) (10,0.5) with data budget 100 100 M and training budget 600 600 M. With these settings, mixed-batch models achieve an average accuracy of 67 % 67\% or more (Section 5 ). With “mono-batches”, none of the models manages to learn (accuracy around 4 % 4\% ). This indicates that mixed batching of samples from each of the two sets plays a central role for the two-set effect .
### C.3 Shifting the small set
In these experiments, we study, in two-set training, the possible impact of overfitting on the small set, by refreshing the small set with fresh examples periodically. This mimics certain aspects of curriculum learning, where the training set is changed over time. On the GCD experiments, with a data budget of 100 100 million, a training budget of 600 600 million, we shift the small set as training proceeds, so that examples in the small set are seen k k times on average. At the beginning of training, the small set is the S S first elements in the train set. After training on k S / p kS/p examples, examples in the small set have been seen k k times, and the small set is shifted to elements S + 1 S+1 to 2 S 2S of the training set.
Table 9 provides performances for two-set training with shift, for different values of p p , S S and k k , for a data budget of 100 100 million, and a training budget of 600 600 million. It is interesting to note that shifting brings no improvement to 2-set training.
### C.4 From two-set to many-set training