Emergent properties with repeated examples — part 2/4

Cleaned-text code points [15405, 31162). Bibliography omitted here; source.md includes it.

On the GCD problem, we consider data budgets of 1 , 5 , 10 , 25 , 50 1,5,10,25,50 and 100 100 M distinct examples, and an “unlimited data” setting, where new examples are generated on the fly and DB ≈ \approx TB 2 2 2 For GCD and modular multiplication, input pairs are uniformly sampled integers from 1 1 to 1 million. In the unlimited data case, this gives rise to infrequent repetitions: over ∼ 1 \sim 1 billion input pairs, our largest data budget, no elements are repeated 3 3 or more times, and about 500 500 thousand are repeated twice. . For each data budget, we train 5 5 models with a training budget of over 1 1 billion examples, and report their average performance (number of correctly predicted GCD), as the TB increases (Figure 2 Left).

For a modest training budget of 30 30 million, the models with the smallest DB ( 1 1 and 5 5 million, 1 1 M and 5 5 M-models henceforth) achieve the best performance ( 20 20 GCD vs 13 13 for all other DB). As TB increases, the 1 1 M-models start overfitting, as shown by the increasing test losses in Figure 2 (Right), and their performance saturates at 21 21 correct GCD. The performance of the 5 5 M models keeps improving to 36 36 GCD, for a TB of 150 150 million examples, then saturates around 38 38 GCD as the models overfit. For TB of 150 150 and 300 300 million examples, the best performing models are the 10 10 M. As training proceeds, they are outperformed by the 25 25 M models, which achieve the best performance for TB from 450 450 million to 1.05 1.05 billion examples (with the 50 50 M-model a close second at 1 1 billion). Throughout training, the models trained on small data budgets learn faster. However, past a certain TB, they overfit their training data, and their performance saturates.

Note. Overfitting is an overloaded term. In this paper, we define it by its empirical consequences: a model overfits when its test loss starts increasing, while the train loss continues to decrease . The relation between learning and overfitting is further studied in Appendix A .

Conversely, models trained with large or unlimited DB perform the worst. For a TB of one billion examples, the 25 25 M-models predict 62 62 GCD on average, and the 50 50 M-models 60 60 . The 100 100 M-models only predict 37 37 GCD and models trained on an unlimited data budget, where all training examples are seen only once, predict 27 27 GCD, way worse than models trained on 25 25 M distinct examples, repeated 42 42 times on average. Summarizing, smaller data budgets and more frequent repetition allow for faster learning, but also for much better performance.

We observe a similar behavior for modular multiplication . For a TB of 600 600 million, we train 5 5 models for small DB, and 25 or 30 for larger DB, to zoom on this interesting region (Table 1 ). Models trained on an unlimited data budget perform at “chance level”: they always predict 0 0 and achieve about 3 % 3\% accuracy. Models trained on data budgets of 100 100 million examples fare little better, and models trained on 10 10 million examples or less overfit and do not learn.

Models trained on DB of 25 25 M and 50 50 M (for an average repetition of 24 24 and 12 12 ) achieve 40 40 and 60 % 60\% accuracy and exhibit a different behavior. On this task, learning happens in sudden steps, separated by flat plateaus (see the empirical learning curves in Figure 7 in Appendix B ), the two last plateaus corresponding to 51 % 51\% and 99 % 99\% accuracy. About 25 % 25\% ( 7 7 out of 25 25 ) of 50 50 M models achieve 99 % 99\% accuracy (i.e. fully learn the task), and almost 90 % 90\% ( 22 22 / 25 25 ) achieve 50 % 50\% (i.e. one learning step away). On this task, learning emerges through repetition . Models trained on smaller data budgets can perform tasks that models trained from large or unlimited data budget cannot learn. To probe whether learning eventually occurs, we trained 19 19 models with unlimited and 100 100 M DB on increased TB of 2 2 billion examples. None of the unlimited DB models could learn modular multiplication, but one 100 100 M model out of 19 19 achieved 99 % 99\% accuracy after 1 1 B TB, and 5 / 19 5/19 after 2 2 B.

Finally, on the eigenvalue problem , Charton (2022) trained models with unlimited data budgets (DB ≈ \approx TB) and observed that whereas 4 4 -layer transformers can learn to compute the eigenvalues of 5 × 5 5\times 5 matrices, deeper models are required for larger problems: 6-layers for 8 × 8 8\times 8 matrices, 8 8 for 10 × 10 10\times 10 and 12 12 layers for 12 × 12 12\times 12 matrices. Even with large training budgets, 4 4 -layer models where unable to learn the eigenvalues of 10 10 or 12 12 dimensional matrices.

In our experiments, we wanted to study whether smaller DB could induce small models to learn large problems. We trained 4 4 -layer transformers to predict the eigenvalues of 10 × 10 10\times 10 matrices. We trained 5 5 models for each data budget of 1 , 5 , 10 , 25 , 50 1,5,10,25,50 and 100 100 M, and 5 5 for an unlimited DB (one pass over the training data), with TB up to 500 500 million. As expected, none of the models trained on unlimited DB did learn: all test accuracy remained close to 0 0 . However, 4 4 of the 30 30 models trained on smaller DB achieved 99 % 99\% accuracy: 3 3 models trained on 50 50 million examples (repeated 10 10 times), and one model trained on 10 10 million (repeated 50 50 times). Scaling even further, to 12 × 12 12\times 12 matrices, still using 4 4 -layer transformers, with a TB of 420 millions, 2 2 models (out of 35 35 ) begin learning: a 10 10 M model achieved 21 % 21\% accuracy, and a 5 5 M 3.5 % 3.5\% . As in previous experiments, for a given training budget, smaller data budgets and repeated training examples prove beneficial, but on this task, small datasets improve model scaling . With small DB, problems that required 8 8 -layer or 12 12 -layer transformers can be learned by 4 4 -layer models.

This first series of experiments clearly indicates that repetition helps learning. On three different tasks, for a fixed training budget, models trained on a small data budget, i.e. fewer distinct examples repeated several times, achieve much better performance than models trained from examples used only once or repeated a small number of times, as is customary in most recent works on language models ( Muennighoff et al., 2023 ) .

This phenomenon applies in different ways for different problems. On the GCD task, small DB allow for faster learning and higher accuracy. For modular multiplication, we observe emergence: a task inaccessible to models trained with large or unlimited DB is learned with small DB. Finally, for eigenvalues, small DB allow for better model scaling: tasks that normally require 8 8 or 12 12 -layer transformers are learned by 4 4 -layer models. But in all cases, the repetition achieved by small DB prove beneficial: smaller data budgets with repetition can elicit “emergent learning” .

## 5 Two-set training

The previous experiments demonstrate that for a fixed training budget, the optimal data budget is not the largest possible, as commonly practiced. On all three tasks, training from a set of distinct examples an order of magnitude smaller than the training budget, repeated many times, improves performance. We now turn to a different but related problem: how to best use a given data budget?

As we have seen, repeated examples help the model learn. Training from a small subset of the available data should therefore be beneficial, since it would increase repetition. However, models trained from very small datasets will eventually overfit their data, causing their accuracy to saturate. Yet, this can be prevented by increasing the size of the training set. To address these contradictory requirements – a small train set to increase repetition vs a large train set to avoid overfitting – we propose two-set training . We randomly split the training sample into a small set of examples that will be repeated many times during training, and a large set of examples that will be seen a few times only. By doing so, we hope that the small set fosters learning, while the large set prevents overfit.

Specifically, for a data budget of N N distinct examples, we randomly select S < N S<N examples that will form the repeated set – in practice, we shuffle the training set, and assign the S S first examples to the repeated set. During training, examples are selected from the repeated set with probability p p , and from the N − S N-S others with probability ( 1 − p ) (1-p) . As a result, a model trained with a training budget of T T will see p ​ T pT examples from the repeated set, repeated p ​ T / S pT/S times on average, while the N − S N-S remaining examples will be repeated ( 1 − p ) ​ T / ( N − S ) (1-p)T/(N-S) times on average. The repetition levels in both samples can be adjusted by choosing the values of S S and p p . Note that the limiting cases p = 0 p=0 and p = 1 p=1 correspond to one-set training, with a data budget of N − S N-S and S S examples respectively.

On the GCD problem , models trained on a single set, with a data budget of 100 100 million examples and a training budget of 600 600 million, predict 27 27 GCD on average (Figure 2 (Left)). Experimenting with two-set training for different values of S S and p p , we observe that models trained on a repeated set of 250,000 250,000 examples or less, with a probability p p of 0.25 0.25 or 0.5 0.5 , predict more than 62 62 GCD on average, a much better performance than their one-set counterparts. For S = 50,000 S=50,000 and p = 0.25 p=0.25 , models predict 69 69 GCD on average, a better performance than the best models trained on a single set, with a larger training budget of 1 1 billion examples. For these parameters, the 50 50 k examples in the small set are repeated 3,000 3,000 times on average, and the rest of the training examples 4.5 4.5 times on average. On a 100 100 M data budget, two-set training clearly outperforms single set training.

These results can be extended to unlimited training sets, by creating a fixed set of S S examples, selected with probability p p , and generating (unlimited) random examples with probability 1 − p 1-p . The best choices of p p and S S are roughly the same as with a DB of 100 100 M (Figure 8 in Appendix B ). In particular, with p = 0.25 p=0.25 and S = 50,000 S=50,000 , two-set training on unlimited data achieves an average performance of 67 67 GCD on 6 6 models, a spectacular improvement over models trained on unlimited (single) datasets, which predict 25 25 GCD on average.

Therefore, for large and unlimited data budgets, frequent repetition of a tiny number of random examples, lost in a sea of single-use examples, unlocks surprising performance gains. Note the synergistic nature of this effect: training on the tiny sample alone (with large repetition), or one-set training on the same data budget, result in much lower performance than what two-set training provides by mixing them together (see also Appendix C.2 : during training, mixing repeated and single-use examples into the same mini-batches is required for two-set training to happen).

We observe similar behavior for smaller data budgets. Figure 4 compares single and two-set training performance, for data budgets of 10 10 , 25 25 and 50 50 million example, and training budgets up to 600 600 M. For a given training budget, two-set training always achieves better performance than single-set training, and the benefit of two-set training increases as DB get larger. On this problem, two-set training accelerates learning. With large enough TB, single-set models sometimes catch up with the performance of their two-set counterparts with large enough TB (for 400 400 M TB for 10 10 M models, 600 600 M for 25 25 M, 1 1 B TB for 50 50 M models, see Figure 9 , Appendix B ). Still, most two-set models retain a marginal advantage 3 3 3 and note that S S and p p might no longer be optimal for this larger training budget over models trained on a single set.

For modular multiplication , experiments with large and infinite data budget, for a training budget of 600 600 M (Figure 5 ), indicate that larger repeated samples and smaller repetition, are needed, compared to GCD. With a DB of 100 100 M, S S should be selected between 2.5 2.5 and 10 10 million examples, and p p between 0.25 0.25 and 0.5 0.5 , for a small set repetition between 30 30 and 60 60 (vs 3000 3000 for the GCD experiments). For unlimited DB, S = 25 S=25 M and 0.75 ≤ p ≤ 0.9 0.75\leq p\leq 0.9 , a repetition between 18 18 and 22 22 , seem optimal. Note also that in this problem, the choice of parameters S S and p p is more sentitive: only a few combinations allow for good performance (empirically, constant ratio between repetition on the small and large sample ( p ⁡ ( N − S ) ( 1 − p ) ​ S ≈ 10 \frac{p(N-S)}{(1-p)S}\approx 10 ).

However, with a careful choice of p p and S S , two-set training achieves better performance than single set training for all data budgets from 25 25 M to unlimited. Table 2 presents the proportion of models, trained on single and two sets, that learn to compute multiplication modulo 67 67 , after a training budget of 600 600 M. With two set training, 50 50 to 58 % 58\% of the models learn multiplication with 99 % 99\% accuracy. With single set training, 24 24 to 28 % 28\% learn for DB 25 25 and 50 50 M, and none for larger DB. In these experiments, two-set training improves accuracy for all data budgets. However, its impact on learning speed (observed for GCD) is less conclusive (Table 7 in Appendix B ).

Finally, on the eigenvalue problem for 10 × 10 10\times 10 matrices, we train models with an unlimited data budget and a training budget of 500 500 M. With these parameters, models trained on single sets do not learn (see Section 4 ), but two-set training achieves significant accuracy. For p = 0.25 p=0.25 we run 15 15 models each for 7 7 different sizes of S S between 15,000 15,000 and 960,000 960,000 . 9 9 models out of 105 105 learn to predict with more than 60 % 60\% accuracy (Table 3 ). We see that injecting small, frequently repeated random subsets into the training data causes emergence of learning, where uniform repetition fails! Note, again, the synergistic effect: neither training on the small set alone, nor training with unlimited data budget in one epoch would allow any learning at all - it is the combination of both that makes two-set training powerful!

Overall, our experiments indicate that, for a given data budget, two-set training – repeating a small set of randomly selected during training – greatly improves model performance, either by accelerating learning (GCD), or increasing model accuracy (modular multiplication, eigenvalues). The size of the repeated set appears to be problem dependent: small for GCD, larger for modular multiplication and eigenvalues.

## 6 Ablations and variations

In this section, we discuss possible improvements to two-set training. Detailed ablation results can be found in Appendix C .

#### Curating the repeated sample.

In two-set training, repeated examples are randomly sampled from the available training data. We now experiment with a possible improvement: selecting the repeated examples. Perhaps what really matters is the repetition of a particular class of “informative” examples, as in curriculum learning. The GCD problem is particularly well suited for this type of investigation. Charton (2024) showed that increasing the proportion of small integers, or oversampling the tails of the distribution of GCD in the training set ( Prob ​ ( GCD = k ) ∼ 1 k 2 \text{Prob}(\text{GCD}=k)\sim\frac{1}{k^{2}} ), greatly improved model performance.