Emergent properties with repeated examples — part 1/4

Cleaned-text code points [0, 15405). Bibliography omitted here; source.md includes it.

# Emergent properties with repeated examples

###### Abstract

We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.

## 1 Introduction

When training neural networks, it has become customary to use the largest and most diverse datasets available, and to limit example reuse as much as possible. This tendency is manifest in large language models. GPT ( Radford & Narasimhan, 2018 ) was trained for 100 100 epochs (each example was seen 100 100 times on average), BERT ( Devlin et al., 2019 ) on 40 40 and GPT-2 ( Radford et al., 2019 ) on 20 20 . In recent models, most examples in the pre-training corpus are seen only once, a few specialized datasets are iterated 2 2 or 3 3 times, and fine-tuning examples are seen once or twice. Meanwhile, data budgets are on the increase: GPT-2 was trained on less than 10 10 billion tokens, GPT-3 ( Brown et al., 2020 ) was pre-trained on 300 300 billion, Chinchilla ( Hoffmann et al., 2022 ) and Llama ( Touvron et al., 2023 ) on 1.4 1.4 trillion, Llama2 ( Touvron & et al., 2023 ) on 2 2 trillion, and Llama3 ( Dubey & et al., 2024 ) on 15.6 15.6 trillion. Whereas the use of large train sets is grounded in theory ( Vapnik & Kotz, 2006 ) , the practice of not repeating training examples is less motivated. It reflects the belief that, when availability permits fresh data is superior to repeated use of a corpus ( Komatsuzaki, 2019 ; Raviv et al., 2022 ; Hernandez et al., 2022 ; Muennighoff et al., 2023 ) . This belief is grounded in the idea that memorization of repeated examples hinders generalization ( Zhang et al., 2017 ) . From a human learner point of view, this is counter-intuitive. When faced with a situation we never experienced, we recall similar instances ( Proust, 1919 ) , and use them as anchors to navigate the unknown. If memorization benefits human learners ( Ambridge et al., 2015 ) , why should it hinder machines?

In this paper we challenge the view that the repetition of training examples is undesirable, and that for a given training budget (TB, the total number of training examples), one should maximize the data budget (DB, the number of distinct training examples). We explore the impact of repeated samples in three controlled settings using generated data: computing the greatest common divisor (GCD) of two integers ( Charton, 2024 ) , modular multiplication of two integers, and calculating the eigenvalues of symmetric real matrices ( Charton, 2022 ) . These settings allow for perfect control over the distribution of repeated examples, unlike natural datasets (e.g. text from the web) which may feature unintended duplication and redundancy.

Our experiments uncover two striking phenomena: 1. Repetition Helps: For fixed training budgets (300M to 1B examples), models trained from small data budgets (25 to 50M examples) outperform models trained on large DB. This sometimes gives rise to “emergent” phenomena: properties only learned by models trained on small DB.

2. Two-Set Training: For fixed data budgets, learning speed and performance are significantly enhanced by randomly selecting a subset of training examples, and repeating them more often during training. The “two-set effect” is all the more surprising as the repeated examples are not curated, and only differ from the rest of the training data by their frequency of use.

In ablation experiments, we show that the performance of two-set training cannot be improved by curating the set of repeated examples, or refreshing it as training proceeds. This sets us apart from curriculum learning , and strengthens the observation that repetition of a few random examples is really all we need. We also show that mixing repeated and non-repeated examples in the same mini-batches is required for two-set training to work. Finally, we propose a smooth extension of two-set training, by introducing a probability distribution on the training set.

Our work isolates an interesting phenomenon in a clean setting. The three tasks we study each exhibit idiosyncratic structure that allows to test a variety of hypotheses. For instance, the GCD dataset exhibits an inverse polynomial distribution of results, reminiscent of Zipf’s law in natural language ( Zipf, 1935 ) . This allows us to test whether amplification of the tail of the distribution can benefit learning, by incorporating it into two-set training (while an attractive hypothesis, our ablations show that this seems not to be the case). In contrast, the results of modular multiplication are almost uniformly distributed, indicating that our conclusions do not depend on the existence of a power-law. Finally, the eigenvalue problem features non-linear, approximate calculations on reals.

In all three cases, the benefits of repetition are significant, but come in different flavors, from improving performance and accelerating learning (GCD), to allowing a new task to be learned (multiplication), or to be accessible to smaller models (eigenvalues). Alternatively, small random subsets of the data repeated at high frequency can elicit similar effects. These findings have profound implications and should lead to a paradigm shift where the training set size becomes a mere hyper-parameter, not solely governed by the availability of data and the belief that more is always better.

Note. Training budget is known as compute budget in other works ( Power et al., 2022 ; Muennighoff et al., 2023 ) . We use training budget to distinguish it from the compute cost arising from model size.

## 2 Background and Related Work

In this paper, we focus on relatively small transformer models performing mathematical tasks, placing it into a long established corpus of works that study interesting phenomena in a controlled setting, and advance our understanding of the underlying mechanisms in larger models in the wild, see e.g. Power et al. (2022) ; Garg et al. (2022) ; Charton (2024) ; Dohmatob et al. (2024) .

One such example is the study of “grokking” , first observed with modular arithmetic - a phenomenon where models generalize long after achieving 100 % 100\% accuracy on their (small) training set ( Power et al., 2022 ; Liu et al., 2022b ; Liu et al., 2023 ) . On the surface, grokking shares similarities with our work: a small training dataset is iterated for many epochs, the phenomenon is isolated in clean experiments on synthetic data, and it contradicts traditional wisdom regarding overfitting ( Mohri et al., 2018 ) . But there are important differences: in grokking, delayed learning occurs, we observe no such delay; grokking occurs for “tiny” training samples (hundreds or thousands of examples), our models use millions (even for modular multiplication); grokking is very sensitive to the optimizer used, our findings are robust across optimizers (Appendix C.5 ), and, of course, no two-set approach is documented in the grokking setting.

Another related setting is “benign overfitting” ( Bartlett et al., 2020 ; Belkin, 2021 ; Bartlett et al., 2021 ) , where an over-parametrized model perfectly fits noisy data, without harming prediction accuracy. One could argue that our work presents a quantitative manifestation of benign overfitting, inasmuch as decreasing the data budget increases model over-parametrization. However, this would not account for the decrease in performance once the data budget falls below a certain number (one could argue that overfitting is no longer benign, then), nor for the possibility of two-set training.

Prior works have studied the role of data reuse in language models. Hernandez et al. (2022) study data repetition in models with up to 800 800 M parameters with training budgets of 100 100 M tokens to exhibit detrimental impact of repetition of a subset of the training data. In the context of scarcity of training data, Muennighoff et al. (2023) find for LLMs of contemporary size (up to 9 9 B) that with constrained data for a fixed training budget, training with up to 4 4 epochs of repeated data yields negligible changes to loss compared to having unique data; any further repetition decreases the value of additional training. A limitation of these works is the lack of control over repetition in the training set: partial copies, of sentences, paragraphs sometimes whole documents, abound in pre-training corpora. Allen-Zhu & Li (2024) undertake a controlled study on synthetic language data in the context of knowledge retrieval and find that knowledge augmentation - repeated inclusion of reformulated variants - of a small subset of the data leads to performance improvement; an effect somewhat akin to what we observe in two-set training.

Our work is related to, but different from, curriculum learning (CL) ( Bengio et al., 2009 ; Wang et al., 2022 ) , where training data is presented in a meaningful order, usually from “easy” to “hard” samples. Two-set training differs from curriculum learning in at least two important ways: in CL, datasets are curated, our subsets are completely random; in CL, the training distribution shifts over time, while our subsets are static. Our ablations show that curating the repeated set, or changing it over time, as in CL, brings no improvement in performance (and may even have an adverse effect).

Lastly, our work touches upon the expansive area of out-of-distribution (OOD) generalization ( Gulrajani & Lopez-Paz, 2021 ; Lopez-Paz, 2025 ) , which studies generalization when train and test distributions differ. Curiously, while our two-set approach increases the frequency of some training examples, because the repeated set is chosen at random , the training set remains distributionally equivalent to the test set. Thus, our study falls outside the usual framework of OOD studies.

## 3 Experimental settings and baselines

We focus on three problems of mathematics: computing the greatest common divisor, multiplication modulo 67 67 , and computing the eigenvalues of real symmetric matrices. The GCD and eigenvalues were studied in prior work ( Charton, 2022 ; Charton, 2024 ; Dohmatob et al., 2024 ; Feng et al., 2024 ) .

Greatest common divisor. The model is tasked to predict the GCD of two integers uniformly distributed between 1 1 and 1 1 million, encoded in base 1000 1000 . Following Charton (2024) , who observes that throughout training almost all pairs of integers with the same GCD are predicted the same, we evaluate model performance by the number of GCD below 100 100 predicted correctly, measured on a random test sample of 100,000 100,000 pairs: 1000 1000 pairs for each GCD from 1 1 to 100 100 . Charton reports a best performance of 22 22 correct GCD for a model trained on uniformly distributed inputs. Note. We prefer this test metric over a more standard accuracy on random input pairs, because the GCD are distributed according to an inverse square law. In particular the probability that a GCD is 1 1 is about 62 % 62\% . As a result, the accuracy metric results in overly optimistic model performances.

Modular multiplication. Modular arithmetic plays an important role in many public key cryptography algorithms ( Diffie & Hellman, 1976 ; Regev, 2005 ) , and is known to be a hard problem for neural networks ( Palamas, 2017 ) . Modular addition was studied in several previous works, in the context of grokking ( Power et al., 2022 ; Liu et al., 2022a ) and mechanistic interpretability ( Zhong et al., 2023 ) 1 1 1 Power et al. (2022) also study modular division , equivalent to modular multiplication. . While modular multiplication over ℤ / p ​ ℤ × \mathbb{Z}/p\mathbb{Z}^{\times} is mathematically is equivalent to modular addition mod p − 1 p-1 , these problems differ computationally , due to the hardness of the discrete logarithm ( Diffie & Hellman, 1976 ) . In most previous works on arithmetic modulo p p , model inputs are sampled from integers between 0 0 and p p , which results in a very small problem space for small p p . In this work, we study the multiplication modulo 67 67 of two integers from 1 1 to 1 1 million. This allows for a much larger problem space, and training sets. Model accuracy is evaluated by the percentage of correct predictions of a × b mod 67 a\times b\mod 67 , on a test set of 10,000 10,000 examples (a new test set is generated at every evaluation). In this problem, all outcomes from 1 1 to 66 66 are uniformly distributed, while 0 0 appears nearly twice as often.

Eigenvalue calculation. This problem was introduced to deep learning by Charton (2022) , who showed that transformers can learn to predict the eigenvalues of real symmetric matrices with independent and identically distributed entries, rounded to three significant digits. The eigenvalue problem is arguably a harder problem than the previous two, non-linear and typically solved by iterative algorithms. Note also that because matrix entries and eigenvalues are rounded, this problem features noisy inputs and outputs. Model accuracy is evaluated as the percentage of model predictions that predict the correct eigenvalues of a test matrix with less than 5 % 5\% relative error (in ℓ 1 \ell^{1} distance). It is measured on a test set of 10,000 10,000 samples, generated afresh at every evaluation.

Models and tokenizers. In all experiments, we use sequence-to-sequence transformers ( Vaswani et al., 2017 ) with 4 4 layers in the encoder and decoder ( 4 4 -layers encoders and 1 1 -layer decoder for eigenvalues), an embedding dimension of 512 512 , and 8 8 attention heads. Models have 35 35 million parameters for GCD and modular multiplication, and 22 22 million for eigenvalues. They are trained to minimize a cross-entropy loss, using the Adam optimizer ( Kingma & Ba, 2014 ) , with a learning rate of 10 − 5 10^{-5} , over batches of 64 64 . The integer inputs and outputs of the GCD and multiplication problems are tokenized as sequences of digits in base 1000 1000 , preceded by a separator token. The real numbers in the eigenvalue problem are encoded as floating point numbers, rounded to three significant digits, and tokenized as the triplet ( s , m , e ) (s,m,e) – sign, (base 1000 1000 ) mantissa, (base 10 10 ) exponent – i.e. f = s ⋅ m ⋅ 10 e f=s\cdot m\cdot 10^{e} (P1000 encoding from Charton (2022) ). All experiments are run on one NVIDIA V100 GPU with 32 32 GB of memory.

## 4 Repetition Helps

We now embark on a systematic study of the impact of data budget on performance, for various training budgets. In other words, we compare the performance of models trained on datasets with a fixed number of examples (data budget), for increasing amounts of time (training budget).