Emergent properties with repeated examples — part 4/4
Cleaned-text code points [46315, 52240). Bibliography omitted here; source.md includes it.
Two-set training with a small randomly selected subset S S amounts to assigning different probabilities to elements in the training set. For a randomly shuffled training set of size N N , two-set training amounts to selecting the first S S elements with probability p / S p/S (with replacement) and the N − S N-S last with probability ( 1 − p ) / ( N − S ) (1-p)/(N-S) , a step-function distribution over { 1 , … , N } \{1,\dots,N\} . We now generalize this approach by introducing a probability law P P such that P ( i ) P(i) is the probability of selecting the i i -th example in the training set. Our motivation is to obtain a smooth, possibly more principled, distribution than the step-function induced by the two-set approach. Pragmatically, a one-parameter family of smooth distributions eliminates the need to tune both S S and p p . Lastly, we can study whether a smooth decay in frequency might be even more beneficial than a non-continuous two-set partition.
In this section, we consider a discrete exponential distribution: P ( i ) ∼ β e − β i / N , P(i)\sim\beta e^{-\beta i/N}, with β > 0 \beta>0 , suitably normalized 5 5 5 The normalization factor is ( 1 − e − β ) − 1 (1-e^{-\beta})^{-1} . In our calculations we will approximate it by 1 1 to simplify computing S eff S_{\text{eff}} . For the range of β \beta we consider, the resulting approximation error is negligible. In general, for fixed p p , to compute the size of the set S ( p ) S(p) of first elements that carry probability mass p p , we can use β ≈ − ln ( 1 − p ) N / | S ( p ) | \beta\approx-\ln{(1-p)}N/|S(p)| . . If β \beta tends to 0 0 , P P tends to the uniform distribution, and implements the single-set strategy of Section 4 . As β \beta becomes large, a small fraction of the full training set is sampled ( 99 % 99\% of the probability mass lies on the 4.6 N / β 4.6N/\beta first elements, 99.99 % 99.99\% on the first 9.2 N / β 9.2N/\beta ). For intermediate values of β \beta , the model oversamples the first elements in the training set, and undersamples the last: we have a continuous version of two-set training. To allow for comparison with two-set training, we define S eff S_{\text{eff}} such that the first S eff S_{\text{eff}} examples in the training set jointly are sampled with probability 25 % 25\% . In this setting, 10 % 10\% of the probability mass is on the 0.37 S eff 0.37S_{\text{eff}} first training examples, and 99 % 99\% on the first 16 S eff 16S_{\text{eff}} .
For GCD, we experiment with values of β \beta ranging from 5.8 5.8 to 1152 1152 ( S eff S_{\text{eff}} from 25,000 to 5 5 million) 6 6 6 Note that for these values of β \beta the distinction between DB 100 100 M and unlimited DB becomes essentially meaningless, as the tails of the training set are sampled exceedingly rarely. . Table 5 shows that for our training budget of 600 600 million examples, the best model ( S eff = 3 S_{\text{eff}}=3 M) predicts 65 65 correct GCD, slightly less than what was achieved with two-set training (Section 5 ).
For modular multiplication, we need lower β \beta (i.e larger S eff S_{\text{eff}} ) for our training budget of 600 600 M. We report the number of models (out of 25 for each setting) that learn to accuracy above 50 % 50\% and 95 % 95\% respectively (Table 10 ). Again we see that these results are comparable to two-set training (Section 5 ).
We conclude that the benefits observed in two-set training do not pertain to the specific two-set partition of the training set; rather, it seems that the core of the effect lies in the non-uniform sampling frequency distribution over the (randomly ordered) training set, with a range of frequencies.
### C.5 Varying the optimizer
Some effects observed in deep learning depend on the optimizer, with grokking being a prominent example ( Power et al., 2022 ) . Here we provide experimental evidence to show that our findings hold for a variety of optimizers and are thus robust and universal . We rerun models used for the GCD problem with different optimizers. Specifically, we trained models to predict GCD, with a training budget of 600 600 million examples, single and two-set training (with | S | = 50,000 |S|=50,000 and p = 0.25 p=0.25 ), and data budgets of 25 25 million, 50 50 million and unlimited. We considered four optimizer settings: • Adam without dropout or weight decay,
• Adam with weight decay 0.01 0.01 ,
• Adam with dropout ( 0.1 0.1 ) in the feed-forward networks of the transformer,
• AdamW with weight decay 0.01 0.01 .
Table 11 presents the best performance of 5 5 models for each configuration. On average, dropout has an adverse effect on learning, but there is no clear benefit of using weight decay, or AdamW over Adam. Importantly, the separation in performance between single-epoch unlimited training, training on smaller data budgets with more repetitions and two-set training persists across optimizers: the effects we present are robust.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .