Emergent properties with repeated examples

arxiv:2410.07041

Original source · Paper JSON

Data, training, optimization · 2024-10-09

Cites (2) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Emergent properties with repeated examples
• Abstract
• 1 Introduction
• 2 Background and Related Work
• 3 Experimental settings and baselines
• 4 Repetition Helps
• 5 Two-set training
• 6 Ablations and variations
• Curating the repeated sample.
• 7 Discussion
• Appendix
• Appendix A Learning dynamics and overfitting in math transformers
• Appendix B Additional figures and experiments for modular multiplication
• Appendix C Ablation results
• C.1 Curating the small sample
• C.2 Batching in two-set training: mixed batches are needed
• C.3 Shifting the small set
• C.4 From two-set to many-set training
• C.5 Varying the optimizer
• Instructions for reporting errors