{"id":"arxiv:2410.07041","title":"Emergent properties with repeated examples","date":"2024-10-09","section":"Data, training, optimization","quality":0.087466,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/","source_url":"https://arxiv.org/abs/2410.07041","text_available":true,"authors":["François Charton","Julia Kempe"],"outgoing":[{"source":"arxiv:2410.07041","target":"arxiv:2309.14316","evidence":"title","id":"arxiv:2309.14316","title":"Physics of Language Models: Part 3.1, Knowledge Storage and Extraction","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b87376d90e99d2bb590e1a02ed50a55c90881591ed7d3e00c3d5be2aaacbb2af/"},{"source":"arxiv:2410.07041","target":"arxiv:2407.21783","evidence":"id","id":"arxiv:2407.21783","title":"The Llama 3 Herd of Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de3cebb4c28bc5d41a843279f219dca14476284991ea5c7c0aeab7068a40e306/"}],"incoming":[{"source":"arxiv:2503.21676","target":"arxiv:2410.07041","evidence":"id","id":"arxiv:2503.21676","title":"How do language models learn facts? Dynamics, curricula and hallucinations","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/"}],"text_parts":[{"start":0,"end":15405,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-1.html"},{"start":15405,"end":31162,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-2.html"},{"start":31162,"end":46315,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html"},{"start":46315,"end":52240,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-4.html"}],"contents":[{"title":"Emergent properties with repeated examples","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-1.html#text"},{"title":"Abstract","offset":46,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-1.html#text"},{"title":"1 Introduction","offset":949,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-1.html#text"},{"title":"2 Background and Related Work","offset":6328,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-1.html#text"},{"title":"3 Experimental settings and baselines","offset":10544,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-1.html#text"},{"title":"4 Repetition Helps","offset":15100,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-1.html#text"},{"title":"5 Two-set training","offset":22378,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-2.html#text"},{"title":"6 Ablations and variations","offset":30337,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-2.html#text"},{"title":"Curating the repeated sample.","offset":30495,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-2.html#text"},{"title":"7 Discussion","offset":34219,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"Appendix","offset":37105,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"Appendix A Learning dynamics and overfitting in math transformers","offset":37118,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"Appendix B Additional figures and experiments for modular multiplication","offset":39825,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"Appendix C Ablation results","offset":40254,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"C.1 Curating the small sample","offset":40286,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"C.2 Batching in two-set training: mixed batches are needed","offset":43743,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"C.3 Shifting the small set","offset":45227,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"C.4 From two-set to many-set training","offset":46272,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-3.html#text"},{"title":"C.5 Varying the optimizer","offset":50000,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-4.html#text"},{"title":"Instructions for reporting errors","offset":51276,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/part-4.html#text"}],"text_code_points":52240,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/63a5d833b43fae5ed1f9f44f26b90813fa8a37dd44870e8e409d7a7c34fabfec/source.md"}