Data, training, optimization — page 1

• ADAM Optimization with Adaptive Batch Selection
  url:iclr.cc/virtual/2025/poster/30565 · 2024-10-01 · metadata only
• Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
  arxiv:2312.10549 · 2023-12-16 · text available
• Continual Backprop: Stochastic Gradient Descent with Persistent Randomness
  arxiv:2108.06325 · 2021-08-13 · text available
• Continual Learning and Catastrophic Forgetting
  arxiv:2403.05175 · 2024-03-08 · text available
• Cramming: Training a Language Model on a Single GPU in One Day
  arxiv:2212.14034 · 2022-12-28 · text available
• Cyclical Learning Rates for Training Neural Networks
  arxiv:1506.01186 · 2015-06-03 · text available
• Deep Double Descent (OpenAI)
  url:openai.com/index/deep-double-descent · 2020-01-23 · metadata only
• Emergent properties with repeated examples
  arxiv:2410.07041 · 2024-10-09 · text available
• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
  arxiv:2607.27372 · 2026-07-29 · text available
• Grokfast: Accelerated Grokking by Amplifying Slow Gradients
  arxiv:2405.20233 · 2024-05-30 · text available
• gzip Predicts Data-dependent Scaling Laws
  arxiv:2405.16684 · 2024-05-26 · text available
• How much do language models memorize?
  arxiv:2505.24832 · 2025-05-30 · text available
• How much LLM training data is there, in the limit?
  url:educatingsilicon.com/2024/05/09/how-much-llm-training-data-is-there-in-the-limit · 2024-05-09 · metadata only
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
  arxiv:2009.11848 · 2020-09-24 · text available
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
  arxiv:1904.00962 · 2019-04-01 · text available
• Learning in High Dimension Always Amounts to Extrapolation
  arxiv:2110.09485 · 2021-10-18 · text available
• Learning Vision from Models Rivals Learning Vision from Data
  arxiv:2312.17742 · 2023-12-28 · text available
• Maintaining Plasticity in Deep Continual Learning
  doi:10.1038/s41586-024-07711-7 · 2023-06-23 · text available
• Measuring Catastrophic Forgetting in Neural Networks
  arxiv:1708.02072 · 2017-08-07 · text available
• MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
  arxiv:2402.02342 · 2024-02-04 · text available
• Nested Learning: The Illusion of Deep Learning Architectures
  arxiv:2512.24695 · 2025-12-31 · text available
• No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
  arxiv:2307.06440 · 2023-07-12 · text available
• NorMuon: Making Muon more efficient and scalable
  arxiv:2510.05491 · 2025-10-07 · text available
• On the Information Bottleneck Theory of Deep Learning (Saxe et al.)
  openreview:ry_WPG-A- · 2017-10-27 · text available