Data, training, optimization — page 2

• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
  arxiv:2508.11408 · 2025-08-15 · text available
• Optimizing Millions of Hyperparameters by Implicit Differentiation
  url:paperswithcode.com/paper/optimizing-millions-of-hyperparameters-by · 2019-11-06 · metadata only
• Overcoming catastrophic forgetting in neural networks
  doi:10.1073/pnas.1611835114 · 2017-03-14 · text available
• Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
  arxiv:2405.20541 · 2024-05-30 · text available
• Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
  arxiv:2608.27370 · 2026-08-27 · text available
• Reinforcement Pre-Training
  arxiv:2506.08007 · 2025-06-09 · text available
• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations
  arxiv:2405.18392 · 2024-05-28 · text available
• Scaling Laws for Neural Language Models
  arxiv:2001.08361 · 2020-01-23 · text available
• Scaling Laws for Reward Model Overoptimization
  arxiv:2210.10760 · 2022-10-19 · text available
• Self-Improving Pretraining: using post-trained models to pretrain better models
  arxiv:2601.21343 · 2026-01-29 · text available
• Sharpness-Aware Minimization for Efficiently Improving Generalization
  arxiv:2010.01412 · 2020-10-03 · text available
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
  arxiv:2305.14342 · 2023-05-23 · text available
• Step-size Optimization for Continual Learning
  arxiv:2401.17401 · 2024-01-30 · text available
• Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates
  arxiv:1708.07120 · 2017-08-23 · text available
• Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
  arxiv:2507.12856 · 2025-07-17 · text available
• Symbolic Discovery of Optimization Algorithms
  arxiv:2302.06675 · 2023-02-13 · text available
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
  arxiv:2009.11243 · 2020-09-23 · text available
• The AdEMAMix Optimizer: Better, Faster, Older
  arxiv:2409.03137 · 2024-09-05 · text available
• The Loss Does Not See the Basis, but Adam Does
  arxiv:2608.05136 · 2026-08-05 · text available
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
  arxiv:1803.03635 · 2018-03-09 · text available
• The Road Less Scheduled
  arxiv:2405.15682 · 2024-05-24 · text available
• Training compute of frontier models grows 4–5x/year (Epoch AI)
  url:epochai.org/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year · 2024-05-28 · metadata only
• Why the Third Axis Is Freedom
  doi:10.5281/zenodo.21979851 · 2026-08-01 · text available