2 OLMo 2 Furious

arxiv:2501.00656

Original source · Paper JSON

LLMs: architectures, context, training · 2024-12-31

Cites (5) · Cited by (5)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7

Contents

• 2 OLMo 2 Furious
• Abstract
• 1 Introduction
• 2 OLMo 2 Family
• 2.1 Model Architecture
• 2.2 Tokenizer
• 2.3 Base Model Training Recipe
• Stage 1: Pretraining
• Stage 2: Mid-training
• Model Merging or “Souping”
• Overall
• 2.4 Base Model Data
• 2.4.1 Pretraining data: OLMo 2 Mix 1124
• 2.4.2 Mid-training data: Dolmino Mix 1124
• 2.5 Evaluation and Results
• Base Model Evaluation:
• Post-Training Recipe and Evaluation
• 3 Deep Dive: Pretraining Stability
• 3.1 Repeated n-Grams
• 3.2 Model Initialization
• Gradient and activation growth
• Hyperparameter transfer across width
• Spike score
• Empirical results
• 3.3 Architecture Improvements
• 3.3.1 Nonparametric layer norm and RMSNorm
• 3.3.2 Reordered norm and QK-norm
• 3.3.3 Z-Loss
• 3.4 Hyperparameter Improvements
• 3.4.1 ϵ \mathbf{\epsilon} in AdamW
• 3.4.2 Weight decay on embeddings
• 4 Deep Dive: Mid-training Recipe
• 4.1 Learning rate annealing
• 4.2 Data Curriculum: Dolmino Mix 1124
• 4.3 Dolmino Mix 1124 : High Quality Sources
• High quality web
• Instruction data and Q&A pairs
• Code
• Academic, encyclopedic and other reference content
• Math
• 4.4 Dolmino Mix 1124 : Math Mix
• 4.4.1 Math Sources
• TuluMath
• DolminoSynthMath
• TinyGSM-MIND
• MathCoder2-Synthetic
• ProofPile OWM-Filtered
• GSM8K-Train
• 4.4.2 Evaluating Math Data with Microanneals
• Microanneal experiment 1: domain specific data is helpful even in small proportions
• Microanneal experiment 2: some duplication is beneficial
• Microanneal experiment 3: rewriting can help dramatically
• 4.5 Final Midtraining mix and Checkpoint Soups
• Mid-training model merging or “soups”
• 5 Deep Dive: Post-training Pipeline
• Supervised Finetuning (SFT)
• Preference Finetuning (PreFT) with DPO
• Reinforcement Learning with Verifiable Rewards (RLVR)
• Hyperparameter selection
• Evaluation of OLMo 2-Instruct
• 6 Deep Dive: Infrastructure as a Research Catalyst
• 6.1 Clusters
• 6.1.1 Jupiter
• Compute
• Storage
• Interconnect
• Cooling
• 6.1.2 Augusta Cluster
• Compute
• Storage
• Interconnect
• Cooling
• 6.2 Beaker
• Portability
• Isolation
• 6.3 Stability and Operations
• GPU health checks
• Cordoning
• Active monitoring
• 6.4 Maximizing hardware utilization
• Taking advantage of compilation
• Minimizing host-device syncs
• Asynchronous bookkeeping with a separate backend
• Explicit Python garbage collection
• 6.5 Environmental Impact
• Conclusion
• Author Contributions
• Acknowledgments
• Appendix A OLMo 2 Evaluation Framework
• A.1 Base Model Eval
• Multiple-choice tasks
• Generative tasks
• Held-out tasks
• A.2 Instruct Model Eval
• Instruct tasks
• Appendix B OLMo 2 1B
• B.1 Difficulties with OLMo 2 1B
• Training
• Base Results
• Analysis
• Appendix C Additional Instruct Details
• C.1 Additional Hyperparameters
• C.2 Additional RLVR Learning Curves
• C.3 OLMo 2-Instruct Preview Models
• Appendix D Additional Hyperparameters
• Appendix E Annealing Data Details
• Instructions for reporting errors